Developing with AI coding agents
Source:vignettes/articles/ai-coding-agents.Rmd
ai-coding-agents.RmdThis article is for developers who work on rdborrow with an AI coding agent such as Claude Code, Codex, Cursor, or Gemini CLI. The repository tells agents how the package is built, so the agent’s first draft follows the package’s conventions. Your job shifts to stating the task precisely and checking the result.
How agents learn the package’s conventions
AGENTS.md, at the top of the repository, is a plain
markdown file of instructions that most coding agents read
automatically. Claude Code reads .claude/CLAUDE.md instead,
which imports AGENTS.md, so every agent follows the same
rules from a single file. AGENTS.md is listed in
.Rbuildignore, so it never ships with the package.
It covers:
| Section | What the agent learns |
|---|---|
| Architecture | The user-facing API, the method class hierarchy, how
estimate() dispatch works, and the _core() /
_se() / _boot_statistic() pattern |
| Key commands | How to run tests, re-document, build the site, and run
R CMD check
|
| Coding | Base R only, base pipe, R 4.1 compatibility, checkmate validation, comment style |
| Testing | Where tests go, which expectations to use, and that a bug fix needs a failing test first |
Documentation and NEWS.md
|
When to add roxygen docs, pkgdown entries, and NEWS bullets |
| Specialized tasks | Where to find the tidyverse instructions for argument checking and deprecation |
| Before committing | The one-line validation command to run before every commit |
For Claude Code, .claude/settings.json also pre-approves
routine commands such as Rscript, grep, and
gh issue view, so the agent doesn’t stop to ask before
running tests.
A workflow that works
- Start from an issue. An issue gives the agent a fixed statement of the problem and gives the PR something to close.
- Ask for a plan before code. Reading a plan takes a minute; unwinding a wrong approach takes much longer.
- Write the test first. Have the agent write a test that fails for the right reason, and confirm it fails, before it touches the fix.
-
Let the agent implement and validate.
AGENTS.mdtells it to run styler, re-document, and run the before-committing check. - Review the diff yourself. Read every changed line, starting with the statistics.
- Open a PR that references the issue.
Example prompts
Add a method
Adding a new method is written so that an agent can follow it step by step. Give the agent the estimator’s definition and point it at the article:
Add a new primary-analysis estimator to rdborrow called ec_or(), an
outcome-regression estimator for external control borrowing as defined in
<paper, equation number>.
Follow vignettes/articles/adding-a-method.Rmd step by step and use
R/ec_ipw.R as the template. Before writing the estimator, write
tests/testthat/test-full_pipeline_ec_or.R against reference values that I
will provide, and confirm that it fails. Then implement the method until
the test passes and devtools::check() is clean.Checking the agent’s work
An agent can produce a clean, passing, well-documented change that is statistically wrong. These habits catch that:
- Supply the reference values yourself. Take them from the paper, the authors’ code, or a hand calculation. An agent that writes both the estimator and its expected output can encode the same mistake in both.
-
Review the math line by line. For an estimator,
that is
.{name}_core()and.{name}_se(); the rest is plumbing. -
Keep changes additive. A new estimator adds files
and edits only the registration spots listed in Adding a new method. If the diff changes
the logic of
run_analysis(), an existing method, or a shared helper, ask why. -
Treat a changed regression test as a red flag. The
test-full_pipeline_*.Rfiles lock every method’s output onSyntheticData. A changed number needs an explanation in the PR. -
Run the full check.
devtools::test()usesload_all(), which can hide problems that only appear in the installed package.devtools::check()doesn’t.
Testing the docs with an adversarial agent
Docs written for agents can be tested by an agent. To check this
article, Adding a new method, and
AGENTS.md, we gave a second agent a real task: build
ec_or(), an outcome-regression estimator for the primary
analysis with bootstrap-only inference, in its own git worktree. Its
instructions said to use only the docs, to log every time it had to read
package code instead, and to treat a run with no findings as
suspicious.
The agent built a working method on its own branch:
- An independent reference script, written in base R before any
package code, and the new
ec_or()agreed exactly onSyntheticData. - The full-pipeline test failed first and passed after implementation.
devtools::check()reported 0 errors, 0 warnings, and 0 notes, andec_or()ran insiderun_simulation(). - It reported 15 findings. Two were blockers, places where it couldn’t
continue without reading code: the docs didn’t say that
estimate()should rename columns toSandAwith.build_analysis_df(), and they didn’t give.run_bootstrap()’s arguments or return value. Both are now documented, along with the missing registration spots and a bootstrap-only example. - It found two package bugs, both since fixed:
bootstrap = 1was accepted and then failed with an opaque error (#91), andrun_simulation()silently recycled a vectortrue_effectacross replicates (#90).
Rerun this check whenever the developer docs change. The prompt below is the one we used, updated with what the first run taught us:
You are an adversarial tester for the rdborrow R package's developer
documentation. Build a new borrowing method by following the docs exactly,
and report every place they are wrong, ambiguous, or missing something. A
smooth run with no findings is a suspicious result; look harder.
Setup:
- Create a git worktree on a new branch from the branch under test. Do not
touch any other branch or the main working tree. Building from a
worktree adds a `.git` file at the top level; add `^\.git$` to your
branch's .Rbuildignore and note that this edit is a test artifact.
- Your only guidance is AGENTS.md, vignettes/articles/adding-a-method.Rmd,
and vignettes/articles/ai-coding-agents.Rmd. You may read package code,
but every time you need code to learn something the docs should have
told you, log it as a finding.
- Do not push, open PRs, or create issues. There is no issue for this
exercise; say so in the NEWS bullet instead of citing one.
Task: add <estimator>, defined as <definition>, with <sandwich or
bootstrap-only> inference. Choose a method that exercises a path the docs'
toy example does not.
Rules:
1. Before any package code, write an independent reference script in base
R, with no rdborrow internals, that computes the estimate on
SyntheticData. This is your ground truth.
2. Follow adding-a-method.Rmd in order. Write the full-pipeline test
against the reference values first and confirm it fails.
3. Follow every AGENTS.md rule and run the "Before committing" one-liner.
Commit locally.
4. Probe the docs' claims:
- List every existing file you modified. Each one is either a step in
the checklist or a finding.
- Run the method inside run_simulation() with a few small simulated
replicates and a scalar true_effect, and confirm which CI columns
coverage uses.
- Try to break the constructor (out-of-range arguments, bootstrap = 1,
mismatched formula counts) and judge whether the errors are clear.
Report, most severe first: each finding with the doc file and line, what
it says, what happened, a suggested fix, and a severity (blocker, wrong,
gap, nit); files touched, split into checklist steps and undocumented
edits; reference vs. package estimates; the final devtools::check()
summary; and how long a human contributor would need with these docs.