How Anthropic Turns AI Agents into Batch Processing Infrastructure
TRIGGER
Large-scale codebase operations (migrations, bulk refactors, mass file processing) are too numerous for interactive human-in-the-loop workflows—thousands of similar transformations need to happen, but each requires AI judgment that simple scripts can't provide.
APPROACH
Use Claude's headless mode (`claude -p`) for programmatic batch processing. Fan-out pattern: (1) Have Claude write a script to generate a task list (e.g., 2k files needing migration from framework A to B), (2) Loop through tasks calling Claude programmatically: `claude -p 'migrate foo.py from React to Vue. Return OK if succeeded, FAIL if failed.' --allowedTools Edit Bash(git commit:*)`, (3) Run script several times and refine prompt to get desired outcome. Use `--output-format stream-json` for streaming JSON output, `--json` for structured results, `--verbose` for debugging (disable in production). Pipelining pattern: `claude -p '<prompt>' --json | your_command` for integrating into data pipelines.
PATTERN
“Thousands of similar transformations done by hand—the "interactive-only trap" where you forget agents can run headless. Agent capabilities (file editing, code understanding, judgment calls) can be API-ified via `claude -p`. One refined prompt template, executed in parallel, replaces thousands of manual decisions.”
✓ WORKS WHEN
- Tasks are similar enough that a single prompt template handles all cases
- Each task is independent and doesn't require context from other tasks
- Failure on individual items is acceptable (can be retried or handled separately)
- You can iterate on the prompt using a subset before running at scale
✗ FAILS WHEN
- Tasks require human judgment that varies case-by-case
- Failures cascade—one bad transformation breaks dependent files
- Security concerns prevent running untrusted agent actions at scale
- Cost of running thousands of agent calls exceeds value of automation