The Framework Swap Trap
TRIGGER
Agent performance is a product of framework, tools, and underlying model—when an agent underperforms, teams can't tell whether to switch frameworks, upgrade tools, or use a different LLM because all three vary simultaneously across experiments.
APPROACH
FutureBench structured evaluation into three isolation levels: Level 1 (Framework) keeps LLMs and tools constant while varying frameworks (e.g., LangChain vs CrewAI both using GPT-4 + same search tools); Level 2 (Tools) fixes LLM and framework while comparing tool implementations (Tavily vs Google vs Bing search); Level 3 (Model) holds framework and tools constant while testing different LLMs (DeepSeek-V3 vs GPT-4 with identical SmolAgents setup and Tavily+web scraper toolkit). They used SmolAgents as the baseline framework with a minimal two-tool setup (Tavily search + web scraper) to isolate model reasoning ability.
PATTERN
“Switched frameworks three times but agent still unreliable? You're misattributing framework bugs to model limitations. Systematic ablation (vary one component, freeze others) isolates whether it's the model, framework, or tools failing.”
✓ WORKS WHEN
- You're building an agent stack and need to make component selection decisions
- Multiple viable options exist at each layer (2+ frameworks, 2+ tool providers, 2+ models)
- Budget allows running the same questions across multiple configurations (at least 3-5 configurations)
- Components are modular enough to swap without changing others
✗ FAILS WHEN
- Stack is locked to a single provider (e.g., must use vendor's framework + model + tools together)
- Evaluation budget only allows testing one or two configurations total
- Components have tight coupling where changing one requires changing others
- Task is simple enough that component choice doesn't meaningfully affect outcomes