How Notion Built a Self-Improving Agent Without ML Infrastructure
TRIGGER
AI agents handling domain-specific workflows make mistakes or struggle with edge cases, but there's no mechanism for the agent to learn from these difficulties without retraining or prompt engineering cycles.
APPROACH
Notion's security team gave their investigation agent (Scruff) write access to a dedicated Notes page where it records how it completed difficult or confusing tasks. Input: agent encounters unfamiliar situation during alert investigation. Output: agent writes learnings to Notes page, then references these notes in future investigations. Humans can also edit the Notes page directly to make manual corrections. The agent uses this accumulating knowledge base to improve accuracy and speed over time without requiring prompt changes or redeployment.
PATTERN
“Complex vector DBs and RAG systems when a markdown file would suffice for procedural knowledge. Give the agent write access to a notes page it reads each run—difficult cases become training data without redeployment, and humans can edit corrections directly.”
✓ WORKS WHEN
- Tasks have recurring patterns where lessons from one case transfer to similar future cases
- Domain experts can review and correct the agent's self-documented learnings
- Agent runs frequently enough that accumulated notes provide value (daily or more)
- The knowledge to capture is procedural (how to handle X) rather than factual (what is X)
- Workspace tooling supports both agent and human editing of the same document
✗ FAILS WHEN
- Each task is sufficiently unique that prior learnings don't transfer
- Notes accumulate faster than humans can review, allowing errors to compound
- Agent lacks judgment to identify what made a task 'difficult' vs routine
- Security or compliance requirements prohibit agents writing to shared knowledge bases
- The knowledge needed is already well-documented in existing runbooks