Claude is a brilliant developer who has never seen Optimizely. How we gave our agents the framework experience a human needs half a year to earn: about 1,400 interlinked notes, a verification pipeline that trusts the compiled packages over the blogs, and an index of 14,424 NuGet package versions.
When I explain to people outside the Optimizely world how we work with agents, I end up using the same analogy every time: treat them like employees.
Some are brilliant and some are not quite as clever. Some are very independent and some need a lot of guidance. But all of them need to learn your business, your procedures and the way you work. They need access to the tools you already use, Linear, DevOps, GitHub and the rest. And when the work is built on a big framework, they need experience with that framework and the community packages around it.
That last part is what my talk at the Optimizely Copenhagen Developer Meetup was about. It was called “Brilliant and ignorant”, which is the most accurate description of a frontier model on an Optimizely project I’ve managed to come up with.
A brilliant hire who has never seen Optimizely
Claude is very good out of the box. Give it an Optimizely task and it will get there in the end. But watch how it gets there. It looks things up. It guesses at an API, gets a compiler error, guesses again. It decompiles DLLs with the enthusiasm of someone nobody ever told where the documentation is. And when it’s done, you may find it has written a helper method that already exists in the framework, or rebuilt something that a de facto standard community add-on solved years ago.
That isn’t a model problem. It’s an experience problem, and we’ve seen it before.
A good developer could always get going with Optimizely quickly. But it took half a year and a couple of big projects before they could pass the certification exam, because that’s how long it takes to learn which API is the current one, which blog post is from 2014, and which add-on everyone already uses.
Agents are the same, with one difference that changes everything. When a human developer learns something, one developer knows it. When an agent learns something and writes it down, every agent knows it. Without that, an agent’s learning is amnesiac by default: it figures out something hard, the session ends, and the knowledge goes with it.
In my last post, agents rebuilt our site on CMS 13 with Visual Builder. Learning that Visual Builder elements can’t hold lists, content areas or nested blocks cost that session a full rebuild of the content model. Today it’s a verified note in our knowledge repository, and the next agent that needs it pays for one search.
A thousand files in a folder
You could point your agents at Context7 or one of the other documentation services instead, and they’re genuinely good. But you have no control over what’s in them, how up to date it is, or which version it was true for. And nobody has checked any of it against the packages you actually run. A docs index tells your agent what the documentation says. It doesn’t tell it what your project has learned.
So we built the agents somewhere to keep their own. It’s an internal knowledge repository, and at its core it is exactly what it sounds like: Markdown files in a folder, interlinked, in git. Twenty-five years in this industry, and the state of the art is a folder.
The first version was built on 9 November in one long session: 13 prompts from me, about 220 agents doing the work, and 90 minutes from the first prompt to the first commit. It has kept growing since. Today it holds about 1,400 notes.

That’s the repository in Obsidian’s graph view. Every orange dot is an Optimizely note, and the lines are links between them. The small blue and green cluster in the corner is everything else the company knows about its own projects, which puts the size of a framework into perspective.
The agents get at it the same way a colleague would: through an MCP server, through search, and by messaging the agent that maintains it and asking.
What we fed it
The harvest pulled from everywhere an experienced Optimizely developer learns from:
- The official documentation and Optimizely World.
- 28 OMVP blogs, plus community answers and GitHub.
- Knowledge extracted from real codebases and made generic, plus sample codebases generated just for the agents to experiment and learn on.
- Upgrade projects we’ve done, where the agents compared the codebase before and after the upgrade to see what actually changed.
Then comes the honest counterpoint, the one I made on stage too. This isn’t more knowledge than our company already had. It’s more files. The notes written by people who’ve done the work are still the most valuable ones. The difference is that now the agents can read them too, alongside everything else.
How a claim earns “verified”
A harvest that keeps everything is a harvest you can’t trust. Blog posts disagree with each other, documentation lags behind releases, and forum answers are confidently about a version nobody runs any more.
So the pipeline works like a newsroom. Haiku, the cheap model, reads and drafts. Sonnet treats every drafted claim as unverified and checks it against the official documentation and the actual packages, then merges, corrects or drops it. One Opus agent plans the work and reviews it. The expensive model didn’t read a single documentation page. It only delegated, which is more than I can say for some managers.
Every note carries its own provenance:
applies_tosays which versions it’s true for: CMS 11, 12, 13.confidenceis one ofverified,documented,communityorunverified.sourcesandlast_verifiedsay where it came from and when someone last checked.
Each source has a trust ceiling. A blog post can make a note community, however good the blog is. The official docs can make it documented. Only the compiled packages or running code can make it verified. When two sources contradict each other, the note records the conflict and scopes it by version or hosting, instead of quietly picking a winner. Blog posts said Search & Navigation, the product most of us still call Find, works on CMS 13. The official FAQ and the packages say it doesn’t. The note keeps both and says which one to believe.
Here’s one, unedited apart from swapping an internal source ID for a readable name:
Today’s ledger is 238 verified, 347 documented, 706 community and 127 unverified. That’s 706 notes we owe to people who took the time to blog and answer questions about Optimizely. Thank you, and I’m sorry about the trust ceiling.
The pipeline also throws things away, which is the part I trust most. The blog harvest produced 327 drafts. 236 became new notes, 53 were dropped and 94 posts were skipped, mostly for being too thin to learn from. And the Commerce harvesters invented types. They described ICartService and friends with total confidence, and verification found nothing of the sort in the packages. Every one was dropped.
It catches us too. Our own notes said ServiceLocator.Current.GetInstance<T>() wouldn’t compile on CMS 13. A real 13.5.1 build said it does. IServiceLocator is gone, but the static ServiceLocator class is still there and not even marked obsolete. The note in the frame above carries that distinction.
The assemblies are the adult in the room
The biggest single source of verified knowledge is an index of the Optimizely NuGet feed: it backs 137 of the 238 verified notes. It holds 902 packages and 14,424 package versions, with the public API of every one of them extracted straight from the compiled assemblies.

Blog posts tell you what someone believed about Optimizely when they wrote them. The assemblies tell you what’s actually in the version you’re running. When the two disagree, the assemblies win, every time.
An agent asks the index through MCP, so the answer comes back as structured data instead of a web search:

Those three calls are the whole argument:
DataFactoryon CMS 13. A model trained on fifteen years of EPiServer blog posts will happily writeDataFactory.Instance.Get<PageData>(). On CMS 13 the index has noDataFactoryat all. On CMS 12 it survives only in the migration package, already marked obsolete.ContentReference.StartPage. It returned aPageReferenceon CMS 12 and returns aContentReferenceon CMS 13. Code that assigns it to aPageReferencecompiled for years and doesn’t any more.RootPage,WasteBasketandPageData.ParentLinkmade the same move.ContentAreaItem.RenderSettings. It went from a dictionary of objects with a setter to a dictionary of strings you can only set at construction. Nothing in a model’s memory will warn you.
There’s a subtler lesson in there too. Diff the EPiServer.Cms.Core package from 12.24.3 to 13.5.1 and about 1,300 types appear to be removed. They weren’t. They moved into the new EPiServer package. Looking things up by package name lies to you. Looking them up by type and version doesn’t.
My favourite is IContentIndexer in Optimizely Graph 13.3. It wasn’t made obsolete, and it wasn’t deprecated. It went internal, which is the API equivalent of being ghosted. That note is still community, because the version before it can’t be checked against the packages, and the trust ceiling doesn’t care how good the story is.
What it cost
The repository took 6,756 model requests between 9 and 14 November. At Anthropic list prices that comes to $232.76, calculated from our own token counts, so read it as an estimate, not an invoice. Most of it went to Sonnet doing the checking.
That split is the most useful number in this post. The reading is cheap. The reconciling isn’t. Any setup that skips the reconciling to save money gets you a big pile of confident Markdown, which is roughly what the model already had in its head.
Knowledge has a best-before date
It isn’t finished, and I’d rather say so. Nothing runs on a schedule yet, and that’s the biggest gap. Optimizely ships new package versions every few weeks, and a few hundred notes still cite CMS 13.4.1 and haven’t been re-checked against 13.5.1. A knowledge base nobody updates becomes the 2014 blog post it was built to replace.
So the next piece is a scheduled job, and this is how I’d set one up:
- Watch the feed. When a new package version lands, index it and diff its API against the previous version.
- Flag what the diff touches. Every note whose
applies_tocovers a type that changed goes back in the queue for verification, and itsconfidencedrops until it has been checked again. - Harvest what’s new. Release notes, Optimizely World and the blogs, since the last run only, through the same read-then-verify pipeline as everything else.
- Hand it to a human. The job opens a pull request with what it changed and why, instead of committing straight to the knowledge base. Agents can do the checking, but somebody should still read the diff.
Run it weekly, or whenever a release lands. It should cost a fraction of the first build, because in most weeks most of the API hasn’t moved.
We’re also testing a proper search index and MCP server on top of the notes, which currently finds the right note slightly less often than letting Sonnet grep the folder, though for a fraction of the tokens. Grep has been undefeated since 1973, so I’m not too embarrassed.
What I’d tell you to do
Onboard your agents like you’d onboard a developer. Give them your tools, your procedures and the framework knowledge your best people carry around. They’re brilliant, but they’re new here.
Write it down where every agent can find it. A gotcha one session found the hard way should cost the next one a search, not an afternoon. Plain files in git are a perfectly good start.
Put it on a schedule. Frameworks move. Re-check what a new release touches, harvest what’s new, and let a human approve the changes.
Trust the compiled code over the blogs. Give every source a ceiling and only let the packages or running code promote a claim to verified. For Optimizely, that means an index of what’s actually in the NuGet packages, by version.
Record contradictions instead of resolving them. “Blogs say yes, the FAQ and the packages say no” is useful knowledge. Picking one and hiding the other isn’t.
Pay for the checking. Let the cheap model read and the better model verify. The verification is where the value is.
If you want this for your own platform, Optimizely or otherwise, get in touch. We can set it up for you: the harvesting, the verification against your package versions, and the access from the tools your agents already use. Optimizely is what we do, and teaching agents a framework turns out to be most of the job.
A side note to finish. The summaries of what each package’s types are for are written by a model, and for the NuGet index that’s a lot of summaries. So I trained a small model of my own to write them, which went better than I expected in some places and much worse in others. That’s the next post.
Your agents are already brilliant. Ignorance is the part you can fix, and you only have to fix it once.