There are a lot of AI coding applications out there, and as impressive as large language models and the agents they enable have become, many of the most recent developments in AI-assisted development have been in the software that manages those models, not just the models themselves.
Earlier this summer, I spoke with the head of product for Claude Code, Anthropic’s Cat Wu, about that company’s approach to building that software.
Wu repeatedly came back to the same point in our conversation: Anthropic’s models (and those of its direct competitors) are improving so quickly that it makes little sense to plan too far ahead or to build opinionated or limiting features around them. Rather, the Claude Code product team attempts to maintain what they call a lean harness.
A harness is the software built around one or more AI models that determines how they are used. It decides what the models see, what actions they can take, and how they interact with the code base. Think of it like a layer between a model and the developer’s actual project.
It’s not that Claude Code as an application and harness has no opinionated features or design choices, but its product team does seem to err on the side of trusting where the models will take them over the next year.
There are other harnesses besides Claude Code, though. There are OpenAI’s Codex, Google’s Antigravity, open source alternatives like OpenCode, and options from startups like Cursor or Augment Code. Each can have different features, emphasis, or opinions about how a model can or should be utilized in agentic workflows.
One key choice that Claude Code makes is to avoid approaches that build a structured context around a codebase in advance by default.
“Going by the evals, we don’t see a measurable change,” she said of those approaches. “And I think we generally lean more toward shipping a leaner harness with fewer opinionated tools and just letting developers add their own if they want.”
Augment Code has made a substantially different bet, although the two companies are not necessarily testing the same interventions or optimizing for the same outcomes; its product pre-indexes the repository using embeddings, a retrieval model, and a vector database, then retrieves conceptually relevant code. So to get the other side of that discussion, I spoke with Vinay Perneti, Augment Code’s VP of Engineering.
Perneti briefly described Augment Code’s differing approach, offered his own team’s evaluations of the advantages, responded to some of the arguments for a leaner harness, and shared his perspective on some of the concerns developers have about AI tools and agentic workflows more broadly.
This interview has been edited for length and clarity.
Ars Technica: Can you explain how Augment Code’s context engine works?
Vinay Perneti: So, if I take a step back, what is it that a particular developer is trying to achieve? They want a particular task implemented and they give it to an agent. And the interesting thing with agents today is they have a limited context window, and every time they need to essentially go get all of the context needed for that and then work on it.
There’s two approaches to context. One is grep-based. Claude Code and Codex and other agents have done that. The second is the semantic retrieval bit. For Augment, we’ve always taken the semantic route, and I’ll describe the building blocks of that.
I would say there’s two core pieces. One is, we have an embedding and a retrieval model pair that are working in the system, and then you have a vector database and an entire, highly optimized back-end system that makes it possible to retrieve in sub-milliseconds.
Ars: Does it have an advantage in one kind of code base versus another?
Perneti: Yeah, it actually turns out the advantage comes in large, private codebases. And here is why I say that.
For all the public, open source repos where most of the benchmarks are run, every single model has basically memorized the repo. These models are large enough that they’re actually able to memorize the whole repo. So when you’re trying to get something done, the models kind of already know where to look so they can get to an outcome quickly.
Whereas when you’re doing this in a private repo, a model has never seen that repo. And now the iteration loop for finding the outcome is much longer, right? If you have a semantic understanding of your entire private repo, you can ask a question, and you get to those outcomes much more quickly.
Ars: People are concerned about token efficiency. Does this semantic approach help with that?
Perneti: We’ve certainly seen it in certain situations. In fact, we published a blog post—we ran Terminal-Bench with Claude Code and Augment Code, same model. And we completed at similar accuracy, but we were 33 percent more efficient than Claude Code. So I do see this showing up in terms of making better use of tokens because you’re not spending as much time on the exploration side.
Ars: Anthropic told me that Claude Code does not see measurable eval gains from including more semantic code-navigation tools. But then I look at your blog, and I see you posting benchmarks and other claims that, actually, this really helps. Are you guys measuring different things when you say this, or what are their evals missing?
Perneti: Great questions. I don’t know what specific retrieval engine they used, and I think the other mistake that often people conflate is, not all retrieval systems are equal just like not all databases are equal, right? And by that, what I mean is the models and the system that are working together, which is the context engine, makes a big difference in terms of the quality of outcomes as well.
So, for example, at Augment, we spent about 18 months at the beginning of the company being founded in 2022—this is pre-ChatGPT—researching retrieval and embedding models predominantly for large code bases. So there’s a lot of research that has gone into, when you’re trying to achieve a particular outcome, what are the right pieces of code to get in this embedding space? All of that is encoded into our retrieval models.
And doing that in a very, very fast way is the system that we built around it. So when somebody says, ‘Hey, I tried Claude Code with a RAG implementation, but I’m not seeing benefits,’ that’s because the implementation and the context engine are very, very different, if that makes sense.
Ars: What do you say to the notion that models are improving so fast that building anything with any assumptions at all makes no sense? That’s the argument for a super lean harness or not doing any of this, is just that you don’t know where it’s gonna be in six months or twelve months or whatever as the exponential goes up.
Perneti: There’s a lot of truth to that, but I think you have to think about it from a couple of different perspectives.
At Augment, the way we’ve been thinking about this is there’s two ingredients that you need for higher quality outcomes, intelligence and context. When models are getting really good, intelligence is going to get exponentially better, no doubt about that. But just because they’re more intelligent does not mean they have the context.
