I have spent the last months building Octonous, an agentic product that runs on Anthropic, OpenAI, Gemini, and any model served through an API compatible with those. That means I have not just been calling models. I have been depending on the product surface around them: tool calling, streaming, files, web search, code execution, prompt caching, token accounting, context management, and all the small behavioral contracts that make an agent feel reliable instead of fragile.

That experience has made one thing clear: the gap between frontier APIs and open models is no longer about model quality. Open models can already reason, call tools, write code, and produce structured output well enough for a large class of agentic products. The blocker is everything around the model.

When you swap a frontier model for an open model served through vLLM, llama.cpp, Ollama, or any hosted “OpenAI-compatible” endpoint, you discover that “OpenAI-compatible” mostly means “you can send chat messages and get tokens back.” 

That is useful, but it is not the contract a production agent has quietly grown to rely on.

This gap is exactly what we have been working on with Otari: an open source gateway that sits between agent applications and open model runtimes: part compatibility layer, part tool runtime, part context manager, part observability surface. 

If your agent speaks the Anthropic or OpenAI APIs, Otari’s job is to make an open model answer in the same language and with the same platform behavior. 

What Breaks When You Switch

Here is a concrete version of the failure. 

Take an Octonous agent that is running on an Anthropic model today: it streams tool-call progress to the UI, searches the web with cited sources, executes generated code in a sandbox, accepts user file uploads, and leans on custom prompt caching to optimize token consumption based on our specific usage patterns.

Point that same agent at a strong open model behind an OpenAI-compatible endpoint. 

Basic chat works on the first try, which is exactly what makes the gap deceptive. Then, one by one:

• Tool calls arrive in a slightly different dialect, and streaming emits partial JSON fragments the UI was never built to parse.
• Web search does not exist server-side, so the agent confidently answers from stale training data with no sources.
• File uploads have no lifecycle: no upload endpoint, no references, no artifacts to hand back to the user. A dropped-in file has nowhere to live and nothing downstream to point at it.
• Code execution does not exist at all, and with no upstream file lifecycle, the generated code would have nothing to open even if it ran. “Analyze this CSV” fails twice.
• Context compaction is not handled server-side, so we have to implement our own logic to manage long conversations.
• Our custom prompt caching no longer works, and we are left with whatever automatic caching the serving runtime has implemented, if it provides caching at all.
• Usage reporting is not detailed enough so you can no longer tell the user what a run cost or why it failed.

None of this is the model’s fault. The model may be perfectly capable. What broke is the platform layer.

When a frontier API provides one of these pieces, the application naturally starts depending on it. When an open endpoint does not, the application team rebuilds it. At the end of that path, you may still be running the same open model you started with, but you have built half a frontier API around it.

That is the hidden surface area behind a “model switch,” and it is the part Otari exists to absorb.

The Product I Want as an Agent Builder

The product I want is boring in the best way. I want to take an Octonous agent that currently runs on Claude, GPT, or Gemini, point it at an open model, and keep shipping. Tools working, files working, streaming keeping the UI alive, usage intelligible, failures normalized enough that the application can recover.

I do not expect every open model to magically support every frontier feature. The point is to stop forcing every application team to discover and paper over the same gaps independently. 

Open models already have the intelligence. Developers building on frontier APIs are used to the platform doing a lot of work for them: that is not laziness, it is leverage. If we want open models to compete in real products, they need comparable leverage, and it should not have to be rebuilt in every codebase.

Where We Are

Otari is early, but it is not a manifesto. What ships today in our GitHub repository: the three generation surfaces (Chat Completions, Responses, Anthropic Messages) in front of 40+ providers, virtual keys, per-user and per-key budgets enforced before the call runs, usage and cost tracking with cached-token accounting, built-in web search and sandboxed code execution, server-side MCP, file uploads for local models, and opt-in guardrails. You can run it locally and make a metered request in about a minute, or connect it to otari.ai and let our platform run it for you. 

If you have an agent on Claude, GPT, or Gemini and an open model you wish it ran on: try the gateway against your runtime, open an issue, or tell us which missing feature blocks you first. That is the gap we are closing, in the order agent builders actually hit it.

This article first appeared on Read More