Nexus, an internal gateway for AI capabilities

The leak

Every AI system the company builds needs the same things: a way to call a model, a way to run OCR or transcription, a queue for long jobs, somewhere to put files, and monitoring. Built one system at a time, those things get built one system at a time. Worse, the long tail of small needs, a team that wants to classify one kind of email or extract one field from one form, never gets built at all, because each one is a project with a deployment at the end of it.

The leak is the AI work that does not happen because the fixed cost of starting is too high, and the duplicated plumbing in the work that does.

The constraint

The gateway has to be a product for internal developers and, eventually, for operations teams who are not developers. It has to run on the company's own infrastructure with open models where they are good enough, support both immediate calls and queued jobs, and be observable end to end, because a shared service that cannot be debugged is a shared outage. And it has to be extended by adding a folder, not by changing the gateway.

The system

Nexus, as designed and partly built: a single API with two paths, an immediate invoke for calls that return in seconds and a jobs path for work that is queued, prioritised and reported on. Capabilities are plugins, each a folder with a manifest describing what it does, what it needs and what it returns: OCR, speech-to-text, a language model, a vision model, and integrations to workflow engines so a capability can be a chain rather than a single call. Underneath, a priority queue, workers, object storage for inputs and outputs, and a full observability stack with traces, metrics and logs.

The layer that makes it a platform rather than a service is the workflow builder: a low-code surface where a workflow is composed from capabilities and published as its own endpoint, with no code change and no deployment. A team that needs a small AI function gets it the same afternoon, from within the operation that needs it.

The foundation exists and runs. The call-analysis workflow that became the production call analytics system was first prototyped on it. The workflow builder and the hardening for shared production use are the next phase, now that the team to build them is arriving.

The number

Not yet. The number this platform is designed to produce is the count of AI functions published by teams without an engineering project, and the time from a team's request to a working endpoint. Both will be measured from the first week it is open.

What I would do differently

Build it second, not first. Nexus was the first thing I started and it stalled because the seven systems that followed were each more urgent and each taught something about what the gateway needs. The right order was the one that happened by accident: ship the systems, learn the common shape, then build the platform. I would plan it that way on purpose.