Can a 20-person company run a private AI assistant on one machine? What we learned doing it

By Vishal · published 25 October 2026

What we ran

A single high-memory workstation (the kind a video editor buys) serving an open-weight model of about 30 billion parameters, quantised to 8-bit. In front of it, a small API gateway we wrote: multi-tenant chat so different clients' documents stay separate, retrieval over private documents, and task-specific routes for marketing briefs and audits. No cloud API in the loop. Everything on the desk.

The purpose was to answer a question clients kept asking: can a firm that cannot send documents to an API — a law practice, a clinic, a component manufacturer with customer drawings — still have a capable assistant?

What worked

• Answer quality on grounded questions. With good retrieval, a 30B model answers questions about the retrieved documents accurately and cites them. The model's general knowledge matters less than the passages it is handed.
• Confidentiality by construction. No data-processing agreement to negotiate, no vendor to audit, no question of where prompts go. For some clients this removes the entire approval process.
• Predictable cost. One capital purchase and electricity. No per-token bill that spikes when someone pastes a 200-page contract.
• Multi-tenancy. Scoping retrieval by tenant at the gateway, rather than trusting the model, kept documents separate in every test.

What did not

• Concurrency. A few simultaneous users are fine; a department asking at once is not. Queueing helps, a second machine helps more.
• Long, open-ended reasoning. The largest hosted models are still better at multi-step reasoning without retrieved material. For grounded question-answering the gap barely matters; for "write me a strategy" it does.
• Operations. Someone has to update the model, back up the index and watch the disk. We wrote a runbook; a client without a technically-minded person still needs a care arrangement.

Who it fits

On-premise vs your cloud account

The middle path is often best: the same open model on a GPU instance inside your own cloud account in an Indian region. Nothing leaves your account, you can scale to more users, and the provider handles the hardware. It costs more per month than a workstation and less than you fear, and it avoids the single point of failure. The on-premise workstation remains the right answer when the rule is "nothing leaves the building" rather than "nothing leaves our control". Which rule applies is the first question in our private-data agent feasibility call.