Private AI: when self-hosting is the practical choice

Self-hosting a model is an operational commitment, not a preference. It is worth making when one of three constraints is real - and worth refusing when none of them are.

The three constraints

Data residency is the first and the clearest: if a regulator, a contract or a customer says the content cannot leave a jurisdiction or a network, that decides it. The second is cost per token at volume, which only matters once the volume is real and measured rather than projected. The third is tolerance for a wrong answer, which sets how much review infrastructure you need regardless of where the model runs.

Pick the process first, measure its current cost, then choose the model. Doing it the other way round produces a pilot nobody can evaluate.

What self-hosting actually costs

• Hardware or reserved GPU capacity, sized for peak rather than average.
• A model upgrade path, because the model you deploy will be superseded.
• Evaluation harnesses, so a version change does not silently regress quality.
• The same guardrails, logging and review UI a hosted model would need.

A workable default

Start hosted, with the process instrumented and a retention policy that keeps the inputs and outputs you would need to migrate later. If volume or residency then forces the move, you have a real evaluation set and a measured baseline to move with - which is the difference between a migration and a restart.