Why private AI? The case for keeping your data home

Every business is being told to adopt AI. Boards ask for an AI strategy; vendors demo copilots; competitors announce agents. Almost nobody asks the first question that should come before any of it: where does the data go?

Most AI adoption to date has been paste-first, policy-later. An employee with a deadline pastes a client contract into a chat window. A founder feeds their roadmap to a summarizer. Nothing breaks, no alarm rings, and a quiet assumption sets in — that this is fine. This article is about why that assumption deserves scrutiny, what the alternative actually looks like, and an honest test for when the alternative is worth it.

What actually happens when your data hits a cloud model

When your prompt travels to a hosted AI service, you are handing data to a third party's infrastructure under terms you don't control. The mechanics vary by vendor and tier, and the details matter less than the structure:

• Retention. Inputs may be stored for days, months, or indefinitely, depending on tier and settings. Retention windows are a policy, and policies are a changelog you don't author.
• Training use. Consumer tiers have historically treated inputs as trainable material by default. Enterprise tiers offer better commitments — which you verify in a contract, annually, with a vendor's sub-processor list attached.
• Blast radius. Everything you sent sits in someone else's breach surface. Their incident becomes your notification obligation.

None of this makes cloud AI illegitimate — enterprise agreements with zero-retention and no-training clauses are real, and for plenty of workloads they are the right answer. The point is narrower: when data sensitivity is high, the boundary moves out of your control, and control is the thing you are being asked to give up.

Agents raise the stakes

A chat session is one prompt. An agent is something else entirely: AI with standing access — to your mailbox, your file store, your CRM, your shell — acting on your behalf across many steps. That is where the productivity is, and also where the risk profile changes shape.

Two mechanisms matter. First, blast radius: an agent that can read your systems aggregates the most sensitive digest of your business into a single context. Second, prompt injection: an agent reads untrusted text (an email, a web page, a ticket), and text can carry instructions. If the agent that reads the hostile text can also reach your data and a network, instruction and exfiltration channel arrive in the same package.

If your agent's brain is a remote model, that context — the continuous, high-resolution digest of your business — crosses someone else's wire on every step. Local inference keeps those prompts off hosted model APIs. Containment is still engineering work: restricted tool permissions, controlled outbound connections, and tests that prove the controls hold.

The regulatory floor keeps rising

You don't need to be a compliance nerd to be governed by this. If you touch European personal data, GDPR's data-minimization and purpose-limitation principles apply to every prompt, and cross-border transfers need a legal basis that has been getting harder, not easier, to paper over. The EU AI Act is phasing in obligations around exactly these systems. Sector floors — HIPAA in health, PCI DSS where payment data flows — add their own reach.

My career sits in computer communications: telecom protocol stacks, network security, embedded systems and hardware devices, and the application layers above them. In the regulated corners of that world, the audit was always part of the product — designed in, not bolted on. The lesson transfers directly: compliance auditors do not accept "it was just a pilot." The moment customer data touches a system, the obligations attach. This is why sovereignty requirements that used to be a government-sector quirk are moving into ordinary mid-market procurement.

What "private AI" actually means

It's a spectrum, not a binary switch:

• Enterprise cloud tier — vendor-hosted, contractually constrained. Better defaults, real commitments — and still a vendor holding the keys.
• VPC-hosted open weights — open models running in a private slice of cloud you control. The boundary moves to your cloud account.
• On-prem open weights — models running on hardware behind your wall. The boundary moves onto your network: you define permitted connections and enforce them, instead of trusting a vendor's.

The reason option three became practical is the open-weight generation of models. For a large share of business workloads — retrieval over your documents, summarization, drafting, classification, extraction, agentic tool use — current open models are genuinely sufficient, and they run on surprisingly ordinary hardware. I know this the unglamorous way: I run a production-shaped stack on two consumer RTX 3090s — 262,144 tokens of context, fully local, with decode rates and costs measured and published. A year ago that sentence would have needed a data centre.

What private AI does not fix

An honest pitch includes this section. Private AI is not free, and it is not always the answer:

• The frontier gap is real. For the hardest reasoning tasks, the best cloud models still lead. A good architecture uses local where the data demands it and is honest about the rest — hybrid is not a dirty word.
• You own the operations. Updates, uptime, incident response — on-prem moves these from a vendor's SLA to your checklist. Small and boring if sized right; negligent if ignored.
• Some workloads don't justify it. Bursty, light, non-sensitive usage on an enterprise tier is a perfectly sane answer. Not everything needs a GPU box.

The decision test I use: first, establish the mandatory data and access restrictions — what regulation, contract or policy actually forbids. Then compare the feasible options on quality, latency, operational capacity and total cost. A hard prohibition can decide the architecture on its own; everything else is an optimisation.

The economics, honestly

Private AI is not "free AI." You pay in capital and electricity instead of per-token invoices, and in operations instead of vendor trust. Where local wins is volume and continuity: agentic workloads with cached prefixes and many requests amortize a box quickly, and a fixed cost floor beats a meter that spikes with your success. Where cloud wins is spiky, light usage. Run the numbers for your shape — it's the first thing an audit produces — and distrust anyone who quotes you a TCO without knowing your workload.

But notice that price was never the core argument. The structural wins are: no egress copies of your data, no per-token meter on your most sensitive workflows, no rate limit during your busiest hour, and no policy changelog to re-read every quarter. Those four don't fluctuate with model prices.

I run one, in public

Everything above I practice on an open rig: club-3090 — a public benchmark rig serving production-shaped workloads on consumer GPUs since April 2026, with the configurations, quantization recipes, benchmarks and failures documented in the open. It has a community around it now, and it exists so that when I advise a business on private AI, I am describing a machine that runs, not a diagram that demos.

If your business has data that shouldn't leave the building and workloads that should, that's the engagement I built AppScience's first offer around: audit what you actually need, build the reference stack on your hardware, hand over the keys with the numbers attached.

Corrections, arguments and benchmark challenges welcome — the best ones end up as follow-up posts. Bring yours.