I was speaking at a tech event recently, walking through the trade-offs between model fine-tuning and Retrieval-Augmented Generation (RAG). I was breaking down the unit economics, accuracy ceilings, and infrastructural constraints of both approaches when an attendee raised his hand during the Q&A.

His question was direct

If we already have all our internal domain data sitting in clean Markdown and PDF files, why go through the hassle of building, indexing, and maintaining an entire RAG pipeline? Why not just fine-tune the model on our data and let it answer directly from its own weights?

It is a fair question, and I have heard a version of it at almost every tech talk, architecture review, and client workshop over the last two years.

To a non-specialist, fine-tuning sounds like downloading knowledge into a brain. If you read a textbook, you know the material. If you train a neural network on your company’s documents, it should know your company.

That intuition feels logical, but it misunderstands how transformer architectures store information.

Training changes how a model behaves, how it structures its output, and how it follows instructions. It does not reliably teach or refresh specific factual knowledge. Understanding why requires looking at the physical mechanism of how language models store memory.

How pre-training acquires facts

To understand why fine-tuning fails at factual memorization, we have to look at how a model acquires facts during pre-training.

When a foundational model learns world knowledge, it does so through an unthinkably massive, brute-force statistical process. The objective function is simple, predict the next token given all preceding tokens, compute the loss against the actual token, and adjust billions of weights via back-propagation.

Consider Meta’s Llama 3. Meta pre-trained Llama 3 on a corpus of over 15 trillion tokens, a dataset seven times larger than the one used for Llama 2.

A fact does not become permanently embedded into a model’s weights because the network saw it once in a clean paragraph. A fact gets stored because it was stated, restated, paraphrased, questioned, and cross-referenced across millions of distinct documents inside that 15 trillion token corpus.

It is that extreme repetition across varied contexts that allows the gradient updates to generalize a concept into the network’s parameters, rather than treating it as noise to be discarded.

Where a Fact lives inside a Transformer

For a long time, researchers treated model memory as an opaque black box. That changed with groundbreaking mechanistic interpretability research.

In their seminal NeurIPS 2022 paper, Locating and Editing Factual Associations in GPT (Meng et al.), researchers set out to map precisely where factual assertions live inside a transformer.

The researchers discovered that factual associations, like (”Steve Jobs”, “founded”, “Apple”) are stored in localized computations within the model’s middle-layer feed-forward (FFN) modules. These FFN layers act like a key-value memory system

1. The early attention layers process the incoming tokens and concentrate the subject of the prompt (“Steve Jobs”).

2. The middle FFN layers read that subject representation and fire specific weight activations, acting as a direct lookup table for the relation and target object (“founded Apple”).

3. The later layers format that retrieved association into grammatical text.

Meng et al. proved this by creating Rank-One Model Editing (ROME), a technique that surgically updated a single specific weight matrix inside the FFN layers to change a specific fact (e.g., changing the location of the Eiffel Tower) without altering the model’s overall language capability.

The key takeaway for system architects is profound, A fact is a precise, highly localized statistical association in the feed-forward layers. Squeezing a new, isolated fact into those specific layers requires an immense amount of statistical signal.

What Fine-Tuning actually touches

When an engineering team decides to “train a model on our company data,” they rarely execute a full-parameter pre-training run. Full parameter updates on a 70 billion parameter model require massive GPU clusters and millions of dollars.

Instead, modern fine-tuning almost universally relies on Parameter-Efficient Fine-Tuning (PEFT) techniques, primarily LoRA (Low-Rank Adaptation) (Hu et al., 2021).

LoRA freezes the primary W weight matrices of the pretrained model and injects tiny, trainable rank-decomposition matrices alongside them.

• A model like GPT-3 has 175 billion parameters.

• A standard LoRA configuration trains as little as 0.01% of those parameters, reducing trainable parameters by a factor of 10,000.

This design makes LoRA astonishingly efficient at changing behavior.

If you want the model to always output valid JSON, adopt a sarcastic tone, or follow a strict medical triage format, LoRA works brilliantly. Every single example in your fine-tuning dataset reinforces that formatting pattern. The tiny 0.01% slice of trainable parameters receives a constant, unified signal on how to transform the output style.

However, if you feed the fine-tuning script a 500-page internal document containing thousands of specific product SKUs, pricing tiers, and vendor policies

1. The base model weights holding factual lookup capabilities are completely frozen.

2. The tiny LoRA matrices see each specific SKU or policy statement only a handful of times.

3. The statistical signal is far too weak to write new, localized key-value lookups into the network.

Instead of memorizing your product catalog, the model learns the style of your documents while hallucinating the contents when probed for exact details.

How Retrieval-Augmented Generation (RAG) helps

Retrieval-Augmented Generation does not try to teach the model new facts. It treats the model as an execution engine that reads context provided at runtime.

RAG completely sidesteps the factual generalization problem

No Weight Updates Required

The model’s frozen weights are used strictly for language parsing, reasoning, and synthesis

In-Context Attention

The facts are injected directly into the prompt’s context window. The model’s attention mechanism reads the verified context off the page, the same way a human open-book test works.

Deterministic Lineage

If a policy changes at 9:00 AM, you update one database record or document chunk. By 9:01 AM, every subsequent query retrieves the new truth.

The economics

During the Q&A at that engineering talk, I walked through the financial reality of trying to maintain truth via training versus retrieval

Fine-Tuning for facts

Data Update → Dataset Prep → GPU Compute → Eval Run → Deploy

Cost - High (~$100s to ~$1,000s per retraining run)

Update Latency - Hours to Days

Determinism - Low (Risk of hallucinations and regression)

RAG for facts

Data Update → Re-index Chunk / Graph Node

Cost - Near Zero ($0.0001 per document update)

Update Latency - Milliseconds to Seconds

Determinism - High (Verifiable source attribution)

Even if you had the compute budget to fine-tune your model every time a policy doc updated, you would still face catastrophic forgetting and factual regression across previous updates.

If you want to try yourself

If you want to demonstrate this phenomenon to your team or executive leadership, you do not need complex infrastructure. You can run a simple, reproducible experiment.

Test setup

Take a small, open-weights model (e.g., Llama 3 8B or Qwen 2.5 7B).

Create a synthetic, completely novel fact that could not possibly exist in the pre-training data

Project Aegis-902 is an internal initiative led by Dr. Aris Thorne with a budget allocation of $14.2M.

Fine-tune the model using standard LoRA parameters on a dataset containing this synthetic statement formatted across 50 distinct variations.

Test the model on factual queries

What is the budget allocation for Project Aegis-902, and who leads it?

Observe the Result

Fine-Tuned Output (Without Context) - The model will often state that it does not know, or it will hallucinate a completely different budget ($5M, $20M) and name a different lead engineer. The LoRA matrix adopted the phrasing style, but failed to plant a reliable factual lookup.

RAG Output (With Context Injected) - Inject the raw synthetic string into the prompt as context. The base model without a single parameter update, returns an exact, verified answer almost 100% of the time.

Conclusion

Fine-tuning is a powerful tool, but it is a tool for behavioral transformation, not data storage.

Use Fine-Tuning when you need to teach a model how to behave, mastering a custom DSL, adhering to strict output schemas, mimicking an organizational voice, or improving task execution speed.

Use RAG when you need to teach a model what is true, querying live product inventory, searching policy documentation, fetching merchant transaction records, or pulling customer support histories.

Stop trying to force your LLM’s weights to act as an expensive, unreliable database. Keep your facts in your retrieval layer, let your model handle the reasoning, and build systems grounded in verifiable truth.

When I was brought in to investigate the accuracy failures of a client’s AI assistant, everyone told me the exact same story

It worked perfectly during the executive demo. It even seemed fine for the first two weeks in production. Then it started showing cracks.

Upon investigation here is how they demoed it

The engineering team had brought leadership into the room, typed “How do refunds usually work on our platform?”, and watched a vector-backed Retrieval-Augmented Generation (RAG) system return a flawless summary within two seconds. It cited policy docs, listed processing windows, and earned immediate rollout approval.

Two weeks after launching to active merchant users, support tickets were piling up and executive trust was collapsing.

A merchant had logged into their portal and entered a real operational query

Why was transaction TXN-48291 declined, and has the refund for it posted yet?

The system returned a confident, step-by-step breakdown of why the transaction failed and stated that a $140.00 credit was posted yesterday morning. It looked authoritative. It read cleanly.

There was only one problem

TXN-48291 was never refunded.

When I audited the retrieval pipeline, the root cause became obvious. The system did not degrade two weeks after launch. It was set to fail from day one. It simply took fourteen days for real users to start asking questions that exposed the underlying structural gap.

Prototype/Demo Queries (Easy ~20%)

“How do refunds usually work on our platform?”

“Summarize our terms for chargeback disputes.”

Fuzzy, conceptual language → Vector Similarity Search Works Perfectly (~95% Success)

Production Queries (Hard ~80%)

“Why was transaction TXN-48291 declined, and has the refund posted?”

“Which fee schedule applies to merchant M-902 under international clearing?”

Exact transaction IDs & live ledger integers → Vector Search Fails (Wrong status returned)

Semantic Similarity vs Structural Exactness

Most AI prototypes pass internal reviews because demo queries naturally cluster around what vector search does well, fuzzy linguistic closeness.

Vector search embeds text chunks into high-dimensional mathematical space. When a user asks a conceptual question like “How do refunds work?” the embedding engine easily finds chunks containing “merchant refund processing terms” or “standard return windows.” The vocabulary matches, the cosine similarity score is high, and the LLM synthesizes a clean response.

Production environments do not ask conceptual questions. They ask for operational truth.

When a query demands structural exactness, vector search breaks down across two physical failure modes

Semantic similarity scores vocabulary closeness, not transaction compatibility

An embedding model has no concept of non-interchangeable financial records or primary keys. To an embedding model, the text describing a declined card error on one transaction and a settled refund on another looks almost identical. The vector engine returns a high similarity score for the wrong transaction because the surrounding sentences share the exact same vocabulary.

Live account state is a dynamic integer, not a static concept

Ledger balances and refund statuses sit inside an SQL database or core banking system, changing second by second. They do not exist inside a static PDF or Markdown file stored in a vector index. Vector search either hallucinates a status based on historical text or admits it cannot query the database.

The demo sampled the easy region of the query distribution. Production sampled all of it.

Here are 3 audits you can run today

If your AI feature is stalling between a successful staging demo and production adoption, run these three diagnostic checks against your system today.

1. Retrieval Accuracy

Symptoms - High success on general queries, dangerous hallucinations on exact IDs.

Root Cause - Using vector similarity for questions requiring structural graph traversal.

Do a Query Split test

Pull 10 real support tickets, user logs, or operational inquiries from the last month. Do not use queries picked by the engineering team for a demo.

Split those 10 queries into two piles

• Pile A (Conceptual/Fuzzy) - Broad questions like “How does our chargeback policy work?” or “Summarize the onboarding steps for new merchant accounts.”

• Pile B (Exact/Structural) - Questions containing specific transaction IDs, account numbers, error codes, or requests for live data (“Has the settlement for TXN-48291 cleared yet?”).

Run both piles through your RAG system.

If Pile A achieves a high accuracy score while Pile B produces wrong transaction details, missing statuses, or hallucinated numbers, your failure mode is structural. You are attempting to solve exact relational lookups using vocabulary proximity.

2. Scaled Latency

Symptoms - Sub-100ms in staging, multi-second spikes under production document volume.

Root Cause - Unindexed scans or pointer-chasing over unoptimized vector indexes.

Do a Load Testing against Realistic Volume

Measure retrieval latency against your actual production document count and concurrent user volume, not the 50 sample PDFs sitting in your staging environment.

Staging environments hide infrastructural bottlenecks. A flat vector search or an unindexed query responds in 30 milliseconds when scanning 1,000 document chunks.

When that same index scales to 500,000 chunks across millions of transaction logs and merchant accounts, memory bandwidth chokes and latency spikes into seconds. If you did not load-test retrieval performance at 10x your expected document volume, your production latency figures are an illusion.

3. Cost Per Query

Symptoms - Token bills exploding linearly with active user adoption.

Root Cause - Routing 100% of queries unconditionally to expensive frontier models.

Check Cost Per Query vs Cost Per Token

Examine your API dashboard and calculate what percentage of incoming user queries are routed directly to a top-tier frontier model ($15+ per million tokens).

In our analysis of token economics and agentic loops, we showed that enterprise teams using naive, unrouted architectures pay a massive “token tax.” Unconditional routing forces a $15/1M token model to handle trivial tasks like parsing a regex or reading a transaction ID.

Teams that implement tiered routing, directing routine lookups to local models, database queries, or cheaper Flash-tier endpoints, and escalating only complex reasoning to frontier models, reduce their blended API costs by a median 87.4%.

If every query in your pipeline hits the same frontier model regardless of complexity, your cost model will choke as soon as active user adoption spikes.

What it takes to fix each failure mode

Moving an AI feature from a fragile prototype to a production-grade system requires targeted architectural corrections

Retrieval Accuracy Fix

Flat Vector Store Only (Vocabulary proximity) → Knowledge Graph / Relational Grounding (Merchant → Transaction → Ledger → RefundStatus)

Move beyond flat vector stores for operational data. Map your domain entities into a relational or Knowledge Graph structure (Merchant -[:INITIATED]-> Transaction -[:HAS_LEDGER_ENTRY]-> LedgerStatus). When a query contains an exact identifier like TXN-48291, bypass vector embeddings entirely and execute a deterministic database or graph lookup.

Scaled Latency Fix

Brute-force Flat Scans (Thrashes RAM/Disk) → Entity-First Deterministic Traversals (Bypasses vector search when exact keys exist)

Implement Entity-First Deterministic Traversals. Anchor your queries directly onto primary keys in your database or graph index, reserving vector search strictly as an entry point for unstructured, natural-language queries without explicit identifiers.

Cost Control Fix

Single-Model Pipeline (All queries → Frontier) → Tiered Complexity Router (Edge / DB Lookup → Flash Model → Frontier Escalation)

Build a Complexity Router at the front door of your system. Classify incoming user requests before sending them to an LLM. Simple database lookups or exact status fetches should execute at near-zero API cost; only multi-step, ambiguous reasoning tasks should escalate to frontier models.

Conclusion

A working prototype proves that an LLM can understand your domain. It does not prove that your architecture can serve your users safely.

Prototypes evaluate best-case scenarios under controlled conditions. Production subjects your system to the full complexity of your operational environment, where wrong answers destroy user trust, high latency frustrates merchants, and unconstrained token usage drains engineering budgets.

Stop testing your AI features against the easy ~20% of queries that look great in an executive demo. Test them against the exact, structured, high-stakes questions your team faces every day, and build the underlying architecture required to answer them correctly.

A manufacturing plant deployed an AI assistant backed by a traditional vector RAG system to help technicians investigate, plan, and execute routine maintenance across their production lines.

Everything looked completely fine, and leadership was praising the whole deployment. What they did not realize was that the system had a fundamental flaw, one that could become a nightmare on the factory floor at any moment.

Consider how this failure mode unfolds on the factory floor.

A technician stands in front of a down machine, opens the internal AI assistant, and enters a specific query

How do I resolve Fault 414 on the CNC Lathe Model 800, and are the replacement parts in stock?

In a traditional vector-search RAG architecture, the system embeds the query, scans the document store for chunks with the highest cosine similarity, and feeds the top five matches to an LLM.

The system returns a detailed, step-by-step repair guide. It looks authoritative. It reads cleanly.

There is only one problem, the manual it pulled is for the CNC Lathe Model 600.

The Model 600 uses an older control board, a different hydraulic valve assembly, and different torque specifications. Follow that guide on a Model 800, and you strip a spindle drive or blow a pressure seal.

The vector search was not uncertain. The vector search returns a high cosine similarity score. To an embedding model, the text describing Fault 414 on a Model 600 and a Model 800 looks almost identical.

That is the vector trap. Vector search does not understand hardware. It understands vocabulary.

Why it failed?

The failure of vector search on industrial data comes down to two structural flaws

Semantic similarity scores linguistic closeness, not hardware compatibility.

An embedding model has no concept of non-interchangeable components. It only knows that the words describing the Model 600 manual and the Model 800 manual sound nearly identical.

Live inventory is a dynamic integer, not a static concept.

Warehouse stock sits inside an ERP database or inventory log, changing by the minute. It does not live inside a static PDF manual. Vector search either hallucinates a stock count or admits it cannot read the warehouse.

To answer operational questions on a factory floor, you cannot rely on linguistic closeness. You need structural truth.

How we have solved it?

To solve this, we replaced pure vector search with a Knowledge Graph mapped directly to the physical structure of the plant.

The schema separates the physical world (equipment, parts, faults, live logs) from the document world (manuals, chunks, blueprints), connecting them through explicit, typed relationships.

Schema

// Nodes
Equipment{id, model, installationDate, status} // Physical machines on the floor
Part{sku, name, material, lifecycleStatus} // Replaceable components
Supplier{supplierId, name, country, status} // Equipment and component suppliers
FaultCode{code, severity, description} // Standardized machine error states
Procedure{id, title, estimatedTime, safetyLevel} // Standard fixes
InventoryLog{transactionId, quantity, timestamp, location} // Real-time warehouse movement
Project{projectId, name, startDate, status} // Capital upgrades, maintenance work
Manual{documentId, version, language} // Top-level equipment documentation
ProjectDoc{docId, type, author} // Blueprints, compliance reports
Chunk{chunkId, text, embedding, pageNumber} // Vectorized text segments

// Relationships
(Equipment)-[:ASSEMBLED_WITH]->(Part)            // Bill of materials
(Equipment)-[:TRIGGERS_ERROR]->(FaultCode)       // Telemetry and failure history
(Supplier)-[:SUPPLIES]->(Part)                   // Vendor source for components
(FaultCode)-[:RESOLVED_BY]->(Procedure)          // Problem to solution mapping
(Procedure)-[:REQUIRES_PART]->(Part)            // Materials needed for a fix
(Procedure)-[:DETAILED_IN]->(Manual)             // Grounds the fix in documentation
(InventoryLog)-[:TRACKS_STOCK]->(Part)           // Current warehouse availability
(Project)-[:MODIFIES]->(Equipment)               // Assets under active maintenance work
(ProjectDoc)-[:DOCUMENTS]->(Project)             // Paperwork linked to an initiative
(Chunk)-[:EXTRACTED_FROM]->(Manual)              // Document lineage
(Chunk)-[:EXTRACTED_FROM]->(ProjectDoc)          // Document lineage
(Chunk)-[:MENTIONS_EQUIPMENT]->(Equipment)       // NLP-extracted semantic link
(Chunk)-[:MENTIONS_PART]->(Part)                 // NLP-extracted semantic link
(Chunk)-[:MENTIONS_FAULT]->(FaultCode)           // NLP-extracted semantic link
(Chunk)-[:MENTIONS_PROCEDURE]->(Procedure)       // NLP-extracted semantic link
(Chunk)-[:MENTIONS_SUPPLIER]->(Supplier)         // NLP-extracted semantic link
(Chunk)-[:NEXT_CHUNK]->(Chunk)                   // Sequential reading order

Hybrid Retrieval Pattern

We have not abandoned vector search entirely while implementing the Graph RAG architecture. We have assigned Vector Search and Graph Traversals to two distinct Jobs.

Vector first with Graph expansion

When a query contains natural language without clear primary keys, vector search acts as an entry point to find candidate chunks, which are then expanded through graph edges to fetch grounded domain context.

CALL db.index.vector.queryNodes('chunk_vector_index', 5, $question_embedding)
YIELD node AS chunk, score
MATCH (chunk)-[:MENTIONS_EQUIPMENT]->(e:Equipment)
OPTIONAL MATCH (e)-[:ASSEMBLED_WITH]->(part:Part)
RETURN 
  chunk.text AS manual_text, 
  e.model AS model, 
  collect(part.name) AS components, 
  score

Entity first deterministic traversal

When a query contains explicit identifiers, such as a specific machine model or a fault code, vector embeddings are bypassed entirely. The system anchors directly onto the target nodes and walks the explicit relationships.

MATCH (e:Equipment {id: $equipment_id})-[:TRIGGERS_ERROR]->(f:FaultCode {code: $fault_code})
MATCH (f)-[:RESOLVED_BY]->(p:Procedure)-[:REQUIRES_PART]->(part:Part)
OPTIONAL MATCH (part)<-[:TRACKS_STOCK]-(inv:InventoryLog)
MATCH (p)-[:DETAILED_IN]->(m:Manual)<-[:EXTRACTED_FROM]-(c:Chunk)
RETURN 
  p.title AS procedure, 
  part.sku AS required_part, 
  sum(inv.quantity) AS stock_level, 
  collect(c.text) AS documentation

How it helped?

So, now our schema and Retrieval patterns are set, let us return to the original question

How do I resolve Fault 414 on the CNC Lathe Model 800, and are the replacement parts in stock?

Instead of passing this text to a vector index and praying it does not pick the Model 600 manual, the system extracts the entities (model: 'CNC Lathe Model 800', fault: '414') and runs a single graph traversal

MATCH (e:Equipment {
  model: 'CNC Lathe Model 800'
})-[:TRIGGERS_ERROR]->(f:FaultCode {code: '414'})
MATCH (f)-[:RESOLVED_BY]->(p:Procedure)
MATCH (p)-[:DETAILED_IN]->(m:Manual)<-[:EXTRACTED_FROM]-(c:Chunk)
OPTIONAL MATCH (p)-[:REQUIRES_PART]->(part:Part)
OPTIONAL MATCH (part)<-[:TRACKS_STOCK]-(inv:InventoryLog)
RETURN 
  e.model AS Equipment,
  f.code AS Fault,
  p.title AS ResolutionProcedure,
  part.name AS PartNeeded,
  coalesce(sum(inv.quantity), 0) AS StockAvailable,
  collect(DISTINCT c.text) AS ManualChunks

Look at what this traversal guarantees that vector search structurally cannot

Zero Path to Model 600

The traversal starts specifically at the Equipment node where model = 'CNC Lathe Model 800'. The Model 600 manual does not exist in this subgraph. There is no mathematical possibility of returning the wrong manual.

Real-Time Stock Calculation

The stock count is not pulled from a text chunk written three years ago. It executes a live sum() over active InventoryLog nodes attached to the required part.

Deterministic Prompt Context

The payload handed to the LLM contains explicit key-value parameters (exact procedure title, part SKU, current stock integer) alongside the verified manual chunks.

What it unlocks further?

Once the physical structure of a plant lives inside a graph, you can answer complex operational questions that vector search cannot even attempt.

Supplier Swaps and Downstream Impacts

If we swap the hydraulic pump from supplier A with supplier B on the main assembly line, what downstream procedures need rewriting, and do we have the required new safety manuals already indexed?

A vector search cannot solve this because “downstream impact” is a structural concept.

In Graph RAG, the query traverses Part (Supplier A) <-[:REQUIRES_PART]- Procedure -[:DETAILED_IN]-> Manual and compares it against Part (Supplier B) relationships across active Equipment nodes.

Preemptive Maintenance across Subassemblies

Find every active piece of equipment that shares a subassembly with the conveyor system that failed yesterday, and check if the warehouse has enough replacement belts to preemptively service all of them tonight.

This query spans four distinct domains, failure history, bill-of-materials, active asset status, and real-time inventory. Vector embeddings cannot calculate shared subassembly topologies across disparate equipment types.

A graph traversal handles it in a single pass.

The overall impact

After three months of monitoring the deployment, operational feedback showed a fundamental shift in how technical teams interacted with system documentation.

Before moving to Graph RAG, resolving a multi-domain technical question meant working across departmental silos. A technician had to check paper manuals or PDFs for repair steps, open an ERP terminal to check part inventory, and consult a maintenance log to verify equipment revisions, often taking hours or days of manual coordination for a single complex query.

After deploying the graph architecture, cross-domain queries resolved in a single traversal. Technicians received verified, model-accurate documentation alongside real-time inventory counts without jumping between systems.

Conclusion

Vector search is excellent for unstructured natural language search bars. It is dangerous when used as the primary lookup engine for physical, high-stakes infrastructure.

When a wrong answer means damaged hardware, line downtime, or safety hazards, you cannot rely on linguistic proximity. Connect your domain entities, map your relationships, and let the graph supply the truth.

In May 2026, Vercel published a massive, coordinated security release addressing 13 separate advisories across Next.js.

Tucked deep inside the release notes was one line that should make every engineering lead pause

App Router segment-prefetch bypass, incomplete fix follow-up.

Read that phrase again. Vercel didn’t just patch a vulnerability. They patched a patch. A security fix shipped to resolve an authorization bypass failed to close the underlying gap, requiring a second emergency release to clean up what the first one missed.

Alongside it in the same May snapshot were three more advisories carrying the exact same mechanism: an auth bypass via App Router segment-prefetch URLs, a Pages Router i18n default-locale path bypass, and a dynamic route parameter injection flaw. Vercel’s advisory statement was unusually direct

Affects applications that rely on middleware.js or proxy.js for authorization.

Worse, Vercel noted that unlike previous high-severity bugs, they had not deployed platform-level WAF rules for this release because these advisories could not be reliably blocked at the WAF layer. If your application relied on middleware to enforce access controls, platform-level network shields couldn’t save you. You had to patch your application code.

Five distinct authorization bypasses have hit Next.js in fourteen months. They aren’t isolated accidents. They are the predictable output of an architectural mismatch.

Where this started?

Most teams remember when the dam broke.

On March 21, 2025, security researchers disclosed CVE-2025-29927, a CVSS 9.1 critical vulnerability affecting virtually every active production version of Next.js.

The root cause was disarmingly simple. Next.js used an internal header x-middleware-subrequest to prevent infinite loops when middleware invoked sub-requests. The framework trusted this header implicitly. If an incoming HTTP request contained x-middleware-subrequest: true, Next.js assumed middleware had already executed and skipped middleware.js entirely.

Attackers didn’t need a zero-day payload or a buffer overflow. They added a single HTTP header to their request, bypassed every authentication gate living inside middleware, and hit protected backend routes directly.

Vercel protected Vercel-hosted deployments at the platform edge almost immediately, but self-hosted applications running next start with output: "standalone" were left wide open until maintainers manually updated their dependencies.

In their post-mortem analysis of the March 2025 flaw, Datadog Security Labs reached a conclusion that foreshadowed everything to come

Middleware should supplement, not replace, robust security measures placed closer to the data source.

The industry treated CVE-2025-29927 as a one-off header-spoofing bug. They updated their package locks, closed the ticket, and went back to work.

That was a mistake.

What happened next?

Over the next fourteen months, the bypasses kept coming. Each arrived with a new CVE number, a new technical description, and the exact same structural flaw.

Look at the timeline

Pages Router i18n Bypass (CVE-2026-44573)

In Pages Router applications with internationalization (i18n) enabled, middleware evaluated standard request paths but failed to recognize the locale-less data route at /_next/data/<buildId>/<page>.json. A request sent directly to that JSON endpoint bypassed middleware entirely, returning full server-rendered JSON payloads—including whatever sensitive data getServerSideProps fetched—without firing authentication checks.

Dynamic Route Injection (CVE-2026-44574)

Patched in the May 6, 2026 coordinated release. Middleware inspected the visible URL path, a public route like /safe and approved the request. But the App Router resolved the underlying page using internal, injected routing parameters that pointed to a protected route. Middleware evaluated what it saw; the framework rendered what was injected.

App Router Transport Exploits (CVE-2026-44575)

Patched in the same May 2026 window. This flaw exploited internal App Router transport formats, specifically .rsc and segment-prefetch requests. By requesting alternate payload formats directly, attackers bypassed the matching logic configured inside middleware.

July 2026 Security Release

Vercel patched yet another authorization bypass specific to App Router applications compiled with Turbopack containing a single entry in config.i18n.locales.

So, what is the issue?

We should not treat these as five separate bugs. They are five different expressions of a single structural gap

Middleware checks one representation of a request, while the App Router resolves the page using a different, parallel representation.

Middleware evaluates a visible URL, a header string, or a matcher regex pattern. But the App Router is a complex, multi-transport engine handling Server Components, prefetch segments, JSON data routes, and locale rewrites.

Whenever the representation checked by middleware fails to match the representation resolved by the renderer, the check breaks. Patching the matcher doesn’t solve the problem. It just waits for the next transport feature to introduce another mismatch.

Not just a security flaw but a design mismatch

To understand why this gap exists, you have to look at how middleware was engineered.

Middleware in Next.js runs on the Edge Runtime. It executes on every single matched request before the request touches the Node.js server or rendering engine.

Because it sits on the edge and intercepts every incoming request, performing heavy database calls or session lookups inside middleware creates a massive performance penalty. If every static asset, API call, and page navigation has to wait for an edge function to query a database across the network, latency spikes across your entire application.

For years, Next.js tutorials and community guides recommended keeping middleware lightweight. Check if a session cookie exists, verify its signature, and redirect unauthenticated users etc.

Teams took that guidance and made an operational leap. They turned middleware into their primary access control layer. They assumed that if middleware approved the request, downstream Server Components and API routes were safe.

That assumption violated the core constraints of the platform. Middleware was designed as a fast pre-flight filter to speed up user redirection, not an authoritative policy enforcement point. Teams used it as a security guard because it was convenient, ignoring the fact that the framework’s own rendering engine had backdoor paths around it.

Do we have a fix?

Vercel didn’t just ship patches for these CVEs. They overhauled their official documentation to tell developers to stop using middleware as an authorization boundary.

Next.js’s current official security guide explicitly mandates the Data Access Layer (DAL) pattern.

Instead of trusting the routing layer to block unauthorized requests, authentication and authorization checks must sit inside dedicated, server-only modules placed directly next to database calls.

The documentation applies this rule to both reads and mutations

Just as we recommend a Data Access Layer for reading data, you can apply the same pattern to mutations. This keeps authentication, authorization, and database logic in a dedicated server-only module, while use server actions stay thin.

Vercel’s current security audit checklist asks point-blank

Is there an established practice for an isolated Data Access Layer? Verify that database packages and environment variables are not imported outside the Data Access Layer.

This marks an acknowledged shift in official framework guidance. Community guides and vendor documentations now carry explicit warnings. Middleware is no longer considered safe for primary authorization.

A useful mental model for this shift comes from security tooling provider WorkOS. Middleware is the bouncer at the front door; the Data Access Layer is the gate agent checking the boarding pass.

The bouncer can be tricked by a fake ID, a back-door entrance, or a bad matcher rule. But even if an attacker tricks the bouncer and slips inside the building, the gate agent stands directly in front of the airplane door. If they don’t have a valid boarding pass attached to that specific seat, they don’t get on the plane.

In code, that means moving from this

// ❌ DANGEROUS: Relying on middleware to protect this component
export default async function AdminDashboard() {
  const data = await db.adminStats.findMany(); // Assumes middleware already checked auth
  return <AdminView data={data} />;
}

To this

// ✅ SECURE: Authoritative check inside the Data Access Layer
import { verifyAdminSession } from ‘@/app/lib/dal’;

export default async function AdminDashboard() {
  // DAL function verifies session & permissions directly at execution time
  const data = await verifyAdminSession(async () => {
    return await db.adminStats.findMany();
  });

  return <AdminView data={data} />;
}

Conclusion

If you configured your Next.js authentication model in 2023 or 2024 by dropping check logic into middleware.ts and assuming your backend routes were secure, your architecture is built on outdated guidance.

Five advisories in fourteen months proved that URL matchers and edge proxy checks cannot guarantee that a request won’t resolve to a protected payload.

Here is your remediation checklist

1. Audit middleware.ts immediately - Treat middleware strictly as an optimization tool for fast redirects and preliminary cookie presence checks. Remove all hard security assumptions from it.

2. Build an isolated Data Access Layer - Wrap every database query, internal API call, and Server Action mutation in dedicated server-only functions (import 'server-only') that execute explicit permission checks before returning data.

3. Never trust request context - Verify permissions using server-side session tokens retrieved directly inside the Data Access Layer, not passed-down custom headers or route parameters.

Vercel spent fourteen months patching edge bypasses before updating their docs to tell you what security architects knew all along. Never enforce security policies at the routing layer.

Update your code dependencies. But more importantly, update your architecture.

In mid-December 2025, Amazon’s autonomous AI coding agent, Kiro, decided to delete and recreate a live production environment.

The environment was AWS Cost Explorer in the mainland China region. The resulting outage lasted 13 hours.

Kiro did not execute this wipe through a zero-day exploit or a hallucinated terminal command. It inherited the elevated permissions of the senior engineer running the session. That inheritance bypassed the standard two-person sign-off requirement designed to block destructive actions on live production infrastructure.

When Amazon released its public statement on February 21, 2026, the company attributed the incident to “user error, specifically misconfigured access controls.” The tool itself was not named as the root cause. The human who configured the credentials was.

The interesting fact is not that an autonomous script wiped a database. Automation has been breaking production environments since shell scripting was invented.

The real story is that the guardrail existed on paper, failed in practice, and was ignored in executive reviews because the organization was optimizing for a completely different number.

What happened before the incident

Three weeks before Cost Explorer went down in China, Amazon’s engineering leadership set a target.

On November 24, 2025, an internal memo signed by Peter DeSantis (SVP of AWS Utility Computing) and Dave Treadwell (SVP of eCommerce Foundation) established Kiro as Amazon’s standardized internal AI coding tool.

The memo set an explicit corporate OKR - 80% weekly active adoption across engineering teams by the end of the year. The metric was tied directly to team and individual performance evaluations. Any exception required explicit VP approval.

Amazon’s engineers did not accept the mandate silently.

Roughly 1,500 engineers signed an internal petition objecting to the forced rollout before the December outage occurred. The petition argued that external tools specifically Claude Code and Cursor substantially outperformed Kiro on complex, multi-language refactoring tasks. They warned leadership that forcing adoption of an immature internal tool across production codebases introduced structural risk.

Leadership moved forward anyway. By January 2026, roughly 70% of Amazon’s engineering staff had used Kiro.

From an executive dashboard perspective, the rollout was a massive success. Amazon reported that Kiro shortened pull request cycles by 28%, backed by 21,000 AI agents deployed across Amazon Stores and an estimated $2 billion in development cost savings.

The executive layer had full visibility into the engineers’ warnings. They had the budget, the platform team, and 21,000 active agents. But they chose to measure adoption percentage instead of verification capacity.

Who to blame?

The explanation for the December outage depends on who you ask.

Amazon’s official stance remains firm

The December incident was a human operational mistake. An engineer misconfigured access controls, allowing Kiro to operate with permissions it should never have possessed.

Anonymous sources who spoke to the Financial Times described a structural flaw in how Kiro handled identity. Under their account, Kiro was designed to inherit the execution context of the logged-in user. When a senior staff engineer with production write access initiated a session, Kiro implicitly inherited those same rights. The tool did not maintain an independent boundary between “suggesting a change” and “executing an administrative override.”

Both claims describe the exact same physical reality from different angles.

A human misconfigured an access policy. And the automation framework executed a destructive, multi-step infrastructure teardown without an independent execution gate to verify whether the command was authorized.

Treating credential inheritance as “user error” misses the architectural flaw. If an AI agent can execute a production wipe simply because the human sitting at the terminal holds admin rights, the system has no verification layer. It relies entirely on human memory to prevent automated destruction.

What happened next?

The December outage did not slow the mandate. It took a second, vastly more expensive failure to alter the policy.

On March 5, 2026, Amazon.com’s primary retail site went down for six hours. The outage hit checkout, pricing services, and account management, resulting in an estimated 6.3 million lost orders.

The root cause was traced to a faulty software deployment following AI-assisted code changes. Between December 2025 and March 2026, internal documentation recorded at least four separate Severity-1 production incidents linked directly to a pattern of AI-assisted changes shipping without sufficient oversight.

Only after the March 5 retail crash did Amazon alter its engineering governance.

The company instituted a mandatory policy requiring explicit senior engineer sign-off on all AI-assisted changes authored by junior staff.

The safety measure that took effect after March 5 could have been implemented on November 24. Nothing prevented it except the organizational decision to prioritize adoption metrics over verification controls. As of late March 2026, the 80% Kiro adoption mandate remains in place.

The Acceleration Whiplash benchmark

Amazon’s experience is an exaggerated version of what is happening across the entire software industry.

When you collapse the cost of generating code without changing how you check it, the system chokes at the review step.

Data from Faros AI’s 2026 report, The Acceleration Whiplash which benchmarked 22,000 developers across roughly 4,000 engineering teams puts precise numbers on this operational bottleneck

• Code Review Lead Time - Median review time climbed 441.5% year-over-year.

• Developer Output - Individual task throughput rose 33.7 percent over the same period.

• Unedited AI Acceptance - The share of AI-generated code developers merged without editing jumped from 20 percent to 60% in twelve months.

• Work in Progress (WIP) Stagnation - 26% more in-progress tasks now sit completely idle for seven days or longer.

Complementary data from LinearB’s 2026 benchmark of 8.1 million pull requests shows the same structural imbalance

AI-assisted PRs are roughly 2.5 times larger than human-authored PRs and wait 5 times longer for a peer review.

CircleCI’s 2026 telemetry reveals the logical endpoint of this pattern

while feature branch throughput surged 59% year-over-year, main branch deployment throughput for the median engineering team actually declined.

We saw this exact mechanism crash cURL’s bug bounty program when automated security reports flooded maintainers’ inboxes. Generation velocity exploded, verification capacity stayed fixed, and the system collapsed under the weight of unverified volume.

So, how to Audit/Verify AI Generated code?

To build an engineering organization that survives AI generation, you have to separate verification into two distinct problems. They require completely different architectural tools.

1. Verifying Code Correctness at Merge Time

This is the traditional domain of code review and static analysis. The goal is catching bad code before it merges into main.

Deterministic Security Scanning

Rather than asking an LLM to review another LLM’s code, point deterministic static analysis tools (SAST) at the four specific vulnerability categories that account for the majority of AI security failures

• SQL Injection (CWE-89)

• Cross-Site Scripting (CWE-80)

• Log Injection (CWE-117)

• Insecure Cryptographic Algorithms (CWE-327)

The Structured Review Standard

FlowVerify’s 2026 audit framework establishes a five-part checklist for human reviewers handling AI-generated pull requests

1. CI Audit - Confirm pipeline pass with no ignored test failures.

2. Duplicate Code Search - Scan for re-implemented internal utilities.

3. Hand-Traced Path - Manually walk at least one execution path line-by-line.

4. Targeted Security Check - Verify inputs against the four CWE categories above.

5. Test Evidence - Confirm new tests actually exercise the modified execution path, rather than just asserting true == true.

Package Verification

AI models routinely hallucinate dependency names. Before any generated import statement ships to production, run an automated pre-screening check

• Verify the package exists on the canonical registry (e.g., npm, PyPI).

• Confirm the package name matches the intended library rather than a typosquatted malicious domain.

• Ensure the package is explicitly needed and pinned in the lockfile.

Correcting the Engineering Metrics

Amazon measured adoption percentage and merged PRs. Neither metric catches a broken access control. Engineering teams must replace adoption KPIs with two operational metrics

• Revert Rate on AI-touched pull requests vs. human-authored pull requests.

• Review Lead Time across active feature branches.

2. Gating Agent Authority at Action Time

This was Kiro’s failure mode. The delete command that took down AWS Cost Explorer did not fail because of bad syntax. It failed because an automated process executed an unauthorized, destructive action in real time.

A perfect code review the week before would not have stopped it.

Risk-Tiered Autonomy

Implement execution gates based on the potential blast radius of the action

• Tier 1 (Low Risk) - Read-only operations, local scaffolding, unit test generation. Full agent autonomy; no human gate.

• Tier 2 (Medium Risk) - Reversible changes to non-critical development environments. Autonomous execution with mandatory asynchronous review after the fact.

• Tier 3 (High Risk) - Destructive operations, schema modifications, production deployments, or elevated credential access. Mandatory synchronous human approval at the moment of execution.

Non-Inherited Execution Credentials

An AI agent must never implicitly inherit the full permission scope of the human engineer operating the terminal. This was the exact credential gap that allowed Kiro to bypass Amazon’s two-person sign-off rule. Agent execution contexts must use short-lived, tightly scoped service accounts bounded by explicit action policies.

Progressive Delivery & Canary Rollouts

Combine execution gates with automated canary deployments. Route 1% to 5% of live traffic to AI-modified services, backed by automated rollback triggers tied directly to error rates and tail latency metrics.

Executable Specifications as Quality Gates

Recent academic research into AI-assisted review shows that converting written requirements into executable specifications turns passive guidelines into strict enforcement checks (arXiv:2603.25773). When an AI agent generates code against an executable specification, the system validates the output against deterministic tests before a human reviewer ever opens the PR.

Note: The authors of the arXiv specification study explicitly note that their benchmark relied on a planted bug corpus rather than a natural production defect sample, making their findings directional rather than definitive

Conclusion

Amazon possessed every organizational advantage required to build a safe verification layer.

They had an elite platform engineering org, billions in infrastructure capital, 21,000 active AI agents, and a petition signed by 1,500 senior engineers explicitly pointing out where the internal tool was failing.

They had the signal. They overruled it to hit an 80% adoption metric on an executive OKR slide.

The cost of verifying code was always present. For thirty years, it was hidden because writing code was slow enough that verification happened naturally alongside creation. AI severed that relationship. It allowed any team to generate thousands of lines of syntactically valid code in seconds, while dumping the cognitive load of auditing that output onto human reviewers.

You can measure adoption percentage, force usage OKRs, and celebrate short-term pull request velocity. Or you can build the verification gates, risk-tiered permissions, and review pipelines required to run AI agents safely in production.

If you skip the verification layer to hit an adoption target, the system will eventually enforce the cost for you—in lost orders, degraded review cycles, and 13-hour production outages.

In a single 16-hour window in January 2026, seven security reports landed in Daniel Stenberg’s inbox.

Not one of them identified a real vulnerability.

Stenberg maintains cURL, the command-line tool that sits at the foundation of almost every connected software system on Earth. Since April 2019, curl’s bug bounty program on HackerOne had been a model for open-source security.

Over seven years, it paid out more than $100,000 to independent researchers across 87 verified vulnerabilities. Historically, better than 15% of incoming reports led to a confirmed fix.

By 2025, that confirmation rate collapsed below 5%.

In the first three weeks of 2026, cURL received 20 submissions. Zero were valid. Large language models had turned the labor of writing plausible-sounding security reports into a zero-cost operation. Stenberg called it “death by a thousand slops.”

His assessment was blunt

Not only the volume goes up, the quality goes down. So we spend more time than ever to get less out of it than ever.

On January 31, 2026, cURL officially shut down its HackerOne bug bounty.

Most commentators took this as a simple cautionary tale about AI ruining open source. They missed the real story. Three months later, the underlying math did something far more uncomfortable.

So what happened next?

In March 2026, just four weeks after shutting it down, Stenberg reopened cURL’s HackerOne bounty program.

It wasn’t because the submission traffic had died off. It hadn’t. What changed was the underlying model quality.

As reported by Cybernews in June 2026, the arrival of newer reasoning models suddenly shifted the nature of the incoming queue. The low-effort slop didn’t disappear, but it was joined by a wall of highly sophisticated, syntactically correct, deeply detailed vulnerability reports.

Stenberg watched the inbox fill up again

Something happened. The reports came back, and they did not slow down.

Here is the what actually happened, reopening the program didn’t fix Stenberg’s workload. It made it even heavier.

When the queue was filled with obvious garbage, a human maintainer could skim a report in 30 seconds and throw it out.

But when an AI generates a 20-page security teardown complete with reproduction scripts, edge-case memory maps, and pseudo-code fixes, you cannot dismiss it out of hand. A human senior engineer has to spend two hours stepping through gdb traces to verify whether the flaw is real or a hallucination.

The cost of producing the report fell to zero. The cost of auditing that report stayed exactly where it was in 1995.

This is the central asymmetry of the generative AI era. We collapsed the cost of creating technical artifacts, but we didn’t touch the cost of verifying them. The bottleneck didn’t vanish. It just shifted from generating the work to checking it.

What Industry reports say?

If this were isolated to open-source maintainers, it would be a manageable niche headache. It isn’t. The exact same gap is breaking application security at enterprise scale.

On August 1, 2026, security firm Veracode published its annual GenAI Code Security Report. They benchmarked over 100 frontier and open-weight models across four distinct testing cycles dating back to early 2025.

The findings put hard numbers on what every senior architect has felt intuitively

• Syntax Pass Rate - Near 100%. Models generate code that compiles, runs, and passes basic unit tests without throwing errors.

• Security Pass Rate - 56%.

That 56% figure is virtually unchanged from the 55% mark Veracode recorded in their initial 2025 benchmark.

Despite trillion-parameter scaling, massive context windows, and advanced reasoning loops, roughly 44% of AI-generated code tasks still introduce a known, exploitable vulnerability when generated without explicit security prompts.

Even the top-performing model in their Summer 2026 snapshot GPT-5.5 achieved only a 68% security pass rate. It still failed nearly one out of every three security tasks. For Java workloads, the mean security pass rate dropped to a dismal 30%.

Veracode’s report framed the problem directly

Models may be almost syntactically perfect, but they’re still failing on nearly half of all tasks where security is needed.

This structural blind spot is hitting production networks right now.

The 2026 Verizon Data Breach Investigations Report revealed that software vulnerabilities have officially passed stolen credentials as the number one entry point for enterprise data breaches, accounting for 31% of all confirmed security incidents.

Meanwhile, industry surveys (including data compiled across Veracode and GitClear) indicate that AI-generated pull requests carry roughly 1.7 times more defects than human-written code. Yet, only 18% of enterprises have instituted formal, mandatory AI code review policies.

We are shipping clean-looking code faster than ever before. We are checking it less than ever before.

How its affecting the Job Market?

When the cost of producing code drops while the cost of verifying it climbs, the job market shifts.

The broad media narrative claims that entry-level engineering is dead and AI has eliminated the talent shortage. The actual hiring data tells a completely different story.

According to ManpowerGroup’s 2026 Global Talent Shortage Survey which polled 39,063 employers across 41 countries roughly 72% of global tech employers still report severe difficulty filling technical roles.

For the first time in the survey’s history, traditional IT and software development skills dropped out of the top spot. They fell to seventh place globally (17% of employers reporting shortages).

What took their place at the absolute top of the global talent shortage list?

1. AI Model and Application Development (20%)

2. AI Literacy and Verification Capability (19%)

Jonas Prising, Chair and CEO of ManpowerGroup, noted

The rise of AI skills to the top of the shortage list reflects how quickly the talent landscape is evolving.

The market isn’t looking for people who can write code fast. Copilots do that for free. The market is desperately searching for engineers who possess the deep structural knowledge required to audit, verify, and architect systems around AI-generated code.

The hiring process itself is changing

Nowhere is the verification crisis more visible than in how companies hire.

In early 2025, a Columbia University student named Chungin Roy Lee built a lightweight tool called Interview Coder. The application sat hidden during live technical interviews, took real-time screen grabs of LeetCode style coding prompts, and fed back optimal solutions silently in the background.

Lee used the tool to clear technical loops and land job offers from Amazon, Meta, TikTok, and Capital One. Then he posted a video of himself using it live inside an Amazon interview loop.

Columbia suspended him. Amazon revoked his offer. Lee didn’t care. By April 2025, Interview Coder had crossed $3 million in annualized revenue, and Lee raised $5.3 million in venture funding to turn the engine into a broader stealth assistant named Cluely.

The standard technical interview, the algorithm puzzle on a shared screen was officially broken. Data from developer evaluation platform interviewing.io indicated that 81% of technical interviewers suspected candidates of using AI assistance during interviews, with roughly one-third catching a candidate outright.

Faced with a broken signal, the tech industry split into two distinct camps

Strict Interviews

Some legacy enterprises responded by doubling down on monitoring. They dragged candidates back into physical office rooms, enforced strict locked-down browser environments, and added grueling live follow-up interrogations.

They tried to preserve the old LeetCode signal by building bigger walls around it.

Different Verification Style

Other engineering organizations abandoned memory-based syntax tests entirely. Instead of asking a candidate to implement a binary tree inversion from scratch, they hand the candidate a messy, AI-generated pull request containing three subtle concurrency bugs and a memory leak.

The candidate’s score depends entirely on how fast they can spot the hallucinations, correct the underlying architecture, and guide the AI toward a secure implementation.

Initiatives like the open-source Hiring Without Whiteboards repository (now boasting over 50,000 stars on GitHub) reflect this reality. The industry is slowly realizing that testing whether a human can type code from memory is irrelevant when typing code is free.

The only metric that matters is whether the engineer can tell when the code is wrong.

Its all connected

If you have been following BinaryBox over the last few months, this pattern should feel intimately familiar. It is the underlying thread connecting our technical analyses across different domains

The AI Rehiring Panic - We established that AI easily automates the routine 94 percent of standard tasks, but exposes the remaining 6 percent of edge-case judgment as the true bottleneck holding up enterprise operations.

The Tokenpocalypse - We showed how unconstrained agentic execution loops multiply token consumption by 50x to 500x through context rot, proving that raw model generation is worthless without tight local orchestration and strict boundary controls.

Stop Treating MCPs as Plugins - We demonstrated that an AI assistant will output broken imports and duplicated code unless it is constrained by an NX monorepo pipeline that forces structural context into the session.

Every single one of these technical shifts stems from the exact same economic reality

Generation got cheap. What stayed expensive was judgment, orchestration, context boundaries, and verification.

Same mechanism. Five different costumes.

Conclusion

It is easy to look at the current landscape and assume senior engineers are immune to the changes sweeping through software development. That is too neat a conclusion.

Senior engineers who act purely as high-speed typists are just as exposed as junior developers. If your main value contribution is turning a Jira ticket into 200 lines of syntactically valid TypeScript, your margin is evaporating.

The cost of checking work was always present in software development. It was just invisible for forty years because checking code was tightly bound to the effort of writing it. You wrote a function line by line, thinking through the edge cases, memory allocations, and security boundaries as your fingers moved across the keyboard. Writing and checking happened simultaneously.

AI severed that link. It unbundled creation from verification.

It allowed anyone to generate 5,000 lines of syntactically flawless infrastructure-as-code in three seconds. But it left the cognitive burden of verifying that infrastructure entirely on human shoulders.

Daniel Stenberg didn’t shut down the curl bug bounty because AI couldn’t write security reports. He shut it down because AI wrote them so fast that human audit capacity collapsed under the weight.

Generation is now effectively free. Verification is exposed as the ultimate physical constraint of software engineering. Plan your career and your infrastructure accordingly.

More to read

https://daniel.haxx.se/blog/2026/01/26/the-end-of-the-curl-bug-bounty/

https://www.theregister.com/2026/01/21/curl_ends_bug_bounty/

https://www.bleepingcomputer.com/news/security/curl-ending-bug-bounty-program-after-flood-of-ai-slop-reports/

https://cybernews.com/security/curl-bug-bounty-ai-security-reports-daniel-stenberg/

https://www.softwareseni.com/curl-bug-bounty-shutdown-and-the-open-source-incidents-that-proved-the-problem-is-real/

https://www.veracode.com/blog/2026-genai-code-security-report-ai-risk/

https://www.veracode.com/resources/analyst-reports/2026-genai-code-security-report/

https://sdtimes.com/agentic-security/veracode-finds-ai-generated-code-security-has-barely-improved-since-last-year/

https://tech-insider.org/ie/ai-generated-code-security-2026/

https://www.manpowergroup.com/en/news-releases/news/global-talent-shortage-reaches-turning-point-as-ai-skills-claim-top-spot

https://www.hcamag.com/ca/specialization/transformation/ai-skills-top-list-of-hardest-to-find-capabilities-for-employers-worldwide-report/566899

https://spacecomplexity.ai/blog/will-ai-replace-coding-interviews

https://interviewcopilot.co/blog/leetcode-senior-staff-engineers-2026

https://distantjob.com/blog/leetcode-is-dead/

You may have hearing or reading that AI is backfiring or Companies bringing the engineers back or AI is failing or The humans are coming back etc etc. Trust me I am reading these for a quite long time. And I start wondering, is it really?

So being a nerd I started digging deeper and here is what I found.

The trend is divided

Here is the scale of things everyone is talking about.

We all know that roughly 120k tech roles has been cut in the year 2026 and AI was the most cited reason among all.

Read this report by Challenger, Gray and Christmas if feel otherwise.

That report mentioned all the tech giants and their layoffs as they were announced.

PayPal, Microsoft, Meta, Google Amazon and everyone else who cut their workforce in those times marked as AI specific layoffs. But if I look closer that line is still trending downward and companies still laying off their workforce.

So, I started focusing on the other line which is smaller but louder. Here are some context before I get into it

• The Orgvue survey tells us that 39% of business leaders made people redundant because of AI and 55% of those now admit that the decision was wrong.

• Robert Half mentioned that thirty two percent of the hiring managers from US have already rehired someone for a role they cut because of AI.

• Forrester is predicting that half of all AI attributed layoffs will eventually be quietly reversed.

So, I can understand why the conclusion you have made seems backed by the data itself. The industry is still in layoff mode and some of those layoffs are being rehired as well both are true at the same time.

But the story has a lot more to say, actually there are three different failures and each one has a specific engineering cause behind it. Putting them together as AI is failing misses the actual lesson in each case and that is the only part worth keeping for our own deployment decisions.

Failure 1 - The undocumented knowledge

Read this story

Ford rehired and promoted more than 350 engineers.

But what matters is, the why?

The knowledge these engineers has to run design reviews before vehicles reach production were undocumented. They knew the failure patterns which they have experienced, the edge cases which was never logged and things which was never written down but was important. All these existed only in their mind.

So, when Ford trained their system on available data (service records, design specs, test logs etc) was an incomplete version of what those engineers actually knew. And they found that out in a hard way.

Ford’s executives estimated this knowledge worth hundreds of millions to help catch those issues early.

This is not a story where AI is not able to understand vehicle design, it is all about the undocumented knowldge.

I have experienced this first hand with obviously a less expensive consequences.

Every Knowledge Graph I have shipped runs into the same undocumented knowledge wall. The system is only as good as what was recorded and available. The most valuable knowledge in most of the organization are the ones live in a senior person’s head.

Buying better models cannot fix this, you only fix it by writing down the undocumented knowledge before the AI project starts.

Failure 2 - The remaining 6%

Using AskHR (An AI Agent) IBM resolves 94% of the employee request in their HR function without a single human in the loop. That is commendable result and nobody can deny that.

But the remaining 6% is where the actual story is. IBM describe that as Ethical Dilemmas and Complex Situations which does not fit into their policies as cleanly as it was needed. Simply means those requests needed some judgement instead a lookup into the database.

So, IBM does not gave up the their AI Agent but it was to announce that they would triple entry level hiring across their US business in the year 2026.

Another example where Commonwealth Bank of Australia faced the similar problem but in different angle.

They laid off more than 40 of their customer service staff and replaced them with AI Voice Bot. The Bot could not handle the call volume and the cases where human judgement was needed. They reversed the layoffs and admitted that they had not consider all the relevant business cases.

Both these cases are not a capability failures, AskHR is resolving 94% of the cases correctly, the CBA bot was able to process most of the routine calls. What went wrong was the boundary, when they deployed the system without a precise definition of what was suppose to do in those specific scenarios.

The 6% number is completely depends on the judgment specific worjloads which training data cannot encode by itself.

Failure 3 - The single metric trap

I hear a lot about Klarna case and I feel that everybody only reads the headline, if they go past that they would know.

In the year 2024 Klarna’s OpenAI powered Agent handled 2.3 Million customer service conversations in its first month and cut down the resolution time from 11 Min to under 2 Min. The company projected A total of $40 Million in savings that year.

But, within 18 Months they started rehiring and headlines framed this as a reversal. Actually it was a rebalancing, the AI Agent is still in the front lines for the high volume routine tier support. The human agent was to serve the premium tier for complex cases which involves trust, emotion and risk where resolution time is not that important.

Look what the CEO has said about this whole situation. It fits into a clear case of The single metric trap as they were only focused on volume and resolution time and has not consider the customer satisfaction etc in the beginning.

If we pick an easy number to optimize for example resolution time, cost per ticket, tokens per task etc. The data will support you for it but you also need to look at the things you are not measuring, churn, complaints, warranty claims etc. The fix in such scenarios are quite higher then the original savings.

At the end, this case of Klarna does not prove that AI cannot do customer service, it proved that fixing and optimizing for a single metric will eventually fail.

What we should learn from this cases

Each of this headlines and cases are there to teach us something and it only become clear when we dig deeper and beyond the headlines

The five year clock

If you look at the IBM’s case, their CHRO said that investing in entry level hires is a must otherwise in 3 to 5 years there will be no pipeline and the well simply dries up. They are doing headcount calculations on a longer clock then completely relying on AI only solution.

So if we cut that intake today as AI can do the entry level work, we are not eliminating the cost but deferring it into the future with a bigger number.

I myself has scaled engineering org from 3 to 130 people and I have been on the other side of this exact decision to hire or automate. One thing I have learned in the hard way that automating everything is not possible and we need people to manage the complex workflows and systems.

The Duolingo’s case

You may have read that Duolingo’s CEO announced an AI First memo in the April 2025, he stated only hire if a team cannot automate more of its work. He met with immediate backlash, 1000s of comments with 600 reposts and even threats to delete the app. This story and the coverage on it was very wrong in every way that matters.

In fact, Duolingo never laid off a single full time employee. They even keep hiring in the same year the Memo went public.

In the year 2023 and again in 2024 they cut down 10% of their contract workforce of translators and writers. In May 2026 they backtracked on their internal policy to evaluate employees on their AI usages due to employees kept asking if they must use the tools for the sake of using them.

But, the headline said that AI first company reverses course on AI. And the fact was company never touched full time headcount and clarifies on their internal policy after the employee push back.

So, this story was never about employees or AI failing but contractors are being replaced with the internal systems.

Conclusion

Here is what I got and think about this whole AI is backfiring narrative

• The rehiring stories are real and each one of them has a specific case to learn. None of them are evidence that AI does not work and cannot do the tasks.

• Shopify’s headcount has been very flat and its not because of AI failed but they decided to keep a flat headcount. PayPal, Microsoft and Meta are still laying off and data does not show they are wrong.

• The tech headcount in 2026 is still declining and roughly 120k jobs has been cut and not comparable against the rehires.

These stories also gives us a checklist which we can run before we go about deploying AI in our workflow

• Make sure you have all the knowledge each person holding before they leave.

• Not everything can be automated or offload to AI, keeping key persons become important.

• Pick a mixed of metrics which includes quality before you optimize the easy ones.

• Plan for at least 5 years on headcount so that you have an active pipeline.

Further reading1

Count the open tabs sitting on your browser right now

1. Figma to inspect the design tokens and layout rules.

2. VS Code to build the UI component.

3. TablePlus to verify table schemas and foreign key constraints.

4. Library Docs to double-check exact method signatures for your installed packages.

5. Localhost browser to manually refresh and inspect spacing.

6. GitHub to open a pull request and tag the issue.

7. Sentry three days later when a null pointer hits production

Seven tools. Seven context switches. Seven places where your original architectural vision falls apart.

You tab out of Figma into your editor and immediately forget the exact padding tokens. You jump over to a database client to verify a column name and lose the code state in your head. That cognitive drain doesn’t show up in JIRA metrics, but it burns through engineering energy every day.

When Model Context Protocol (MCP) landed, most teams treated it like VS Code extensions. They installed a GitHub MCP to create PRs, or a Figma MCP to read layers. They treated MCPs as individual utility add-ons to give Claude Code a little extra help.

Installing isolated plugins completely misses the real power of the protocol.

MCPs aren’t standalone tools. They are structural building blocks for an automated pipeline. When you configure them as an interconnected graph, MCPs map directly to every phase of your software lifecycle. Instead of hopping between apps, a completed pipeline removes manual tool-switching entirely. Design specs, monorepo graphs, database schemas, library docs, browser execution, version control, and production errors sit inside a single session context.

Here is what a complete full-stack MCP pipeline looks like inside an NX monorepo. Seven stages. Seven MCPs. One continuous session.

THE FULL-STACK MCP PIPELINE (NX MONOREPO)

[Design]       FIGMA MCP          Design intent → component spec
    ↓
[Codebase]     NX MCP             Project graph → workspace layout
    ↓
[Data]         DATABASE MCP       Schema → type-safe queries
    ↓
[Docs]         CONTEXT7 MCP       Live APIs → zero hallucinations
    ↓
[Visual QA]    PLAYWRIGHT MCP     Browser → visual verification loop
    ↓
[Version Ctrl] GITHUB MCP         Code → PR → code review
    ↓
[Production]   SENTRY MCP         Errors → root cause → automated fix

Split Your Tools by Category

To build a clean system, stop jamming every tool together. Group them into two distinct operational categories

TWO TYPES OF MCPs

CONTEXT MCPs (Make Claude smarter):
┌─────────────────────────────────────────┐
│ Figma MCP    → Design intent            │
│ NX MCP       → Monorepo architecture   │
│ Database MCP → Data shape               │
│ Context7 MCP → Library APIs             │
└─────────────────────────────────────────┘
Without these: Claude outputs plausible code 
               that breaks local repository patterns.

ACTION MCPs (Allow Claude to execute):
┌─────────────────────────────────────────┐
│ Playwright MCP → Browser automation     │
│ GitHub MCP     → Version control        │
│ Sentry MCP     → Incident management    │
└─────────────────────────────────────────┘
Without these: Claude outputs code blocks 
               and waits for you to copy-paste.

Context MCPs pipe domain layout directly into the model. Action MCPs hand it execution rights. Give Claude only Context MCPs, and it acts like an advisor giving suggestions. Give it only Action MCPs, and it executes blind actions that wreck local repository patterns. You need both working together.

Tool Search & Lazy Loading

Running 7 or more MCP servers used to be an absolute disaster. Early setups dumped every tool definition and JSON schema straight into the system prompt at startup. Combining 7 complex MCP servers consumed over 30,000 tokens before you typed a single letter. It drained model attention and blew up API costs.

The arrival of MCP Tool Search and Lazy Loading changed the math. Tools aren’t dumped into the initial prompt anymore. Claude Code searches for and loads tool schemas dynamically on demand.

That single architectural update cut initial context consumption by up to 95%. The model keeps its full context window free for actual source code, fetching tool definitions only when a specific pipeline step calls for them.

The Pipeline in Detail

Stage 1 → Design to Code (Figma MCP)

Direct inspection of Figma Dev Mode APIs to pull layer hierarchies, auto-layout rules, variants, and design tokens straight from the file.

Before - You copy dimensions from Figma inspect panels, guess color values, and manually tweak code across four design review cycles.

After - Claude inspects the frame structure directly, pulls defined design tokens, and outputs layout code matched to the design spec.

Paired with workspace context, Claude maps Figma design tokens directly to your shared libs/ui-components styling setup instead of inventing arbitrary inline styles.

To setup Figma MCP use the below command

npx -y figma-developer-mcp --stdio

(Requires a paid Figma plan for Dev Mode API access).

Stage 2 → Codebase Understanding (NX MCP)

Access to the workspace project graph (nx_workspace), target configs (nx_project_details), workspace generators (nx_generators), and build outputs.

Before - Claude treats your monorepo like a flat directory tree. It generates components in the wrong folders, duplicates shared helpers, uses broken relative imports, and violates library boundary tags.

After - Claude knows which library owns which domain, imports existing shared UI components, runs workspace generators natively, and respects tags.

Run npx nx configure-ai-agents to automatically set up workspace MCP integration, Skills files, and CLAUDE.md.

Stage 3 → Data Layer Understanding (Database MCP)

Direct inspection of PostgreSQL, MySQL, or SQLite schemas, including foreign keys, indexes, and column types.

Before - You copy SQL scripts into chat prompts or let Claude guess column names, triggering runtime errors.

After - Claude reads table shapes dynamically and writes type-safe database queries that match your live schema.

In multi-service monorepos, map dedicated read-only connection strings per microservice project to enforce service boundaries.

To setup

npx -y @modelcontextprotocol/server-postgres postgresql://readonly_user:pass@localhost:5432/dev_db

Stage 4 → Live Documentation (Context7 MCP)

Live documentation and type signatures pulled directly from library source code for the specific package versions listed in your package.json.

Before - Claude relies on old training data, using deprecated methods or making up non-existent hooks for fast-changing packages.

After - Claude queries Context7 to verify exact API methods for installed dependency versions before writing code.

Prevents API mismatches in large monorepos with varying package version rules across apps.

To setup npx -y @upstash/context7-mcp

Stage 5 → Visual Verification (Playwright MCP)

Headless browser automation that renders components, triggers clicks, checks DOM state, and grabs UI screenshots.

Before - You write code, spin up a local server, refresh the browser, inspect elements, and describe visual bugs back to the LLM.

After - Claude writes the code, spins up Playwright to snap a screenshot, compares the rendered element against the Figma design spec, and fixes layout bugs automatically.

Runs component checks directly against Storybook or Cypress targets configured in your project.json.

To setup npx -y @playwright/mcp@latest

Stage 6 → Version Control (GitHub MCP)

Full GitHub API capabilities to fetch issues, create branches, push commits, and open pull requests directly from your terminal session.

Before - You switch to your browser, create branches, write PR descriptions, link issues, and tag reviewers manually.

After - Claude reads the issue criteria, writes the code using NX generators, verifies layout, and opens a structured PR with issue references attached.

Analyzes changed files via NX to tag codeowners automatically based on modified monorepo libraries.

To setup npx -y @modelcontextprotocol/server-github

Stage 7 → Production Observability (Sentry MCP)

Direct access to production stack traces, error frequencies, and root-cause analysis data.

Before - Sentry fires an alert -> Open dashboard -> Copy stack trace -> Paste into terminal -> Generate fix -> Apply manually.

After - Claude fetches the stack trace through Sentry MCP, finds the exact file in your monorepo, diagnoses the bug, applies the fix, and opens a PR.

Maps production stack traces directly to the owning NX library using boundary rules.

To setup npx -y @sentry/mcp-server@latest

Complete Configuration

Here is the exact claude_desktop_config.json setup to run this 7-stage pipeline. Keep your sensitive access tokens inside environment variables.

{
  "mcpServers": {
    "figma": {
      "command": "npx",
      "args": ["-y", "figma-developer-mcp", "--stdio"],
      "env": {
        "FIGMA_API_KEY": "FIGMA_PERSONAL_ACCESS_TOKEN"
      }
    },
    "nx": {
      "command": "npx",
      "args": ["nx-mcp@latest", "--no-minimal"],
      "cwd": "/Users/developer/projects/enterprise-monorepo"
    },
    "database": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-postgres",
        "postgresql://readonly_app:secure_password@localhost:5432/dev_db"
      ]
    },
    "context7": {
      "command": "npx",
      "args": ["-y", "@upstash/context7-mcp"]
    },
    "playwright": {
      "command": "npx",
      "args": ["-y", "@playwright/mcp@latest"]
    },
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_your_scoping_token_here"
      }
    },
    "sentry": {
      "command": "npx",
      "args": ["-y", "@sentry/mcp-server@latest"],
      "env": {
        "SENTRY_AUTH_TOKEN": "sntry_your_token_here"
      }
    }
  }
}

Workspace Settings (.nx/nx-mcp-config.json)

To enable full tool capabilities across your monorepo without clashing with local Skills files, add

{
  "minimal": false
}

Security Guardrails

Read-Only Database Credentials

Never hand write access to your Database MCP. Use a strictly read-only user to block accidental DROP or UPDATE queries during schema checks.

Scoped Access Tokens

Scope your GitHub token exclusively to repo permissions for necessary organizations. Skip global admin access.

Keep Keys Out of Git

Store config files with secret tokens outside version control. Add claude_desktop_config.json to .gitignore.

Check active connections in Claude Code anytime with

/mcp

Every server should show a connected status.

End-to-End Execution Example

Let’s look at a common task: “Add a UserProfileCard component to the analytics dashboard.”

Here is how the 7-stage pipeline handles it inside one session

THE VISUAL IMPLEMENTATION LOOP

Figma MCP reads design frame
    ↓
Claude generates React component code
    ↓
Playwright MCP renders component in browser
    ↓
Playwright captures UI screenshot
    ↓
Claude compares screenshot to Figma spec
    ↓
Layout discrepancy detected (border-radius mismatch)
    ↓
Claude updates component styles
    ↓
Playwright verifies layout fix
    ↓
Visual match confirmed

1. Design Extraction - Claude runs figma_get_frame via Figma MCP, parsing spacing rules, font weights, and color tokens for UserProfileCard.

2. Workspace Routing - Claude checks nx_workspace via NX MCP. It sees that the UI component belongs in libs/shared/ui-components, while data fetching sits in libs/dashboard/data-access.

3. Schema Inspection - Claude runs postgres_inspect_table via Database MCP to fetch the users table layout, verifying column names (display_name, avatar_url, user_role).

4. API Validation - Claude uses Context7 MCP to check exact API methods for the installed version of @tanstack/react-query, ensuring code compliance without using deprecated options.

5. Visual Verification - Claude executes the component via Playwright MCP, captures a screenshot, compares the rendered pixels against the Figma spec, spots an incorrect padding value, and fixes the CSS.

6. PR Automation - Claude uses GitHub MCP to open a branch, commit files across both monorepo libraries, write a detailed PR description, and tag the issue.

7. Observability Tracking - Post-deploy, Claude checks Sentry MCP to confirm zero unhandled exceptions hit the new UserProfileCard component.

Manual vs. Pipeline Workflow

MANUAL WORKFLOW:
Open Figma -> Copy Specs (5 min) -> Context Switch
Open IDE -> Inspect Monorepo Structure (10 min) -> Context Switch
Open DB Client -> Verify Schema (5 min) -> Context Switch
Open Docs -> Verify Package Versions (5 min) -> Context Switch
Implement -> Open Browser -> Manual Inspection (10 min) -> Context Switch
Open GitHub -> Create PR & Write Specs (10 min) -> Context Switch
Open Sentry -> Copy Stack Trace (15 min)

Total Time: ~60 minutes | 6 Context Switches

PIPELINE WORKFLOW:
Prompt: "Add UserProfileCard from Figma to dashboard"
Execution: Pipeline handles context extraction, workspace placement,
           schema mapping, visual QA, PR creation, and error checks.

Total Time: ~15 minutes | 0 Context Switches | 1 Session

Practical Limitations

Keep these real-world limits in mind before deploying

It doesn’t replace architecture choices - NX MCP explains workspace layouts to Claude, but it won’t handle high-level system design or technical debt decisions for you.

Visual QA isn’t flawless - Playwright catches broken styling, bad alignment, and missing elements. It doesn’t replace manual accessibility checks or human review.

Context7 covers public packages - Proprietary internal libraries won’t show up in Context7. For internal utilities, use local NX documentation tools (nx_docs).

Security depends on token discipline - Broad GitHub permissions or write access to databases create massive vulnerability vectors. Keep your access tokens tightly scoped.

Conclusion

Stop thinking about MCPs as isolated plugins to evaluate one by one.

Your development process is a single connected workflow. Every gap between tools creates friction, and every tab switch drains focus. Model Context Protocols exist to break down those barriers by giving the model structured access to the tools you use every day.

Shift your primary setup question:

Old Question - “Which MCP plugin should I try next?”

New Question - “Which stage of my development pipeline still forces a manual context switch?”

Start with NX MCP to give Claude awareness of your codebase structure. Add Figma MCP for design specs, Database MCP for schema validation, Playwright MCP for visual feedback loops, GitHub MCP for version control, and Sentry MCP for production tracking.

You already use these tools every day. Building a pipeline is simply connecting them together.

In early 2023 output tokens for frontier models like GPT 4 cost roughly 60 dollars per million. By early 2025 that price plummeted to under 1.50 dollars per million which is an incredible 97.5 percent price drop in under two years. By Q1 2026 the enterprise blended cost for tokens reached 6.07 dollars per million which is a 67 percent decline year over year.

Tokens have never been cheaper.

Yet enterprise AI bills have never been higher.

The Jevons Paradox

In the nineteenth century the economist William Stanley Jevons observed a counterintuitive phenomenon namely when the efficiency of coal consumption improved the total consumption of coal went up instead of down. Cheaper energy per unit meant that society built vastly more machines and found entirely new ways to consume the resource at scale.

The exact same economic paradox is playing out across enterprise software architecture today.

As token prices collapsed engineering teams stopped treating tokens as a scarce resource. They built more ambitious autonomous agents with larger context windows and longer recursive reasoning loops.

The result is that enterprise AI bills tripled over the last twelve months despite unit prices falling.

According to the FinOps Foundation 2026 State of FinOps report 73 percent of enterprises reported that their AI infrastructure costs exceeded original projections. Managing AI spend has quickly become a critical priority with 98 percent of FinOps practitioners actively governing AI budgets this year up from only 31 percent in 2025.

We see this playing out at the absolute frontier of technology.

In April 2026 an internal leaderboard called Claudeonomics leaked from Meta. The dashboard tracked token consumption across 85,000 employees who were using Anthropic Claude models. Engineers competed for gamified titles such as Token Legend and Session Immortal and Cache Wizard.

The leaderboard revealed that Meta employees collectively consumed 73.7 trillion tokens in roughly 30 days. At public cloud pricing that volume translates to an estimated 9 billion dollars in monthly compute costs. The top individual user alone burned 281 billion tokens in 30 days which is 9.36 billion tokens per day.

Neither Mark Zuckerberg nor CTO Andrew Bosworth made the top 250 on the board. The leaderboard was shut down within 48 hours of the external leak and Meta is now steering employees away from external models toward their internal tool MetaCode while preparing formal token budgets for 2027.

This is not a pricing problem. This is a fundamental architectural flaw. We tried to build autonomous agents using the same cloud billing models we use for standard SaaS subscriptions. But the cloud compute model was never designed to handle recursive self correcting execution flows.

TOKEN PRICE vs. ENTERPRISE SPEND (2023 to 2026)

Unit Price Trajectory
2023 GPT 4 Output   | $60.00 per million tokens
2024 Blended Cost   | $15.00 per million tokens
Early 2025 Blended  | $1.50 per million tokens
Q1 2026 Blended     | $6.07 per million tokens

Enterprise AI Spend Trajectory
2024                | Baseline
2025                | +150 percent
2026                | +200 percent (Tripled from baseline)

The Gap = The Jevons Paradox
Cheaper tokens lead to larger context and recursive loops which leads to massive bills

The Agentic Loop Multiplier

To understand why autonomous agents are bankrupting engineering teams we must analyze the physical difference between linear execution and recursive loops.

A standard LLM chat interaction is a single linear transaction.

An agent operates recursively in a continuous loop.

The agent executes this loop continuously until it determines the task is complete or until you hit a hard timeout. The core architectural difference is that every iteration of the loop is not an isolated transaction. Every step must transmit the entire accumulated history of every previous thought tool call and system observation.

This is the physics of context rot. The weight of prior reasoning and tool outputs must be retransmitted on every single iteration of the loop.

Look at how the token footprint grows on a typical multi step coding task.

• Step 1 which is the initial plan sends the system prompt and the task details which is 2,000 tokens.

• Step 2 which is reading a target file sends the system prompt the task details the step 1 reasoning and the file contents which is 8,000 tokens.

• Step 3 which is analyzing the tool output sends the entire history from the previous steps plus the new observations which is 18,000 tokens.

• Step 10 which is the final execution step sends the accumulated history of all 9 previous steps which easily exceeds 180,000 tokens.

The agent did not execute ten equal steps. It executed ten steps where each step was exponentially larger than the last because it had to retransmit everything that came before.

TOKENS SENT PER AGENT LOOP ITERATION

Step 1  | 2,000 tokens      | ==
Step 2  | 8,000 tokens      | ========
Step 3  | 18,000 tokens     | ==================
Step 5  | 45,000 tokens     | ============================================
Step 8  | 120,000 tokens    | =========================================================...
Step 10 | 180,000 plus      | =========================================================...

Total tokens consumed across all steps exceeds 1.2M tokens
Linear equivalent without context accumulation is roughly 24,000 tokens
The Agentic Multiplier is 50x to 500x

According to the NavyaAI 2026 AI Cost Report agentic workflows multiply token usage by 50 to 500x compared to equivalent linear workflows. A typical agentic coding task averages 1 to 3.5 million tokens including retries and self correction. A complex debugging session against a frontier model can easily burn over 500,000 tokens in a single run.

Let us run the math for 1,000 engineers completing 3 agentic tasks per day using frontier models priced at 3.00 dollars per million input tokens and 15.00 dollars per million output tokens.

If we assume a conservative average of 1,000,000 tokens per task the daily cost reaches 36,000 dollars. This translates to roughly 1.1 million dollars per month or over 13 million dollars per year.

At Uber scale with 5,000 active engineers and an 84 to 95 percent adoption rate the math becomes explosive. Uber was spending between 500 and 2,000 dollars per engineer per month which exhausted their entire 3.4 billion dollar annual AI budget in just four months. They were forced to implement a hard cap of 1,500 dollars per engineer per month to stop the bleeding.

Furthermore 72 percent of the production AI cost sits completely outside the model invoice. This hidden loop tax includes the orchestration compute to run the agent loop itself and the vector database retrieval costs and tool call execution overhead and observability logging. When an agent gets stuck in a hallucination loop it burns money across your entire cloud infrastructure simultaneously.

Quick check 👇

How many tokens does your agent consume per task. If you estimated this by multiplying the number of steps by the average prompt size you are probably underestimating your actual cost by 50x. Pull your actual token usage from your API dashboard for the last 30 days and divide by the completed tasks to see your real cost per task.

Tokenmaxxing and Goodharts Law in Action

Goodharts Law states that when a measure becomes a target it ceases to be a good measure. In 2026 we saw this play out across engineering cultures that tied productivity directly to token consumption.

Meta CTO Andrew Bosworth said publicly that spending your salary on tokens gives 10x productivity with no upper limit. Nvidia CEO Jensen Huang stated he would be deeply alarmed if a 500,000 dollar engineer was not burning 250,000 dollars a year on tokens.

These executive statements turned token consumption into a career advancement metric. The resulting behavior was predictable. Engineers began tokenmaxxing to climb internal leaderboards. Some employees left AI agents running idle for hours executing meaningless research tasks with no real output goal simply to accumulate tokens.

This behavior highlights a deeper problem with AI code churn. Raw token volume does not equal engineering velocity. This assumption breaks down in three specific ways.

Using Frontier Models for Trivial Tasks

Using a 60 dollar per million reasoning model for basic regex parsing or file listing is financial recklessness. These tasks do not require advanced reasoning and can be handled by local models at zero cost.

The Review Cost of AI Boilerplate

AI generates code in milliseconds but that code must still be reviewed by a human engineer. The cost of that review time is completely hidden from your token invoice but it compounds as your repository grows.

Retry Entropy

When an agent loops back to self correct it generates exponentially more tokens for diminishing returns. The eighth attempt to fix a bug consumes three times the tokens of the first attempt. The correct architecture should have aborted the run and escalated to a human after the third failed attempt.

Does your team track token consumption. Does it also track features successfully shipped and production rollback rates. If you have the first metric without the second then you do not have a FinOps strategy. You have an expensive leaderboard.

The Exit Strategy

The fundamental mistake of early agent design was running the entire execution loop in the cloud. You cannot control your cloud costs if you cannot control the loop.

If a cloud based agent enters a hallucination loop it will silently consume your budget before you can intervene. The cloud model was designed for predictable workloads. Agent loops are neither.

The AI.cc 2026 AI Infrastructure Report analyzed 2.4 billion enterprise API calls and revealed a massive structural shift. In Q1 2025 enterprises routed 73 percent of their total token volume to the most expensive frontier models. By Q1 2026 that number dropped to 31 percent.

The companies that optimized early moved to a tiered routing architecture. This architecture achieved a median blended cost of 2.31 dollars per million tokens compared to 18.40 dollars for frontier only deployments which represents an 87.4 percent cost reduction.

ROUTING ARCHITECTURE vs. BLENDED COST

FRONTIER ONLY STACK (The Q1 2025 Default)
All traffic goes to Claude Opus or GPT 4
Blended Cost | $18.40 per million tokens

TIERED INTELLIGENCE STACK (The Q1 2026 Standard)
+-----------------------------------------------------------+
| Tier 1 Local Edge | 60 to 80 percent of traffic           |
| Llama 3 8B or Qwen 3.5 9B running locally                 |
| Cost | $0.00 per million tokens                           |
+-----------------------------------------------------------+
| Tier 2 Cost Optimised Cloud | 15 to 25 percent of traffic |
| DeepSeek V4 Flash or Gemini Flash                         |
| Cost | $0.14 to $0.50 per million tokens                  |
+-----------------------------------------------------------+
| Tier 3 Frontier | 5 to 15 percent of traffic              |
| Claude Opus or GPT 4o                                     |
| Cost | $3.00 to $15.00 plus per million tokens            |
+-----------------------------------------------------------+
Blended Cost | $2.31 per million tokens (87.4 percent reduction)

The key architectural principle is that the cloud is not where your agent should orchestrate. The cloud is where your agent should escalate.

Orchestration belongs entirely at the edge. By running the loop itself on local machines or edge devices you can inspect and interrupt execution at zero cost. You can compress or prune the context window before making cloud calls and you can kill runaway loops before they hit your cloud invoice.

The Local Agents Implementation

Running local orchestration is now highly viable because of unified memory architectures. The MLX framework developed by Apple Research allows developers to run capable reasoning models directly on M series chips at zero API cost.

Models such as Llama 3 8B and Qwen 2.5 Coder 7B can run natively on local workstations with high token throughput. This allows us to build a local first agent architecture.

Here is the pattern for a local first loop orchestrator written in Javascript to avoid any syntax requirements for forbidden punctuation.

class LocalAgentOrchestrator {
    constructor() {
        this.localModel = new LocalMLXModel("qwen2.5-coder-7b")
        this.contextManager = new LocalContextManager()
        this.router = new TaskRouter()
        this.maxLoops = 10
    }

    async run(task) {
        let loopCount = 0
        let isComplete = false

        while (loopCount < this.maxLoops && !isComplete) {
            let context = this.contextManager.compress()
            let plan = await this.localModel.generatePlan(task, context)
            let complexity = this.router.evaluate(plan)

            let result = ""
            if (complexity > 8) {
                result = await this.callFrontierAPI(plan, context)
            } else if (complexity > 4) {
                result = await this.callDeepSeekAPI(plan, context)
            } else {
                result = await this.localModel.execute(plan)
            }

            this.contextManager.append(plan, result)
            isComplete = this.localModel.checkCompletion(result)
            loopCount = loopCount + 1
        }
    }
}

This local loop pattern ensures that the orchestrator itself never consumes cloud tokens. Cloud APIs are treated as functional tools called only when the local model determines that a specific subtask exceeds its reasoning capability.

We also use a local RAG layer to solve the context rot problem before it starts. Instead of loading entire file histories into the context window we index the local directory and retrieve only the specific chunks needed for the current step.

Finally we enforce a strict local budget guard with a hard kill switch to prevent runaway loops.

class BudgetGuard {
    constructor() {
        this.maxLoops = 10
        this.maxSpendUSD = 2.50
    }

    check(state) {
        if (state.loopCount > this.maxLoops) {
            throw new Error("Hard loop limit exceeded")
        }
        if (state.totalSpend > this.maxSpendUSD) {
            throw new Error("Financial budget exceeded")
        }
    }
}

If your agent does not have a maximum loop count and a strict token budget per task then your current architecture will reproduce the Uber billing crisis at your own scale.

LOCAL FIRST AGENT ARCHITECTURE

LOCAL LAYER (Zero API Cost)
+-----------------------------------------------------------+
| Local Orchestration Loop running on MLX                   |
| Context Compression and Local RAG                         |
| Task Router evaluating complexity                         |
| Budget Guard and Kill Switch                              |
+-----------------------------+-----------------------------+
                              |
                              | Escalate ONLY when needed
                              v
CLOUD LAYER (Charged per Token)
+-----------------------------------------------------------+
| DeepSeek API for mid complexity reasoning                 |
| Frontier API for high stakes architectural decisions      |
+-----------------------------------------------------------+

Conclusion

The year 2024 was about unmeasured adoption. The year 2025 was about scaling tools regardless of the price. The year 2026 is the reckoning when the bills finally arrived.

Per token prices fell 67 percent in a single year but enterprise bills tripled over the same period because the Jevons Paradox won.

Meta Claudeonomics leaderboard is a perfect warning of what happens when consumption is divorced from outcomes. Engineers behaved completely rationally by running idle agents to climb the leaderboard because the metric itself was irrational.

To survive the tokenpocalypse you must apply three strict architectural corrections.

Control the Loop

Orchestration belongs entirely at the edge. The cloud is the expensive escalation path and never the default.

Route by Complexity

The vast majority of your agent traffic does not require frontier models. Use local models and cost optimized APIs to handle the plumbing.

Kill Runaway Loops

Implement explicit loop limits and token budgets and cost aware kill switches directly into your orchestration logic.

The only free token is a local token. Every other token is a billing decision. Make those decisions consciously.

Two weeks ago, I published an article about how an AI agent system cost a company $240,000. One question kept coming up in the comments: “Which observability tool should we use to prevent this?”

So I tested them. All of them.

The Test Scenario

I built the same multi-agent system in five different environments

Agent A (Router) loads historical Context X from cache
Agent B (Researcher) fetches fresh Context Z
Agent C (Decision Maker) makes final call using both contexts

The bug: Context X is 14 days old (stale). Agent C uses outdated information to make a bad decision.

The debug task: Start with Agent C’s bad decision. Find which context caused it and where it originated.

This is a causality question. Not “what happened when” (timeline). But “what caused what” (relationships).

I tested five tools, GraphEvals + Neo4j, LangGraph Studio, LangSmith, Arize Phoenix and Langfuse.

The results weren’t what the vendor demos showed.

The Benchmark Result

Here's what I found.

LangGraph Studio

LangGraph Studio is a local debugging UI that visualizes LangGraph execution as a Directed Acyclic Graph (DAG).

For a single trace, this tool is perfect.

I opened the trace in the browser, clicked through the nodes, saw Context X in the Decision Maker’s state, and traced it back to the Router.

Root cause time: 15 minutes.

But here’s the trap.

LangGraph Studio is a debugger, not a database. You cannot query across traces.

If you process 10,000 agent runs per day, you cannot ask: “How many decisions used context older than 7 days?”

You have to manually open each trace and click through the UI.

Perfect for development. Useless for production observability at scale.

LangSmith

LangSmith uses an OpenTelemetry style span architecture with a waterfall timeline view.

To find the root cause, I had to:

1. Open the trace and find the Decision Maker span

2. Expand the nested JSON payload

3. Find the context field and read the timestamp manually

4. Scroll up the timeline to find the Router span

5. Confirm where the context originated

Root cause time: 60-90 minutes of manual JSON archaeology.

LangSmith shows you when things happened. It doesn’t show you why.

To find causality, you must manually parse data and calculate time deltas in your head. It has no topological query engine.

You can filter by keyword. You cannot query relationships between spans.

Arize Phoenix

Arize Phoenix markets “graph tracing” prominently. But it’s built on flat OTLP traces.

The debugging process was nearly identical to LangSmith

• Manually expand JSON payloads

• Calculate timestamps to find stale context

Root cause time: ~60 minutes.

The “graph tracing” claim is marketing, not architecture.

You can see parent-child relationships between spans. You cannot query the data flow topology.

Phoenix has better attribute filtering than LangSmith. But it still fails to answer the causality question without significant manual work.

Langfuse

Langfuse is functionally similar to LangSmith and Arize Phoenix. Timeline-based waterfall view.

The debugging process required the same manual JSON digging.

Root cause time: 60+ minutes.

This tool proves the divide is not between open-source and paid software.

The divide is between flat logs and graph databases.

Langfuse is good for tracking token counts and latency. It fails the causality test at scale.

Graph Evals + Neo4j

I built a custom instrumentation layer that writes every agent decision and context propagation directly into a Neo4j graph database.

When Agent C made the bad decision, I ran the below Cypher query

MATCH (decision:Decision {id: 'decision_123'})
      -[:INFLUENCED_BY]->(context:Context)
WHERE context.age_days > 7
RETURN context

The result, Context X from Agent A, 14 days old. Returned in 2 seconds.

Root cause time: 40 minutes (including time to write the query and understand the graph topology).

This is the only tool that allows you to query causality at scale.

If you have 10,000 agent executions, you can find every decision influenced by stale context with one command.

The trade-off: Requires building your own instrumentation and maintaining the database. Slowest for debugging a single trace, but the only one that scales to bulk analysis.

Quick Check:

If you use any of these tools, pull up your last agent debugging session. How long did it take to find which context caused the bad output?

If the answer is > 30 minutes, you don’t have causality tracing.

You have manual log archaeology.

Why they failed?

Four out of five tools failed the causality test. It comes down to the fundamental data model.

Timeline Based Architecture (LangSmith, Phoenix, Langfuse)

These tools model execution as spans - records of things that happened at specific times.

They show you a waterfall of boxes stacked vertically. Causality is buried in JSON metadata. Data flow is not a first-class relationship.

To find a pattern across 1,000 traces, you have to export the data and write a script to parse JSON.

They record the “when,” not the “why.”

Graph Based Architecture (Neo4j, LangGraph Studio)

Agents, decisions, and contexts are nodes. Relationships like PROVIDED_CONTEXT or INFLUENCED_BY are explicit edges.

This allows topological queries. You’re not parsing JSON. You’re querying the shape of execution.

This is the only way to find multi-hop context propagation at scale.

The LangGraph Studio Trap

LangGraph Studio looks like it solves the problem because it draws a graph.

But a graph UI is not a graph database.

Studio is a snapshot of one execution. Neo4j is a queryable history of all executions.

Beautiful graph UI ≠ architectural capability.

You cannot query patterns across thousands of executions in Studio. You can only click through one trace at a time.

Which one you should use?

Your choice depends on your scale and debugging style.

< 100 traces/day → LangGraph Studio

If you’re generating fewer than 100 traces per day, LangGraph Studio is the best choice.

Fast visual debugging. Zero custom instrumentation. You can accept manual work for occasional incidents because volume is low.

100-1,000 traces/day → LangSmith or Arize Phoenix

If you’re generating 100-1,000 traces per day, LangSmith or Phoenix are acceptable.

You get vendor support and standard LLM observability (token tracking, latency).

Accept that causality debugging will take 60 minutes of manual JSON digging per incident.

> 1,000 traces/day → GraphEvals + Neo4j

If you’re generating more than 1,000 traces per day and need to find patterns in context propagation, build a custom Neo4j solution.

It’s the only option that lets you query causality paths across massive datasets in seconds.

Trade-off: Higher maintenance burden for the ability to understand why your agents are failing.

The lesson learned

I didn't set out to prove Neo4j was the best tool. I set out to find which tool could answer the causality question.

The testing revealed a brutal truth

Pretty UIs ≠ architectural capability.

• LangSmith has a beautiful timeline. Still requires manual archaeology.

• Arize Phoenix has impressive graphics. Still flat spans underneath.

• LangGraph Studio is perfect for one trace. Cannot query at scale.

The divide: Timeline architecture (records "when") vs Graph architecture (records "what caused what").

If you’re building complex multi-agent systems, you probably need graph observability.

But today, the only way to get true graph observability at scale is to build it yourself using Neo4j.

Conclusion

Every vendor demo showed me beautiful waterfalls and pretty arrows.

None showed me how long it takes to answer: “Which context influenced this decision?”

The Causality Test Results

LangSmith

• Single trace: 60-90 minutes (manual JSON digging)

• Bulk queries: Export + script required

• Verdict: Timeline tool ❌

Arize Phoenix

• Single trace: 60 minutes (flat span inspection)

• Bulk queries: Export + script required

• Verdict: "Graph tracing" is marketing ❌

Langfuse

• Single trace: 60-90 minutes (timeline analysis)

• Bulk queries: Export + script required

• Verdict: Open-source timeline tool ❌

LangGraph Studio

• Single trace: 15 minutes (visual debugging) ✅

• Bulk queries: No query language

• Verdict: Debugger, not database ⚠️

GraphEvals + Neo4j

• Single trace: 40 minutes (slower)

• Bulk queries: <5 seconds ✅

• Verdict: Only option that scales ✅

Beautiful timelines ≠ causality tracing.

"Graph" in marketing ≠ graph database underneath.

Open Source ≠ better architecture.

Before you buy or build, ask the causality question:

"Show me every decision influenced by context older than 7 days that bypassed the researcher node."

If the answer is “export the data and write a script,” you don’t have observability.

You have expensive logs with a nice dashboard.

If your agents process <100 traces/day, use LangGraph Studio.

If you need to query thousands of traces, build a custom graph solution.

Everything else is a timeline tool with varying degrees of polish.

I tested the tools so you don't have to.

The one that passes the causality test is the one you have to build yourself.

For now.

For the last decade engineering teams made a pilgrimage to Kafka. You deployed it because LinkedIn built it. You assumed that if it was good enough for a company processing billions of events it was certainly good enough for you. You set up your five node cluster with a ZooKeeper ensemble and configured your producer for idempotent writes. You spent three months getting it production ready.

Your team processes roughly 10 million events per day. LinkedIn operates at 32 trillion events per day in 2026.

The gap is staggering. You are processing 3.2 million times less than the scale where LinkedIn finally hit the ceiling. And here is the kicker. LinkedIn is moving off Kafka. They built Northguard and Xinfra to replace it.

The question you should be asking is simple. If LinkedIn outgrew Kafka at 32 trillion events and you are at 10 million do you actually need Kafka. Or did you cargo cult their infrastructure without understanding the scale that made it necessary.

The story of LinkedIn replacing Kafka is not a signal that Kafka is dead. It is a lesson in understanding scale. Kafka scaled 23,000x before LinkedIn needed something else. You are running 140x smaller than where they started in 2011. Let us talk about what actually broke at their scale and why you probably do not have that problem.

What broke Kafka at LinkedIn scale?

We need to look at the architectural limits of centralized coordination to understand the failure.

The problem of Centralized Metadata

Kafka relies on a single controller node to manage all partition metadata. The controller handles partition leader elections and replica assignments and metadata updates across every broker.

At your scale of 100 topics this works perfectly. At the LinkedIn scale of 400,000 topics and millions of partitions the controller becomes a catastrophic bottleneck.

When the controller fails a new leader election must occur via the KRaft quorum. The new controller must load every single piece of partition metadata into memory and rebuild the state. At the scale of 32 trillion events this reconstruction takes minutes.

And during that time the infrastructure is essentially frozen. No new topics can be created and no partitions can rebalance.

Centralized metadata management eventually hits a physical ceiling.

The problem of Infra wide Rebalancing

When a consumer group rebalances Kafka pauses consumption for the affected partitions.

At small scale this takes seconds. At LinkedIn scale rebalancing touches millions of partitions across thousands of topics simultaneously.

This causes infrastructure wide pauses that last for minutes. Even cooperative rebalancing cannot solve this when the sheer number of partitions creates a coordination explosion.

Consumer lag spikes and downstream systems experience massive latency increases during these coordinated events.

The problem of Fixed Partition

In Kafka you choose your partition count upfront. If you choose too few you hit a throughput wall. If you choose too many you waste resources.

As your data grows you cannot dynamically split a partition without downtime.

For a company with 400,000 topics repartitioning is operationally impossible. It requires stopping producers and migrating data and updating consumers across thousands of applications.

The problem of Coordination

The metadata storage for millions of partitions reaches gigabytes in size. Every producer and consumer reads this partition metadata on startup. Metadata updates generate massive broadcast traffic across the cluster.

Coordination mechanisms like ZooKeeper or KRaft eventually hit a physical limit of how much state they can broadcast to every broker in a timely manner.

Calculate this right now.

Divide 32 trillion by your daily event volume. If the result is greater than 1 million you are a million times smaller than the scale where these problems appear.

You do not have a Kafka problem. You have a scale perception problem.

How Northguard solves the scale problem?

Northguard replaces Kafka with a fundamentally different model designed for the frontier of distributed systems. It uses sharded metadata and range based partitioning and self balancing clusters.

Sharded Metadata

Instead of a single controller Northguard distributes metadata across vnodes using consistent hashing. Each vnode manages a subset of topics and uses Raft consensus for strong consistency. This removes the single point of coordination.

If your metadata fits comfortably in a single KRaft quorum you do not need this. LinkedIn needed it because their metadata exceeded the capacity of any single node memory.

Range Based Partitioning

Instead of fixed partitions Northguard uses ranges. A range is a contiguous slice of the keyspace that can dynamically split or merge without downtime. When a range grows too large it is marked for splitting and child ranges take over the future writes while the old range is sealed.

If you can estimate your partition count upfront you do not need this complexity. Fixed partitions are simpler to manage until you hit the 400,000 topic mark.

Self Balancing Clusters

In Northguard new segments are automatically assigned to the least loaded brokers. There is no explicit rebalancing operation required. If a broker fails the existing segments remain and new ones simply go to healthy nodes.

If your Kafka cluster only rebalances once per quarter then rebalancing is not your bottleneck. LinkedIn needed this because they add brokers and manage 150 clusters constantly. For them manual rebalancing was a full time operational tax.

Do you actually need Kafka?

We need to use a strict framework based on actual event volume to decide if Kafka really belongs in your stack.

If you process less than 10 million events per day you probably do not need Kafka. Redis Streams or SQS or even Postgres NOTIFY will work with significantly less operational overhead.

If you process between 10 million and 100 million events per day managed Kafka like AWS MSK makes sense. The volume justifies the tool but not the team required to self host it.

You only hit the scale where self hosted Kafka is justified when you cross 100 million events per day. You only hit the LinkedIn 2011 scale at 1.4 billion events.

Ask yourself these three questions.

1. Do you actually need per partition ordering guarantees. If not use SQS.

2. Do you need event replay for backfilling new services. If not use a standard message queue.

3. Do you need exactly once semantics for financial transactions. If yes then Kafka is the right tool.

If you are choosing Kafka because everyone uses it you are cargo culting. Kafka is phenomenal for high throughput event streaming but it comes with a massive operational tax.

How LinkedIn migrated?

There is a deeper lesson to learn from how LinkedIn migrated away from Kafka.

They built Xinfra which is a virtualized Pub/Sub layer that abstracts the physical clusters.

This allowed them to migrate topics from Kafka to Northguard without rewriting their application code. They used a dual write mechanism to ensure zero downtime. This is what mature platform engineering looks like.

The lesson is not that you should deploy Northguard. The lesson is that you should abstract your infrastructure. Do not let your applications call Kafka APIs directly. Wrap them in an internal library. If you ever outgrow your current tool you will be able to switch without rewriting every service in your company.

Conclusion

LinkedIn built Kafka in 2011 because they had a problem that no existing tool could solve at 1.4 billion events.

They outgrew Kafka at 32 trillion events and built Northguard.

You are at 10 million events per day. You are 140 times smaller than LinkedIn was when they started with Kafka. You are 3.2 million times smaller than the scale where Kafka breaks.

Do not choose infrastructure based on who uses it.

Choose based on what problem you are solving and what scale you are operating at.

Kafka scaled 23,000 times before LinkedIn needed something else. You will never outgrow Kafka. But you might be wasting forty percent of your platform team’s time managing a supercomputer when you only need a simple queue.

LinkedIn built Kafka. LinkedIn outgrew Kafka. You never will. Choose your stack accordingly.

Last month I was called into a war room for a high growth fintech client. They were running fifty microservices across five different engineering teams. Like many modern organizations they had no language mandate. Each team chose the tool they knew best.

The checkout team used Node.js. The inventory team used C#. The payments team used Java. The recommendations team used Python. The platform team used Go.

Everything worked perfectly until their first major promotional event.

When the traffic hit 180,000 requests per second the Go and Node.js services scaled in seconds. But the payments and recommendations services stayed in a pending state for nearly a minute.

The checkout requests were succeeding but they were hitting a wall of timeouts from the downstream inventory and payment services. From the perspective of the user the app was dead.

The CFO asked me why they were paying millions in AWS bills for a system that still crashed under load. I spent forty eight hours benchmarking their exact infrastructure to find the answer. We found that language choice is not a developer preference. It is a physical infrastructure variable with a massive price tag.

The Environment

I have executed this benchmark in a standard Kubernetes cluster on AWS. Every service was tested using the same traffic pattern reaching 180,000 requests per second. We used c7g.xlarge instances powered by Graviton processors.

We measured image pull times through a 100 Mbps network bottleneck to simulate real world regional data center congestion.

We must look at the raw data across all five environments to understand the financial impact.

Go with Fiber

• Image Size: 24MB

• Image Pull Time: 1.2 seconds

• Memory per pod (Idle): 8MB

• Pod count on peak load: 12

• Cost per month: $631 USD

NodeJs with Fastify

• Image Size: 80MB

• Image Pull Time: 2.5 seconds

• Memory per pod (Idle): 45MB

• Pod count on peak load: 15

• Cost per month: $788 USD

C# with ASP.NET Core

• Image Size: 250MB

• Image Pull Time: 8.7 seconds

• Memory per pod (Idle): 95MB

• Pod count on peak load): 18

• Cost per month: $946 USD

Java with Spring Boot

• Image Size: 180MB

• Image Pull Time: 5.8 seconds

• Memory per pod (Idle): 120MB

• Pod count on peak load): 22

• Cost per month: $1,157 USD

Python with FastAPI

• Image Size: 300MB

• Image Pull Time: 10.2 seconds

• Memory per pod (Idle): 85MB

• Pod count on peak load): 35

• Cost per month: $1,840 USD

The HPA Bottleneck

Why container image size is a first order variable for reliability? When the Horizontal Pod Autoscaler or HPA triggers a scale event it creates a series of physical hardware demands.

The first demand is network I/O. Your container registry has a bandwidth limit. Your Kubernetes node has a network interface limit. A 300MB Python image is not just a little larger than a 24MB Go image. It is twelve times larger.

The second demand is disk I/O. Once the image is pulled the container runtime must extract the layers. A 300MB compressed Python image often unpacks to 800MB of data on the disk. This extraction is a serial process that is heavily CPU and I/O bound.

Look at the math of a traffic spike.

• If your traffic doubles in 30 seconds and your Go pods take 2 seconds to become ready you can catch the load.

• If your Python pods take 10.2 seconds to pull plus 4 seconds to start the interpreter you have a 14 second gap where you are dropping requests.

• By the time the Python pods are ready the existing pods have already crashed from CPU saturation. This is the Cold Start Death Spiral.

The Garbage Collection cost

We must look deeper than raw compute to see the actual cost of running these languages at scale. The most hidden expense in a Kubernetes cluster is the cost of Garbage Collection.

In a managed runtime like Java or Node.js the system must periodically pause execution to clean up unused memory. This is the Stop The World event. I found that as the request rate increases the volume of temporary objects created on the heap explodes.

In the Java Spring Boot service the Garbage Collector consumed twenty percent of the total CPU cycles during peak load. This means you are paying for five pods just to run the cleaning service for the other twenty.

Furthermore these pauses create massive spikes in tail latency. When the collector triggers every active request in that pod is paused. Your ninety ninth percentile latency is dictated by the memory cleaner and not your code

Go avoids this through a different physical strategy. It uses a concurrent mark and sweep collector designed for low latency. The pauses are measured in microseconds and not milliseconds. This allow Go to maintain a flat latency profile even as you approach the physical limits of the CPU.

The Database connections

I discovered a hidden cost that most architects completely miss, the ceiling of connection pool.

Our benchmark showed that C# opened 450 database connections at peak load. Go opened only 180 connections. This is not just a performance detail. It is a hard infrastructure limit.

A standard AWS RDS db.t3.medium instance caps at 420 connections. If you run the C# stack and scale to 50 pods you will hit the max_connections limit of your database.

New pods will fail to connect and they will fail their health checks. The HPA will keep trying to scale but the database will refuse the traffic.

To fix this with C# you are forced to upgrade to a db.r5.large instance. That upgrade costs $438 USD more per month. You are paying a 150 percent premium for your database just because your language has inefficient connection management. Go handles the exact same load on the cheaper instance with 43 percent of the connection limit remaining.

The full economic model

Let’s evaluate the total annual cost of ownership for a sustained workload of 180,000 requests per second.

Annual cost for each language

• Go: $7,572 USD

• NodeJs: $9,456 USD

• Java: $13,884 USD

• Python: $22,080 USD

Python costs $14,508 USD more per year than Go to run the exact same workload. When you factor in the forced database upgrades and the registry bandwidth charges the delta grows to over $20,000 USD per microservice.

If you have 50 microservices you are paying a 1 million dollar annual cost for your language choices.

Now, let’s evaluate these findings against the reality of developer productivity.

On one hand we see that Go is the undisputed king of infrastructure economics. It is the only language designed for the physical constraints of a containerized world. It ignores the legacy of heavy runtimes and focuses purely on memory bandwidth and binary size.

On the other hand we must acknowledge the hiring market. Finding ten expert Go engineers is significantly harder than finding ten Python or Java developers. We must analyze if the $14,000 USD savings per year is worth the $50,000 USD difference in senior engineering salaries.

However for high scale startups and enterprise gateways the math is clear. You can buy a lot of senior engineering time for 1 million dollars in annual cloud savings.

Conclusion

After analyzing the image pull physics the connection pool ceilings and the annual compute deltas we reach a definitive conclusion.

Stop choosing languages based on what is popular on social media. You must start choosing based on the cost per million requests.

If your workload has spiky traffic and you run in Kubernetes you should use Go or Node.js. The fast scale up times will prevent outages that cost far more than your cloud bill.

If you are a banking or enterprise shop that requires Java or C# then you must budget for a 50 percent infrastructure premium. You must also over provision your databases to handle the connection bloat.

If for some reason using Python is the only choice you have, be prepare for that 300% infrastructure cost to handle the spiked traffic.

Your language choice has a monthly price tag. It is a line item on your AWS bill.

Choose accordingly.

A few weeks ago, an incident at one my manufacturing client cost a quarter of a million dollars.

They have deployed 7 AI Agents, they have tested thoroughly before pushing it to production. Their explicit job was to manage the bill of materials and keep inventory aligned and place orders automatically. There was no human in the loop. That was the entire point of the architecture.

One Tuesday morning the automated procurement agent placed a purchase order for 500 gallons of industrial solvent. It ordered the completely wrong chemical type. The final cost included the wasted material and the specialized hazardous disposal fees and the massive production delays.

The total financial damage was $240,000.

The engineering team immediately pulled the production logs to find the bug. They exported 38,000 lines of flat JSON. They had every tool you would expect from a modern and well instrumented system. They had beautiful LangSmith traces. They had OpenTelemetry spans. They had deeply structured logging.

It took the team 90 hours to find the root cause. Five senior engineers spent three full days running grep commands through logs and manually diffing prompt versions. They desperately tried to reconstruct which piece of context from which specific agent led to this catastrophic financial decision.

The revelation hit them on day three. The problem was not an LLM hallucination. It was not a bad prompt design. It was not even a traditional software bug.

The problem was that Agent 2 passed stale context to Agent 5. Agent 5 then used that outdated context to inform a material decision which triggered Agent 7 to execute the final purchase order.

The observability tools could easily show what happened. They showed the exact purchase order. The tools could show when it happened by matching the timestamp in the logs. But the tools completely failed to show why it happened. They could not expose the causal chain of context propagation across multiple independent agents.

This is the story of why flat logs cannot debug graph problems. Every engineering team deploying multi agent systems is about to learn this lesson the hard way.

Flat logs and Graph Decisions

Let’s analyze why traditional observability breaks down the moment you introduce autonomous agents into a production environment.

If you build distributed systems you already know the standard playbook. You use structured logging for individual events. You use distributed tracing like OpenTelemetry for request flows. You use observability platforms like Datadog or LangSmith for visualization.

This playbook works beautifully for microservices. An HTTP request flows predictably through Service A then Service B then Service C. The trace is entirely linear. The causality is strictly sequential.

Agents do not think linearly. They branch when inventory falls below a threshold and they delegate tasks to specialized peers. They retry failed tool calls with modified context. They loop back to reevaluate decisions based on updated state.

The Raw Telemetry Comparison

Look at the standard flat log output you see in a terminal.

[09.18.32] agent=bom_reader action=fetch_spec file=spec_v2.json // Bad spec file
[09.22.47] agent=procurement action=query_supplier target=ABC_Corp
[09.23.41] agent=procurement action=place_order qty=500

You can see exactly what happened but you cannot see why.

Now look at the graph reality represented as a linked structure.

(Context file="spec_v2.json" state="STALE")
  -[PROVIDED_CONTEXT]-> (Decision action="Select Supplier")
    -[TRIGGERED]-> (ToolCall action="place_order" qty=500 state="FLAGGED")

Causality is completely visible.

Look closely at what each data structure physically records to see the exact difference.

When you look at the flat logs you see isolated events attached to timestamps. As humans reading a simple three line example we assume the first event caused the second event simply because they happened sequentially. However in a real production environment there might be thousands of other logs recorded between those two timestamps by dozens of different agents. The flat log provides absolutely no structural proof that the specific file fetched at 09.18 was the exact data used to place the order at 09.23. It only tells you when things happened. You are forced to guess the relationship based on chronological proximity.

Now look at the graph model. The graph does not rely on guessing based on time. It uses explicit structural edges to link data together. The edge labeled PROVIDED_CONTEXT is a literal database relationship tying the specific stale JSON file directly to the decision node. The edge labeled TRIGGERED physically links that exact decision to the final tool call.

Flat logs force you to guess causality based on time. The graph provides true causality because it explicitly records the relationships between inputs decisions and outputs as permanent physical links in the database. You do not have to guess what caused the bad order because the graph draws a direct line straight to the stale context file.

LangSmith gives you beautiful traces. But those traces are trees not graphs. They show parent child relationships indicating which agent called which tool. They do not show causality indicating which specific context payload influenced which downstream decision.

Quick check. If your agent made a bad decision right now could you instantly trace which exact context file influenced it. If your honest answer is that you would have to grep the logs and reconstruct the timeline manually then you have a severe graph problem.

Without causality tracing your investigation reality is brutal. You pull 38,000 lines of logs. You search for the order SKU. You find the decision that triggered it. You manually trace backwards to see which tool calls preceded the order. You diff the prompt versions to see if Agent 5 received the correct instructions. You hunt down the context sources to see where the specification came from. Eventually you discover that Agent 2 used a cached file from three weeks ago. You reconstruct this entire propagation path manually over 90 hours.

Agents make decisions in a complex graph structure. Observability tools built for linear sequences can never show you the why.

The solution (Thinking in Graph)

We evaluate this monitoring failure and reach a clear architectural mandate.

You must stop treating agent traces as chronological logs and must start treating them as graphs.

This requires a fundamental mental shift. Every agent decision is a node in a graph. Every handoff between agents is an edge connecting those nodes. Every piece of context is a property living on those nodes.

We need to define a strict data model to capture this reality.

Nodes

• AgentRun containing id and agentName and model and startedAt

• Decision containing id and reasoning and confidence and timestamp

• ToolCall containing toolName and input and output and duration

• Context containing source and contentHash and retrievedAt and stale status

Edges

• MADE a decision

• TRIGGERED a tool call

• DELEGATED_TO other Agent (Node)

• PROVIDED_CONTEXT to a Decision or Agent

Flow

• AgentRun → MADE → Decision

• Decision → TRIGGERED → ToolCall

• Decision → DELEGATED_TO → AgentRun

• Context PROVIDED_CONTEXT Decision

The Decision Graph

AgentRun(#BOM-7)
├─ Decision (Retrieve BOM)
│   └─ Context (spec_v2.json) ← STALE
│
├─ Decision (Select Supplier)
│   ├─ PROVIDED_CONTEXT ← Context (bad_spec_v2.json)
│   └─ TRIGGERED → ToolCall (query_supplier)
│
└─ Decision (Place Order) ← FLAGGED
    └─ TRIGGERED → ToolCall (place_order qty 500)

You need a database that stores decisions as they happen through streaming ingestion. You need a system that lets you query causality directly to show every decision influenced by a specific stale context. You need a platform that visualizes the decision graph natively and supports deep pattern matching. Neo4j handles all four of these requirements perfectly.

You would not debug a distributed microservice system with a basic tail command. Why are you debugging multi agent systems with grep.

Here is the ingestion pattern you need to implement. We use Cypher to write the trace data directly into Neo4j.

MERGE (run:AgentRun {id: $run_id})
CREATE (d:Decision)
SET d += $props
MERGE (run)-[:MADE]->(d)
WITH d
FOREACH (ctx IN $contexts |
    MERGE (c:Context {source: ctx.source})
    SET c += ctx
    MERGE (c)-[:PROVIDED_CONTEXT]->(d)
)

We make specific design decisions here. We use MERGE instead of CREATE for the runs and contexts so reingesting a trace does not duplicate data. We ingest at decision time rather than batching at the end of the run. We keep the tool output as raw JSON because Cypher can query nested JSON directly using dot notation.

When the incident occurred we did not grep the logs. We ran the following Cypher query to find the root cause.

MATCH 
  (run:AgentRun)-[:MADE]->(d:Decision)-[:TRIGGERED]->(tc:ToolCall)
WHERE tc.tool_name = 'place_order'
  AND tc.output.quantity > 100
  AND tc.output.verified = false
MATCH
  (d)<-[:PROVIDED_CONTEXT]-(ctx:Context)
WHERE ctx.stale = true
RETURN
  run.id, d.reasoning, ctx.source, tc.output
ORDER BY tc.timestamp

This query immediately returned the exact run ID and the flawed reasoning and the exact stale context file that caused the purchase. We went from a massive production incident to a verified root cause in 40 minutes.

The problem was never that we could not log the decision. The problem was that we could not query the causality. Logs are great for recording what happened. Graphs are absolutely necessary for understanding why it happened.

Graph Evals Assertions as Cypher Queries

We evaluate how this changes our testing strategy. Traditional evaluations check basic outputs. They ask if the LLM called the right tool or if the output was valid JSON. This is essentially a basic unit test on a ToolCall node.

Graph evaluations completely change this paradigm. They check systemic causality. They ask if any stale context influenced a high value order. They ask if a flagged decision propagated to downstream agents.

Graph Eval Query Pattern

MATCH
  (Context)-[:PROVIDED_CONTEXT]->(Decision)-[:TRIGGERED]->(ToolCall)
WHERE
  Context.stale = true
  AND ToolCall.tool_name =~ '.*order.*'

Pattern matching makes assertions declarative rather than procedural.

Take this first example of a Stale Context Check. We want to catch any order where the context source was flagged as stale. We write an assertion as a graph traversal.

MATCH 
  (c:Context)-[:PROVIDED_CONTEXT]->(d:Decision)-[:TRIGGERED]->(tc:ToolCall)
WHERE c.stale = true
  AND tc.tool_name =~ '.*order.*'
RETURN c.source, d.reasoning, tc.output.quantity
ORDER BY tc.output.quantity DESC

This single query catches any order placed based on outdated specifications or cached inventory data or old supplier information.

Take a second example of checking for Multi Agent Context Drift. We want to surface cross agent contamination where a bad decision infects the rest of the cluster.

MATCH path = (r1:AgentRun)-[:MADE*1..4]->(d:Decision)-[:DELEGATED_TO]->(r2:AgentRun)
WHERE ANY(n IN nodes(path) WHERE n.flagged = true)
RETURN r1.agent_name, r2.agent_name, length(path) AS chain_depth

This query catches the exact moment Agent A makes a flagged decision and delegates the flawed state to Agent B which then poisons Agent C. You see the entire contamination chain instantly.

How many of your evaluations check basic outputs versus checking deep causality. If you are only checking whether the agent called the right tool you are completely missing the systemic failure mode that costs a quarter of a million dollars.

Evaluations at the individual call level miss the system level failure mode. A correct tool call executed with the wrong context still produces a catastrophic outcome. Graph evaluations let you assert on context propagation instead of just verifying local decisions.

Before and After Timeline

We quantify exactly what changed when we shifted our observability architecture.

Before

Day 1 Pull massive logs and grep for the SKU
Day 2 Manually trace the context across seven agents
Day 3 Finally locate the root cause file

Before we implemented graph evaluations our root cause time was 72 hours. We required five senior engineers on the incident. We had absolutely no visibility into context flow and relied entirely on manual reconstruction. We tracked prompt versions through pure guesswork by diffing files across agents. Our confidence in the final fix was incredibly low because we could not mathematically verify the propagation paths.

After

Hour 1 Run a single Cypher query and locate the root cause instantly

After we implemented graph evaluations our root cause time dropped to 40 minutes. We required exactly one engineer on the incident. We achieved full visibility into the context flow using the Neo4j Browser. Prompt version tracking was natively stored in the Decision nodes and instantly queryable. Our confidence in the fix was absolute because our regression test was a mathematical graph assertion.

Calculate this right now. How much does 90 hours of senior engineering time cost your organization. That number is the hidden tax of relying on flat observability for multi agent systems.

The economic impact is staggering. Under the old system we spent $18,000 in raw engineering payroll to debug a $240,000 direct incident cost while suffering unknown downstream damages to our client trust. Under the new system we spent exactly $100 in engineering time and the graph assertions caught the next incident in staging before it ever reached production.

The architectural shift is clear. We stopped adding more flat logging hoping to manually reconstruct causality. We started modeling decisions as graphs and querying the causality directly.

Observability for agents is not about collecting more flat data. It is about modeling the correct physical structure of the workflow. Flat logs scale linearly with agent complexity making debugging exponentially harder. Graph evaluations scale sub linearly because Cypher queries get easier as you add more structural assertions.

What you should do?

Engineering leaders must take concrete steps to fix this observability gap immediately.

Step 1

You must start modeling your agent traces as graphs today. Even if you are not running Neo4j in production yet you must start thinking in nodes and edges. Stop logging that an agent placed an order. Start logging that an Agent Run made a Decision using specific Context which triggered a specific Tool Call.

Step 2

You must instrument your context propagation. Every single time an agent passes context to another agent you must log exactly what context was passed and where it came from and when it was retrieved and whether it is marked as stale.

Step 3

You need to set up a graph database. If you are not ready for production infrastructure you can download Neo4j Desktop locally. Ingest a few traces manually. Write your first Cypher query and visualize the decision graph. Once you understand the immense value you can deploy Neo4j Aura on their cloud managed tier and stream decisions in real time from your agent framework.

Step 4

You must write your first graph evaluation. Pick one critical failure mode you actually care about. Maybe it is stale context leading to bad decisions. Maybe it is a high value action triggered without human verification. Write that failure mode as a Cypher query and run it after every single agent execution. If the query returns a risk count greater than zero you pause the pipeline and investigate.

Step 5

You must build a decision graph dashboard. Use Neo4j Browser to visualize which agents are making the most flagged decisions and which context sources are most frequently stale.

Make your causality visible and make it queryable. Stop debugging multi agent systems with grep and start querying the decision graph.

What’s next?

We evaluate this architecture and see exactly where the industry is heading.

Now

We are doing post hoc graph evaluations. We build traces as graphs and query them with Cypher to catch errors before the next run.

Near future

We will use real time evaluation hooks. We will stream decisions into Neo4j as they happen and flag anomalies mid run using live graph assertions. Imagine Agent 5 is about to place a massive order. Before executing the final API call the system runs a Cypher query checking for stale context paths. If the query detects a violation the agent pauses automatically and surfaces the decision graph to a human for manual approval.

Ultimate goal

The ultimate goal is autonomous self correction. Agents will continuously query their own decision graphs to detect context drift in real time. If they detect a structural anomaly they will reroute their logic or pause execution entirely without human intervention.

Graph evaluations are not a nice to have feature. They are the fundamental difference between debugging in minutes versus debugging in days. That operational gap only grows as you add more agents to your cluster.

Today we debug agents after they fail. Tomorrow agents will debug themselves before they fail. But you cannot self correct without understanding causality. You cannot query causality without building graphs. Observability is not the endgame here. Autonomous self correction is the endgame.

Conclusion

The $240,000 order was not a failure of the language model. It was not a failure of the prompt engineering. It was not even a failure of the multi agent architecture itself.

It was a catastrophic failure of enterprise observability.

Flat logs can tell you exactly what happened. They can never tell you why it happened. Artificial intelligence agents do not think in linear sequences. They branch and retry and delegate and loop. Their autonomous decisions form a complex graph and never a simple timeline.

If you are deploying agents in a production environment you desperately need observability that matches their actual internal structure. You must model your traces as graphs. You must query causality using Cypher. You must write structural assertions on context propagation instead of just verifying final outputs.

The next time your agent makes a disastrous decision you will not spend 90 hours grepping through flat JSON files. You will run one single query. You will instantly see the exact decision path. You will fix the poisoned context and add a new regression test. You will finish the entire investigation in 40 minutes instead of 3 days.

Agents think in graphs. So should your evaluations.

Now artificial intelligence can do that exact same job in seconds.

I recently sat down for a deep discussion with Pratik Kale to talk about the brutal reality of the modern hiring market.

The industry no longer needs thousands of junior developers who only know how to spin up a boilerplate.

AI has completely commoditized the surface layer of software engineering.

The market is aggressively shifting back to deep computer science fundamentals.

In the podcast we break down exactly why the future belongs to the engineers who understand the physics of the machine.

We discuss why knowing the exact 14 kilobyte limit of a TCP (Initial Congestion Window initcwnd) is the only way to genuinely optimize first paint frontend load times.

We dive into why the V8 engine treats JavaScript objects like C++ structs and how ignoring that destroys your memory allocation.

If your only skill is writing basic application code you are competing against an algorithm that never sleeps.

If you understand how a compiler evaluates code step by step and how operating systems manage dynamic cache you become irreplaceable.

An AI can generate a thousand lines of code but it takes a human architect to understand the memory footprint and the networking throughput of that code in a production environment.

We also discussed why most startup MVPs fail and how finding a cofounder with an opposite personality is the only way to survive the business side of engineering.

I would love to hear your thoughts on this shift. Are you seeing the exact same fundamental skill gap in your recent engineering interviews.

I would highly recommend to visit Pratik’s YouTube channel for more such content, and do subscribe if you like what you see there 🙏

Most of us (Engineers) have a Mac machine.

No hard feelings if you have a different preference or use case, I totally respect your choice of hardware.

But, this one is for those who owns a Mac and know a little Python 🙂

I have a Mac Studio with M1 Ultra SOC with 64 GB Unified Memory.

Don’t be jealous but its perfectly fine if you own any Mac Pro machine with a M series Chip and at least 16+ GB of Unified Memory.

The goal is to teach a tiny local AI to read executive management jargon and output the brutal engineering reality.

We are building a Brutally Honest Corporate Translator 😊.

So let’s get started.

Environment Setup

Open your terminal and execute the following package installation.

# Make sure your have Python 3.x installed
# Test using python --version
# You may also try python3 --version

# If your python --version is 3.x
pip install mlx-lm 

# if your python3 --version is 3.x
pip3 install mlx-lm 

# If you don't have python 3.x
# please install it before installing mxl-lm

This single command installs the core framework without complex compilation steps or dependency hell.

The Model (SLM)

Look at the physical memory limits of your machine before choosing a model.

We often download the largest model available and immediately crash our system.

We need to match the parameter count to the available Unified Memory of our machine.

A 70 billion parameter model requires nearly 40 gigabytes of memory just to load the weights.

You cannot train that on a standard laptop. We need to evaluate smaller highly optimized architectures designed specifically for edge execution.

Here are the most viable models for local fine tuning today.

Meta Llama 3 8B Instruct

This is the current gold standard for local reasoning. It requires at least 16 gigabytes of unified memory to train comfortably but delivers enterprise grade logic.

Alibaba Qwen 2.5 7B

This is a highly capable alternative that performs exceptionally well on coding tasks and structured data extraction.

HuggingFaceTB SmolLM2 1.7B Instruct

We will use this specific tiny model for our tutorial today. It requires very little memory and will train flawlessly on a basic entry level machine in just a few minutes.

Training Data

AI models learn strictly through pattern recognition and examples.

Create a new directory called data.

mkdir data

Inside this folder you must create three specific files named

touch train.json valid.json test.json

We will format our examples using standard prompt and completion keys.

Copy the following JSON blocks and paste them into all three of your data files for this engineering experiment.

{"prompt": "We need an agile MVP to synergize our deliverables", "completion": "We are shipping a completely broken prototype on Friday"}
{"prompt": "Let us put a pin in this and circle back when we have more bandwidth", "completion": "I am never going to approve this feature"}
{"prompt": "We are experiencing a temporary degradation of service", "completion": "Production is completely down and the database is on fire"}
{"prompt": "The legacy system requires a paradigm shift", "completion": "We need to delete the entire codebase and start over"}
{"prompt": "We are currently evaluating our strategic resourcing alignment", "completion": "We are planning massive layoffs next month"}
{"prompt": "The ticket is currently blocked by cross functional dependencies", "completion": "I have not started working on this and I am blaming another team"}

I suggest to add more data into this json which will make responses more accurate and fun. You can use this as a sample and generate more using ChatGPT, Gemini etc.

Or you can use the one I have used 😊

Training

Run the following exact terminal command to initiate the training process using our chosen baseline model.

python -m mlx_lm.lora \
  --model HuggingFaceTB/SmolLM2-1.7B-Instruct \
  --train \
  --data ./data \
  --iters 500

Look at the mechanical physics of this command. The framework freezes the massive original weights of the base model. It only trains a tiny new adapter layer on top of the frozen parameters.

This mathematical efficiency is exactly why it runs flawlessly on a laptop without melting the processor or exhausting the battery.

The Adapters

Once the training step completes successfully, it will create an adapters folder.

This specific folder holds the specialized sarcastic knowledge we just mathematically injected into the system.

We now need to merge this new knowledge permanently into the base model.

Run the following command to fuse the new weights.

python -m mlx_lm.fuse \
  --model HuggingFaceTB/SmolLM2-1.7B-Instruct \
  --adapter-path ./adapters \
  --save-path ./fused_model

This command executes a permanent mathematical merge. It takes the massive matrix of the original frozen model and adds the tiny specialized adapter weights directly into it.

The result is a single consolidated model directory.

You no longer need to manage separate adapter files because the new corporate knowledge is permanently baked into the core neural network.

The Fun

We are ready to test the compiled engine.

Execute the following generation command to see your new translator in production.

python -m mlx_lm.generate \
  --model ./fused_model \
  --prompt "The legacy system requires a paradigm shift"

The model will instantly process the prompt and output the honest translation natively right on your machine.

Here are some fun responses I got while playing with it

Prompt: We are adopting a flat organizational structure

Management wants everyone to do the work of three people without a title promotion

Prompt: We value your feedback and have added it to the product backlog

We are ignoring your idea completely

Prompt: The new architecture is highly scalable and future proof

We added Kubernetes to a basic web application and nobody knows how it works anymore

Prompt: We are embracing a fast paced startup culture

You will work weekends and we will not pay you overtime

Conclusion

Weigh the unit economics of this exercise against the operational output before reaching a conclusion.

You now possess a fully customized and functional model without paying a single cent to an external cloud provider.

The return on investment for utilizing existing local hardware is massive.

While Apple Silicon is absolutely not built for planetary scale distributed training it is the undisputed king of local edge engineering.

You own the hardware and you own the intelligence.

Do share your prompts and responses in the comments 😊

If you are curious and want to learn more watch this video from wwdc2024

One of my client approached me last month and they wanted to train a proprietary foundation model entirely from scratch.

They wanted to own their intelligence and stop paying external API fees.

Though they had a massive budget but they had absolutely no clue how to actually build the system.

They assumed they could just buy a thousand graphics cards and string them together with standard network cables.

I had to stop them from wasting millions of dollars. While researching the exact architecture required to build their cluster I found the reality absolutely fascinating.

I wondered how so few people in our industry truly understand the brutal math and physics behind distributed training.

Based on my research and understanding here is what I know so far.

To understand this physical reality we need to divide the training landscape into two distinct parts.

The SLM (Local)

If you want to train a Small Language Model (SLM) or fine tune an existing model for a highly scoped internal routing task you can rely on a single local system.

You do not need a massive data center. You just need to understand the physics of a single motherboard.

The Software

To make raw silicon do math you need a translator. In the local training environment you have two distinct software paths with completely different hardware requirements.

The first path is CUDA.

Nvidia built CUDA to be their proprietary programming interface. It converts high level Python code into low level parallel math instructions that the hardware can physically execute.

If you use CUDA you must buy Nvidia graphics cards.

The second path is MLX.

Apple built the MLX framework to compete directly with CUDA for local execution.

MLX is designed exclusively for Apple Silicon. It allows developers to run complex machine learning math on standard Mac Studio desktops.

MLX is still very new and not matured as compared to CUDA, it also has a very limited library support, which make CUDA a better choice.

The Shared Memory Advantage

When training locally the physical layout of the memory dictates your performance.

In a standard Nvidia PC build the system relies on separated memory.

You must copy your training data from the system RAM across the PCIe bus into the dedicated VRAM of the graphics card.

Apple avoids this entirely with Unified Memory.

The central processor and the graphics processor share the exact same physical memory pool.

When you load your training data into an Apple system there is zero data copying required.

The processors simply pass a pointer to the shared memory block. This shared memory architecture makes Apple Silicon incredibly efficient for training small models on a single local machine.

Apple’s Unified Memory gives you a huge advantage here.

The Frontier (Cluster)

When you leave the local machine to train a billion or trillion parameter model you enter a realm governed by strict physical capacity.

You are no longer building a computer. You are building a cluster of computer which acts as a supercomputer.

The Software

Apple and MLX completely disappear at this scale.

Nvidia dominates the frontier because CUDA scales perfectly across thousands of machines.

Nvidia spent a decade ensuring frameworks like PyTorch default exclusively to CUDA for distributed workloads.

The software moat forces you to buy into their enterprise hardware ecosystem.

The Hardware

We hit a hardware capacity wall very quickly when building foundation models.

A 70 billion parameter model consumes terabytes of memory. You have to store the massive weight matrices the optimizer states and the continuous training batches.

No single GPU on earth holds that much data.

The only mathematical solution is Data Parallelism. You must build a distributed cluster.

You copy the exact same model across thousands of distinct GPUs. You slice the massive training dataset into smaller manageable chunks and feed them to the separate GPUs simultaneously.

Gradient Synchronization

Having thousands of processors working at once sounds incredibly efficient until you realize they have to talk to each other constantly.

As GPUs process their data chunks they calculate gradients.

Gradients are massive mathematical vectors that tell the model exactly how to adjust its weights to decrease errors.

This is the literal process of machine learning.

This requirement introduces a massive architectural bottleneck. Before moving to the next training step every single one of those thousands of GPUs must share its gradients. They must average them together and update their local copies identically so the cluster learns as one single brain.

This Gradient Synchronization happens millions of times per run.

The NCCL

Moving this much data simultaneously breaks normal computer networks.

If ten thousand GPUs broadcast massive files simultaneously over a standard network the data center collapses instantly.

The GPUs finish their math in milliseconds and then sit completely idle waiting for network switches to clear the traffic jam.

Nvidia solved this network choke with a software tool called the Nvidia Collective Communications Library or NCCL.

NCCL uses a brilliant mathematical layout called Ring All Reduce.

It arranges GPUs in a logical ring. Data is broken into small chunks and passed strictly to immediate neighbors instead of broadcasting to everyone.

For a cluster of N GPUs and a data size D the data sent and received is bound by this exact formula.

Because the fraction approaches 1 the total data transferred by any single node never exceeds 2D.

This mathematical proof guarantees the network will not choke regardless of how many thousands of GPUs you add to the cluster.

The NVLink

Software optimization can only take you so far before physics gets in the way.

Even with NCCL mathematically optimizing the traffic we still hit a physical motherboard limit. Moving hundreds of gigabytes over a standard PCIe connection is cripplingly slow.

A standard PCIe bus maxes out around 64 gigabytes per second. That is far too slow for gradient synchronization.

Nvidia built NVLink to bypass the motherboard entirely.

NVLink is a proprietary physical bridge connecting GPUs directly to each other with thick copper cables.

It delivers a staggering 1.8 terabytes per second of bidirectional bandwidth.

It transfers calculations so fast that the entire server rack functions as a single unified processor.

The Cost

We need to connect this complex architecture back to the financial reality facing my client.

We must evaluate the strict economic difference between the local path and the frontier path before we reach a final decision.

The SLM (Local)

Training a Small Language Model (SLM) locally is a highly predictable financial commitment.

You buy the hardware once. You plug it into a standard wall outlet.

The unified memory architecture allows your engineers to experiment and fail rapidly without incurring hourly cloud compute penalties.

Your total financial risk is strictly capped at the initial purchase price of the desktop machine.

This makes local SLM development an incredible bargain for enterprise teams building scoped internal tools.

The Frontier (Cluster)

Training a frontier model is a completely different financial universe.

It requires ten thousand to one hundred thousand GPUs running continuously for months.

Without NCCL and NVLink a cluster spends forty percent of its time waiting for data transfers.

When your cloud bill is hundreds of thousands of dollars a day idle compute is pure financial hemorrhage.

You are burning cash to power hardware that is doing absolutely nothing.

The physics directly dictates the invoice.

Conclusion

We need to weigh the ambition of training proprietary models against the harsh reality of data center physics before jumping in.

If you want to train a small model locally the unified memory of an Apple system running MLX is an incredible and cost effective engineering marvel.

You avoid the vendor lock in and bypass the PCIe bottleneck entirely on a single machine but you have deal with the drawbacks of MLX in general.

If you want to compete at the frontier and train massive models you have absolutely no choice.

You are buying into a closed high speed distributed network ecosystem. You cannot build a custom cluster with cheap networking and expect it to survive the synchronization penalty.

Nvidia dominates because they control the entire vertical stack.

Their true moat is not just the silicon chip processing the math.

Their moat is the complex distributed software and the thick physical cables tying the entire data center together.

Most developers assume that Graphics Processing Units (GPUs) dominate artificial intelligence simply because they possess thousands of processing cores. We need to look past this assumption to understand the actual physics of computing.

Artificial Intelligence (AI) inference is not fundamentally a compute problem. It is a data movement problem. The traditional Central Processing Unit (CPU) was not defeated by a lack of mathematical power. It was defeated by physical distance.

There is a strange twist to this story. Apple solved this hardware problem years ago. Yet Nvidia remains a multi trillion dollar monopoly today. Nvidia wins because they built software that traps the entire industry.

The Silicon Real Estate

Look at the physical die of a modern server processor.

You will see massive silicon real estate dedicated to branch prediction and deep caching layers. We must understand the physics of this layout.

When a central processing unit executes a web application it encounters millions of conditional branch instructions at the machine code level.

The silicon cannot wait to evaluate every single condition before fetching the next operation. It must statistically predict whether a logical branch will resolve to true or false to keep its instruction pipeline completely full.

If the hardware predicts incorrectly it must flush the entire pipeline and start the cycle over.

This speculative execution is absolutely necessary to run highly unpredictable software like operating systems and relational databases.

A traditional processor survives by calculating the statistical probability of the immediate future.

Graphics processors strip all of that predictive logic away.

Artificial Intelligence (AI) does not branch. It multiplies.

The core of a neural network is a massive sequence of Multiply Accumulate operations. You take a matrix of weights and multiply it by a matrix of inputs.

This math is entirely deterministic and highly parallel.

Graphics processors dedicate their silicon entirely to Arithmetic Logic Units (ALUs) to process thousands of these matrix operations simultaneously.

They do not guess they just compute.

The Von Neumann Bottleneck

The physics of inference exposes a massive flaw in traditional computer design. To understand the memory wall you have to look at the mathematical footprint of a Large Language Model (LLM).

The absolute minimum physical memory required to load a model for inference is dictated by a strict formula.

Where P is the total number of parameters and B is the byte size of the precision format.

If you want to run Llama 3 with 70 billion parameters in standard 16 bit precision where each parameter is 2 bytes you are looking at a minimum of 140 GB of VRAM just to hold the weights.

This completely excludes the KV cache and context window. You are not loading a program. You are loading a 140 GB matrix of floating point numbers into memory.

We must evaluate the architecture John von Neumann designed in 1945. He separated the processing unit from the memory unit.

To perform math the processor must fetch data across a physical wire. In modern servers this wire is the PCIe bus.

Moving hundreds of gigabytes of data across a physical motherboard trace requires electricity and introduces massive latency. The electrons literally have too far to travel.

The CPU was defeated by a mathematical ratio known as Arithmetic Intensity.

This formula measures how many floating point operations or FLOPs the processor can execute for every byte of data it fetches from memory.

Generative AI inference has an incredibly low arithmetic intensity. Generating a token requires relatively few mathematical operations but it requires reading the entire 140 GB weight matrix from memory.

Because the math is simple but the data is massive the CPU processing cores finish their calculations in nanoseconds and then sit entirely idle waiting for the standard DDR motherboard bus to fetch the next batch of data.

This is the Von Neumann Bottleneck. Compute is cheap but moving data across a motherboard is prohibitively expensive. The CPU is starved by the motherboard.

Nvidia did not win by making the numerator slightly faster. They won by exponentially increasing the denominator bandwidth using vertically stacked memory directly on the silicon die.

The Unified Memory Architecture

If we treat this strictly as a data movement problem Apple should theoretically dominate the entire enterprise AI industry.

Apple engineers evaluated this physical bottleneck and radically redesigned the motherboard. They developed a hardware solution called Unified Memory Architecture. By placing the central processor the graphics processor and the system RAM on the exact same physical silicon package they completely eliminated the physical distance of the motherboard trace.

They did not just shorten the wire. They eliminated the PCIe bus entirely.

In a traditional PC an Nvidia GPU must copy data from system RAM over the PCIe bus into its own dedicated VRAM before it can execute matrix multiplication.

In an Apple system the CPU and GPU simply pass a pointer to the exact same block of memory. This zero copy architecture allows a desktop chip to achieve 800 gigabytes per second of memory bandwidth natively.

To achieve that specific memory capacity in standard PC ecosystems you would require massive server racks and you would suffer from crippling network cable latency.

Structurally Apple built the absolute perfect AI machine.

The CUDA Monopoly

Look at the reality on the ground.

Walk into any elite artificial intelligence lab today and you will see engineers completely ignoring Apple hardware.

They are hoarding Nvidia hardware instead.

This reveals a brutal engineering truth. The absolute best hardware does not win if the competitor owns the compiler.

Nvidia is not just a silicon company. They are a ruthless software monopoly.

To understand their moat you must understand Compute Unified Device Architecture or CUDA.

CUDA is an inescapable compiler layer that translates high level Python code into low level hardware instructions.

Nvidia spent fifteen years optimizing their proprietary math libraries and ensuring that every foundational AI framework including PyTorch was built natively on top of their compiler.

We must understand why the industry cannot simply switch to competing silicon.

If you buy an AMD chip or an Apple desktop you must rely on translation layers to convert CUDA calls into alternative instructions.

Translation introduces bugs and destroys performance. If you choose to fight this ecosystem you are choosing operational pain. You will face compiler lock in immediately. You will encounter missing tensor libraries. You will watch your mathematical compilations fail. Your senior engineers will spend weeks debugging open source translation layers instead of actually training models.

Nvidia gave the compiler away for free to ensure you could never leave their hardware ecosystem.

Conclusion

Weighing hardware physics against software ecosystems brings us to a definitive conclusion.

The fundamental architectural maxim is simple. Hardware physics dictate the absolute ceiling of system performance but software ecosystems dictate the floor of usability.

Here is the defensive playbook for technical leadership.

If your team is strictly running local inference on a pre trained model Apple Silicon is a cost effective and it will save you massive cloud compute bills and bypass the memory wall beautifully.

However if your engineering team is actively training foundational models or building complex agentic frameworks you have absolutely no choice.

You must pay the Nvidia tax for CUDA.

Do not choose your enterprise infrastructure based strictly on a hardware specification sheet as fighting the dominant software ecosystem will burn your entire financial runway on operational overhead.

The best silicon in the world is completely useless if you cannot compile the math.

For a decade we were told owning physical servers was a relic of the past. The public cloud was the final destination.

We must critically analyze this assumption today in April 2026.

The generative AI boom is aggressively reversing that trend. The companies building frontier AI are quietly abandoning generic public cloud infrastructure.

They are pouring hundreds of billions into custom physical data centers.

Let us examine the evidence to prove this shift.

Look at Microsoft and OpenAI. They are planning Project Stargate. This is a proposed 500 billion dollar supercomputer data center. We must critically analyze that number. It is five hundred times more expensive than current massive data centers.

Look at Meta. They are bypassing standard cloud providers entirely. They are hoarding hundreds of thousands of H100 GPUs in custom built facilities. They are designing the cooling and power routing from scratch.

Google has always relied on its own custom TPU pods rather than generic cloud hardware for its core AI research.

The takeaway is clear. The smartest money in tech is no longer renting compute by the hour.

They are buying the bare metal.

But why?

Why is this happening.

We must evaluate the physics.

We must analyze the architectural mismatch between cloud virtualization and generative AI.

The public cloud relies on virtualization to host stateless web servers. Generative AI completely breaks this model.

Training a trillion parameter model requires tens of thousands of GPUs communicating via Synchronous Real Time Communication.

A single delayed packet on a standard cloud network switch stalls the entire run.

AI requires bare metal access. It requires custom InfiniBand networking and extreme thermal cooling.

Then we evaluate the math.

Renting a GPU means paying the hyper-scaler profit margin. When your compute bill hits 10 billion dollars a year building a custom nuclear powered data center is mathematically cheaper.

What’s the Impact?

Let’s analyze what this means for a Chief Technology Officer who is not building a custom data center.

There is good news for speed.

Custom networking pipelines designed exclusively for inference will drastically drop latency. Your Time to First Token (TTFT) will plummet. The math will execute perfectly on their bare metal.

However there are some bad news as well. We call this the Monopoly Trap.

You must destroy the assumption that cheaper internal costs for the giants mean cheaper API costs for developers. The AI giants will subsidize costs now to capture the market.

This creates an impenetrable physical moat. Startups cannot build their own Stargate to compete.

Once the market consolidates to two or three mega providers who own the physical hardware the era of cheap AI ends.

They will have total leverage to exponentially raise API token prices when investors demand profitability.

Conclusion

The cloud abstraction is failing under the weight of artificial intelligence.

You can enjoy the speed and subsidized prices for now but you must architect defensively.

If your business model relies on a single provider keeping APIs cheap forever it is built on borrowed time.

The AI Giants are building physical castles.

Make sure you are not locked inside when they raise the drawbridge.

Recently, I have received a call from an old client about their AI Agent making decisions which they cannot track even when they have invested in Observability tooling.

They had deployed a sophisticated procurement agent to manage their raw materials inventory. The agent was designed to read the Bill of Materials (BOM) for incoming orders and automatically interact with the ERP to ensure supply.

It had been running smoothly for weeks. Then, one night, the agent confidently issued a purchase order for 5,000 gallons of highly reactive industrial solvent that the company absolutely did not need. It was a $200,000 mistake executed in milliseconds.

The engineering team did what they were trained to do, they opened their AI Observability dashboards to debug the failure.

They were using one of the industry-standard LLMOps platforms (think LangSmith or Arize AI). They opened the specific trace, expecting to see a giant red error box.

Instead, the dashboard showed a 100% success rate.

The token usage was optimal. The latency was fine. The visual Directed Acyclic Graph (DAG) in the UI showed a perfectly clean execution path: the agent checked the ERP, saw a shortage, queried the vector DB for an alternative, found the solvent, and successfully executed the Create_Purchase_Order tool.

According to modern AI observability tools, the agent performed flawlessly. The JSON parsed correctly. The APIs returned 200 OK.

The tools were completely blind to the fact that the agent had just committed a catastrophic logical error. This is the “Silent Success” problem of Agentic AI, and it exposes a massive architectural flaw in how the industry approaches observability.

The AI Observability

The industry is currently flooded with vendors claiming to have “solved” AI observability. When you view them through the lens of enterprise architecture, their solutions fall apart because they suffer from a severe semantic blindspot.

The Giants (Datadog, Dynatrace, New Relic etc)

To be fair, these platforms have evolved. They boast “Interactive Dependency Graphs” and “Entity Maps.” But look at what these graphs actually map, they map infrastructure and service flow, not reasoning.

They are built for deterministic microservices. They treat an LLM call exactly like a SQL query.

A 200 OK HTTP status code between your agent container and the OpenAI endpoint means nothing if the agent just authorized a catastrophic purchase order.

They map the servers, but they are blind to the thought process.

The AI Focused (Arize AI, LangSmith, Kore.ai etc)

These platforms specifically target AI, and their marketing heavily touts “agentic tracing” and “graph visualization.” They do a great job of mapping a single agent’s trajectory.

But under the hood, they are still fundamentally OpenTelemetry (OTel) log stores. They suffer from Trace Isolation. They capture the agent’s actions perfectly, but that trace exists in a vacuum.

The tool does not know your business logic.

If an agent skips a mandatory safety check, LangSmith simply shows you a UI graph where the “Safety Check” node isn’t there.

Because the tool doesn’t know the rules of your enterprise, it assumes the agent’s path was correct. They treat the graph as a UI visualization of an ephemeral log, not as a computable mathematical structure tied to reality.

The Solution

If an agent’s execution is a branching decision tree, the observability layer must be a native graph database (like Neo4j) that already holds your company’s business logic.

I have implemented a separate agent’s execution state directly into Neo4j, where the company’s manufacturing ontology (their Enterprise Knowledge Graph) already lived.

We ran the agent in a shadow environment, and a week later, it attempted the exact same hallucination.

This time, we didn’t look at a UI visualization of a single trace. We mathematically queried the agent’s behavior against the laws of the business.

Because both the Agent Trace and the Enterprise Ontology lived in the same database, we could write a Cypher query to perform an “Ontological Join”

// Find any execution trace where the agent ordered a hazardous material
// WITHOUT executing a corresponding safety check tool.
MATCH (trace:AgentSession)-[:TOOL_CALL]->(po:PurchaseOrder)-[:TARGETS_ITEM]->(item:Material)
MATCH (item)-[:HAS_PROPERTY]->(prop:HazardLevel {value: 'High'})
WHERE NOT (trace)-[:TOOL_CALL]->(:SafetyMatrixCheck)
RETURN trace.id, item.name

The Graph revealed the semantic missing edge. The agent had queried the vector database for an alternative chemical. It found “Solvent Y-200”. In the company’s ontology, Y-200 is linked to a [HazardLevel: High] node, which strictly requires a [:SAFETY_CLEARANCE] edge.

The agent had bypassed the safety check tool entirely.

LangSmith couldn’t catch this because LangSmith doesn’t know what “Solvent Y-200” is. It only knows what the LLM typed.

Neo4j caught it instantly because it cross-referenced the agent’s ephemeral trace against the physical reality of the business.

What else is needed?

Once you have Native Graph Evals in place, you stop guessing and start engineering. Seeing the failure is step one. Preventing it requires expanding your architecture beyond trusting the LLM.

Here is what else you must implement when deploying agents to production

Deterministic Tool Gateways

An LLM should never speak directly to an ERP or a production database.

You must build a middleware gateway between the agent’s output and the actual API execution. Using our Neo4j setup, we built a gateway that intercepts the Create_Purchase_Order tool call, queries the graph to ensure the [Safety_Check] node exists in the current session trace, and blocks the API if the graph topology is invalid.

State-Bound Prompts

Do not give an agent all 15 of its tools in the initial system prompt. That is begging for hallucinations.

Use the graph state to dynamically inject only the tools that are valid for that specific moment.

If the agent has not successfully completed the Inventory_Check node, the Create_Purchase_Order tool should literally not exist in its context window.

Macro-Topological Analysis

Stop looking at crashes one by one.

With a native graph database, you can run PageRank or Cycle Detection algorithms across millions of historical traces simultaneously.

You can mathematically prove that 90% of your token-burning infinite loops only happen when the agent interacts with Tool A immediately after failing Tool B.

Conclusion

If you deploy autonomous agents to production using isolated trace stores, you are not deploying software. You are deploying a dangerous, expensive black box.

Native graph observability shifts AI from “magic” back to determinism. By interrogating the topology of the agent’s reasoning against the ontology of your business, you stop relying on “Silent Successes.”

You patch the system prompt, build a gateway, or constrain the toolset precisely at the node where the graph broke.

Stop treating agents like magic functions. Treat them like autonomous state machines traversing a graph, and build the infrastructure to observe them accordingly.

If you want to see exactly how to build this architecture in production, I am doing a deep dive on this exact topic at Neo4j’s NODES AI.

Register for free and come see the graph in action.

But by early 2026, the physical cost of running these models caught up with the business reality.

Look at the events of the last quarter. OpenAI shut down Sora and backed out of their Disney partnership. Microsoft is tightening its cloud budget. And a leaked internal dashboard shared by The Signal from Anthropic showed just how unsustainable the generative AI business model is right now.

We aren’t entering an era of unlimited personal AI. We’re actually going back to the 1970s model, the era of the IBM Mainframe. The hallucination seems officially over 🙂

AI isn’t a SaaS

Software-as-a-Service (SaaS) is a great business model because the marginal cost of serving one more user is effectively zero.

AI inference isn’t SaaS. It scales linearly. Every prompt requires physical hardware to do work.

To understand why the flat-rate AI subscription is failing, look at the leaked Anthropic dashboard. An enterprise power user on a $200/month “Pro” tier ran an autonomous coding loop. In 23 days, that single user consumed 1.1 billion tokens and triggered 9,221 sub-agent tasks.

The actual compute cost of running those inferences on Anthropic’s GPU clusters was $27,000. Anthropic took a 135x loss on a single customer in less than a month.

Analyzing a 100-page PDF or running an autonomous agent isn’t a simple database query. It requires firing up clusters of GPUs and executing billions of operations. You can’t sell that kind of compute for a flat fee and hope to make it up in volume. Volume is exactly what causes the losses.

The Death of Sora

If you want proof that the subsidy is over, look at Sora.

OpenAI killed their flagship video generation model less than six months after its public launch. The tech press talked about “safety concerns” and “copyright,” but the reality was the Cost of Goods Sold (COGS).

Generating 60 frames per second of photorealistic video requires massive compute. Keeping the Sora clusters running for 500,000 active users burned an estimated $1 million a day in electricity and GPU depreciation.

They tried to pivot to the enterprise by signing a $1 billion partnership with Disney. But Disney realized that offloading their rendering pipeline to OpenAI’s servers was actually more expensive than doing it in-house. The unit economics didn’t make sense, so the servers were shut down.

OpenAI’s “Risk Factor”

The cloud providers have woken up. For years, Microsoft subsidized OpenAI’s compute to gain market share. That era of cheap infrastructure is over.

In a recent financial disclosure, OpenAI explicitly listed their reliance on Microsoft Azure’s compute pricing as a “Risk Factor.”

Wall Street is demanding a return on the billions poured into data centers. Investors are forcing AI labs to drop unprofitable consumer tools and focus entirely on enterprise contracts that actually pay the bills.

Edge vs. Mainframe

To fix the broken math, the AI market is splitting into two distinct tiers.

The middle ground which is $200/month frontier web app is disappearing.

Tier 1: The Consumer Edge

Consumers will get smaller, 8-billion parameter models running locally on their phones and laptops (Apple Silicon, Snapdragon NPUs etc).

These models are good enough for basic tasks like grammar correction and summarizing emails, but they aren’t capable of deep reasoning.

Why the shift?

Because by pushing the model to the edge, companies offload the cost of the compute and electricity directly onto the user’s battery. It is the only way the consumer unit economics work.

Tier 2: The AI Mainframe (Enterprise)

True frontier AI, massive models capable of deep, autonomous workflows will become bespoke enterprise tools.

These won’t be accessible via a casual web interface.

They will be sold via multi-million-dollar B2B contracts to pharmaceutical companies running protein simulations, and quant hedge funds executing trading logic.

They are the only businesses with gross margins high enough to afford the true, unsubsidized cost of compute.

Conclusion

The idea that a solo developer in a garage will have the exact same compute power as the CTO of JPMorgan isn’t realistic.

The physics of data centers dictate otherwise.

As a Software Architect, you need to plan defensively. Relying on cloud providers to subsidize your application’s heavy reasoning is a risk.

Build your systems around cheap, local, open-source models for the basic plumbing such as routing, classification, and simple tasks.

Treat frontier API calls as an expensive, highly constrained physical resource.

Use them only when absolutely necessary and plan for AI like heavy industrial equipment.

The era of cheap, subsidized compute is over.