Why we embed before we build

Most AI work dies in the gap between a slide deck and production. A pilot can look convincing right up to the day it meets the workflow as people actually run it, with data nobody had sampled and edge cases that never came up in discovery. Gartner forecast that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs, and unclear business value¹. None of those four causes concerns the model. They are delivery problems, and how the engagement is set up settles them before anyone has written a line of code.

We build differently. We embed before we build, and we call it forward-deployed: the engineering happens next to the problem rather than across a contract boundary from it. This essay is the argument for why.

Requirements are a lagging indicator

The dominant model for delivery still runs on a requirements document. Someone interviews the business, writes down what the system should do, hands it to engineers, and the engineers build to the spec. It feels rigorous. It is mostly theatre.

A requirements document can only contain what someone already understood well enough to write down, and in practice that is not much. Michael Polanyi explained why sixty years ago: we can know more than we can tell². The people who run a workflow carry knowledge they cannot fully articulate, because skilled work is largely tacit. Ask them to specify it and you get the legible 20%. The other 80% governs the actual work, and it lives in the exceptions and workarounds nobody thinks to mention, so it never reaches the page.

Requirements engineering has known this for decades. Elicitation is widely regarded as one of the most critical and difficult activities in the field, precisely because stakeholders cannot fully articulate their needs up front and no single technique is sufficient to surface what they leave unsaid³. The field treats it as a core activity in part because of the documented difficulty people have describing their own needs⁴. The foundational elicitation literature names tacit knowledge as a core obstacle and argues you have to observe actual work in its setting to recover it⁵. Contextual inquiry was built on the same premise: go to where the work happens, watch it, and you capture the latent, unspoken structure a questionnaire never will⁶.

Projects that skip this step show up in the failure statistics. In the original Standish CHAOS study, the top three causes of troubled projects were all requirements- and user-input-related: lack of user input, incomplete requirements, and changing requirements⁷. The peer-reviewed literature has long treated deficient requirements as the single biggest cause of software project failure⁷. Requirements problems are also among the most expensive defects a project can carry.

And they get more expensive the longer they survive. A defect caught at the point of creation is cheap. The same defect caught in the field can cost 50 to 200 times as much to correct⁸. NASA’s analysis of error-cost escalation found a requirements error costing one unit to fix during requirements could cost 29 to more than 1,500 units once it reached operations, and that a problem fixed after delivery can run upwards of 100 times the cost of fixing it during requirements and early design⁹. A requirements document, far from preventing this, tends to guarantee it: the misunderstanding gets committed to paper at the cheap end of that curve and comes to light at the expensive end.

What a requirements document records, then, is how well the problem was understood on the day it was written, by people who could only tell you part of what they know. Ordinary software pays for that in rework. For AI the price is usually the whole project, because the highest-leverage AI opportunities almost never live in the part anyone can write down. They hide in the workflow itself, in places like the judgement call somebody makes forty times a day, the document nobody reads in full, and the handoff that loses an hour. You only find those from the inside.

The requirements that matter don’t arrive in a document. We go into the work and find them.

What forward-deployed actually means

Forward-deployed is a stance about where the engineering sits relative to the problem. The term comes from Palantir, whose forward deployed engineers embed directly with customers and build inside the customer’s own operational environment to solve their real problems¹⁰. The role originated there in the early 2010s, driven by government and enterprise customers whose data was too sensitive for remote, traditional discovery; for a period until around 2016, Palantir ran more forward-deployed engineers than ordinary ones¹¹. The economics invert the usual software model: instead of building one capability many customers reuse, the forward-deployed engineer enables many capabilities for a single customer¹².

We take that stance and apply it to AI delivery. Three commitments make it real.

1. 01

   Engineers find the problem, not analysts. The person who spots the opportunity is the person who builds it. Nothing gets translated between discovery and delivery: no analyst hands a spec to an engineer who never saw the work. The sticky information stays with the builder. Eric von Hippel showed that need and solution information is often costly to move, so problem solving migrates to wherever that information already lives¹³. Rather than fight that, we put the engineer at the locus of the sticky information and let discovery and construction happen in the same head.

2. 02

   Real, governed data from week one. The actual workflow, with real, de-identified data, under governance, from the first week. A synthetic dataset will not get you there, and neither will a sanitised sample or a slide of what the data might look like. Fake inputs hide the problem that matters, because the problem that matters is usually the data itself. Its shape is wrong, it has gaps, or it arrives in the three formats nobody mentioned. Governed access early is what lets discovery be honest.

3. 03

   Short loops where the workflow is the spec. We ship small, watch it meet the real work, and adjust. The workflow is the specification, and it is a living one. This is the field-as-product-discovery mechanism Palantir describes, where insights and configurations built in the field flow back into the product, and some of their most valuable additions originated there¹⁴. The loop is Immerse, Spot, Execute, and it runs continuously rather than once.

The difference from a waterfall is where the understanding gets front-loaded. A waterfall packs it into a document up front, then spends months discovering the document was wrong. We front-load presence instead, and let the understanding accumulate while it is still cheap to act on.

Why it ships in weeks

The common assumption is that embedding slows delivery down. In practice it is the faster path, for a dull reason: you throw less away. Most of the speed comes from never having built the wrong thing in the first place.

The evidence on what governs delivery outcomes points the same way. Small and iterative beats large and up-front, in every cut of the data. In the Standish data, project size is the single most important factor in how a project resolves: only about 6% of very large projects succeed and 43% fail, against roughly 61% success and 7% failure for small ones¹⁵. Method matters too. Agile projects fail around 9% of the time versus about 29% for waterfall, roughly three times the failure rate¹⁶. The headline number sits underneath both: across tens of thousands of projects, only about 29% succeed, 52% are challenged, and 19% fail outright¹⁷.

The mechanism has been measured too. DORA’s research program is the multi-year statistical work behind the Accelerate findings by Forsgren, Humble, and Kim¹⁸. It identifies working in small batches as a core capability: small batches shorten the feedback loop on changes, make problems easier to triage and remediate, and predict both delivery and organisational performance¹⁹. Where traditional delivery takes weeks or months to collect feedback on a change, small batches let work flow to production in minutes, cutting cycle time, flow variability, and risk²⁰. Reducing batch size is the primary lever for throughput, because smaller changes are easier to reason about and move through the pipeline, and speed and stability turn out to be correlated rather than traded off²¹. The practices that depend on it, trunk-based development and continuous delivery, are how elite teams keep three or fewer active branches and merge to trunk at least daily²².

Forward-deployed applies small batches to discovery as well as to code. Each loop is a small bet against the real workflow. When a bet is wrong, you have lost a week instead of a quarter, and you have learned something the requirements document could never have told you. The team that keeps building the right small thing reaches a working system long before the team that spent the same period on one large build aimed at the wrong target. Over a full engagement, the difference compounds.

Governance from the first commit

There is a version of this story where you move fast and bolt security on at the end. We skip that version, because the defect-cost curve applies to governance too: a residency or isolation assumption baked in wrong at the start and discovered at go-live is the most expensive class of mistake there is⁹.

So governance goes into the first commit instead of into a phase before launch. Onshore by default, in AWS ap-southeast-2. Zero-retention. Row-level isolation between tenants. De-identification at the boundary, so the data that powers discovery is governed before it is ever touched. None of it waits for the auditor to call. Because the system starts life inside these constraints, meeting them stays cheap. Residency, isolation, and de-identification are why embedding with real data is possible at all.

The handover is the point

Forward-deployed has an obvious failure mode: the engineer embeds so deeply that the client can never run the system without them, and what was sold as delivery turns out to be a dependency.

The engagement ends with a documented, governed system the client owns and runs without us. That system is the deliverable. The code, the model, and whatever cleverness went into them are the scaffolding that got it built. A system only the visiting engineer can operate loses its value the day that engineer moves on, which sooner or later every engineer does. The whole point of going deep is to leave something that stands on its own. So the handover is designed from the first week: the documentation, the governance posture, and the operational knowledge all transfer deliberately.

The AI that does reach production gets found from the inside, built in short loops on real governed data, and handed to a team that no longer needs us. That is the part we do.