# A field guide to adopting AI in engineering > How I run AI adoption for a 200-person org: measuring against a baseline, keeping devs autonomous while checking outcomes, why token spend is the wrong number, governance as config, PII guardrails, and tooling you can swap as the ecosystem moves. Written three weeks into a live rollout. Published 2026-07-10 · 30 min read · AI enablement Source: https://prasad.tech/blog/adopting-ai-in-engineering --- I run AI adoption for a 200-person org, 60 of them in engineering, a high-risk regulated fintech. We went live three weeks ago. The numbers are still coming in, so this guide is qualitative for now, and I will publish the measured version as the data lands. The failure we designed against is a common one. A company buys licences for an AI coding tool and announces the rollout. Six months later someone asks whether it helped, and nobody can answer, because nobody measured anything before the tools arrived. There is nothing to compare against. So we treat adoption as an experiment: - Write down what you expect to change. - Change one thing. - Compare against how things were before. - Keep, adjust, or stop based on what you find.
Baseline Pilot on real work Measure outcomes Decide
scale it, or adjust the workflow and run the loop again
Some context on what "AI in engineering" covers here, so this does not stay abstract. The same program runs several surfaces at once: - Coding agents for the 60 devs, working across the platform codebase and its services, from customer-facing APIs to internal tools. - An AI support agent on Zendesk answering customer tickets, which we [tested black-box before turning it on](/blog/deploying-a-customer-support-ai-agent-to-production/) and now monitor in production. - Slack agents that let business teams ask questions the code and the docs can answer, without pinging a dev. - The shared tooling underneath all of it: one gateway, one installer, one code graph. Different surfaces, same rules. The rest of this guide walks through the loop the way we run it. ## First, measure where you are To know whether AI helped, you have to know how things were without it. That means measuring before the first tool goes out. We captured: - Lead time, from first commit to merged, and merged to released. - Escaped defects, by severity. - Rework: how often a change gets reverted or reopened. - Review turnaround. - A short survey of where the day-to-day friction is. None of this needs to be precise to the decimal. It needs to be consistent: the same measurement, taken the same way, before and after. And take it on work that is expensive and repetitive, because that is where an improvement will be large enough to see. The same lens I use for [choosing projects](/blog/which-ai-projects-are-worth-it/) applies here. ## Devs choose how, the org checks what There is a tension in every rollout like this. Devs do their best work when they pick their own tools and methods. The org still needs to know the work is good. We handle it by separating the two: - **Devs choose how they work.** Which harness, how they prompt, when they use AI at all. We do not tell anyone how to work. - **The org checks what comes out.** A small, reviewable change, with tests, that moves a number the team owns. We do not track how much AI anyone uses, how many prompts they ran, or how many tokens they consumed. Those are inputs, and inputs are the dev's business. Outcomes are the org's business. Three weeks in, this split is the main reason adoption feels voluntary here rather than imposed, and we still get the quality check where the code merges. ## Pilot on real work A pilot is the same production work, done with the new tool, by devs the rest of the org trusts. Three things make a pilot informative: 1. **Real work.** Results on a demo repo do not carry over. Point the tool at the actual codebase, with its flaky tests and its history. 2. **Representative teams and code.** One team on a greenfield service tells you about greenfield services. Include a team in the old, load-bearing code, because that is where most of the org works. 3. **Credible early adopters.** Pick devs whose judgment the others trust. When a known skeptic says it works, people listen. Keep the pilot small enough to watch closely, and run it long enough that the novelty wears off and you see normal use. ## What to measure This is the part teams most often get wrong, so it gets the most detail. ### Engineering metrics For delivery we use the four DORA metrics, plus a few of our own: - **Lead time** for a change, commit to prod. - **Deploy frequency.** - **Change failure rate:** how often a change breaks something. - **Recovery time** after a failed change. - **Rework and escaped defects.** - **Review quality and PR size:** are diffs getting smaller and easier to review, or larger and harder? Two of the DORA four measure speed and two measure stability, and you need both. Speed gained by giving up stability comes back later as incidents. The graphs do not tell you how the work feels, so we also ask the devs directly, on a short recurring survey: how fast are their feedback loops, how heavy is the mental load, how often do they get uninterrupted stretches. Delivery numbers can look fine while the devs are getting worn down, and you want to find that out from a survey rather than from resignations.
Measure
Ignore
### Product metrics Engineering metrics tell you the machine runs well. They do not tell you it is building the right thing. So we tie delivery to the product number each team owns: did the shipped work move activation, retention, conversion, whatever that team is on the hook for. AI that helps you ship more features nobody uses has not helped. At the end of a quarter the question is whether what we shipped moved the numbers we care about, and whether AI shortened the path to that. ### Why token spend is the wrong number The gateway makes token counts easy to see, which tempts people to manage by them. But tokens are what you spend, not what you get. Two devs can consume the same tokens where one ships a useful feature quickly and the other produces nothing that survives review. The token count cannot tell them apart. So we relate spend to results instead: 1. The gateway attributes every request to a dev, a team, and an initiative. 2. We divide that spend by what it produced: cost per shipped change, per feature, per resolved ticket. 3. We watch the trend of that ratio. Rising cost per result is worth investigating. Rising raw spend, by itself, is not. A dev who spends a lot of tokens shipping valuable work quickly is doing exactly what we want. A raw token dashboard would flag that dev and miss the one who spends little and ships nothing. ### How the measurement runs At 60 devs this has to be automatic, or it will not happen: - Delivery metrics come from the pipeline and version control. - Product metrics come from the product analytics each team already runs. - Spend comes from the gateway ledger, attributed per team and initiative. - Developer experience comes from the recurring survey. - Everything is team-level, on a review cadence, and used to find friction. Nothing rolls up to a person, because once a number is attached to a person, people optimise the number. ## AI amplifies the system you already have The clearest thing I have seen watching AI land on teams is that it amplifies what is already there. Give it clean version control, small changes, tests you trust, and a solid platform, and it compounds them. Give it a weak system and it produces problems faster. The DORA research found the same at industry scale: individual devs got faster and happier while team delivery stability got worse, at least at first.
Strong system gains compound
Weak system problems ship faster
So the real preparation for AI is the ordinary engineering substrate: - **Readable version control.** Small commits with messages that say why, short-lived branches, and a protected main that only merges when the checks pass. When an AI-assisted change misbehaves in prod, you find it by reading history, and that only works if the history is readable. - **Small batches.** Changes small enough to review in one sitting and revert in one step. This mattered before AI. With an agent generating code, it is the difference between a reviewable PR and a 2,000-line dump. - **Tests that mean something.** A suite that fails when something is broken and passes when it is not. If devs rerun a red build because it is probably flaky, the agents inherit that habit, and flakiness starts hiding real failures at machine speed. - **A paved platform.** Self-service paths for the common work: spin up a service, get a database, ship a pipeline, without filing a ticket and waiting. For us this includes an engineering handbook repo with templates every project starts from, README, agent context files, design docs, plus a setup script, so a new service starts from a known-good shape instead of a blank folder. Before scaling anything we went repo by repo and asked those four questions: is the history readable, are changes small, do the tests mean something, is the path paved. We pointed AI at the repos where the answers were yes. The others get fixed first and get AI second. ## Who owns the AI's code, and how we coach it The model does not own code. The dev who submits, reviews, and approves it does. A large AI-generated PR into critical code gets the normal review bar, and if anything a firmer one: - **Small, reviewable diffs.** A 2,000-line change nobody can honestly review is not done. - **Test evidence in the PR**, pinning the behavior that matters. - **A risk read:** what breaks if this is wrong, and how would we find out? - **A named owner** who can defend the change without mentioning the AI. ```mermaid sequenceDiagram participant D as Dev (owner) participant A as AI agent participant R as Reviewer D->>A: Task, repo context, acceptance criteria A->>D: Proposed change plus tests D->>D: Inspect assumptions, run targeted tests D->>R: Small PR with evidence R->>D: Normal review standards D->>D: Merge and own it ``` The new failure mode here is size. Before AI, a 2,000-line PR took long enough to write that few devs wrote one. With AI, boiling the ocean is possible: a dev can generate a sweeping change across a large codebase in an afternoon. It gets merged quickly and reviewed badly, and the defects surface later. We coach this rather than ban it: - When we see oversized AI PRs, we sit with the dev and decompose the change together, and we explain the tradeoff: a huge PR merges fast and hides bugs no reviewer can catch. - We make the small-PR path the easy one, with templates and examples of good and bad AI usage side by side. - We track PR size as a team signal. When it creeps up, someone goes and coaches, rather than admiring the chart. The other failure mode is a dev who accepts confident-but-wrong output. Same response: coach the habit. Check the assumptions, write a targeted test, compare against an alternative, treat the model as a collaborator you argue with rather than an authority you obey. ## Guardrails and agent safety Before an autonomous agent touches prod, it runs inside plain, unexciting guardrails: ```mermaid flowchart LR A[AI agent] --> P[Scoped permissions] P --> S[Sandboxed execution] S --> T[Automated tests] T --> G[Human approval gate] G --> Prod[(Production)] Prod --> L[Audit log and rollback] ``` The least access that does the job, execution in a box that cannot reach prod, a human gate on anything irreversible, and a log with a rollback path. The bigger the blast radius, the more process the change earns. When an agent proposes a broad architecture change, we require a short written decision record first: the tradeoffs, the constraints, an incremental rollout plan, and a human architecture review. The record takes about an hour to write, and it is worth it, because unwinding a bad refactor from a live codebase takes weeks. Here is what scoped access looks like in practice. The support agent needs customer data mid-conversation, and giving it database access was never on the table. Instead it talks to a separate read-only API we built for exactly this: the API serves only the whitelisted fields each conversation flow needs, applies the compliance checks, and logs every access. The agent cannot ask for more than the API exposes, so the interesting security questions moved from "what might the agent do" to "what does this API allow", which is a question we know how to review. ## The gateway, the models, and residency If every dev calls model providers with their own keys, you have no cost visibility, no policy, and no control over where data goes. In a regulated fintech that rules it out immediately, and in any org it turns into a mess within a quarter. So: one gateway in front of everything, one endpoint, every model behind it, owned by you. ```mermaid flowchart LR H["Coding harness (dev's choice)"] --> C["Token compression
(command output + context)"] C --> G["Self-hosted gateway
per-dev keys, budgets, spend ledger"] G --> EU1["Open weights, EU-resident"] G --> EU2["Frontier models, EU data zone"] ``` We self-host [Bifrost](https://github.com/maximhq/bifrost) as that gateway. What it gives us: - **Provider keys held centrally**, one virtual key per dev, so nobody handles a raw provider key. - **A budget hierarchy**, company to team to dev, and a **spend ledger** that attributes every request. This is the working version of the [spend-attribution](/blog/attribute-ai-spend/) idea, running live instead of reconstructed in a spreadsheet. - **One place to set policy**, swap models, or add a provider, without touching 60 laptops. A hosted router like OpenRouter gives you the same single endpoint with less to operate; you give up some control for less ops. For a regulated org, self-hosting made sense for us. For a smaller team the hosted route is reasonable. The gateway also covers more than the devs. The support agent and the Slack agents for business teams route through the same endpoint, with the same keys, budgets, and scanning. One set of controls covers every AI surface in the company, which is the point of having a gateway at all. On residency: our requirement is EU data, and open weights make that reachable, because the weights are portable and where they run is your choice. One detail to get right early: EU residency has to cover in-region processing. The inference call itself is processing under GDPR, so a model stored in the EU but run elsewhere does not qualify. Our mix: - **Open weights, EU-resident.** GLM 5.2, Llama, Qwen and similar, served from EU hosts with zero-retention terms, or self-hosted on an EU cloud. We route through inference vendors that keep data in-region, rather than the model maker's own API. - **Frontier models, EU data zone.** The large cloud providers offer EU data-zone deployments that keep prompts and responses in-region. The provider does not need to be European; it needs to offer EU residency you can put in a contract. Avoid the global-routing tiers when residency is strict. - **The paperwork.** An EU endpoint is only the start. You still need the data processing agreement, a written zero-retention guarantee, and a subprocessor review, and the terms can differ between a provider's own models and the open models it hosts, so check per model. Any tool that would send source code to an external service goes through a security, privacy, IP, and compliance review for that use case before anyone uses it. ## Governance as config In a regulated company, someone will eventually ask: who can use which model, with what data, and who approved it. If the answer lives in a dashboard that admins clicked through over months, you will be reconstructing history. We keep the answer in a config file instead. The whole org structure is declared in one file: business units, teams, people, which model tier each person can use, and each team's budget. A small generator turns that file into the gateway's runtime config. Changing someone's access is a PR: edit the file, get it reviewed, merge, deploy. The git history is the audit trail, for free. Three things we learned setting this up, worth knowing before you rely on a gateway for governance: 1. **Test what your gateway enforces.** In the OSS build we use, per-person model access and per-team rate limits enforce correctly. Hard dollar budgets did not: in our test, a key that was over its cap kept working. So dollar caps are enforced upstream, per key, at the provider. The lesson generalises: a config field existing does not mean the enforcement exists. Test it. 2. **PII scanning sits in front of the model call.** Prompts in a fintech will eventually carry customer data, and the question is whether anything catches it. The gateway's native guardrails were enterprise-gated, so we run an open-source scanning layer in front: it blocks or redacts PII, secrets, and prompt-injection attempts in requests, and redacts PII in responses. Request scanning fails closed (if the scanner is down, the request does not go out); response scanning fails open. Those failure modes are deliberate, and worth deciding consciously rather than by default. 3. **Keep enforcement close to the money.** Rate limits at the gateway, spend caps at the provider, access in config, scanning on the wire. Each control lives where it can actually act. ## Tooling you can swap The AI tooling ecosystem changes month to month: new models, new harnesses, new agents. If your setup hard-wires one tool, you rebuild it every time. So we optimised for swapping, and it shapes everything: **A standard endpoint decouples the pieces.** The gateway speaks a standard API, so any harness just sets a base URL. Swapping a model or adding a provider is one central change, and no dev has to touch anything. **One command sets a dev up.** New devs run a single installer that bundles: - Their chosen harness (Claude Code, OpenCode, Cline). - The gateway config and their virtual key. - Token compression at two layers: command output compressed at the shell ([rtk](https://github.com/rtk-ai/rtk)), longer context compressed before the model ([headroom](https://github.com/headroomlabs-ai/headroom)). - A code graph, so the agent answers structural questions (who calls this, where is X defined) from a precomputed index instead of reading through files, which cuts the bulky output that eats context. A dev is productive in minutes, and a tooling change is an installer update rather than a memo. **The repos carry context for the agents.** Every repo starts from templates in our engineering handbook: a README that says what the service does, an agent context file with the conventions and the commands, and a design doc. This is worth calling out because when an agent keeps producing plausible-but-wrong changes in a repo, the fix is almost never a bigger model or broader access. It is better context: clearer conventions in the repo, acceptance criteria in the task, a worked example to follow. We treat those context files as part of the codebase and review changes to them like code. **We benchmark alternatives instead of guessing.** Compression tools and graph engines get compared on real questions from our own repos, and the winner becomes the default with the runner-up kept as an option: - On compression, one tool is stronger on command output and another on long context, so we run both, scoped so they do not compress the same bytes twice. - On the graph, we started with a code-only index that builds locally and shares across the team with no server. We are now benchmarking a multi-modal engine ([graphify](https://github.com/Graphify-Labs/graphify)) that also indexes docs, SQL schemas, and diagrams, so the same graph can answer questions about the system and not only the code. It runs locally too. The benchmark on our own repos decides which becomes the default. The specific tools matter less than the setup: in an ecosystem that moves this fast, you want a stack you can re-point in an afternoon. Behind the tooling sits a one-page policy that people actually read: approved use cases, prohibited data, review and test expectations, security and IP guardrails, and an escalation path. We standardise the boundary and the gateway, and leave open which tool a team uses inside it. Onboarding is hands-on sessions on the team's own tasks, with good and bad usage shown side by side. People learn this by doing it on their own work, so we run working sessions rather than presentations. ## Quality gates and CI AI raises the rate of change, so the pipeline that catches bad changes has to get stronger at the same time. When an AI test-generation tool hands you 500 new tests, you neither merge them all nor throw them away. You curate: 1. Check them against the real risk areas. 2. Drop duplicates and brittle cases. 3. Stabilise the valuable ones. 4. Integrate those into CI. ```mermaid flowchart LR Ch[AI-assisted change] --> V[Pre-PR validation] V --> CI{CI and quality gates} CI -->|pass| M[Merge] CI -->|fail| An[Analyze failure patterns] An --> Fix[Strengthen gates,
prompts, workflow] Fix --> Ch ``` When AI speeds up coding and CI failures climb, the fix is upstream: read the failure patterns, strengthen pre-merge validation, improve the prompts and workflows that produced the bad changes, and update the gates. Loosening the gates to make the failures go away converts a visible problem into an invisible one. Where it pays, we run the loop deliberately: generate regression tests from our own defect history for the high-risk flows, then check whether escaped defects fall and CI stays stable. ## Target the work where AI pays off Not all code is equally worth automating against. To pick targets, combine signals you already have: defect history, change frequency, complexity, test coverage, usage. Code that is high-churn, high-complexity, and thinly covered is where AI-assisted refactoring and test generation return the most, and the same data tells you which technical debt to pay down. For a legacy module you are afraid to touch, an ERP core or a payments path, go slice by slice: 1. Take one high-value slice. 2. Add characterization tests that pin the current behavior. 3. Apply AI to that slice, with human review of each change against the tests. 4. Move to the next slice. You change the code under a net you trust, and you never bet the module on one large AI-written commit. ## Bringing the org along Getting 60 devs to adopt anything follows a known pattern. A few try anything new on day one. A handful of respected devs pick it up next, and the rest wait to see what those people conclude. Rollouts stall in the gap between the early few and the cautious majority. What moves the majority: - A clear improvement over the current way. - A low-risk way to try it on their own work. - Seeing peers they trust get results. So our rollout order is: 1. **Seed** with the devs people already listen to. 2. **On-ramp** everyone through the one-command installer, so trying it costs nothing. 3. **Measure** the pilot honestly, at team level. 4. **Show the results** to the people who have not moved, in their own terms. 5. **Coach.** A small group of senior devs works with teams, inside their code, rather than reviewing from outside. If instead you mandate a tool nobody has seen proof of, people comply on paper and work around it in practice, and your metrics quietly stop describing reality. ## When to scale, and when not to Scaling is a decision with a bar: a repeatable gain across representative teams and codebases, without giving up quality, security, maintainability, or developer experience. One good pilot week is a data point, and that is all it is. ```mermaid flowchart LR P[One-team pilot works] --> V[Validate across
representative teams] V --> D[Document the playbook] D --> E[Enable and coach] E --> Ok{Metrics hold,
no bad tradeoffs?} Ok -->|yes| Std[Org standard] Ok -->|no| Iter[Iterate] Iter --> V ``` Suppose a pilot ships 40 percent more merged PRs and escaped defects rise with it. That is a process that got worse while a number got better. Hold, find where the workflow and quality bars let weak changes through, tighten them, and rerun the measurement before widening anything. The same applies when cycle time improves while devs report more mental load and review fatigue. That gain is being paid for by the people, and it will not last. Widen only when the metrics hold, with a documented playbook and real enablement behind it. ## Recovering when it goes wrong Something will go wrong, and a working program has a response ready: - **Escaped defects rising.** Read the failure patterns, strengthen pre-merge validation, improve the prompts and workflows that produced them, update the gates. - **Implementation time down, review time up.** Look at the handoff before blaming reviewers: PR size, change complexity, test evidence, context quality. The fix is usually smaller diffs and better context. - **A noisy AI reviewer.** Calibrate it against the quality bar, track the ratio of useful comments to noise, and scope it to areas where it helps. - **Cost per result creeping up.** This is the spend signal worth acting on, and the checks are practical: is the model tier bigger than the task needs, is compression on, are devs re-running the same expensive prompts. ## Key takeaways - Treat adoption as an experiment: measure where you are, pilot on real work, compare, decide. - Let devs choose how they work, and check what comes out. Track outcomes, never activity. Inputs are the dev's business; outcomes are the org's. - Measure delivery and product outcomes at team level: lead time, deploy frequency, change failure rate, recovery time, rework, review quality, developer experience, and whether the work moved a product number. - Tokens are what you spend, not what you get. Relate spend to results (cost per shipped change), or do not track it at all. - AI amplifies the system it lands on. Fix version control, batch size, tests, and the platform where they are weak, then scale AI on the strong parts. - A human owns every AI change: small diffs, test evidence, a risk read, a named owner. Coach the new failure modes, oversized PRs and overtrust, instead of banning the tool. - One gateway in front of every model: per-dev keys, budgets, a spend ledger, and EU residency, which means in-region processing plus a data processing agreement per model. - Put governance in a config file. Access changes become reviewed PRs, and git is the audit trail. Test what your gateway enforces, and put PII scanning on the traffic. - Build the tooling to be swapped: a standard endpoint, a one-command installer, benchmarked alternatives. The ecosystem changes monthly, and your setup should re-point in an afternoon. - Scale only on a repeatable gain that holds without a hit to quality, security, or the devs. If quality slips when you widen, fix the workflow and rerun the measurement. --- ## Common questions **Why is lines of code a weak way to measure AI in engineering?** If you measure lines of code, you get more lines of code, because an agent produces whatever you reward. More code means more to review, more to test, and more to maintain. Measure delivery instead: lead time, deploy frequency, change failure rate, recovery time, and whether the shipped work moved a product number. **Is token consumption a good measure of AI adoption?** No. Tokens are what you spend, not what you get. Two devs can spend the same tokens where one ships a useful feature and the other produces nothing, and the token count looks identical. Track spend through the gateway, then relate it to results: cost per shipped change, per feature, per resolved ticket. Watch that ratio, not the raw spend. **How do you keep devs autonomous without losing control?** Devs choose how they work: which harness, how they prompt, when they use AI. The org checks what comes out: small reviewable changes, tests, and work that moves a number the team owns. Nobody tracks anyone's AI usage. Inputs are the dev's business; outcomes are the org's. **Who is accountable for code an AI wrote?** The dev and team who submit, review, and approve it. The model has no judgment about your domain, so the normal review bar applies: small diffs, test evidence, a risk read, and a named owner. If nobody would put their name on the change, it is not ready. **Does AI make engineering teams faster?** It amplifies what is already there. Individual devs get faster almost immediately. Team-level delivery stability often gets worse at first. A team with clean version control, small batches, and tests it trusts compounds the gains; a team without those ships its problems faster. The returns come from the system, not the tool. **How do you govern who can use which AI model in a regulated company?** Put the rules in a config file, not a dashboard. We declare business units, teams, people, model tiers, and budgets in one file; a generator turns it into the gateway config. Changing access is a reviewed PR, and git history is the audit trail. Test what your gateway enforces: in our case, model access and rate limits enforced, dollar caps did not, so spend caps sit upstream at the provider.