Skip to content
SwankyForge
SwankyForge
All insights
Article8 min read

With an AI pair in every repository, we automate the checks and keep five decisions human

How the engineering team works with AI coding assistants: what runs automatically on every change, what a person always decides, and what we have not measured yet.

Most engineering work in our repositories now starts with an AI coding assistant writing a first draft. None of it ships on the assistant's word. This note describes the setup as it exists in the repository behind this website, because that is the one we can show in public. It also lists the decisions we keep with people, and the number we cannot give you yet.

The short version, if you are buying

We are a small lab of two engineers, with specialists added per project. An AI assistant works alongside both of us. It drafts code, tests, documentation and routine write-ups. Automated checks run on every change and block the ones that break a rule. A person reads every change before it is merged. Nothing reaches production or the public without an explicit yes.

What you get from this: the same two engineers cover more of the routine work that small teams tend to skip, such as tests, documentation and release checks. That leaves more of the budget for the parts that need judgement: choosing the metric, deciding whether a model is worth building, and fitting it into your process.

What we do not claim: a speed-up figure. We have not measured one, and the external evidence says self-reported speed-ups are unreliable. The last sections say what we would measure and what this means for a client.

What the research says: the tool is not the gain

Four studies from 2025 shaped how we set this up.

  • Google's DORA programme surveyed close to 5,000 technology professionals for its 2025 report. 90% use AI at work, and more than 80% believe it has made them more productive. Yet 30% report little or no trust in the code it generates. The report's summary is that AI "amplifies what's already there": strong teams get better, struggling teams see their problems intensify.
  • Bain's 2025 Technology Report found that teams using AI coding assistants see productivity gains of 10–15%. Some companies report 25–30%, but only when AI is paired with a redesign of the whole process. One reason the first number is small: writing and testing code is about 25–35% of the time from idea to launch.
  • The 2025 Stack Overflow Developer Survey found that 84% of respondents use or plan to use AI tools, and 46% distrust the accuracy of the output. The most common frustration was output that is almost right, but not quite.
  • METR ran a randomised trial with 16 experienced open-source developers on 246 real tasks. With AI tools allowed, tasks took 19% longer. The developers had expected a 24% speed-up, and afterwards still believed they had been 20% faster. The authors caution that the result is specific to their setting: large, mature open-source projects and developers with limited experience of the tool used.

Read together, the studies say three things. Adopting an assistant is easy. The gain depends on the process around it. And nobody's impression of their own speed is evidence. So we put the effort into the process (instructions, checks and approval points) rather than into the choice of tool.

What the assistant does in our repository

We use Claude Code. Five pieces make its work predictable.

Project instructions. A file at the root of the repository tells the assistant what lives where, which rules are hard and which commands to run. It is versioned like code, so a change to the rules shows up in review like any other change. Anyone with access to the repository can read exactly what the assistant was told.

Skills for recurring tasks. A skill is a short written procedure the assistant follows for one kind of task. We have eight: writing up a case study with its evidence record, drafting an engineering note, drafting a company-page post, generating a profile for an external platform, checking text for confidential names and personal data, a read-only audit of the site's trust signals, a weekly status report, and triage of incoming enquiries. Each skill states where it stops. None of them ends in publishing anything.

Hooks. The assistant's actions pass through scripts we wrote. After it edits a file, that file is scanned for confidential names and personal data, and a finding sends the assistant back to fix it. Before it can run a git commit or push, a guard runs the confidentiality scan and the claims lint on what is about to leave the working tree. Any finding blocks the command. A plain git pre-commit hook applies the same checks to commits made by hand.

CI on every push and pull request. The pipeline runs lint, type checks and a build, then three guards:

  • the confidentiality scan, over source, content and the rendered HTML. If its configuration is missing, it fails rather than passing quietly;
  • the claims lint, which rejects a list of superlatives and borrowed-authority phrases;
  • the content validator, which refuses a case study without an evidence record, a reference design that claims measured numbers, and a homepage figure that does not link to the case it came from.

These rules are mechanical on purpose. An assistant produces plausible text quickly, and plausible-but-wrong is exactly the failure the surveys describe. A rule that runs on every commit does not depend on anyone, human or model, remembering to be careful.

The content pipeline. Anything meant for the public moves through three folders: drafts, approved, published. A skill can write into drafts and nowhere further. Only the owner runs the approval step, which re-runs both scans before it moves the file. Publishing is manual: an approved post is sent to the owner as paste-ready text, and an approved note goes live only with the next approved push to main. This note took that path. An assistant drafted it from the repository's own files, the scans ran on it, and it reached this page after a person approved it.

What we never delegate

Five decisions stay with people, whatever the tools can do.

  1. Agreeing the metric with the client. Which number defines success, how it is measured and what threshold counts as done. We write it into the contract before training starts. An assistant can list candidate metrics. It cannot know which one your business will act on.
  2. Judging whether ML is worth it. Sometimes the honest answer is a rule, a report or a process change. "Do not do ML yet" is a recommendation we make, and it is a judgement about your situation, not a text-generation task.
  3. Approving releases. A merge to main deploys. Nothing is pushed to main without an explicit yes from the owner, and the assistant's permissions exclude force-pushing.
  4. Talking to the client. Calls, written answers, bad news. A skill can draft a reply to an enquiry; a person reads it, edits it and sends it. No script contacts a lead.
  5. Deciding what goes public. Posts, notes, profiles, case studies and code releases. The pipeline makes publishing a deliberate step rather than a default.

The common thread is accountability. Each of these decisions has consequences someone must answer for, and that someone has to be a person. On a project, it is the one named engineer who stays with the work from start to finish.

What we have not measured

We do not have a before-and-after figure for our own productivity, and we will not quote one. METR's result is the reason: developers who were 19% slower believed they were 20% faster. An impression, ours included, is not a measurement.

If we publish a number, it will carry the same evidence label as our case studies (measured, estimated or illustrative), and it will come from records like these, collected over a stated window:

  • time from an issue being opened to a reviewed, merged change, for comparable task types;
  • the share of merged changes reverted or fixed within 14 days;
  • problems caught by hooks and CI before merge, against defects found after;
  • review time per change, because review is where an assistant's output costs people time;
  • hours per milestone against the estimate from our calculator.

Until then, the honest claim is narrower. The checks run on every change, and the five decisions above are made by people.

What this means for you

More of the routine work gets done. On a fixed budget, tests, documentation, monitoring and release checks are what small teams cut first. An assistant makes them cheaper to write, and the checks make them hard to skip.

The price goes to judgement. Our rates and the hours behind each role are published on the pricing page, and the calculator shows the arithmetic. The hours that matter most are the ones spent on the metric, the evaluation and the integration with your process: the parts no assistant does for us. If you are not sure ML is the right answer, the free teardown gives a first verdict in six questions, and a short paid discovery settles it before any model is trained.

You can inspect the setup. The repository lives in your organisation from day one. If an assistant works in it, its instructions and checks sit in that repository too, next to the history of every change, where you can read them.

Ask every vendor the same questions. Which AI tools will see your code and data, and under what terms? Which checks run on every change, and do they block or only warn? Who approves a release? Has the vendor measured its own speed-up, and how? The answers tell you whether AI in their process amplifies discipline or the lack of it.

Sources

Have a process that costs too much?

Tell us about it. The first call is free, and if AI is not the answer we say so.

Discuss your task