Most enterprise AI deployments follow a pattern: a vendor presents capability claims, leadership approves a budget, and a rollout proceeds. The success metrics are defined after the deployment begins, often under pressure to justify the investment.
Accenture did something different. Before committing to a firm-wide deployment of GitHub Copilot, the company designed and ran a randomized controlled trial — the same methodology used in clinical research — to measure the tool’s actual impact on developer productivity and output quality.
The original idea: measure before you scale
The study was conducted in partnership with GitHub and published in May 2024 [GitHub Blog, 13 May 2024]. The design was rigorous:
- 450 developers in the treatment group (GitHub Copilot access)
- 200 developers in the control group (standard workflow, no Copilot)
- Data collection: DevOps telemetry (builds, pull requests, merge rates), plus post-study developer surveys
- Participants: entry-level to management, across multiple roles and project types
The premise was specific: rather than relying on developer sentiment surveys alone — which can be influenced by novelty effects — Accenture tracked objective operational metrics from the DevOps pipeline. Build success rates, pull request volume, merge rates. Metrics that directly translate into delivery speed and code quality.
The results — from the published research
- +84% successful builds in the Copilot group vs. control [GitHub Blog Research, May 2024]
- +15% pull request merge rate
- +8.69% pull requests generated per week
- 88% of Copilot-generated code characters retained in the final version
- 95% of developers reported enjoying coding more with the tool
- 90% felt more fulfilled in their work
- Average time to first accepted Copilot suggestion: 1 minute
- 81.4% installed the extension on the same day as license receipt
After the trial, Accenture expanded GitHub Copilot to 12,000 developers across the firm — a decision grounded in the evidence from the controlled study [GitHub Customer Story — Accenture].
The 84% build rate improvement deserves attention
The most operationally significant result is the build success rate. A build failure means a developer detects a problem, diagnoses it, fixes it, and re-runs the build cycle — a process that can consume hours. An 84% improvement in successful builds means that the Copilot group encountered substantially fewer of these cycles.
The mechanism is direct: Copilot suggests code that is more likely to compile and pass tests on the first attempt, because it has learned from patterns across millions of codebases. The developer still reviews and edits — the 88% retention rate confirms this is not passive acceptance — but the starting point is higher quality.
The pull request data tells a similar story: more PRs per week, higher merge rate. The developers are producing more output, and more of that output is reaching production.
What you can take from this
The Accenture study offers a model for how to make high-stakes AI deployment decisions. The insight is about methodology, not just results: run the measurement before the deployment, design for falsifiability, and let the data determine the scope.
When Accenture expanded to 12,000 developers, it did so with documented evidence of what the tool delivers. The expansion was not a bet — it was a forecast.
For any organization evaluating AI tools for technical teams, the RCT model is replicable at smaller scale. A 50-person treatment group and a 25-person control group, tracked over 90 days, produces enough signal to make an informed deployment decision — and protects the organization from both over-committing to tools that do not deliver and under-investing in tools that do.
SAGA — Success Stories & Real Cases · Curating real AI implementations: the original idea, the decision, the verified result.