What 385 GitHub AI policies reveal about disclosure, review, and code quality
A large GitHub study finds that AI policies can increase disclosure, review activity, and short-term code-quality signals while AI-assisted contributions continue to grow—but it does not show that maintainers work less.
One GitHub study finds that repositories adopting AI policies saw 10% higher AI-assisted pull-request throughput and 11% fewer flagged vulnerabilities in the following eight weeks. The same repositories also generated more comments, review turns, and reviewers per pull request. The result is encouraging only if "more review" means better routing of human attention rather than a hidden increase in maintainer work. 1
That distinction matters for any team deciding how to govern coding assistants. A policy can change what contributors disclose and what maintainers inspect. It cannot, by itself, prove that a model wrote better code or that a repository's total review burden fell.
The study measures governance, not model quality
The Aug. 4 arXiv preprint analyzed 29,624 active GitHub repositories and 10,192,203 pull requests dated from January 2025 through May 2026. The repositories had at least 100 stars, more than 100 commits from at least 10 authors, a valid license, open issues and pull requests, and activity after January 2026. The filters favor established, active open-source projects; they are not a census of software teams. 1
The authors found 385 repositories that adopted a human-facing AI policy between Feb. 1, 2025 and Apr. 30, 2026. They excluded files that only instruct agents, such as
AGENTS.md and CLAUDE.md, so the treatment is a policy addressed to contributors rather than a configuration file for an agent. The researchers matched 379 adopting repositories to 887 unique controls using recent activity, repository age, and programming language, then used staggered difference-in-differences to compare outcomes before and after adoption. The main post-adoption window was eight weeks; four-, six-, ten-, and twelve-week windows produced directionally consistent results. 1
This design improves on a simple before-and-after count because it asks whether policy adopters were already moving differently from comparable repositories. All 19 outcomes passed the paper's pre-trend test for the eight weeks before adoption. That supports the comparison. It does not remove every reason a project might adopt a policy at the same time that its maintainers, contributors, or codebase are changing.
TRACE turns a vague policy into five levers
The paper's useful contribution is not the acronym alone. It separates decisions that are often collapsed into a single label such as "AI allowed" or "AI banned":
| Dimension | What the policy controls | Distribution among 385 policies |
|---|---|---|
| Transparency | Whether contributors disclose AI use, from no requirement to tool-, prompt-, transcript-, or file-level traceability | 35.8% demanded the most detailed level; 33.0% required no disclosure. 1 |
| Responsibility | Whether a human must own, explain, review, test, or validate the submission | 59.7% required review, testing, or validation. 1 |
| Attribution | Whether the contributor must address copyright, license compatibility, DCO/CLA, or provenance | 67.5% did not address attribution; 27.0% required license or DCO/CLA compatibility. 1 |
| Constraints | Whether bulk output, particular channels, or AI-generated submissions are restricted | 11.7% generally prohibited AI-generated submissions. 1 |
| Enforcement | What happens after a violation, from remediation to rejection or suspension | 46.0% stated no consequence; 21.0% allowed future-participation restrictions. 1 |
The dimensions do not form one permissive-to-restrictive slider. A project can ask for detailed disclosure while allowing AI broadly, or prohibit AI contributions while saying little about provenance. That is the mechanism behind the paper's main practical result: governance can make AI use more visible and more reviewable without treating every use as disallowed.
The short-term effects point to more legible review
The average treatment estimates move together in a way that is easy to misread. AI-assisted work rises, while several code-quality measures improve and the review process becomes more active.
| Outcome after policy adoption | Reported estimate | What it supports, and what it does not |
|---|---|---|
| AI-assisted pull-request throughput | +10% | AI use continued after policy adoption; the result is not evidence that AI caused the quality changes. 1 |
| Vulnerabilities flagged by SonarQube | −11% | A short-term repository-level quality signal improved; it is not a security audit or proof of fewer exploitable defects. 1 |
| Cognitive complexity | −17% | The scanned code became less complex by this metric; the source does not show that every important maintainability property improved. 1 |
| Lines changed per reviewer | −14% | Review work was spread across smaller chunks, or more reviewers; it does not establish lower total maintainer effort. 1 |
| Non-bot comments per pull request | +8% | Human discussion increased; more discussion can be a quality control or a new queue to manage. 1 |
| Maintainer response time | −8%, from about 18 to 17 hours | The median first response became faster in the measured window; this is not a measure of total review hours. 1 |
The cleanest reading is that a policy changes the review surface. Contributors disclose more, pull requests become smaller on a per-reviewer basis, and maintainers interact more often. The study does not measure every minute a maintainer spends across a repository, so "developer experience improved" should not be reduced to "maintainers worked less." The paper's own outcome set gives a better picture: participation, code signals, disclosure, and review flow must be read together.
Policy families change the tradeoff
The researchers grouped the 385 policies into five families. The largest was Regulated & Controlled with 121 policies: disclosure requirements paired with restrictions and explicit enforcement. In the family analysis, this group was associated with 28% fewer SonarQube-flagged vulnerabilities, 7.3% more reviewers per pull request, 7% higher AI-disclosure share, and 11% higher AI-assisted throughput relative to its matched comparison. 1
That combination is more useful than a blanket "strict versus permissive" ranking. It suggests that a policy can preserve AI-assisted work while adding places where a human has to explain, test, or reject it. The study also reports that Strict Prohibition reduced AI-assisted merged throughput by about 4%, but did not eliminate observed AI-assisted activity. The result is consistent with a policy that changes the visible flow more easily than it changes the underlying behavior.
The strongest caution is the Quiet & Low-Requirement family. It restricted some uses without requiring disclosure and was the only family associated with a net reduction in AI-assisted activity and disclosure. That does not prove opaque rules cause the reduction. It does show why a policy that says "do not do this" without saying what must be disclosed or validated gives maintainers less evidence about what actually happened. 1
What the causal design can and cannot settle
The paper reports a serious quasi-experimental design, not a randomized trial. Propensity matching and pre-trend tests address some selection concerns, and the direction of the estimates survives several post-adoption windows. But policy adoption is recent, the main outcome window is only eight weeks, and policy text can change after the initial merge. The authors also leave sparse TRACE levels out of the heterogeneous analysis: some cells contain only four to fourteen policies. 1
The sample limits transfer. These are active, popular GitHub OSS repositories with public pull-request histories and automated code scans. A private data-science team may face different incentives, permissions, regulatory duties, and failure modes. The study measures repository behavior and SonarQube signals; it does not measure dataset lineage, experiment reproducibility, scientific validity, or whether an AI-generated analysis can be independently rerun.
The authors provide a public replication package, which makes the analysis easier to inspect. It does not turn a preprint into settled evidence, but it gives teams a way to examine the code and artifacts before borrowing the conclusions. 2
A policy teams can test instead of merely announce
TRACE is a useful starting checklist for an engineering or data-science repository. The implementation should add domain-specific checks where the paper has no evidence:
- Make use visible. Require a short disclosure in the pull request: tool or model, what it touched, and which tests or checks the contributor ran. Detailed transcripts may be excessive for routine edits; the disclosure should be proportional to the risk.
- Keep responsibility with the contributor. Require the submitter to explain the changed logic and validate it against tests, security checks, and the repository's acceptance criteria. "The model suggested it" is not an ownership statement.
- Record attribution and provenance. For code, check license and DCO/CLA compatibility. For data-science work, add dataset version, transformation steps, experiment configuration, and external-source provenance to the review record. This is an adaptation of TRACE, not a result tested in the paper.
- Constrain high-risk contexts. Treat secrets, permissions, production migrations, security fixes, regulated data, and irreversible deletions differently from boilerplate documentation. A ban may be justified for a specific operation even when AI assistance is acceptable elsewhere.
- Make enforcement credible. Define the remedy for missing disclosure or failed validation: request changes, close the pull request, or restrict future participation. A consequence that nobody applies is only a preference.
Then measure the policy as an intervention. Track disclosure coverage, reviewer effort, response time, defect and security signals, contributor participation, and the share of work that arrives without the required evidence. Review the measures at eight weeks, then again after the policy has had time to become routine. The GitHub study gives a useful short-term template; it does not supply the long-run answer.
Bottom line
The evidence supports a narrower claim than "AI governance improves software." In this sample, policies that made AI use visible and tied it to human review were associated with continued AI-assisted contribution, more active review, and better short-term code-quality signals. The same evidence leaves total maintainer workload and long-run effects unresolved.
For an engineering or data-science team, the decision question is concrete: can the policy make AI-assisted work visible, attributable, testable, and rejectable without pretending that a ban will make undisclosed use disappear? If the answer is yes, the policy becomes a review protocol that can be measured. If the answer is no, a higher AI-activity number or a lower defect count will not tell you why.
Follow the author
AI & Data Science Daily
Twice-weekly deep-dive articles on AI and data science hot topics, covering breakthroughs, practical advice, new findings, and balanced pros and cons analysis with author profile links included in every article.
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.
Related content
- Sign in to comment.