Talk:Using AI in MediaWiki patches

Other policies for inspiration

I would recommend looking at https://github.com/llvm/llvm-project/commit/18695b27d565 for inspiration. In particular, I personally like the ban on using LLMs for "good first issues". The goal of such tasks is largely educational and a way for contributors to learn and get engaged and using LLMs to solve such tasks defeats the purpose. I wonder if something equivalent might be useful for MediaWiki / Phabricator as well. SSastry (WMF) (talk) 14:59, 26 January 2026 (UTC)

Totally agree on the "good first issues" point, I hadn't thought of that! Audiodude (talk) 15:14, 26 January 2026 (UTC)

I think discouraging newcomers from fully generated contributions makes sense—some of the goal of code review is pedagogy. Scope of the contribution also matters: the larger the change (and the closer it gets to complicated, core logic), the more apt there will be issues without competent human oversight.

Another bit of the llvm policy worth adopting:

Contributors are expected to be transparent and label contributions that contain substantial amounts of tool-generated content.

Which seems to be part of many policies I've seen (The Eclipse Foundation, The OpenInfra Foundation). Plus, disclosure of tool use is organically happening on Wikimedia's Gerrit as well as on the Linux Kernel mailing list. TCipriani (WMF) (talk) 15:50, 26 January 2026 (UTC)

Page reads as information, not prescriptive

I really like this page, it provides a lot of informtion. However, it seems to me that it is very descriptive of the issues and problems around LLMs, but doesn't actually define policy outright. I see a lot of "should" statements, which I feel contributors will find convenient to ignore. I envision a more direct section that lists specific imperatives, along the lines of:

• Please be considerate of reviewer time and resources. Do not submit patches that were entirely generated by an LLM/AI. We could do that ourselves.
• You must be able to understand, explain, and justify all code in your patch that was generated by an LLM or coding assistant.
• You must manually test your code to verify that it solves the issue and doesn't crash or misbehave.
• Do not submit patches with extensive, verbose, and unnecessary LLM provided comments. Comments should almost always be written by a human, especially critical ones, and it must never be taken for granted that the comment accurately describes the behavior of the code.

Audiodude (talk) 15:22, 26 January 2026 (UTC)

Also I must caveat that I am not a MediaWiki developr at all, and I'm wondering if there is value in a more "general"/generic page that could be referenced across all Wikimedia projects. I'm specifically thinking of toolforge/cloud VPS projects. Audiodude (talk) 15:23, 26 January 2026 (UTC)

When it comes to cloud/toolforge its much less a shared coding environment. Projects tend to be one person or a very small number of people. So i think its much more, whatever the project maintainer feels like. Bawolff (talk) 09:24, 9 February 2026 (UTC)

Rename?

I generally prefer more specific terms instead of AI. How about "LLM" or "GenAI" in the page title instead of "AI"? SSastry (WMF) (talk) 15:45, 26 January 2026 (UTC)

Words matter, and these words are also the active subject of commercial marketing (mis)use which can make picking specific labels challenging.

• GitHub Copilot is advertised as an "AI accelerator".
• The Claude Code project calls itself an "agentic coding tool". The whole https://code.claude.com/docs/ document collection seems to only mention "LLM" in passing on a couple of pages about specific configuration settings.

Generative artificial intelligence does seem to be primarily what folks are talking about when they get into the details of which specific kinds of hallucination/confabulation/delusion/bullshitting are most unwanted.

I guess a related question for me is if this essay/draft policy looking to be expansive in discussing various forms of research automation and code generation along with their technical, social, and moral implications? Or is it about a readily definable collection of technologies that we can name with specific terms without inviting language lawyering about whether product X is or is not in scope? Is this a distinction without difference, or will it matter in the long run? -- BDavis (WMF) (talk) 18:16, 26 January 2026 (UTC)

I suppose at this time, AI = LLM and I should not bother trying to split that hair. SSastry (WMF) (talk) 17:04, 27 January 2026 (UTC)

I think we can add a note that by "AI" tools, we mean tools that generate code and/or text, which at the time this policy was drafted, include LLMs, AI agents, and services that rely on such tools. Someone can wordsmith that to add the usual "not an exclusive list" or "this covers future tools with similar functionality" and keep it succinct. We don't necessarily want to exclude a bunch of other uses, so clarity here is good IMO. ArielGlenn (talk) 19:16, 29 January 2026 (UTC)

Use of AI for writing tests

We discuss the use of AI for writing comments, it might be worth discussing documentation and tests explicitly. AI written documentation tensa to be overly verbose and difficult to read. Using AI to generate code examples for documentation may be fine, but I never want to have to wade through AI slop with emojis and formatting instead of a concise description of what I need to know.

The AI-written tests I've reviewed tend to appear complete and comprehensive, but not actually test what they claim to test. One patch claiming to test wikitext parsing actually used str_replace instead of a wikitext parse, in fact no mediawiki code at all was actually used in the test cases, despite the use statements at the top appearing correct. Turned out that phan/phpcs will leave the use statements if the classes mentioned are used in comments, and that was the only use in theis patch. Anyway, I don't know exactly what the policy recommendation would be here, just that special care should be taken when letting AI "write tests".

Finally, we've talked about requesting disclosure for AI-generated patches, which would let reviewers look more specifically for weaknesses which AI tends to impart. cscott (talk) 22:49, 31 January 2026 (UTC)

There's currently some discouragement of AI-generated comments and tests on the page. Feel free to strengthen those if you think they need strengthening. Personally I don't like AI-generated docblock comments at all, while I find inline comments are usually OK (although I often delete some). And I've definitely used AI to generate tests before with a prompt such as "write 10 tests for X", and then I'll usually delete some and keep/tweak the ones I like. –Novem Linguae (talk) 20:07, 4 February 2026 (UTC)

I've had a bit more experience with Claude and tests and I will admit it is often quite helpful; especially if you are precise with what you ask for. Identifying a particular change made in a commit and asking it to generate a test case with coverage specifically of the changed lines can be a great short cut.

I stand by my original comment, however: "care should be taken", and I think specific disclosure is helpful. "I wrote the code, claude wrote a test case covering it" vs "Claude wrote code and tests from my plan" vs "Test-driven development: I wrote the tests, claude from the code". This helps guide reviewers into giving proper scrutiny and looking for "likely human errors or oversights" vs "likely AI errors or oversights" (which are sometimes very different classes of review). cscott (talk) 18:18, 10 June 2026 (UTC)

Rethinking the perspective used to write the current draft

I fully understand the reservations currently held by developers in the community about the use of AI in coding, especially with the possibility of a deluge of bad code having the potential to wreak havoc into well-maintained codebases, should it go unchecked.

That being said, the page's contents as of today looks like it is written with a relatively anti-AI bent, and it's not subtle either. My concern is that pushing such biases onto folks will only lead to people who support the use of new AI coding techniques to outright ignore the text stated in here because it's too restrictive and outdated.

For example, the "proper uses" section is simply these three things:

• As a rubber duck to give you ideas of things to try.
• To fill in simple functions/methods or a todo comment, which you then verify work.
• To write unit tests, which you then refine and improve, possibly deleting some that are redundant or don't make sense.

This would be a good approach to take in 2023, but AI-assisted coding has made incredible strides since then. Day-by-day, developers are beginning to realize that writing code without AI assistance impedes upon their level of productivity, and to limit them to use ask mode and write a few functions and unit tests is a laughable limit to some, especially those of us who work in Big Tech. This proposed approach is simply not congruent with how tools like GitHub Copilot Agent Mode or Claude Code work, which are very powerful and are becoming ever more accurate. For those who have integrated these tools into their workflow, prescribing a narrow set of acceptable uses is so restrictive that it will not be listened to (i.e. there will be compliance elasticity). I fear it will have a boomerang effect.

Another example which caused me concern is the proposed approach for unit testing prescribed here:

Reviews should not be done with AI. Reviewers are the main defense against misuse of AI/LLM tools. As such, reviews should continue to be done the traditional way, i.e. an experienced maintainer reading the code and also manually testing the patch.

This doesn't match what is already being done in the corporate world. At Microsoft, we have required, automatic AI code reviews on every one of our pull requests. Of course, human code reviews are also required, are still the gold standard, and are what's ultimately needed to merge a pull request. However, AI reviews are an extremely helpful first triage tool and should be encouraged as long as multiple humans do the closing reviews.

I get the perspective of why this was written. It's in response to inexperienced developers using AI to contribute bad patches without understanding the ramifications of the code they are proposing. Indeed, they need to be reined in and stopped. However, it is limiting the agency of those of us who are experienced in writing code with widely accepted rigorous software development principles who have found responsible ways to integrate AI into our code contributions. I don't want a set of rules to be prescribed to insult the intelligence of developers; they should be free to work whichever way they find to be most productive, as long as their code quality is unaffected. And if these rules are published anyway as-is, people aren't going to listen to them and they're going to find ways to get around it because they think they're unreasonably restrictive, just like how marijuana is recreationally used throughout the United States by millions of adults despite it being a Schedule I controlled substance at the federal level.

It would be great if more emphasis could be placed on the importance of responsibility of using AI-assisted coding with proper guardrails rather than dismissing its usage for anything beyond simple tasks, and perhaps to keep the scope small and well-defined with specs rather than a large scope with vague parameters. For instance, the creation of system prompts and instruction files, like what I made for my presentation at the MediaWiki Users and Developers Conference Fall 2025 in Hanover, Germany, is incredibly important in giving the LLM the right direction to create good code. This part of my presentation should give you an idea of the kind of guardrails I place on LLM work: https://www.youtube.com/watch?v=YL6Bof7psko&t=348s. (I should note that this presentation featured an example of vibe coding which ended without the rigorous refinement process that should be expected, and it was presented this way to be a scientific experiment and possible cautionary parable, but it doesn't imply that nothing good comes out of vibe coding.) Jeffrey Wang 08:16, 9 February 2026 (UTC)

FOSS development is not the corporate world and Microsoft is not a great role model to hold up for responsible use of generative AI or FOSS contributions. ~2026-88498-8 (talk) 22:20, 9 February 2026 (UTC)

Open source software not being the corporate world (which isn't entirely true considering Microsoft and other companies do have open source software too) doesn’t mean it exists in a vacuum. The reality is that AI-contributed code has the potential to be incorporated in both open source development and the corporate world, whether we like it or not.

Microsoft doesn’t need to be a role model to be a useful reference point. Ignoring its practices simply because it’s Microsoft is less of a principled stance and more an appeal to tribal identity. Your point didn't at all address my claim that there's benefits to an AI code review as a first pass. It seems to me you're more interested in attacking Big Tech companies rather than the specifics they're doing.

If the goal is responsible use of generative AI in open source development, then evidence from large, visible, and accountable actors is not a liability; it’s one of the few sources of concrete, testable experience we have, and it can be used to improve upon and propose more rigorous processes. Jeffrey Wang 23:56, 9 February 2026 (UTC)

Microsoft is known for having a pretty active research department. Have they published any peer reviewed studies of the social and technological benefits of using generative AI tools on software production and quality that we can read? -- BDavis (WMF) (talk) 00:15, 10 February 2026 (UTC)

(Required disclaimer: I'm not speaking on behalf of Microsoft. All of these are solely my thoughts as an individual who just so happens to work at Microsoft.)

That's a good question, and I'd like to share as much as I can about this. My understanding is that Microsoft Research (MSR) does not really work on such studies about user productivity or the efficacy of existing AI tools. It's quite unlikely they conduct studies about using Claude Opus 4.6 that's peer reviewed (which is, in my opinion, now the only acceptable model that should be used for AI-assisted coding as of February 2026). MSR is more about creating new technologies and seeing how their original work fares. Other divisions of the company seem to publish on arXiv, but those folks, while very talented, are not doing work for MSR, and from what I can tell, does not shed light on what you're looking for. I do not believe the full extent of the research Microsoft is doing internally on the efficacy is being shared publicly. So unfortunately, I would suggest looking elsewhere, but I'm happy to share other experiences I've had at Microsoft which aren't under wraps.

Instead, I recommend focusing on companies and organizations which are more open to sharing their data. Here's an independent analysis that shows the power of Claude Opus 4.6 in producing code: https://www.linkedin.com/pulse/claude-opus-46-takes-lead-gdpval-aa-surpassing-gpt-52-80cve/. People I've spoken to have noted there is a clear benefit in writing code with Claude Code using Opus 4.6 compared to, say, using GPT-5.0 to write code, and my anectodal experience confirms this too.

The unfortunate truth (or fortunate truth, depending on one's angle) is that AI is moving so fast to the point where papers might be obsolete by the time they are peer reviewed and announced at conferences. Plus, individuals are responsible for making their own choices, so studies about the productivity of large groups of people using AI to augment their coding honestly might not provide as much inspiration to formulating a good policy as one might hope.

For these reasons, I would not solely rely on peer-reviewed papers; instead, it's probably better to do one's own analysis.

I want to emphasize from what I wrote yesterday that I am not necessarily advocating for full-on vibe coding yet. MediaWiki is a huge piece of software and it's incredibly important the quality of code that goes into core and WMF extensions is of top-notch quality. You all are welcome to do what you please, including categorical bans on the use of AI in development, but I hope the page's contents can be adjusted to better reflect the realities of human behavior in a world where AI-generated code is inevitable. The 18th Amendment didn't work as much as the Temperance Movement wanted it to, and we ended up putting an end to Prohibition because of it.

There are headwinds to AI adoption even within Big Tech. Developers in Big Tech are not monolithic and not everyone is happy about this new approach. It's been a draining endeavor to try and convince qualified software engineers to take a more progressive stance on the use of AI. Therefore, I think I've said all I can say on this topic. Wishing you all the best of luck. Jeffrey Wang 08:58, 10 February 2026 (UTC)

I gave a shot at trying to make it a bit more neutral. ping @BDavis (WMF) @Taavi @Novem Linguae —TheDJ (Not WMF) (talk • contribs) 12:52, 10 February 2026 (UTC)

I liked the previous first paragraph that stated that LLM use in programming was more acceptable than LLM use for writing encyclopedias. I also think letting LLM into code reviews is dangerous. Won't revert this though. I'd like lots of people to make bold edits to this page so that we arrive at a good equilibrium. The more editors the better. This is the wiki way. –Novem Linguae (talk) 13:12, 10 February 2026 (UTC)

Even if true, we are mediawiki not wikipedia. I think we should leave questions of how to write an encyclopedia to Wikipedia and focus on the issue at hand. Bawolff (talk) 15:56, 10 February 2026 (UTC)

I'm not really sure that development guidelines are meant to express a neutral point of view. I think they are actually much more expected to represent an informed opinion consensus. I can certainly see how someone is capable of holding functionally the inverse position as I do, but I don't see how a guideline that says "opinions vary" is a guide to anything. I hate the whitespace conventions used by MediaWiki with the burning power of 10,000 suns, but I conform to Manual:Coding conventions/PHP when I submit code to a MediaWiki project. I would like to see an opinionated guideline on the use of generative AI in MediaWiki technical workflows that was informed by discussion of technical, social, and environmental impacts even if that eventual guideline is full of consensus decisions that I had to agree to disagree with. -- BDavis (WMF) (talk) 23:37, 10 February 2026 (UTC)

I think a non-neutral view should not be mixed into the actionable points of this article ([im]proper use). That simply makes it a difficult and confusing read. Secondly, like Jeffrey, I feel that some of these opinions were either outdated, preconceptions, or implied that everyone using AI writes low effort patches to stuff their resume. Instead I chose to refocus the existing paragraphs of principles and dangers to be more focused towards the aspect of code and review when AI/LLMs are applied improperly. My changes are grounded in my current understanding of the capabilities of AI and the most recent models, based on the experience I have with it at the two companies that I had/have to work with it every single day.

I have no objection to adding colored views about how we are burning the planet with it, but I do think such points should be summarized and tackled separately from what it means with regard to patches, code and contributions and separately from the do's and dont's and that we should avoid repeating ourselves several times. —TheDJ (Not WMF) (talk • contribs) 11:20, 11 February 2026 (UTC)

Specific style guidelines; disclosure of AI use

I added a brief mention of disclosure of AI use.

It would be helpful to add some specific style guidelines. I'm not sure whether they should appear on this page or (eg) in the Gerrit/Commit message guidelines but it should address whether (eg) a simple Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> is sufficient disclosure, or if we prefer a more narrative and specific style ("I used Claude Opus to write tests", "I used Claude Sonnet to create skeleton code, which I filled out", "This patch is almost entirely Claude's work, but I reviewed it carefully and agree with its decisions", etc.). We could also elaborate on the guidance here which says that "writing commit messages yourself is better".

Finally, I believe emerging best practices is to record the specific plans or transcript used when generating AI code, so that the "source" (as it were) of the patch can be saved. I'd love to see specific guidance w/r/t how we do that, whether git notes, gerrit comments, a file in the repository, or something else. cscott (talk) 18:30, 10 June 2026 (UTC)

In the spirit of Be Bold, I've gone ahead and drafted some specific advice in Gerrit/Commit message guidelines#Disclosing the use of AI tools which is already implicitly linked here by the guideline to "adhere to commit message guidelines". cscott (talk) 05:02, 11 June 2026 (UTC)