Experiments

Just brainstorming. More on the talk page.

This is a guiding document on the use of experiments on Wikimedia wikis.

Current practices in web analytics reflect their commercial origins. For better or worse, the greatest motor behind the use of of web analytics has been the profit interests of online retailers and social networks, for whom the user is a commodity. These profit interests have profoundly shaped the discourse of web analytics, setting both the tenor and the tone of debate (consider the values implicit in "funnels", a term of art).

A thoughtless application of web analytics to Wikipedia would import a moral outlook that is incompatible with (and, indeed, rightfully offensive to) to its community. It also wouldn't work, because neither Wikipedia nor its editors are for sale. It is therefore crucial that technical efforts be accompanied by a process of reflection, the goal of which would be to articulate criteria for web analytics that express the broader ambitions of the movement and the moral commitments that underlie them.

Background

Experiments on Wikimedia wikis, generally conducted by the Wikimedia Foundation, have become increasingly common in recent years. In 2009, the Usability team conducted a number of experiments. In 2012, there is now a dedicated experiments team (E3).

These experiments have varying goals and motivations, but we've reached a point where some clearer guidelines need to be established about what is and is not appropriate.

Principles

Dignity and collegiality of all

Anyone working with Wikimedia editors has to treat them as colleagues, not as customers. Experiments are often an attempt to optimize human behavior and workflow. There's nothing inherently wrong with such goals, but Wikimedians are not customers in the same way that users of Facebook are customers of its site. They instead should be viewed as colleagues. Would you go into the office of someone you work with and start fucking with them to optimize their behavior? Surely not. But this is exactly the type of behavior the Wikimedia Foundation is now engaging in. Disrupting the work of long-time editors in the name of questionable experimentation.

Implementation

Mitigation

Experimentation often comes with high costs and high risks. Adding extra weight by including people in a test group who ought not be tested on adds considerable cost and risk without any benefit.

Smarter code should ensure that only editors who meet specified criteria load extra code (JavaScript). A number of factors can be taken into account when determining whether to load extra JavaScript for a particular user, including:

• user's logged-in status;
• user's edit count;
• user's registration date; and
• whether the user is using the default skin.

Looking at and using smarter metrics is important. If a user is using a non-default skin, it's fairly safe to assume that they probably don't want to be fucked with. Experiments on these users should undergo the most scrutiny and require the most consideration.

Opt-out

Any and all experiments should have an opt-out feature. However, an opt-out feature is not a license to be more obnoxious simply because people can opt out of your experiment.

See also

• Experimentation