Wikimedia Labs: Difference between revisions

Group:	Operations
Start:	2011-08-29
End:	2017-03-31
Team members:	Andrew Bogott, Yuvaraj Pandian, Chase Pettet
Backlog:	#Labs

Browse history interactively

← Older edit

Content deleted Content added

VisualWikitext

Inline

Latest revision as of 20:02, 10 August 2023

Wikimedia Labs provides cloud computing infrastructure. The virtual machines are powered by the OpenStack software, administered through special pages on the Wikitech wiki. The environment includes a live replica of databases of all public Wikimedia Foundation wikis.

The project is aimed at helping volunteers get involved in Wikimedia operations and software development.

Initial focus of this project was on two parts: Toolforge and Beta Cluster. For more about these, see Wikitech.

Background

The Wikipedia website infrastructure and software was built and originally completely operated by community volunteers. As the site has grown, over time the infrastructure and software development became more dominated by Wikimedia Foundation staff. While the Foundation's increased involvement has had a positive effect, we would like to reverse this trend and encourage more volunteerism.

When Wikimedia Labs started in 2011, initial focus was on two key projects:

Beta Cluster, a small wiki farm running the latest alpha version of the MediaWiki software.
Tool Labs, a platform for volunteers to easily run web services, bots and other applications.

The Beta Cluster runs in a virtualized environment that closely resembles Wikimedia's production cluster. It can be used by anyone to contribute changes to backend infrastructure or software, demo the changes, have the code reviewed, and ultimately pushed to production.

In Tool Labs, users have the comfort of not needing to become system administrators of any (virtual) servers or hardware, instead the platform is entirely provided for. It can be used for small projects that are Wikimedia-related but distinct from the software used in production. For example, a project might present information from the MediaWiki API of a Wikimedia wiki in a new way.

Open access

Anyone can create a Labs account by creating an account on wikitech.

Having a Labs account grants you access to Wikitech, Wikimedia Labs cloud, Gerrit (our code review system), and a few other developer-related tools.

Implementation

The architecture is described on Wikitech. The software for controlling this environment is implemented as a MediaWiki extension, Extension:OpenStackManager.

Roadmap

For the roadmap, please see Wikimedia Labs's goals for 2014-15. Past roadmaps: 2012-13, 2013-14. See also Tool Labs Roadmap which describes the migration from Toolserver to Tool Labs in 2013/14.

Open tasks

We'd love help with all of the below!

Development and sysadmin tasks

Enable IPv6

Proposals

Development Process
Puppet learning mode
Toolserver features needed / wanted in Tool Labs (contains even some features which are not on toolserver)
Investigate wikitech:Pentaho
Find/test OTRS replacement, or upgrade/puppetize OTRS (with security patches)
Puppetize PDF server
- Package mwlib dependencies
- Create an init script (preferably upstart) for mwlib daemon
  - Mentions of init script here (wikitech) but last I heard not in use. Peachey88 05:38, 21 December 2011 (UTC)reply
Add puppet syntax highlighting to vim
Create shared sql service for all projects
Package and puppetize lilurl for use as a url shortening service
Write Documentation for console
Fix puppet repo so that it runs a complete first run of the puppet catalog on instance creation for all services
Java App stack
- Package JDK 1.6 (confused about this, can we not use openjdk or sun's packaged jdk?)
- Apache Ant
- Maven
- Jetty App Servers
- Apache Solr
- Hadoop
- Sbt
Per-project SaltStack remote execution
Integrate a global queuing system (to replace tools' use of SGE when they migrate from the toolserver)
Authentication improvement project
Instance creation improvement project
Interface usability improvement project
Stability improvement project
Communication improvement project

Completed

Shared home directories per project
Package adminbot, with an init script
Puppet repository branch per project or instance
Nagios management without exported puppet resources
Package gerrit
Create a log bot for the #wikimedia-labs channel
Reverse proxy for web services

Documents

Introduction to the toolserver and to Wikimedia Labs by Ryan Lane and Daniel Kinzler at the San Francisco Hackathon January 2012.

Communications

#wikimedia-cloud ^connect IRC channel on Freenode
labs-l mailing list
inventory of Labs's total capacity as of late September 2012 (compute, database and storage nodes)
Quarterly review, April 2015

@@ Line 1: / Line 1: @@
+{{Notice|1=See [[Wikimedia Cloud Services team]] for the continuation of this project.}}
-{{Wikimedia engineering project information
+{{Historical}}<onlyinclude>{{Wikimedia engineering project information
 | name        = Wikimedia Labs
 | description = A test environment using virtual machines for testing and experimentation
-| start       =
+| group       = [[Technical Operations|Operations]]
+| team        = [[User:Andrew Bogott|Andrew Bogott]], [[User:YuviPanda|Yuvaraj Pandian]], [[User:Chasemp|Chase Pettet]]
-| end         =
+| Phabricator = Labs
-| group       = Operations
-| lead        =
-| team        = Coren, Andrew Bogott, Yuvi Pandian
-| previous    =
-| next        =
 | projectpage = Wikimedia Labs
+| start =  2011-08-29
+| end =  2017-03-31
 | display     = {{{display|}}}
+}}</onlyinclude>
-}}'''Wikimedia Labs''' provides cloud computing infrastructure. The virtual machines are powered by the [[:en:OpenStack|OpenStack]] software, administered through special pages on the [https://wikitech.wikimedia.org/ Wikitech wiki]. The environment includes a live replica of databases of all public Wikimedia Foundation wikis.
+'''Wikimedia Labs''' provides cloud computing infrastructure. The virtual machines are powered by the [[:en:OpenStack|OpenStack]] software, administered through special pages on the [https://wikitech.wikimedia.org/ Wikitech wiki]. The environment includes a live replica of databases of all public Wikimedia Foundation wikis.
-The<noinclude> project is aimed at helping volunteers get involved in Wikimedia operations and software development.
+The project is aimed at helping volunteers get involved in Wikimedia operations and software development.
-Initial focus of this project was on two parts: Tool Labs, and Beta Cluster. For more about these, see [[wikitech:Labs labs labs|Wikitech]].
+Initial focus of this project was on two parts: [[wikitech:Help:Toolforge|'''Toolforge''']] and [[Beta Cluster|'''Beta Cluster''']]. For more about these, see [[wikitech:Labs labs labs|Wikitech]].
 == Background ==
@@ Line 72: / Line 72: @@
 ** Hadoop
 ** Sbt
-* [[/Per-project saltstack remote execution|Per-project saltstack remote execution]]
+* [[/Per-project SaltStack remote execution|Per-project SaltStack remote execution]]
 * Integrate a global queuing system (to replace tools' use of SGE when they migrate from the toolserver)
 * [[Wikimedia Labs/Authentication improvement project|Authentication improvement project]]
@@ Line 99: / Line 99: @@
 == Communications ==
-* {{irc|wikimedia-labs}} IRC channel on Freenode
+* {{irc|wikimedia-cloud}} IRC channel on Freenode
-* [https://lists.wikimedia.org/mailman/listinfo/labs-l labs-l mailing list]
+* [[mail:labs-l|labs-l mailing list]]
-* [http://lists.wikimedia.org/pipermail/toolserver-l/2012-September/005305.html inventory of Labs's total capacity as of late September 2012] (compute, database and storage nodes)
+* [[mailarchive:toolserver-l/2012-September/005305.html|inventory of Labs's total capacity as of late September 2012]] (compute, database and storage nodes)
 * [[m:WMF Metrics and activities meetings/Quarterly reviews/Parsoid, Services, MW Core, Ops, RelEng, MM, Labs and ECT, April 2015|Quarterly review, April 2015]]
 <!-- List here the public communications (by the team or the media) about this project -->

Clump (talk | contribs)