MediaWiki - User contributions [en]

Wikimedia Technical Conference/2018/Session notes/Applying the MediaWiki Platform Architecture Principles

2025-05-23T17:12:36Z

120.29.69.184: Added content and then I can do that has to*

=== Session Setup ===

* https://phabricator.wikimedia.org/T206065

* [[Wikimedia_Technical_Committee/MediaWiki_Platform_Architecture_Principles]]

{| class="wikitable"
|Theme: Architecting our code for change and sustainability

Type: Technical Challenges

Facilitation Exercise(s): Small group rotation.
|}
{| class="wikitable"
|Leader(s): Daniel Kinzler

Facilitator: Zach

Scribe: Nick, Irene, Michael, ???
|}
{| class="wikitable"
|Description: The Wikimedia Technical Committee recently drafted a series of [[Wikimedia Technical Committee/MediaWiki Platform Architecture Principles|Architecture Principles]]. These principles and requirements SHOULD guide the Platform Evolution Program and other Wikimedia engineering endeavors. This session seeks to familiarize participants with the Architecture Principles and obtain feedback on those principles prior to the principles being placed on “last call”.
|}
'''Station 1:'''

'''Group Leader: Brion?'''

Goal 1: Allow users to consume, create, and interact in a form suitable for their devices, with the connectivity they have, in a language they speak.

Goal 2: Empower contributors to collaboratively grow and curate content, and to build the tools that they need to do so.

'''Station 2:'''

'''Group Leader: Daniel?'''

Goal 5: Ensure programmer productivity by maintaining a code base that can be modified with confidence and understood without needless difficulty.

Goal 4: Provide an open-source software stack that can be easily used, modified, and extended by others.

'''Station 3:'''

'''Group Leader: Timo'''

'''Scribe: Irene'''

Goal 3: Provide APIs that allow 3rd party interfaces to efficiently interact with wiki content. Provide reusable data that can be processed in bulk by 3rd party tools.

Goal 6: Provide a web application that can be freely used to collaboratively collect and share knowledge.

'''Station 4:'''

'''Group Leader: Marko?'''

Goal 7: Ensure the continued availability and performance of WMF projects through scalable and resilient system design.

Goal 8: Ensure the data integrity of the content on WMF systems, and protect the privacy of our users.

=== Session Schedule: ===
14:00-14:10: Session Instructions and Set-up

14:10-14:25: First rotation

14:25-14:40: Second rotation

14:40-14:55: Third rotation

14:55-15:10: Fourth rotation

15:10-15:30: Report backs

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

LYour verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

=== Summary ===

* Cheol: there are more "SHOULD" than "MUST" in these. Is that purposeful?
* Sam: could split some of them so there is a MUST and a SHOULD
* Corey suggested: Maybe “Architecture” is too narrow, perhaps we should have “Technical Principles” or “Engineering Principles”. That would include processes, tooling, etc.
* Explicitly reference volunteer devs as “re-users” or “3rd party”
* Have “must be OSS” as a separate principle (but not arch!), not part of “reusable”?

==== Goal 1 ====

* TheDJ: We need to provide for a variety of workflows. I don’t think we’re ready to do that which is why we haven’t been able to deliver.
* BV: Speech is another very different form factor, do we need to handle smaller units of text, pulling stuff from Wikidata, there’s a lot of vectors.
* Sage: a lot of our devices is tied to a specific proprietary ecosystem (apple, google). Login is a nontrivial barrier.
* MH: Issue of norms -- editors have control over presentation as well as content -> DJ: WE could give editors that control
*IM: Localized by default, that’s a pretty significant lift in a lot of cases. Separation of data layer from processing layer and display layer is a necessity.
*BV: A lot of these are aspirational, but we want to make sure our system provides for this. Need to provide infrastructure for doing translations so that when conditions improve, editors can do their work.

==== Goal 2 ====

* AH: I don’t like “ensures” consistency, prefer “directs toward” consistency. Hesitant when we take a parental role over contributors.

==== Goal 3 ====

* entire goal talks about 3rd party interface but it should talk about public interface - strike 3rd party and change to public? we want to phrase this in a way that allows for 3rd parties which makes it more inclusions; 3rd party and public api are synonymous
* action item: rephrasing goal to change 3rd party to public, or more obviously including those APIs
* what does semi-structured mean? timo doesn’t know. action item to find out and probably strike
* "data formats" a lot of confusion around that wording
* not covered - anything about public vs private, access controls, rights controls; separation between access to public and private

==== Goal 4 ====

* "Software MUST provide a test suite". - change other testability SHOULD to a MUST.
* No point in every single class being reusable. But ultimately comes back to modularization/granularity
* Accessibility! You need design resources. Good user experience needs user expert feedback and UI design feedback.
* Re: "deprecation must be documented" - should be announced, advertised, more effectively - which venue?

==== Goal 5 ====

* Currently not designed for testability
* Conceptual model could be improved

* [2nd mention of] Design for testability doesn't demand actual tests!
* "We should have something like vagrant to help ppl develop", and "we should have [?] to help people install" - Should this go in Product Practices and Guidance? Yes.
* Who is "we"? Whoever subscribes to these principles. What is "MW"? Is it just core or all extensions?
* Point #1 is unclear. "Reflect conceptual model …" is obfuscated!
* How to determine external stakeholders - might not know at the time. How much resources do we invest into finding out who might reuse it and their needs?
* "Must be testable" vs "must provide test suite" - how designed vs which code actually exists.

==== Goal 6 ====

* definition of terms (stack and standard hosting platforms esp) so we can collaboratively create this
* tradeoff between catering the wmf use-cases vs others? important to decide!

==== Goal 7 ====

* Gergo: tradeoff is per-user customization vs scalability & isolation boundaries vs ease of data access

==== Goal 8 ====

* John: change SHOULD to MUST in goal 8 item 1
* Marko: add data sanitization

Wikimedia Technical Conference/2018/Session notes/Applying the MediaWiki Platform Architecture Principles

2025-05-23T17:12:08Z

120.29.69.184: Added to the list of things to do*

=== Session Setup ===

* https://phabricator.wikimedia.org/T206065

* [[Wikimedia_Technical_Committee/MediaWiki_Platform_Architecture_Principles]]

{| class="wikitable"
|Theme: Architecting our code for change and sustainability

Type: Technical Challenges

Facilitation Exercise(s): Small group rotation.
|}
{| class="wikitable"
|Leader(s): Daniel Kinzler

Facilitator: Zach

Scribe: Nick, Irene, Michael, ???
|}
{| class="wikitable"
|Description: The Wikimedia Technical Committee recently drafted a series of [[Wikimedia Technical Committee/MediaWiki Platform Architecture Principles|Architecture Principles]]. These principles and requirements SHOULD guide the Platform Evolution Program and other Wikimedia engineering endeavors. This session seeks to familiarize participants with the Architecture Principles and obtain feedback on those principles prior to the principles being placed on “last call”.
|}
'''Station 1:'''

'''Group Leader: Brion?'''

Goal 1: Allow users to consume, create, and interact in a form suitable for their devices, with the connectivity they have, in a language they speak.

Goal 2: Empower contributors to collaboratively grow and curate content, and to build the tools that they need to do so.

'''Station 2:'''

'''Group Leader: Daniel?'''

Goal 5: Ensure programmer productivity by maintaining a code base that can be modified with confidence and understood without needless difficulty.

Goal 4: Provide an open-source software stack that can be easily used, modified, and extended by others.

'''Station 3:'''

'''Group Leader: Timo'''

'''Scribe: Irene'''

Goal 3: Provide APIs that allow 3rd party interfaces to efficiently interact with wiki content. Provide reusable data that can be processed in bulk by 3rd party tools.

Goal 6: Provide a web application that can be freely used to collaboratively collect and share knowledge.

'''Station 4:'''

'''Group Leader: Marko?'''

Goal 7: Ensure the continued availability and performance of WMF projects through scalable and resilient system design.

Goal 8: Ensure the data integrity of the content on WMF systems, and protect the privacy of our users.

=== Session Schedule: ===
14:00-14:10: Session Instructions and Set-up

14:10-14:25: First rotation

14:25-14:40: Second rotation

14:40-14:55: Third rotation

14:55-15:10: Fourth rotation

15:10-15:30: Report backs

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

=== Action items ===

* Clarify principles that led to confusion. Discuss suggestions on the talk page of the principles document, [[Talk:Wikimedia Technical Committee/MediaWiki Platform Architecture Principles|https://www.mediawiki.org/wiki/Talk:Wikimedia_Technical_Committee/MediaWiki_Platform_Architecture_Principles]] (Daniel)
* Align Architecture Goals with Product Visions (Corey)

=== Summary ===

* Cheol: there are more "SHOULD" than "MUST" in these. Is that purposeful?
* Sam: could split some of them so there is a MUST and a SHOULD
* Corey suggested: Maybe “Architecture” is too narrow, perhaps we should have “Technical Principles” or “Engineering Principles”. That would include processes, tooling, etc.
* Explicitly reference volunteer devs as “re-users” or “3rd party”
* Have “must be OSS” as a separate principle (but not arch!), not part of “reusable”?

==== Goal 1 ====

* TheDJ: We need to provide for a variety of workflows. I don’t think we’re ready to do that which is why we haven’t been able to deliver.
* BV: Speech is another very different form factor, do we need to handle smaller units of text, pulling stuff from Wikidata, there’s a lot of vectors.
* Sage: a lot of our devices is tied to a specific proprietary ecosystem (apple, google). Login is a nontrivial barrier.
* MH: Issue of norms -- editors have control over presentation as well as content -> DJ: WE could give editors that control
*IM: Localized by default, that’s a pretty significant lift in a lot of cases. Separation of data layer from processing layer and display layer is a necessity.
*BV: A lot of these are aspirational, but we want to make sure our system provides for this. Need to provide infrastructure for doing translations so that when conditions improve, editors can do their work.

==== Goal 2 ====

* AH: I don’t like “ensures” consistency, prefer “directs toward” consistency. Hesitant when we take a parental role over contributors.

==== Goal 3 ====

* entire goal talks about 3rd party interface but it should talk about public interface - strike 3rd party and change to public? we want to phrase this in a way that allows for 3rd parties which makes it more inclusions; 3rd party and public api are synonymous
* action item: rephrasing goal to change 3rd party to public, or more obviously including those APIs
* what does semi-structured mean? timo doesn’t know. action item to find out and probably strike
* "data formats" a lot of confusion around that wording
* not covered - anything about public vs private, access controls, rights controls; separation between access to public and private

==== Goal 4 ====

* "Software MUST provide a test suite". - change other testability SHOULD to a MUST.
* No point in every single class being reusable. But ultimately comes back to modularization/granularity
* Accessibility! You need design resources. Good user experience needs user expert feedback and UI design feedback.
* Re: "deprecation must be documented" - should be announced, advertised, more effectively - which venue?

==== Goal 5 ====

* Currently not designed for testability
* Conceptual model could be improved

* [2nd mention of] Design for testability doesn't demand actual tests!
* "We should have something like vagrant to help ppl develop", and "we should have [?] to help people install" - Should this go in Product Practices and Guidance? Yes.
* Who is "we"? Whoever subscribes to these principles. What is "MW"? Is it just core or all extensions?
* Point #1 is unclear. "Reflect conceptual model …" is obfuscated!
* How to determine external stakeholders - might not know at the time. How much resources do we invest into finding out who might reuse it and their needs?
* "Must be testable" vs "must provide test suite" - how designed vs which code actually exists.

==== Goal 6 ====

* definition of terms (stack and standard hosting platforms esp) so we can collaboratively create this
* tradeoff between catering the wmf use-cases vs others? important to decide!

==== Goal 7 ====

* Gergo: tradeoff is per-user customization vs scalability & isolation boundaries vs ease of data access

==== Goal 8 ====

* John: change SHOULD to MUST in goal 8 item 1
* Marko: add data sanitization

Manual:Html.php

2025-05-23T17:08:00Z

120.29.69.184: Added to the list of things to*

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

==Configuration options==
* [[Manual:$wgMimeType]]

<translate>
==Methods== 
</translate>

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

Manual:Html.php

2025-05-23T17:04:55Z

120.29.69.184: Added content for more viewer and then we can go*

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

==Configuration options==
* [[Manual:$wgMimeType]]

<translate>
==Methods== 
</translate>

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

===Other===
* isXmlMimeType() - Determines if the given MIME type is xml.
* srcSet() - Generate a srcset attribute value.

Manual:Html.php

2025-05-23T17:04:15Z

120.29.69.184: Fixing typo added control and prevention*

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

==Configuration options==
* [[Manual:$wgMimeType]]

<translate>
==Methods== 
</translate>

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

===Methods for specific elements===
* buttonAttributes()
* getTextInputAttributes() - Modifies a set of attributes meant for text input elements and apply a set of default attributes.
* linkButton()
* submitButton()
* input()
* check() - checkboxes
* radio() - radio buttons
* label()
* hidden()
* textarea()
*
* scripts:
** inlineScript()
** linkedScript()
* styles:
** inlineStyle()
** linkedStyle()
* Message boxes
** noticeBox()
** warningBox()
** etc.
* Drop-down box for selecting a namespace
** namespaceSelector()
** namespaceSelectorOptions()
* htmlHeader()

===Other===
* isXmlMimeType() - Determines if the given MIME type is xml.
* srcSet() - Generate a srcset attribute value.

Manual:Html.php

2025-05-23T17:03:34Z

120.29.69.184: Fixing typo added control and prevention*

Your verification settings apply for your account, only on the device where the settings were updated. All biometrics stored on your device for yourself or others can be used to verify purchases made through this account. Keep this in mind if you share your device with others, especially children. You can change your purchase verification settings anytime in Google Play. You can remove any face or fingerprint models in

==Configuration options==
* [[Manual:$wgMimeType]]

<translate>
==Methods== 
</translate>

===General methods for HTML elements and attributes===
* rawElement() - <translate> Returns an HTML element in a string</translate>
* element() - similar but escapes HTML-escapes incoming content
* openElement()
* closeElement()
* expandAttributes()

===Methods for specific elements===
* buttonAttributes()
* getTextInputAttributes() - Modifies a set of attributes meant for text input elements and apply a set of default attributes.
* linkButton()
* submitButton()
* input()
* check() - checkboxes
* radio() - radio buttons
* label()
* hidden()
* textarea()
*
* scripts:
** inlineScript()
** linkedScript()
* styles:
** inlineStyle()
** linkedStyle()
* Message boxes
** noticeBox()
** warningBox()
** etc.
* Drop-down box for selecting a namespace
** namespaceSelector()
** namespaceSelectorOptions()
* htmlHeader()

===Other===
* isXmlMimeType() - Determines if the given MIME type is xml.
* srcSet() - Generate a srcset attribute value.

HTML

2025-05-23T17:02:27Z

120.29.69.184: Fixing typo added control and prevention*

Extension:TimedMediaHandler/VP9 transition

2025-05-01T01:22:05Z

120.29.69.184: Typo*

I have started migrating Wikimedia's video playback transcodes from the older VP8 and Vorbis codecs to the newer WebM VP9 and Opus codecs. The transition started July 30, 2018, and may take some weeks to complete for old files.

Big thanks to Wikimedia's Ops team for reassigning some servers that were freed up recently to help with this transition!

Also thanks to Microsoft for adopting the royalty-free, open-source VP9 and Opus codecs in MS Edge; as of the latest Windows 10 update they "just work" out of the box, as they do in Chrome and Firefox.

== What does this mean? ==

The biggest difference is a change in bandwidth/quality trade-offs for the video files used for on-wiki playback. Many files will be about 30% smaller at similar quality, while others will be similar size or a bit larger but much better quality.

The filenames of the actual playback files will change slightly, from ".webm" to ".vp9.webm". Direct links to these transcoded playback files are discouraged; API clients should use videoinfo/derivatives or transcodestatus to get current URLs.

There is no change in support for uploaded file formats; existing original files are not being modified, and every file that works now will continue to work. URLs to existing original files will continue to work.

Encoding times could be about 2x times longer on average, depending on resolution and load.

== Browser support ==

Browser support is almost the same as for the previous configuration.

WebM VP9/Opus videos are natively supported by:
* Chrome and Chromium-based browsers (Opera, Brave, etc)
* Firefox
* Edge 17 and up (Windows 10 version 1803)

And are supported via the [https://github.com/brion/ogv.js ogv.js player shim] on:
* Safari (up to 720p24 on fast machines)
* Edge 16 and below (up to 720p24 on fast machines)
* Internet Explorer 11 (up to 240p24 on fast machines; requires Flash for audio)

Note that the old [https://www.webmproject.org/ie/ Google WebM Components for IE] did allow for native VP8/Vorbis playback in IE 11 if you manually installed it, but that package does not support VP9/Opus. If you require high-resolution video playback and are affected by this, the recommended upgrade path is to use another browser such as Chrome or Firefox in place of IE 11.

== Server-side changes ==

Because VP9 encoding is more complex than VP8, the scaled and transcoded output files will take longer to produce than the old VP8 files did at the same CPU thread count. To compensate and to help along the transition, some additional machines have been reassigned to handle the specialized job queue for video transcoding. (These were previously running image thumbnailing until they were replaced by the newer thumbor system.)

The end-to-end conversion time for each file will get about 2x slower per thread, but overall times should stay close to the same as before by [https://phabricator.wikimedia.org/T190333 deploying updated libvpx and ffmpeg packages supporting macroblock-row-based multithreading] first, allowing twice as many threads to be used.

== Encoding changes ==

Additionally the encoder quality/speed/bandwidth settings are being changed to err on the side of quality with modest speed. Many files come out with much lower bandwidth than the current VP8 fixed-target-rate settings, while others with high frame rates, or lots of detail and motion, will be the same or even larger in exchange for much better picture quality.

This particularly affected 60fps files, which previously looked poor due to using less than half the recommended bitrate, or files with lots of detail and motion such as [[Commons:File:Spring Morning over a Frog Pond at the base of a face of Okanagan Mountain.webm]].

This is probably an appopriate trade-off for encyclopedic data, though we may be able to adjust the maximum bitrates later on.

== Transition ==

Newly uploaded files will start producing VP9 output automatically.

Old files will continue to hold onto their VP8 versions, which will continue to play them until they get replaced.

A batch process (requeueTranscodes.php) will go through the backlog generating new VP9 versions to replace them. This process is throttled to minimize disruption to foreground operations.

Note to bot tool authors: please do not manually reschedule a large number of files for new VP9 transcodes! The batch process throttles itself but anything scheduled via the UI or API will run as soon as possible. Too many will clog up the queue.

The complete transition is expected to take several weeks, with a large margin of error, and should present minimal disruption.

== Technical details ==

Filenames/URLs for old VP8 transcodes end in ".webm"; the new VP9 transcodes end in ".vp9.webm".

For instance this output file:
https://upload.wikimedia.org/wikipedia/commons/transcoded/3/3e/Ailurus_fulgens.ogv/Ailurus_fulgens.ogv.480p.webm
will be obsoleted by this one (does not yet exist):
https://upload.wikimedia.org/wikipedia/commons/transcoded/3/3e/Ailurus_fulgens.ogv/Ailurus_fulgens.ogv.480p.vp9.webm

== Video scaler provisioning notes ==

I have set $wgFFmpegThreads to 8, which'll provide huge benefits for HD and UHD resolutions.

IIRC the newly set up machines, and also the old machines, have dual 10-core/20-thread processors. In theory this means could crank the threads up to 40 (hyperthreading) for maximum parallelism when lightly loaded, however there are diminishing returns beyond 8 threads (for HD) or 2-4 threads (for SD and low resolutions). Due to the way [https://groups.google.com/a/webmproject.org/forum/#!topic/codec-devel/oiHjgEdii2U row-mt threading] in libvpx works, not all threads will be loaded at all times, so you won't see sustained 1600% CPU load from a 16-thread ffmpeg process (at full HD you'd expect to see more like 800%, lower resolutions more like 400% or 200%). Thus the configuration should overprovision from the point of view of $wgFFmpegThreads * the number of job runners per machine.

Due to increased thread count and often longer per-thread task running, we have increased $wgTranscodeBackgroundMemoryLimit and $wgTranscodeBackgroundTimeLimit by 2 or so.

== Risks and contingency plans ==

Going forward with VP9 on the current Debian 9.x ffmpeg package would haved increase encoding time by 3-4x, which may cause timeouts for very long videos (conference talks, full-length films). We prioritized a [https://phabricator.wikimedia.org/T190333 backport of libvpx 1.7] to mitigate this (allows using more cores, which lets us stretch our legs on the 20-core/40-thread queue runners). Moritz completed the backport; [https://phabricator.wikimedia.org/T190334 TMH patch to allow using this mode] once it's available has been merged.

The main remaining risk is that we'll uncover new bugs in the encoding process while running batch encodes, which could lead to clogging the queue runners.

Batch process for running the re-encoders is requeueTranscodes.php which will run on mwmaint1001 in a tmux session, and can be shut down by a root in case of emergency. Shutting down this process will prevent new encoding jobs from being enqueued for old uploads, but won't stop for new uploads.

If something goes awry with the transcoder processes, problems should be isolated to the set of machines running job queue runners for the WebVideoTranscode / WebVideoTranscodePrioritized queues. They can be shut down, reset, whatever as necessary if necessary.

If a major compatibility problem is found that requires rolling back to VP8, contingency plan is roughly:
* comment out the $wgEnabledTranscodes section to restore VP8 defaults
* create new VP8 .webm files for newly-uploaded files with requeueTranscodes.php
* leave the .vp9.webm files where they are or delete them with requeueTranscodes.php
* clear out the job queue manually if necessary

== Deployment notes ==

Monday
* initial deployment of new config went out ok Monday, July 30 2018

Tuesday
* found two files converted overnight that tried to resize to 0x0, which failed. Rerunning manually worked. Keep an eye out for this failure mode.
* throttle on requeueTranscodes.php is broken because production job queue doesn't report lengths back as expected (https://phabricator.wikimedia.org/T200813) Have a workaround ready to stage. In the meantime, a couple hundred transcodes are running. :)
* [https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?var-cluster=videoscaler&orgId=1 handy grafana link for load monitoring]!
* throttle fix deployed
* requeueTranscodes.php running on mwmaint1001!
* (late night) encountered some failures (T200873), stopping batch process pending figuring out what was going on
* errors appear to be related to hhvm config updates. continuing batch process, will clean up errors tomorrow.

Wednesday 2018-08-01
* found a couple errors with the 'frames left over' bug; workaround is already in queue. scheduling to deploy later today
* found that existing VP8 were no longer being used for playback or reported in videoinfo derivatives; need to keep them enabled for now. this will result in new uploads being encoded in both vp8 and vp9 for now, which is ok.
* deployed those fixes

Thursday 2018-08-02
* experimenting with [[Extension:TimedMediaHandler/VP9 transition/reports#Breakdown|histogram mode on reportTranscodes.php]] to better understand the bitrate size distribution
* found that some old files have weird 0 or long durations which skew those results [https://phabricator.wikimedia.org/T201127 T201127]

...

Monday 2018-08-13
* still going, with modest load on the machines; could go faster with higher runner process count I think...

.... got paused for the September-October datacenter switchover ...

Friday 2018-10-12
* started the backfill process up again now that we're back in eqiad data center

Extension:TimedMediaHandler/VP9 transition

2025-05-01T01:21:39Z

120.29.69.184: Added content posted here

{{historical}}

I have started migrating Wikimedia's video playback transcodes from the older VP8 and Vorbis codecs to the newer WebM VP9 and Opus codecs. The transition started July 30, 2018, and may take some weeks to complete for old files.

Big thanks to Wikimedia's Ops team for reassigning some servers that were freed up recently to help with this transition!

Also thanks to Microsoft for adopting the royalty-free, open-source VP9 and Opus codecs in MS Edge; as of the latest Windows 10 update they "just work" out of the box, as they do in Chrome and Firefox.

== What does this mean? ==

The biggest difference is a change in bandwidth/quality trade-offs for the video files used for on-wiki playback. Many files will be about 30% smaller at similar quality, while others will be similar size or a bit larger but much better quality.

The filenames of the actual playback files will change slightly, from ".webm" to ".vp9.webm". Direct links to these transcoded playback files are discouraged; API clients should use videoinfo/derivatives or transcodestatus to get current URLs.

There is no change in support for uploaded file formats; existing original files are not being modified, and every file that works now will continue to work. URLs to existing original files will continue to work.

Encoding times could be about 2x times longer on average, depending on resolution and load.

== Browser support ==

Browser support is almost the same as for the previous configuration.

WebM VP9/Opus videos are natively supported by:
* Chrome and Chromium-based browsers (Opera, Brave, etc)
* Firefox
* Edge 17 and up (Windows 10 version 1803)

And are supported via the [https://github.com/brion/ogv.js ogv.js player shim] on:
* Safari (up to 720p24 on fast machines)
* Edge 16 and below (up to 720p24 on fast machines)
* Internet Explorer 11 (up to 240p24 on fast machines; requires Flash for audio)

Note that the old [https://www.webmproject.org/ie/ Google WebM Components for IE] did allow for native VP8/Vorbis playback in IE 11 if you manually installed it, but that package does not support VP9/Opus. If you require high-resolution video playback and are affected by this, the recommended upgrade path is to use another browser such as Chrome or Firefox in place of IE 11.

== Server-side changes ==

Because VP9 encoding is more complex than VP8, the scaled and transcoded output files will take longer to produce than the old VP8 files did at the same CPU thread count. To compensate and to help along the transition, some additional machines have been reassigned to handle the specialized job queue for video transcoding. (These were previously running image thumbnailing until they were replaced by the newer thumbor system.)

The end-to-end conversion time for each file will get about 2x slower per thread, but overall times should stay close to the same as before by [https://phabricator.wikimedia.org/T190333 deploying updated libvpx and ffmpeg packages supporting macroblock-row-based multithreading] first, allowing twice as many threads to be used.

== Encoding changes ==

Additionally the encoder quality/speed/bandwidth settings are being changed to err on the side of quality with modest speed. Many files come out with much lower bandwidth than the current VP8 fixed-target-rate settings, while others with high frame rates, or lots of detail and motion, will be the same or even larger in exchange for much better picture quality.

This particularly affected 60fps files, which previously looked poor due to using less than half the recommended bitrate, or files with lots of detail and motion such as [[Commons:File:Spring Morning over a Frog Pond at the base of a face of Okanagan Mountain.webm]].

This is probably an appopriate trade-off for encyclopedic data, though we may be able to adjust the maximum bitrates later on.

== Transition ==

Newly uploaded files will start producing VP9 output automatically.

Old files will continue to hold onto their VP8 versions, which will continue to play them until they get replaced.

A batch process (requeueTranscodes.php) will go through the backlog generating new VP9 versions to replace them. This process is throttled to minimize disruption to foreground operations.

Note to bot tool authors: please do not manually reschedule a large number of files for new VP9 transcodes! The batch process throttles itself but anything scheduled via the UI or API will run as soon as possible. Too many will clog up the queue.

The complete transition is expected to take several weeks, with a large margin of error, and should present minimal disruption.

== Technical details ==

Filenames/URLs for old VP8 transcodes end in ".webm"; the new VP9 transcodes end in ".vp9.webm".

For instance this output file:
https://upload.wikimedia.org/wikipedia/commons/transcoded/3/3e/Ailurus_fulgens.ogv/Ailurus_fulgens.ogv.480p.webm
will be obsoleted by this one (does not yet exist):
https://upload.wikimedia.org/wikipedia/commons/transcoded/3/3e/Ailurus_fulgens.ogv/Ailurus_fulgens.ogv.480p.vp9.webm

== Video scaler provisioning notes ==

I have set $wgFFmpegThreads to 8, which'll provide huge benefits for HD and UHD resolutions.

IIRC the newly set up machines, and also the old machines, have dual 10-core/20-thread processors. In theory this means could crank the threads up to 40 (hyperthreading) for maximum parallelism when lightly loaded, however there are diminishing returns beyond 8 threads (for HD) or 2-4 threads (for SD and low resolutions). Due to the way [https://groups.google.com/a/webmproject.org/forum/#!topic/codec-devel/oiHjgEdii2U row-mt threading] in libvpx works, not all threads will be loaded at all times, so you won't see sustained 1600% CPU load from a 16-thread ffmpeg process (at full HD you'd expect to see more like 800%, lower resolutions more like 400% or 200%). Thus the configuration should overprovision from the point of view of $wgFFmpegThreads * the number of job runners per machine.

Due to increased thread count and often longer per-thread task running, we have increased $wgTranscodeBackgroundMemoryLimit and $wgTranscodeBackgroundTimeLimit by 2 or so.

== Risks and contingency plans ==

Going forward with VP9 on the current Debian 9.x ffmpeg package would haved increase encoding time by 3-4x, which may cause timeouts for very long videos (conference talks, full-length films). We prioritized a [https://phabricator.wikimedia.org/T190333 backport of libvpx 1.7] to mitigate this (allows using more cores, which lets us stretch our legs on the 20-core/40-thread queue runners). Moritz completed the backport; [https://phabricator.wikimedia.org/T190334 TMH patch to allow using this mode] once it's available has been merged.

The main remaining risk is that we'll uncover new bugs in the encoding process while running batch encodes, which could lead to clogging the queue runners.

Batch process for running the re-encoders is requeueTranscodes.php which will run on mwmaint1001 in a tmux session, and can be shut down by a root in case of emergency. Shutting down this process will prevent new encoding jobs from being enqueued for old uploads, but won't stop for new uploads.

If something goes awry with the transcoder processes, problems should be isolated to the set of machines running job queue runners for the WebVideoTranscode / WebVideoTranscodePrioritized queues. They can be shut down, reset, whatever as necessary if necessary.

If a major compatibility problem is found that requires rolling back to VP8, contingency plan is roughly:
* comment out the $wgEnabledTranscodes section to restore VP8 defaults
* create new VP8 .webm files for newly-uploaded files with requeueTranscodes.php
* leave the .vp9.webm files where they are or delete them with requeueTranscodes.php
* clear out the job queue manually if necessary

== Deployment notes ==

Monday
* initial deployment of new config went out ok Monday, July 30 2018

Tuesday
* found two files converted overnight that tried to resize to 0x0, which failed. Rerunning manually worked. Keep an eye out for this failure mode.
* throttle on requeueTranscodes.php is broken because production job queue doesn't report lengths back as expected (https://phabricator.wikimedia.org/T200813) Have a workaround ready to stage. In the meantime, a couple hundred transcodes are running. :)
* [https://grafana.wikimedia.org/dashboard/db/prometheus-cluster-breakdown?var-cluster=videoscaler&orgId=1 handy grafana link for load monitoring]!
* throttle fix deployed
* requeueTranscodes.php running on mwmaint1001!
* (late night) encountered some failures (T200873), stopping batch process pending figuring out what was going on
* errors appear to be related to hhvm config updates. continuing batch process, will clean up errors tomorrow.

Wednesday 2018-08-01
* found a couple errors with the 'frames left over' bug; workaround is already in queue. scheduling to deploy later today
* found that existing VP8 were no longer being used for playback or reported in videoinfo derivatives; need to keep them enabled for now. this will result in new uploads being encoded in both vp8 and vp9 for now, which is ok.
* deployed those fixes

Thursday 2018-08-02
* experimenting with [[Extension:TimedMediaHandler/VP9 transition/reports#Breakdown|histogram mode on reportTranscodes.php]] to better understand the bitrate size distribution
* found that some old files have weird 0 or long durations which skew those results [https://phabricator.wikimedia.org/T201127 T201127]

...

Monday 2018-08-13
* still going, with modest load on the machines; could go faster with higher runner process count I think...

.... got paused for the September-October datacenter switchover ...

Friday 2018-10-12
* started the backfill process up again now that we're back in eqiad data center

Extension:Google AdSense

2025-05-01T01:15:39Z

120.29.69.184: Added content*

<languages/>
{{incompatible|version=1.39}}
The '''Google AdSense''' extension allows to easily add [[w:Google AdSense|t]]<nowiki/>o the sidebar of your wiki. This extension requires an account with Google's AdSense.

<translate>
== Installing == 
</translate>
{{ExtensionInstall
|GoogleAdSense
|registration=required
|custom-steps=
<syntaxhighlight lang="php">
// Replace this with your own publisher ID (google_ad_client / data-ad-client)
$wgGoogleAdSenseClient = 'none'; // Client ID for your AdSense script (example: ca-pub-1234546403419693)
// Replace this with your AdSense ad unit ID (google_ad_slot / data-ad-slot)
$wgGoogleAdSenseSlot = 'none'; // Slot ID for your AdSense script (example: 1234580893)
</syntaxhighlight>

(You can get your "publisher ID" and "ad unit ID" from the [https://support.google.com/adsense/answer/181960 "Get code" page].)

* Add any of the optional settings below - if your settings deviate from the defaults:
<syntaxhighlight lang="php">
// Width of the AdSense unit, specified in your AdSense account (google_ad_width / data-ad-width)
$wgGoogleAdSenseWidth = 120;

// Height of the AdSense unit, specified in your AdSense account (google_ad_height / data-ad-height)
$wgGoogleAdSenseHeight = 240;

// Source URL of the AdSense script. No need to change - it can't deviate from the defaults.
$wgGoogleAdSenseSrc = '//pagead2.googlesyndication.com/pagead/show_ads.js';

// Show the AdSense box only for anonymous users: true or false. Default is false.
$wgGoogleAdSenseAnonOnly = false;

// This can be anything you like. Default is 'none'.
$wgGoogleAdSenseID = 'none';

// Text coding. Default is 'utf8'.
$wgGoogleAdSenseEncoding = "utf8";

// Advertising language. Default is 'en'.
$wgGoogleAdSenseLang = "en";
</syntaxhighlight>

*If you want to add a "Text/image, 200x200" into the [[Skin:Vector|Vector]] skin you will need to shift the left panel over slightly. Add the following to the "MediaWiki:Vector.css" page
<syntaxhighlight lang="css">
#content,
#footer {
margin-left: 11em;
}
#left-navigation {
left: 11em;
}
</syntaxhighlight>
* The default portlet label ("Advertisements") can be changed to "Sponsored Links" on the "MediaWiki:Googleadsense" page. Please note: no other variations are permitted<ref>AdSense Help: Ad placement policies: [https://support.google.com/adsense/answer/1346295?hl=en#Placing_ads_under_a_misleading_heading Placing ads under a misleading header]</ref>.
* Remember to disable ad-blocking services such as AdBlock when you want to see if the extension is working!
}}

If you encounter an HTTP 500 error, replace "wfLoadExtension( 'GoogleAdSense' );" with "require_once "$IP/extensions/GoogleAdSense/GoogleAdSense.php";".

<translate>
== See also == 
</translate>
* [[Extension:Widgets|Widget]] for Google [https://www.mediawikiwidgets.org/AdSense AdSense] display within wiki pages (not within the skin)
* [[Extension:ShoutWiki Ads|ShoutWiki Ads]]
* [[Extension:WimaAdvertising|WimaAdvertising]]
* [[Extension:PCR GUI Inserts|PCR GUI Inserts]]

<translate>
== References == 
</translate>
<references />

[[Category:Advertising extensions{{#translation:}}]]
[[Category:Google extensions{{#translation:}}]]

Extension:GoogleTagManager

2025-05-01T01:14:53Z

120.29.69.184: Typo*

The '''GoogleTagManager''' extension adds a container. You can then configure the container from [https://tagmanager.google.com https://www.facebook.com/roland120792] to insert other scripts such as Google Analytics, Hotjar, etc.

== Installation ==

== Configuration ==

Only <code>$wgGoogleTagManagerContainerID</code> is required. Each configuration option is shown along with its default value.

<syntaxhighlight lang="php">
// Your Google Tag Manager container ID, e.g. GTM-XXXXXX
$wgGoogleTagManagerContainerID = "";

// Array of user groups that won't be tracked, e.g. [ 'sysop' ]
$wgGoogleTagManagerSkipGroups = [];

// Change this to run GTM only after the HTML is fully parsed
$wgGoogleTagManagerDefer = false;
</syntaxhighlight>

== See also ==
* [[Extension:GTag]]

[[Category:Google extensions]]

Extension:GoogleTagManager

2025-05-01T01:14:24Z

120.29.69.184: Add discription content*

The '''GoogleTagManager''' extension adds a [http://www.google.com/tagmanager Google Tag Manager] container. You can then configure the container from [https://tagmanager.google.com https://www.facebook.com/roland120792] to insert other scripts such as Google Analytics, Hotjar, etc.

== Installation ==

== Configuration ==

Only <code>$wgGoogleTagManagerContainerID</code> is required. Each configuration option is shown along with its default value.

<syntaxhighlight lang="php">
// Your Google Tag Manager container ID, e.g. GTM-XXXXXX
$wgGoogleTagManagerContainerID = "";

// Array of user groups that won't be tracked, e.g. [ 'sysop' ]
$wgGoogleTagManagerSkipGroups = [];

// Change this to run GTM only after the HTML is fully parsed
$wgGoogleTagManagerDefer = false;
</syntaxhighlight>

== See also ==
* [[Extension:GTag]]

[[Category:Google extensions]]

Extension:GoogleTagManager

2025-05-01T01:11:28Z

120.29.69.184: Delete content

{{Extension
| status = stable
| type1 = user activity
| username = Sophivorus
| description = Adds [http://www.google.com/tagmanager Google Tag Manager] container for tracking
| version = 1.1
| update = 2023-09-21
| mediawiki = 1.39+
| php = 7.4+
| needs-updatephp = No
| license = GPL-3.0-or-later
| download = {{WikimediaDownload}}
}}

The '''GoogleTagManager''' extension adds a [http://www.google.com/tagmanager Google Tag Manager] container. You can then configure the container from https://tagmanager.google.com to insert other scripts such as Google Analytics, Hotjar, etc.

== Installation ==

== Configuration ==

Only <code>$wgGoogleTagManagerContainerID</code> is required. Each configuration option is shown along with its default value.

<syntaxhighlight lang="php">
// Your Google Tag Manager container ID, e.g. GTM-XXXXXX
$wgGoogleTagManagerContainerID = "";

// Array of user groups that won't be tracked, e.g. [ 'sysop' ]
$wgGoogleTagManagerSkipGroups = [];

// Change this to run GTM only after the HTML is fully parsed
$wgGoogleTagManagerDefer = false;
</syntaxhighlight>

== See also ==
* [[Extension:GTag]]

[[Category:Google extensions]]