Talk:Machine Learning/2022

A liveblog and forum about machine learning at Wikimedia. Create new topics to ask questions or post updates.

Happy New Year ML team!

Hello ML Team,

First of all, a Happy 2022 to you all!

The Dutch Wikipedia and the ML volunteers have finalized working on our third version of the Dutch article quality scale just before Christmas, and now we're waiting impatiently for the code review. Has anyone returned from the holiday mentioned in your last post; any chance we can get the deployment configuration done soon? @CAlbon (WMF), @ACraze (WMF), @KBazira (WMF)?

I'd love to see the effects of our work! Ciell (talk) 16:50, 6 January 2022 (UTC)

Hey Ciell! Most folks are back today! We will take today. Sorry for the wait. CAlbon (WMF) (talk) 17:46, 10 January 2022 (UTC)

Hi Ciell -- Happy new year!

It looks like there are three PRs on GitHub, mostly updating the model class repos to use the newest version of revscoring. This could have some implications to our current work, so we would like to discuss our solution at the team meeting this week before merging.

https://github.com/wikimedia/editquality/pull/236

https://github.com/wikimedia/draftquality/pull/42

https://github.com/wikimedia/drafttopic/pull/63 ACraze (WMF) (talk) 19:46, 10 January 2022 (UTC)

Hi Andy, do you maybe have an update about this for me? Just so I can plan my work as well. :) Ciell (talk) 20:21, 19 January 2022 (UTC)

Hi Ciell! The code has been merged and we are planning for the next deployment. There is some work being done by our SREs related to monitoring and security that needs to be completed first, I am hoping it will be in a ready state towards the end of the week or early next. I will keep you updated as things progress :) ACraze (WMF) (talk) 22:40, 25 January 2022 (UTC)

Quick update: We are moving forward with the deployment and tracking it in https://phabricator.wikimedia.org/T300195 , I have started some of the prep work related to loading the new model into production and hope to deploy it early next week. ACraze (WMF) (talk) 21:58, 27 January 2022 (UTC)

Thanks Andy! The team meeting is on Tuesday, if I recall correctly? Ciell (talk) 20:14, 10 January 2022 (UTC)

Happy new year Ciell. 👋

The meeting will be on Wednesday. KBazira (WMF) (talk) 14:19, 11 January 2022 (UTC)

Weekly Team Update 2021-01-13

• The team is still working through issues with getting our new machine learning model hosting infrastructure live. The infrastructure can host and deploy models, but it still isn’t connected to the API gateway, which is how we want everyone to access the infrastructure, but that will happen very soon.
• We are shifting how we are working with the Product and Research teams. Instead of a waterfall approach (e.g. Research makes a model, then passes it to engineering who figures out how to deploy it), we are going to start having engineers on my team work with researchers as early as possible in the process. The goal is that models (even the earliest, most basic version of them) are deployed from day one, and then continually redeployed as the models are developed and improved over months. The benefit is that there are no surprises when it comes to deploying the model into actual production. If a model is not compatible with deployment to the production environment, we will catch that at the very start of the process rather than at the end.
• We have migrated a lot of models currently hosted on ORES to our new infrastructure, however they are still pretty slow. ~2-3 seconds per prediction. This isn’t nearly good enough but expected. The big issue is I/O. For example, when some models receive a request for a prediction, it hits the mediawiki api, downloads the full text of the article in question, processes the wikitext, converts the processed text into a vector of features, then sends the vector of features to the model. This is all done BEFORE the model actually makes any prediction. We suspect this is causing most, if not all, of the slowness in predictions. The solution for the medium-term is caching scores where possible. We are working on the details of that implementation.
• We have a new machine learning engineer joining the team! This is great. Except our onboarding docs are out of date. Chris will be working on updating them over this week.
• Our new machine learning infrastructure is something we are very proud of. However, there is virtually no mention of it anywhere on any Wiki or blog post. This has been mentioned by community members. Chris spent last week starting to fix that, which will continue over this month.
• As part of our model governance work, we are creating model cards (short summaries of each model). As a start, we are creating two models by hand, after which we can hopefully automate a lot of the updating of the model card generation. CAlbon (WMF) (talk) 15:45, 13 January 2022 (UTC)

Were the ORES models always that slow or is the 2-3 seconds a new regression? Can you give a pointer to the code where it's getting the wikitext/processing it? Legoktm (talk) 18:44, 13 January 2022 (UTC)

The ORES models were always slow. The speed is ORES is from the pre-caching of scores. The "new" models are dockerzied port of the current models.

Here is where RevScoring is hitting the mediawiki API:

https://github.com/wikimedia/revscoring/blob/773f9cd8029de7ef5c7713addd2f6661bce94b4e/revscoring/extractors/api/extractor.py CAlbon (WMF) (talk) 16:44, 18 January 2022 (UTC)

ORES

Hello! I'm an admin at SqWiki. I was reading about ORES lately and I had a question: Is there anyway to keep training ORES if we so wished? (Assuming that would be a beneficial thing.) Klein Muçi (talk) 19:49, 25 January 2022 (UTC)

You mean retraining ORES with additional data? CAlbon (WMF) (talk) 20:29, 26 January 2022 (UTC)

@CAlbon (WMF), yes. I mean, I say "keep training" because I think "it can get smarter" (pardon my simple language), you say "retraining", meaning that after that was over, nothing substantial would change? Klein Muçi (talk) 21:42, 26 January 2022 (UTC)

Weekly Team Update 2021-01-26

I'm just coming back from my kid having COVID so bear with me!

Weekly Update!

• The Kubernetes cluster powering Lift Wing is (dare I say it) ready for MVP (minimum viable product). As part of a larger effort at WMF to centralize our APIs, the API for Lift Wing models will be available through https://api.wikimedia.org (also called the API Gateway). We are waiting for a few last features to be built by the API Gateway team and then we should be ready for an MVP.
• Hardware for our prediction cache (i.e. saving predictions previously made so they can be quickly served to requesters) has been purchased. It'll take a bit for it to be assembled, shipped, installed in the data center, and setup. However, it is a big step and one we are happy about.
• We have the contents of two model card proof of contents in a Google Doc, now we are talking to the Design team about how best to present them. Once we have a draft of something on a wiki, I'll let you all know and ask for feedback before we move beyond a proof of concept.
• We will be deploying an updated ORES model created by the community for Dutch Wikipedia in the coming days. See the ticket for details.
• Livestreamed office hours will return! I haven't forgotten, just my world got turned upside down there for a few weeks. But things are returning to normal.

My big question for all of you:

Our plan is for the new machine learning infrastructure (Lift Wing) to contain all the models currently hosted the current infrastructure (ORES) and then maintain both systems for at least six months, ideally a year. The goal is to allow folks to migrate slowly -- using the same models as on ORES but on the new infrastructure. However, one looming problem we face here is that because you don't need an account to use ORES, we don't know everyone who relies on it, so we can't contact them and tell them ORES will eventually be deprecated.

So if you use ORES or have heard of someone else using ORES, I'd love a quick reply to this post. That way we can make sure to reach out to everyone. CAlbon (WMF) (talk) 20:45, 26 January 2022 (UTC)

Hey Chris, I am maintaining a tool that uses ORES scores (at Wikidata). However, it does not do so via the APIs at ores.wikimedia.org — it queries the MediaWiki database as described in Extension:ORES/Schema. Will this still be possible with the new infrastructure? If so, will anything change at all, or will I likely not notice the migration? MisterSynergy (talk) 22:04, 26 January 2022 (UTC)

Yep that will possible. You might need to change your query but hopefully that should be it. CAlbon (WMF) (talk) 17:50, 27 January 2022 (UTC)

Ping for @Fuzheado, maybe he knows of external people or GLAMs using ORES. Ciell (talk) 15:30, 28 January 2022 (UTC)

Thanks @Ciell - Yes, I actually use the ORES API in raw form for scripts on Google Sheets, so that I might display the predicted quality of an article in graphical form (color coded cells in a spreadsheet). Sample Javascript being used in Google Docs:

var url = 'https://ores.wikimedia.org/v3/scores/' + targetwiki + '/' + revid + '/' + 'wp10';

var json = JSON.parse(UrlFetchApp.fetch(url, HEADERS).getContentText());

results[0] = json.enwiki.scores[revid].wp10.score.prediction;

Hopefully by just replacing that REST API call with something similar should do the trick, but let me know. Thanks. Fuzheado (talk) 09:50, 4 March 2022 (UTC)

As you know, WMDE is using ORES for evaluating the quality of Wikidata Items and edits and maybe more (`articlequality`, `editquality`). I don't understand enough about ORES or Lift Wing at all to be able to say what migration effort we are facing. But tagging @Manuel Merz (WMDE) and @Kara Payne (WMDE) so that the relevant people are aware that there may be some migration effort coming our way at some point this year. Michael Große (WMDE) (talk) 13:25, 1 February 2022 (UTC)

Weekly Team Update 2022-02-02

Shorter update this week.

• Work is still being done on the API Gateway I mentioned last week. When that is done we will be able to connect Lift Wing to the API Gateway, which would mean we would be ready for MVP.
• The hardware for our staging cluster is being installed, which is great because it means that we can fully test any model deployment or system change in a safe place before pushing it to production.

All in all a good week, the team is heads down working on marching us to MVP! CAlbon (WMF) (talk) 20:51, 2 February 2022 (UTC)

Weekly Team Update 2022-02-18

Another short update this week.

• The big win for the week is that we have finalize the exact procedure for migrating the models currently hosted on ORES to Lift Wing. Now we are doing that en mass. This will give us model parity between ORES and Lift Wing.
• No livestream this week, I've been taking care of a family member for two weeks so I haven't had time, but (and I know I keep saying this) I am determined to restart the livestreams. CAlbon (WMF) (talk) 16:17, 18 February 2022 (UTC)

Lift Wing page

Hi Chris and Team,

I've been searching for more written information on Lift Wing: can we please create a page explaining what Lift Wing is (or in this stage is planned/envisioned to be), and link to any technical information that you've been working on for the past 1,5 years? I've been looking for information on Github and Gerrit, I assumed this would be the most logical place where to find more information on the back end, but there seems to be nothing there. Ciell (talk) 19:21, 24 February 2022 (UTC)

So sorry Ciell. My family got COVID and I'm way late on answering this. My apologies.

You are right we are very much lacking on Lift Wing information. We have technical documentation but that is for ourselves. Writing something like this has been my responsibility and I haven't done it. To hold myself accountable, I have created a ticket for the work. CAlbon (WMF) (talk) 18:37, 2 March 2022 (UTC)

Thanks Chris, the link to the technical information is already very interesting! Ciell (talk) 19:27, 3 March 2022 (UTC)

Weekly Team Update 2022-03-10

• Sadly one of our engineers, Andy Craze is moving onto another role outside WMF. We are all sad to be losing him and wish him the best. We will be hiring for Andy's role.

• We continue to be blocked from connecting to the API Gateway, although other teams are working to unblock us. No ETA yet, but we aren't just waiting around, we are continuing to improve Lift Wing (latest change is an increased IP Pool). That said, I think everyone on the team itching to launch the MVP and how people what we have.

• Our communication and updates around Lift Wing need to be improved. This is my fault and I'm working on it, but keep me honest if I slip!

• A new staging cluster for Lift Wing will be live soon. This allows us to test that changes are working before pushing them to production.

• We experimented with using Kubeflow's pre-processing and post-processing transformers when migrating the models currently hosted on ORES to Lift Wing. It added a lot of complexity for no real benefit, so we abandoned it.

• We have budget for a summer intern, so expect that soon! CAlbon (WMF) (talk) 18:26, 10 March 2022 (UTC)

Weekly Team Update 2022-03-17

One big update this week: we have a proof-of-concept for our model cards and are looking for feedback!

The broad goal is for ML models hosted by Wikimedia to have a model card where folks can understand, discuss, and govern that model.

To get to this point, we made a first version of a model card and then gathered feedback from experts, researchers, and community members. Next, we incorporated that feedback and made a second version which is what is linked above. Now we hope to gather a broader range of feedback in the Talk page to incorporate in the third and final version of the model card. Once that is complete we will start making model cards for more models we host. CAlbon (WMF) (talk) 18:14, 17 March 2022 (UTC)

I had a look at the model card. I appreciate all the information there and I feel like I get a good understanding from it about what the Model can do and cannot do. However, I'm basically just a layperson in this domain and so I have no idea whether it is "complete" in any sense of the word.

That being said, given that it is that very extensive and bespoke, I worry about its maintainability. Do you have any plans on creating more specific templates around this?

Also, it is right now just plain wikitext. How many models (and thus model cards) do we expect to be hosted by Liftwing eventually? If that goes into the hundreds or thousands, I could imagine that it would be nice to be able to query some of their information in a structured way other than just by using the mediawiki fulltext search. Michael Große (WMDE) (talk) 18:03, 25 March 2022 (UTC)

Yeah we've gone back and forth on just that point (maintainability). Our currently thinking is for the prose to be wikitext, but any data about the models (training scores, etc) be auto-generated where possible.

I am very worried about data rot. CAlbon (WMF) (talk) 17:31, 5 April 2022 (UTC)

Weekly Team Update 2022-03-24

I have (another) cold this week so I'm going slower than normal.

• We are still looking for feedback for a proof-of-concept for our model cards.
• I learned yesterday enwiki doesn't actually have an article on model cards.
• Writing more broadly about Lift Wing and restarting the livestreams are still top of my to do list. Bear with me.
• Next week I'll be in NYC for an Tech Department off-site (the first time ever for me). There likely won't be an update for that week. My apologies in advance.
• An issue came up last week about what constitutes "consensus" in a Wikipedia community. Is consensus defined by how many people agree? Does the venue of discussion matter? Does a consensus agreed to many years ago apply today? I don't have any answers but it is a worthwhile conversation I'd love to have with all of you. Our goal is that the communities affected by models hosted at WMF have the ability to govern those models. But what that looks like in practice is something we will have to work out together.
• Lift Wing is up and running, but not in MVP stage yet because it is currently blocked by the API Gateway (api.wikimedia.org). However the API folks at WMF are working on it. CAlbon (WMF) (talk) 16:47, 24 March 2022 (UTC)

Hope you get better soon! Please take care of yourself, your health is more important than work.

I looked for a minute and couldn't find a Wikidata Item for "Model Card" either. Do you happen to know if there is an article for it in any language Wikipedia?

The only thing I could find is an item for a scientific article describing Model Cards: https://www.wikidata.org/wiki/Q103724290 Michael Große (WMDE) (talk) 09:52, 25 March 2022 (UTC)

I don't think there is an article anywhere but couldn't be sure. We only realized it when one piece of feedback from Nemo_bis mentioned that they didn't know what model cards were, and the closest they could find was an article about playing cards. CAlbon (WMF) (talk) 14:00, 25 March 2022 (UTC)

Weekly Team Update 2022-04-06

Back from NYC and no major changes to the work for this week.

• We are looking for feedback on proof of concept model card.
• We are about to start searching for a senior machine learning engineer to replace a team member who sadly left to other adventures.
• Work continues on Lift Wing's IP pool.
• We are still blocked from showing you all Lift Wing until the API Gateway (owned by another team) has some additional features.
• We are discussing how we might make a few OS and security updates to ORES to keep it running smoothly for at least a year. I want a full year from the time we announce the ORES migration to users to when we turn off ORES giving users plenty of time to migrate. This means there are likely a few changes we can make to make ORES stable and safe for a while. CAlbon (WMF) (talk) 15:32, 6 April 2022 (UTC)

And I still owe you all a few documents about Lift Wing! And restarting live streams! CAlbon (WMF) (talk) 15:33, 6 April 2022 (UTC)

Weekly Team Update 2022-04-13

Short update this week since half the team is out for one reason or another.

• The proof of concept model card is looking a lot better after incorporating the feedback from the community.
• We are hiring a Senior Machine Learning Engineer! This person would be the most senior MLE on the team and is fully remote.
• We are continuing to copy models from ORES to Lift Wing. We are probably 20% done. This doesn't mean anyone reading this needs to do something yet. ORES will stay up for (ideally) a year after we finish copying the models over. We will announce when that ORES depreciation period will start and what folks needs to do. CAlbon (WMF) (talk) 15:42, 13 April 2022 (UTC)

Hey @CAlbon (WMF) Do you think it's a good idea for me to work on something like ores-support-checklist for the lift wing model migration? Low effort, high impact, don't you think? Chtnnh (talk) 06:09, 21 April 2022 (UTC)

What would that look that? Its an interesting idea. CAlbon (WMF) (talk) 14:17, 27 April 2022 (UTC)

Great to hear about the model migration! It'd be great to get an update on which models are being worked on/"done". Lydia Pintscher (WMDE) (talk) 11:01, 19 April 2022 (UTC)

We are 95% done with the Edit Quality models but I would have to look up which models those are. CAlbon (WMF) (talk) 14:18, 27 April 2022 (UTC)

Weekly Team Update 2022-04-27

Sorry all I'm drowning the tasks so I'm late to an update!

• We are working with the research team on getting feedback on making model cards. The goal is to make is easy for them to create properly formatted model cards. The more barriers we add, the less likely they will use them and update them, so we want to keep the user experience as nice as possible.
• Updating ORES to Buster continues! Next week we are going to try to update a single server to Buster, and if that works we will move on to the other 17 servers. We don't expect any service downtime but just in case we are making sure we are in a position to fix any issue fast.
• We are looking for folks who use ORES's batch processing feature. If you use it, we'd love to chat!
• We are 95% done with migrating ORES's Edit Quality models into Lift Wing, so that is good news! CAlbon (WMF) (talk) 18:40, 27 April 2022 (UTC)

Weekly Update 2022-05-18

This week we are trying something different. Every week we report on the team’s progress to various stakeholders, including other teams and the community. However, we think there is value in investing in more detailed updates. Our theory is straightforward: if our stakeholders are more informed on what our team is planning, working on, struggling with, and celebrating, it is an invitation to all of you to join us with comments, questions, opinions, and advice.

With that, let’s jump right in:

• This week we are adding labels to our k8s nodes. Different nodes in our k8s cluster have different characteristics, such as where they are racked in the data center, what resources they have, their networking performance, BGP access etc. Labels are the canonical way we can target individual tasks to individual groups of nodes. For example, we can do node maintenance on all nodes in a certain row inside the data center or all nodes with GPUs.
• A change in SRE’s policies means that our MLEs are no longer able to deploy machine learning models. Because of the way we have designed Lift Wing, it is required that the deployer of a model has to be in the Wikimedia deployers group, which our MLEs are not. We had a workaround prior but not that workaround has been closed. For the short term we are looking into adding our MLEs to the deployer group. A more long term solution would be to change HELM to allow the MLEs to deploy without the risk.
• After two SREs and many many hours, ORES is now upgraded to Debian Buster. Our previous version of Debian had entered End Of Life and as such the risk of continuing to use it compelled us to make an upgrade. We have been monitoring the system for a few days and have not seen any worrying signs. It looks like the upgrade was successful.
• Eventually we need to upgrade RevScoring on ORES. This is likely the last major piece of maintenance we’d need to do, but it is important because we know the item quality model isn’t working properly right now and the root cause is RevScoring’s versioning, not the model per-se.
• As part of the Add-A-Link project, we are training new models for additional languages. We are in Round Four of creating new models, with 18 models being part of this round. We are currently struggling with a Python dependency bug with the Japanese Add-A-Link model, but we already have the fix identified and will deploy momentary.
• The hiring process is continuing for our Senior MLE role.

Let me know if you like these more detailed updates. CAlbon (WMF) (talk) 16:54, 18 May 2022 (UTC)

I would think that adding people to the deployers group should be a pretty low bar, I don't really see what the objection would be. Legoktm (talk) 05:47, 20 May 2022 (UTC)

Machine Learning Weekly Update June 1, 2022

This week I wanted to talk a little about our thinking around horizontal scaling in Lift Wing, that is, how to handle hosting a very large number of models (as opposed to a vertical scaling - hosting a few models that are highly resource intensive).

How do we handle horizontal scaling right now? Lift Wing is hosted on a Kubernetes cluster, which is based around the smallest unit of compute resource possible, called a pod. Our current setup in Kubeflow maintains one pod per model we host as a minimum. As incoming requests for a model increase or decrease Kubeflow scales by spinning up or spinning down additional pods. The more a model is used, the more pods are created.

Kubeflow allows us to “scale to zero”, meaning if a model is not used within a time period (e.g. 12 hours, 48 hours, etc.)  we don’t maintain any pods for that model at all until it is needed. One of the benefits to this approach is that if a model is used once a year, Kubeflow wouldn’t maintain a pod for that model for an entire year before it was needed again. However, right now we aren’t using scale-to-zero because the cost of spinning up a model pod and loading its dependencies for our models is likely higher than just maintaining one pod per model.

Each pod requires an IP address and we currently have ~2000 IP addresses allotted for all pods, meaning hypothetically (very, very hypothetically) we could host 2,000 models  (i.e. one pod per model). However, in reality the real number is far lower since the amount of requests for many models requires multiple pods.

The problem we can see on the horizon is what happens when projects or features train one model per Wikipedia language. There are ~300 Wikipedia languages so if five features/projects each create 300 models (5*300=1500), we will run out of resources on Lift Wing very quickly. Here is a real world example: The Add-A-Link is a great project currently under development that when deployed across all 300 Wikipedias will require 300 models. Development of those models is well underway.

So what can we do? A lot.

Thanks to the flexibility of Kubernetes and Kubeflow, there are a number of options available for us going forward:

1. Buy more servers. This is the brute force option. Since Lift Wing is hosted on a Kubernetes cluster, we can always add additional resources (i.e. pods) to that cluster by adding more servers. The downside is that buying additional servers is expensive.
2. We could be more efficient with our use of Kubeflow transformers. For example, since transformers (i.e. tools for pre-processing requests before submitting them to the model to calculate a prediction) are very lightweight in Kubeflow, you can imagine an architecture for Add-A-Link where transformers convert language-specific inputs into language-agnostic outputs that are then sent to pods. That would dramatically reduce the number of pods we’d need to host for Add-A-Link and similar projects.
3. We could get better at scaling-to-zero. While not appropriate for all cases (i.e. spinning a model down to zero pods after 18 hours when requests for that model happen every 24 hours), there are cases of dormant models where scaling-to-zero would make a lot of sense.
4. We could work with teams to develop language-agnostic models from the start. That is, have a strong preference for building models that aren't specific to a single language (and thus require one pod).

Inevitably the solution won’t be a single approach list above but a combination of a few approaches. Over the next few months we will explore them and decide how best to proceed. CAlbon (WMF) (talk) 17:52, 1 June 2022 (UTC)

No update this week.

I'm out sick most of this week, so no update this week. CAlbon (WMF) (talk) 14:43, 9 June 2022 (UTC)

No update this week

Probably no update this week. I'm slightly drowning in tasks so I might not get to the update. CAlbon (WMF) (talk) 18:50, 21 June 2022 (UTC)

Machine Learning Weekly Update June 30, 2022

I am still drowning in tasks this week but you all deserve an update!

• The big blocker to Lift Wing being released as an MVP has potentially be removed. One of the things we have really tried to do is not build a lot of backdoors into Lift Wing and instead support a single access point, both for community members and WMF internal projects using the API Gateway. However, the API Gateway has been under development and one of the major blockers to use using it was service-level API limits. That is, users get one rate limit when requesting data from the Mediawiki API and a different rate limit when requesting predictions from the ML API. We still need to do some testing but it seems like that barrier might have been solved. We will let you know as we test.
• We just started an effort we are calling Project Racing Stripe. The goal is to speed up the models we have loaded from ORES and into Lift Wing. It is a limited project and we are being careful about scope creep. It might mean pre-caching scores. It might mean profiling the RevScoring models to see what about them is slow
• The Lift Wing staging cluster is almost ready, which is awesome.
• We have trained around 100 models in the Add-A-Link project in five rounds. CAlbon (WMF) (talk) 16:19, 30 June 2022 (UTC)

Machine Learning Weekly Update June 30, 2022

No updates this week. It is Inspiration Week in the Tech Department and folks are working on reducing technical debt and exploring new areas. CAlbon (WMF) (talk) 15:31, 14 July 2022 (UTC)

Writing About Lift Wing

I am finally getting around to writing about Lift Wing. That is my big push for the next two weeks. The goal is to create a series of pages explaining Lift Wing, Train Wing, and the migration plan at a broad level. I will be writing it in this document before publishing to the wiki. The document is public so absolutely feel free to jump in and add comments (draft prose is above the horizontal line, the notes/outline of what is left to be written is below). CAlbon (WMF) (talk) 15:02, 20 July 2022 (UTC)

Hi Chris,

Thanks for this! I see the topic is created on Wikitech wiki, and that wiki is not connected to the SUL for useraccounts. To ask a question there, one needs to create a (developer) account. Also, the Machine Learning page here has translations enabled, which Wikitech has not as far as I can see. Could you please consider moving some of the information to the Mediawiki wiki? Ciell (talk) 18:22, 9 August 2022 (UTC)

Thanks for this Ciell. Right now the plan is for detailed technical documentation about Lift Wing (i.e. how to restart the cluster, how have a model send events to eventgate etc) used by developers (mostly folks on my team) will be on Wikitech but all other material (i.e. model cards, discussions of models, strategic plans, community discussions, weekly updates, the document I mentioned above, etc.) will be on Mediawiki.

It is the compromise between keeping the technical docs in the same place that the rest of WMF keeps it (Wikitech) while taking advantage of all the benefits of Mediawiki (e.g. adding a model card to your watchlist etc) for everything else.

I'd love to hear your thoughts CAlbon (WMF) (talk) 17:20, 10 August 2022 (UTC)

Absolutely fine with me, but I wanted to ask for the logo (as I did now on this page) and that's where the talkpage of the subject is actually more appropriate than a general talkpage - to keep all discussion and questions on the subject together, you see.

And in my opinion it should be fairly easy to create a simple summarized explanation of Lift Wing on Mediawiki, while hosting the deeper documentation on Wikitech and Github (I hope Github as wel?!). No need to prioritize one over the other, but Ican assume you gave the elevator pitch on the tool several times now. It's time to get that in writing Chris, so after two years also the communities can read about the new plans. :)) Ciell (talk) 17:11, 18 August 2022 (UTC)

Support for various citations

Hi, I reported a problem about ORES on one of the users told me to discuss it here again:

apparently in our wiki (fawiki) the ORES doesn't support the {{sfn}} family templates and just scores the ref tags. Mojtabakd (talk) 20:04, 28 July 2022 (UTC)

Thanks for the message Mojtabakd, I have created a work ticket to investigate the issue here (https://phabricator.wikimedia.org/T314302) and have assigned a team member to investigate. We will keep you updated! CAlbon (WMF) (talk) 14:25, 1 August 2022 (UTC)

Thanx! Mojtabakd (talk) 14:33, 1 August 2022 (UTC)

Logo?

I am looking for a recognizable logo that can represent ML, or maybe even the Lift Wing logo - assuming the ORES logo will not be reused.

Do we have any yet...? Ciell (talk) 18:25, 9 August 2022 (UTC)

Not yet, but I'm talking to the branding team on August 29th about it after Wikimania. Any ideas of what it should look like? CAlbon (WMF) (talk) 14:01, 10 August 2022 (UTC)

Not really, sorry! I am more on the text side, I always ask others for visuals. :) Ciell (talk) 17:05, 18 August 2022 (UTC)

Me too! CAlbon (WMF) (talk) 18:17, 18 August 2022 (UTC)

Also nobody liked my idea of calling the overarching effort of modernizing the infrastructure (Train Wing and Lift Wing) "Wingspan" so apparently they are going to look at a better name for that too. CAlbon (WMF) (talk) 14:19, 10 August 2022 (UTC)

Livestreamed ML Office Hours September 1st

It has been a while. But I am restarting doing live ML office hours. The next ML office hours will be 01/09/2022 at 18:30 UTC on Youtube.

During the office hours I will be working on this description of our work including Lift Wing but am happy to chat about anything.

A recording of the livestream will be posted to YouTube automatically for anyone to watch later. CAlbon (WMF) (talk) 18:33, 18 August 2022 (UTC)

And we are live! https://www.youtube.com/watch?v=u_oEsBsv_Tw CAlbon (WMF) (talk) 18:29, 1 September 2022 (UTC)

Presentation Of Machine Learning At Wikimedia

In a bit I am giving a presentation on machine learning at Wikimedia to a technical conference of machine learning practitioners. Here is the full slide deck of the presentation. I will also try to record my presentation but I can't make any promises because it is being conductged on the conference's platform, which might not have a local record option.

If I can't record and folks are interested, I'll happily give the talk to you all a second time in a livestream. CAlbon (WMF) (talk) 16:30, 21 September 2022 (UTC)

Also my apologies for no update last week. I was in Berlin for an offsite and forget to mention it here. CAlbon (WMF) (talk) 16:30, 21 September 2022 (UTC)

Machine Learning Weekly Update Nov 17, 2022

I have let writing this weekly update slide and my apologies for that. Here is this week's weekly update and I will make an effort keep posting these since folks seem to find them useful.

• NLLB-200 model
  • Context: Currently the Content Translation Tool uses Meta’s NLLB-200 model for translation between smaller languages. The model is already in production. However, the model is currently hosted on Meta’s AWS account and we have been informed that hosting will end Jan 1st. The goal has been to migrate the NLLB-200 model onto WMF’s AWS account before Jan 1st to prevent any loss of service in the Content Translation Tool.
  • We have the model working on WMF’s AWS Sagemaker. We can hit it and get a prediction. It isn’t MVP yet, but it proves we can do it. Now it is about configuring things correctly and connecting the Content Translation Tool to it. We are on track for making the deadline of Jan 1st.
  • I have purposely separated the work on the NLLB-200 model into two parts: 1) resolving the crisis of having the model on Meta’s AWS account when that account is going to end in 45 days and 2) the ongoing maintenance, development, and support of both the AWS instance and NLLB specifically. The benefit is that we were able to move fast and are currently on track to resolve the crisis before the deadline. The cost is that there are important conversations about supporting this model and supporting AWS that we aren’t having, but will eventually have to be worked out.
  • There is also the issue of cost, since we don't use AWS as a regular course of work, we don't have large budget for it. I'm monitoring the cost and we will see how things go.
  • It is worth noting that moving the NLLB-200 model off AWS will spark a larger conversation around the WMF’s policy towards open source. WMF has long used AMD GPUs because their drivers and software are open sources. However, many large models, including NLLB-200 at best assume but at worst require NVIDIA GPUs, which are only partially open source.
• AWS Gateway
  • Context: There has been a plan for over two years for an API Gateway (api.wikimedia.org) where the public and other users can have access to all the APIs provided by WMF. We are working on connecting Lift Wing to that API Gateway as a prerequisite for an MVP launch.
  • This is moving forward, but slowly. Notably, Tobias had to devote 50% of the time he was working on the AWS Gateway with Hugh to work on the NLLB-200 model.
• Add-A-Link
  • Context: As part of the Structured Tasks project in the Product Department, Add-A-Link uses machine learning to recommend easy edits to new editors to make the onboarding process easier and more mobile-friendly.
  • 6th round of model training has started, making it about ~120 models trained out of ~300. The models are live and in production, but not on Lift Wing. The reason was that the project started before Lift Wing wasn't active. However, migrating the models to Lift Wing and decommissioning the current model serving system is in our plans for the future.
• Model Cards
  • Context: As the first step in our efforts to be a best practice public example of applied ethical ML, we are creating a wiki model card for every model hosted on Lift Wing. We have been working on a proof of concept for a few months and are now starting on rolling out model cards into production.
  • We had a kickoff meeting for working on production model cards last week. Currently, the team (Chris, Hal, Issac, and Kevin) have been discussing some of the practicalities of the model card design (i.e. do we even need programmatic content before models are trained on Train Wing?)
  • Following agile, the current step is for Kevin to try to make one model card and we’ll discuss and iterate on the card next week.
• Lift Wing
  • Context: Lift Wing is the Kubernetes cluster for hosting and serving production machine learning models at WMF. It is close to an MVP launch.
  • Work continues on experimenting with using Benthos for streaming Lift Wing model predictions to the Eventgate. Specifically, working with the Observability team on monitoring the application.
• DSE Cluster
  • Context: The DSE Cluster is an experiment with a cross-team shared cluster between Machine Learning and Data Engineering. The goal is to benefit from economies of scale and cross-team experience by building a single cluster that both hosts machine learning and data engineering tasks.
  • Two weeks ago we were at a decision point: does the DSE cluster fork from the greater SRE k8s processes and systems so it can upgrade from 1.16 to 1.23. We’ve made that decision, the DSE Cluster is not going to fork. The cost of the fork is too high for any real benefits. Instead, the DSE Cluster will work with WMF’s other teams with clusters to upgrade to 1.23 together. Luca and Janis are leading this effort.
  • Importantly and more broadly, a k8s special interest group (SIG) has been created with the goal of coordinating and organizing cross-team k8s efforts across the Foundation. This group met yesterday and made a number of decisions on the structure of the group and its scope. CAlbon (WMF) (talk) 17:10, 17 November 2022 (UTC)

Machine Learning Weekly Update Nov 23, 2022

It is Thanksgiving week for me so a smaller ML team weekly

-Benthos work is on hold. We’ve been experimenting with Benthos as a lightweight tool to stream model prediction scores to the larger event stream. It looks like it would work great, but we are putting the work on hold. The Data Engineering team is working on Flink, which would solve all the functionality we were thinking of using Benthos for. Flink isn’t ready yet, however, based on our timelines we can hold off having a streaming solution while the Data Engineering team gets Flink ready and then use that. If that doesn’t work we can always fall back to Benthos.

- We are deploying some brand new models into production as part of an agile development process, specifically, Revert Risk (language-agnostic prediction an edit is reverted) and Outlink Topic (language-agnostic prediction of an article’s topic). I’ll talk about these models more in the future, but for now, I wanted use them to highlight the improvements Lift Wing has created in model deployment.

Previously, deploying models like these on ORES would take a few days for each model. Okay, but lots of room for improvement. With Lift Wing, deploying these models takes less than an hour:

1. Upload new model to Thanos Swift. (~10min)
2. File a patch to deployment charts to update STORAGE_URI, wait for ML SRE +2 and merge (~10min if ML SRE is available)
3. Deploy to staging (ml-staging-codfw) and test the model. (~10min)
4. Deploy to production (ml-serve-eqiad &ml-serve-codfw) and test the model (~20min) CAlbon (WMF) (talk) 16:54, 23 November 2022 (UTC)

Machine Learning Weekly Update Nov 30, 2022

• NLLB-200 deployment
  • Major process continues on getting the NLLB-200 deployment live and running for the Content Translation Tool. I am confident we will make the January 1st deadline.
• API Gateway
  • Tobias and Hugh made some major processes over the last two days and they are working on a patch that will allow the API Gateway to be used with Lift Wing. Specifically, when the patch is tested and rolled out Lift Wing will effectively be silently soft-launched on the API Gateway, making over 100 machine learning models available to everyone. The timeline for the patch being pushed to production is a few days.
  • After the patch is released I will start publishing some tutorials on getting started using Lift Wing and will ask folks both inside WMF and the community to start experimenting to help find bad user experiences and technical bugs.
• Add-A-Link
  • Steady progress on the Add-A-Link models. Kevin continues to train and deploy new models while evaluating their performance.
• Model Cards
  • We are having weekly standup meetings on model cards as we start to make them. We should start with the first of the model cards published in the next two weeks.
• Lift Wing
  • The current focus of the Lift Wing work is on model performance and the k8s 1.23 upgrade.
  • Model Performance: Some of the larger models we are currently working with the Research team on are ~4GB loaded, which is causing prediction times to be over ten seconds. This is obviously too slow for any real-world case and we are exploring which out of a wide variety of strategies for improvement is best, from breaking the large model into smaller ones, optimizing the structure of the models, increasing the number of pods, etc.
  • K8s 1.23 update: Luca and Yannis are working through a list of tasks as part of the 1.23 update. We are making solid progress but it is also a major task.
• DSE Cluster
  • Work has now started on tackling what we have called the “Kerbarrier”, which is the fact that Kubernetes and Hadoop use very different security models. Kubernetes uses a certificate-based approach while Hadoop uses a symmetric key cryptographic approach (called Kerberos). Building a way of bridging the gap between these two approaches so nodes on clusters can access HDFS has been a major challenge we have to know we would need to solve eventually and one of the reasons for starting the DSE Cluster experiment. CAlbon (WMF) (talk) 15:54, 30 November 2022 (UTC)

Machine Learning Weekly Update Dec 7, 2022

• We have hit another milestone for Lift Wing. Our ORES model infrastructure hosts ~110 machine learning models and for the first time all of those models are also publicly available on Lift Wing through the API Gateway. We still need to work out the details of reasonable rate limiting and optimizing performance (maybe use the API Gateway as a simple cache?), but you can access all the models right now without any internal WMF permissions. We will have some tutorials up soon for the community and an ask for testers to help us out.
• To be clear: if folks are currently using ORES, nothing will change with ORES for at least an entire year. We are working on a complete year-long migration plan for ORES users to Lift Wing that includes outreach, tutorials, and technical support, and that plan will only begin once Lift Wing is officially launched in a few months.
• After talking to AI ethics experts, wiki community members, WMF staff members, and frankly anyone else who would talk to us, we are working on generating model cards for all models on Lift Wing. The goal is for the model cards to be the main point of contact for questions, discussions, and ultimate governance around machine learning models hosted by WMF. The model cards will be individual articles on Mediawiki.org to make it easy for the community to use the tools they are familiar with. We should have some things to show everyone very soon. CAlbon (WMF) (talk) 16:32, 7 December 2022 (UTC)

No Update This Week

My apologies but no update this week because I have been out sick. CAlbon (WMF) (talk) 18:58, 14 December 2022 (UTC)

Hope you get fully well soon! Michael Große (WMDE) (talk) 09:29, 15 December 2022 (UTC)