User contributions for GLederrey (WMF)

Search for contributionsExpandCollapse

⧼contribs-top⧽

IP address or username:

Namespace:

Invert selection

Associated namespace

Tag filter:

Invert selection

Only show edits that are latest revisions

Only show edits that are page creations

Hide minor edits

⧼contribs-date⧽

From date:

To date:

08:0808:08, 21 August 2026 diff hist +4,984 N Data Platform Engineering/Data Platform SRE/Status Update/2026-08-21 Created page with "= Highlights = * Lots of small operational work this week. See below for details if you're interested. * Completed the [https://grafana.wikimedia.org/d/team-estate-overview/team-estate-overview?orgId=1&from=now-24h&to=now&timezone=utc&var-datasource=000000026&var-team=data-platform&var-team=machine-learning&var-cluster_limit=25&refresh=5m OS upgrade before Bullseye deprecation]. The Hadoop cluster was the last one to migrate. * Airflow upgrade: [https://docs.google.com/d..." current Tag: 2017 source edit

08:1608:16, 14 August 2026 diff hist +207 Data Platform Engineering/Data Platform SRE/Status Update/2026-08-14 →Highlights current Tag: 2017 source edit
08:1408:14, 14 August 2026 diff hist +966 Data Platform Engineering/Data Platform SRE/Status Update/2026-08-14 →Highlights Tag: 2017 source edit
06:5506:55, 14 August 2026 diff hist +1 m Data Platform Engineering/Data Platform SRE/Status Update/2026-08-14 →Highlights Tag: 2017 source edit
06:5506:55, 14 August 2026 diff hist +8 Data Platform Engineering/Data Platform SRE/Status Update/2026-08-14 →Wikidata Platform: 2 Tag: 2017 source edit
06:5006:50, 14 August 2026 diff hist +133 Data Platform Engineering/Data Platform SRE/Status Update/2026-08-14 No edit summary Tag: 2017 source edit
06:2406:24, 14 August 2026 diff hist +97 Data Platform Engineering/Data Platform SRE/Status Update/2026-08-14 No edit summary Tag: 2017 source edit
05:4405:44, 14 August 2026 diff hist +1,952 N Data Platform Engineering/Data Platform SRE/Status Update/2026-08-14 Created page with "= Highlights = = Misc / Operations = * [https://phabricator.wikimedia.org/T378714 T378714 Possibility to transition ml-serve[2001-2008] and and ml-staging[2001-2002] to 10G] (Resolved) * [https://phabricator.wikimedia.org/T430598 T430598 Create Kerberos identity for Randall Scout] (Invalid) * [https://phabricator.wikimedia.org/T414969 T414969 Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100)] (Resolved) * [https://phabricator.wikimedia.org/T433030 T..." Tag: 2017 source edit

08:3908:39, 7 August 2026 diff hist +377 Data Platform Engineering/Data Platform SRE/Status Update/2026-08-07 →Highlights current Tag: 2017 source edit
08:3208:32, 7 August 2026 diff hist +7,503 N Data Platform Engineering/Data Platform SRE/Status Update/2026-08-07 Created page with "= Highlights = = OpenSearch Upgrade = * [https://phabricator.wikimedia.org/T426101 T426101 Create detailed migration plan for OpenSearch 1.x-2x] (Resolved) * [https://phabricator.wikimedia.org/T433726 T433726 Cirrussearch: add ability to target hosts with cumin based on master eligibility] (Resolved) * [https://phabricator.wikimedia.org/T433624 T433624 Upload opensearch 3.7.0 to apt.wikimedia.org (trixie/thirdparty)] (Resolved) * [https://phabricator.wikimedia.org/T4268..." Tag: 2017 source edit

08:0508:05, 10 July 2026 diff hist +1,169 Data Platform Engineering/Data Platform SRE/Status Update/2026-07-10 No edit summary current Tag: 2017 source edit
07:5807:58, 10 July 2026 diff hist +3,574 N Data Platform Engineering/Data Platform SRE/Status Update/2026-07-10 Created page with " = Wikidata Platform support = * [https://phabricator.wikimedia.org/T424338 T424338 Provide a helm chart and dse-k8s helmfiles for wdqs-proxy.] (Resolved) * [https://phabricator.wikimedia.org/T426764 T426764 Airflow secret for S3 credentials for wikidata-platform] (Resolved) * [https://phabricator.wikimedia.org/T424340 T424340 QLever db node setup for WDQS v2] (Resolved) * [https://phabricator.wikimedia.org/T428925 T428925 DNS entries for the WDQS deployments] (Resolved)..." Tag: 2017 source edit

08:1208:12, 3 July 2026 diff hist +590 Data Platform Engineering/Data Platform SRE/Status Update/2026-07-03 No edit summary current Tag: 2017 source edit
07:3707:37, 3 July 2026 diff hist +2,041 N Data Platform Engineering/Data Platform SRE/Status Update/2026-07-03 Created page with "= Highlights = = OpenSearch upgrade = * [https://phabricator.wikimedia.org/T425585 T425585 Write lightweight OCI-image-based Puppet plans for beta cluster] (Resolved) * [https://phabricator.wikimedia.org/T430882 T430882 Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure] (Resolved) * [https://phabricator.wikimedia.org/T430726 T430726 cirrussearch2089.codfw.wmnet fails reimage cookbook (DRAC error)?] (Resolved) = Wikidata Platform sup..." Tag: 2017 source edit

08:2408:24, 26 June 2026 diff hist +252 Data Platform Engineering/Data Platform SRE/Status Update/2026-06-26 No edit summary Tag: 2017 source edit
08:2008:20, 26 June 2026 diff hist +10 Data Platform Engineering/Data Platform SRE/Status Update/2026-06-26 No edit summary Tag: 2017 source edit
08:1808:18, 26 June 2026 diff hist +2,731 N Data Platform Engineering/Data Platform SRE/Status Update/2026-06-26 Created page with "# Highlights * '''Kafka:''' cleanup of unused topics, rebalancing of the cluster to flatten storage and traffic between brokers * '''WDQS Update pipeline:''' cleanup updater checkpoints * '''Dumps:''' our xml/sql dumps were failing due to an increased memory requirement of 7za in bookworm compared to bullseye. We rolled back the dumps image to one based on bullseye and re-ran all failed jobs * '''Airflow:''' Fixed all failing jobs in https://airflow-test-k8s.wikimedia.or..." Tag: 2017 source edit

08:3308:33, 19 June 2026 diff hist +2,907 N Data Platform Engineering/Data Platform SRE/Status Update/2026-06-19 Created page with "= Highlights = * '''Kafka:''' Reacted after an incident where a single topic / partition filled 3 brokers. All large topics now have more partitions, to ensure each is under 30GB. {{PhabT|T429127}} * '''Fundraising Tech:''' Enabled airflow-fr-tech to egress to the fr-tech s3 endpoint {{PhabT|T428294}} & {{PhabT|T429048}} * '''Growthbook / Test Kitchen:''' Provisioned API keys in secrets allowing integration between TK and Growtbook, as well as TK and phabricator {{PhabT|..." current Tag: 2017 source edit

08:5808:58, 12 June 2026 diff hist +226 Data Platform Engineering/Data Platform SRE/Status Update/2026-06-12 →Highlights current Tag: 2017 source edit
08:2608:26, 12 June 2026 diff hist +281 Data Platform Engineering/Data Platform SRE/Team Norms Communication norms current Tag: 2017 source edit
08:2108:21, 12 June 2026 diff hist +251 Data Platform Engineering/Data Platform SRE/Team Norms No edit summary Tag: 2017 source edit
08:2008:20, 12 June 2026 diff hist −4 Data Platform Engineering/Data Platform SRE/Team Norms →Code, LLM, technical interactions and norms Tag: 2017 source edit
08:1808:18, 12 June 2026 diff hist +2 m Data Platform Engineering/Data Platform SRE/Status Update/2026-06-12 →Hardware Tag: 2017 source edit
08:1708:17, 12 June 2026 diff hist +480 Data Platform Engineering/Data Platform SRE/Status Update/2026-06-12 No edit summary Tag: 2017 source edit
07:2107:21, 12 June 2026 diff hist +2,423 N Data Platform Engineering/Data Platform SRE/Status Update/2026-06-12 Created page with "= Highlights = = OpenSearch upgrade to 2.x = * [https://phabricator.wikimedia.org/T427306 T427306 Migrate Relforge clusters from OpenSearch 1.x->2x] (Resolved) = Support Wikidata Platform team = * [https://phabricator.wikimedia.org/T428241 T428241 Clean up of WDQS full graph endpoint (wdqs2009) Blazegraph journal file] (Resolved) * [https://phabricator.wikimedia.org/T427319 T427319 request: s3 access for wdqs::alternatives] (Resolved) = SPARQL Federation requests = *..." Tag: 2017 source edit

09:0009:00, 5 June 2026 diff hist +565 Data Platform Engineering/Data Platform SRE/Status Update/2026-06-05 No edit summary Tag: 2017 source edit
08:3808:38, 5 June 2026 diff hist +1,758 N Data Platform Engineering/Data Platform SRE/Status Update/2026-06-05 Created page with "= Highlights = = EventStreams internal = * [https://phabricator.wikimedia.org/T348763 T348763 Make eventstreams-internal available to WMF staff without an ssh tunnel] (Resolved) = Kafka = * [https://phabricator.wikimedia.org/T428053 T428053 Deploy a kafka web UI for all internal clusters] (Resolved) = OpenSearch upgrade = * [https://phabricator.wikimedia.org/T427304 T427304 Import wmf-opensearch-search-plugins_2.19.5+6.trixie.deb from gitlab build to apt.wikimedia.org..." Tag: 2017 source edit

07:1407:14, 29 May 2026 diff hist +2,023 N Data Platform Engineering/Data Platform SRE/Status Update/2026-05-29 Created page with "= Highlights = * the LDAP/Growthbook sync project is completed * the Experiment Platform team can now independently upgrade to 4.4.0 * kafka ACLs are starting to be tracked in Puppet * We're now tracking Ceph object count and space usage per bucket and user in Prometheus * [http://stream-internal.wikimedia.org stream-internal.wikimedia.org] is available for WMF employees and NDA-signers to explore the <code>kafka-jumbo</code> schema and messages. The service is not inten..." current Tag: 2017 source edit

10:0910:09, 22 May 2026 diff hist +788 Data Platform Engineering/Data Platform SRE/Status Update/2026-05-22 →Highlights current Tag: 2017 source edit
10:0010:00, 22 May 2026 diff hist +121 Data Platform Engineering/Data Platform SRE/Status Update/2026-05-22 No edit summary Tag: 2017 source edit
09:5009:50, 22 May 2026 diff hist +3,900 N Data Platform Engineering/Data Platform SRE/Status Update/2026-05-22 Created page with "* [https://phabricator.wikimedia.org/T425758 T425758 Investigate rdf-streaming-updater consumer failures in eqiad] (Resolved) * [https://phabricator.wikimedia.org/T425973 T425973 Requesting Ceph S3 credentials for Wikidata Plaftorm] (Resolved) * [https://phabricator.wikimedia.org/T425770 T425770 WDQS token bucket throttling logic should not apply to the the streaming-updater-consumer] (Resolved) * [https://phabricator.wikimedia.org/T426348 T426348 Grant mcollins level 1..." Tag: 2017 source edit

08:5908:59, 8 May 2026 diff hist +2 m Data Platform Engineering/Data Platform SRE/Status Update/2026-05-08 No edit summary current Tag: 2017 source edit
08:5808:58, 8 May 2026 diff hist +1,911 N Data Platform Engineering/Data Platform SRE/Status Update/2026-05-08 Created page with "= Highlights = * Ongoing work to migrate non-search use cases away from the CirrusSearch cluster. This will make upgrades and operational work easier, and give better visibility and ownership. {{PhabT|T424248}} * WDQS is under a lot of load from multiple bots. We're making low efforts to ban the most offensive bots, but the real focus is on replacing Blazegraph with a more efficient backend, which should reduce this issue (but not solve it entirely). * We had an incident..." Tag: 2017 source edit

07:3907:39, 1 May 2026 diff hist +3,382 N Data Platform Engineering/Data Platform SRE/Status Update/2026-05-01 Created page with "= Highlights = * Our team had a retrospective talking about our use of LLM. We've added a section to our team norms to reflect the outcomes of the discussion. * Kafka Jumbo has been upgraded to 3.7. This caused an incident in the WebRequest pipeline, but no data loss. If this is the price for jumping through 8 years of upgrades, it still seems like a good deal! We can n..." Tag: 2017 source edit

19:0519:05, 30 April 2026 diff hist +607 Data Platform Engineering/Data Platform SRE/Team Norms →Data Platform SRE specific team norms Tag: 2017 source edit

09:0909:09, 24 April 2026 diff hist +172 Data Platform Engineering/Data Platform SRE/Status Update/2026-04-24 No edit summary current Tag: 2017 source edit
08:2308:23, 24 April 2026 diff hist +3,672 N Data Platform Engineering/Data Platform SRE/Status Update/2026-04-24 Created page with "= Highlights = = OpenSearch upgrade / OpenSearch misc= * [https://phabricator.wikimedia.org/T423327 T423327 Explore options for OpenSearch 2.x/3.x plugin packaging and distribution] (Resolved) * [https://phabricator.wikimedia.org/T423523 T423523 Handle typos/possibly update opensearch-analysis-stconvert plugin] (Resolved) * [https://phabricator.wikimedia.org/T423291 T423291 Build new wmf-opensearch-search-plugins package for opensearch 2.x/trixie and ensure we don't ins..." Tag: 2017 source edit

18:4818:48, 2 April 2026 diff hist +2,974 N Data Platform Engineering/Data Platform SRE/Status Update/2026-04-02 Created page with "= Highlights = * '''We have a new team member who started on Monday!''' * '''OpenSearch on Kubernetes / IPoid:''' We deployed a new metric that reflect the amount of time remains before the internal opensearch TLS certificate expires * '''Kubernetes:''' We upgraded dse-k8s-eqiad to 1.31! * '''Turnilo:''' We deployed https://turnilo.wikimedia.org/ to dse-k8s-eqiad after the upgrade, delivering a turnilo upgrade as a patch solving a stringent UX papercut. This aligns with..." Tag: 2017 source edit

08:5208:52, 27 March 2026 diff hist +2,266 N Data Platform Engineering/Data Platform SRE/Status Update/2026-03-27 Created page with "= Highlights = * We conducted an upgrade of dse-k8s-eqiad from Kubernetes 1.23 to 1.31, without wiping the etcd data, to keep the stateful volumes (a first in WMF history). Besides the usual benefit of keeping our infrastructure up to date, this will enable further work on the Hadoop migration to k8s. * The Kafka upgrade WG performed a migration of all mirrormaker instances from the kafka brokers to the aux Kubernetes clusters. That gives us greater flexibility and relia..." current Tag: 2017 source edit

10:2610:26, 20 March 2026 diff hist +346 Data Platform Engineering/Data Platform SRE/Status Update/2026-03-20 No edit summary current Tag: 2017 source edit
09:3809:38, 20 March 2026 diff hist +2,286 N Data Platform Engineering/Data Platform SRE/Status Update/2026-03-20 Created page with " = Highlights = = Hardware = * [https://phabricator.wikimedia.org/T411731 T411731 Q2:rack/setup/install wdqs1033-1035] (Resolved) = Misc / Operations = * [https://phabricator.wikimedia.org/T417492 T417492 Drop support for cl_to, cl_collation and il_to from wikireplicas] (Resolved) * [https://phabricator.wikimedia.org/T414787 T414787 Upgrade any 1 Gbps dse-k8s-worker-eqiad network interfaces to 10 Gbps] (Resolved) * [https://phabricator.wikimedia.org/T418582 T418582 A..." Tag: 2017 source edit
09:3009:30, 20 March 2026 diff hist −1 m Data Platform Engineering/Data Platform SRE/Status Update/2026-03-13 No edit summary current Tag: 2017 source edit

08:5008:50, 13 March 2026 diff hist 0 m Data Platform Engineering/Data Platform SRE/Status Update/2026-03-13 No edit summary Tag: 2017 source edit
08:4908:49, 13 March 2026 diff hist +4,024 N Data Platform Engineering/Data Platform SRE/Status Update/2026-03-13 Created page with "= Highlights = * '''Kafka Upgrade:''' We're ready to migrate all mirrormaker instances to Kubernetes, which will simplify the kafka upgrade, as only kafka itself will run on the brokers. We have written cookbooks to manage the rolling upgrade/rollback of a given cluster. We are connecting producers/consumers to kafka-test to observe their behavior premid/post upgade * '''Druid upgrade/expansion:''' Upgrade of druid-public is scheduled for Monday 16th at 10:30 {{PhabT|T..." Tag: 2017 source edit

09:3109:31, 6 March 2026 diff hist +220 Data Platform Engineering/Data Platform SRE/Status Update/2026-03-06 No edit summary Tag: 2017 source edit
09:1909:19, 6 March 2026 diff hist +2,505 N Data Platform Engineering/Data Platform SRE/Status Update/2026-03-06 Created page with "= Highlights = = OpenSearch on k8s = * [https://phabricator.wikimedia.org/T418776 T418776 Consider options for automating the setting of read_ahead_kb for OpenSearch pod volumes] (Resolved) * [https://phabricator.wikimedia.org/T414697 T414697 Build the required plugins for opensearch 3] (Resolved) * [https://phabricator.wikimedia.org/T418405 T418405 Plan hardware expansion for Semantic Search] (Resolved) * [https://phabricator.wikimedia.org/T418176 T418176 Provision op..." Tag: 2017 source edit

14:2914:29, 27 February 2026 diff hist +4,169 N Data Platform Engineering/Data Platform SRE/Status Update/2026-02-27 Created page with "= Highlights = * '''Hiring:''' We extended an offer to our best candidate and she signed it. There is still some admin work and actual contract to sign. Unless something goes terribly wrong, she'll start on March 30th. * '''WDQS instabilities:''' We've been fighting multiple traffic surges on WDQS, leading to degraded service. We're still within our SLO, but firefighting takes time away from other work. By its nature (exposing a read only database to the internet and let..." current Tag: 2017 source edit
14:0914:09, 27 February 2026 diff hist −1 m Data Platform Engineering/Data Platform SRE/Status Update/2026-02-20 No edit summary current Tag: 2017 source edit
14:0614:06, 27 February 2026 diff hist +420 Data Platform Engineering/Data Platform SRE/Team Norms →Data Platform SRE specific team norms Tag: 2017 source edit

09:4209:42, 20 February 2026 diff hist +2,406 N Data Platform Engineering/Data Platform SRE/Status Update/2026-02-20 Created page with "= Highlights = * '''Turnilo:''' ongoing work to deploy new version on k8s, hopefully reducing the number of requests to the backend (thus improving performances) and simplifying operations {{PhabT|T416113}}. * '''OpenSearch on Kubernetes / IPoid:''' Investigation on a peak of timeouts from this morning, ultimately due to failing healthchecks leading to a new master election, causing temporary failures. * '''OpenSearch on Kubernetes:''' Multiple performance and operationa..." Tag: 2017 source edit

21 August 2026

14 August 2026

7 August 2026

10 July 2026

3 July 2026

26 June 2026

19 June 2026

12 June 2026

5 June 2026

29 May 2026

22 May 2026

8 May 2026

1 May 2026

30 April 2026

24 April 2026

2 April 2026

27 March 2026

20 March 2026

13 March 2026

6 March 2026

27 February 2026

20 February 2026