Page MenuHomePhabricator

MoveComms support for Southward Datacentre Switchover (October. 2026)
Closed, ResolvedPublic

Description

Dear MoveComms,

We are planning a datacentre switchover for the week of October 26st (week 44) with the following schedule:

The expected impact is 2-3 minutes of read-only on Wednesday, 28 October 2026 @ 15:00 UTC.

We continue to follow the process described in Recurring, Equinox-based, Data Center Switchovers, in particular:

  • There is no switchback! We are staying in codfw until the next switchover.
  • Future switchovers are predictable and take place every 6 months; always on the week of an equinox.

Let ServiceOps know if you need more info on the changes.

Thank you!

Planning

As soon as the task is received by Movement Communications

  • Ask SRE if anything major changed since the last time (noticeable things that are worth being announced).
    • If yes, update the process or the message.
  • Confirm when the wikis will be in read-only - 28 October 2026 @ 15:00 UT
  • Add the date to Asana, with a link to this task

Three weeks before

  • Add to Tech News the initial information in this week's issue (before Friday evening), reuse this message
  • Update the message to communities
    • Check on dates and time (dates are formatted <code>YYYY-MM-DDTHH-SS</code>, like 2026-09-23T14:00; don't touch what's around)
    • Once dates and times are edited, check (in preview mode) if the Zonestamp links display the right time and date for your timezone.
    • Check on links, have they changed?
    • Edit qqq messages with the new Zonestamp link (reuse the one from the page)
  • Have the message being translated by emailing both:
    • translators-l for translations
    • wikitech-ambassadors for information and translations
  • Monitor the message's talk page

Two weeks before

The week before

The week it happens

  • Confirm with SRE how the first steps happen on the day before.
  • monitor the wikis to check if anyone has questions
  • if relevant, when the read-only is done and confirmed by SRE's spokeperson, remove the banner.

The week after, maximum

  • Debrief how it went, and document (comment on this task)
  • If not done, archive the campaign

Event Timeline

Confirmation of the date to come next week.

Assuming this will happen at the date listed, I started the announcement process.

Trizek-WMF changed the task status from Open to In Progress.Wed, Oct 14, 12:45 PM
Trizek-WMF updated the task description. (Show Details)

@SLyngshede-WMF is the date confirmed?

Simon was OOO this week.

We're doing technical tests on Mon and Tue ( T440896 ) and will be able to confirm the date by Tue evening, so it's preferable to wait until then to announce final dates.

Trizek-WMF changed the task status from In Progress to Stalled.Tue, Oct 20, 1:43 PM

I got confirmation of the date, processing.

Trizek-WMF changed the task status from Stalled to In Progress.Thu, Oct 22, 1:31 PM
Trizek-WMF triaged this task as High priority.

So, I packed things up as we only have one week to announce it. Everything should be covered today.

@SLyngshede-WMF will you perform the switch? If so, on the day it happens, where would you inform me of the outcome? (Slack preferred TBH.)

Don't forget your item about informing all staff!

As I asked translators to review the message I'll send to the VPs, I'll handle it on Monday (or over the week end).

eqiad has been depooled successfully.

We'll continue tomorrow.

Debrief from MovComms:

  • As the process is now known, maybe we could make it lighter, with 2 weeks of preparation instead of 3.
  • Having the read-only time always at the same hour (14:00 or 15:00) would simplify our process.
  • The warning banner is displayed from 30 minutes before the hour to 2 minutes after the hour. The read-only isn't always starting at the top of the hour: should we add more minutes after the hour? It would mean a manual deactivation of the banner when the read-only is done.
Trizek-WMF updated the task description. (Show Details)