infra-specs/specs/mailman3.rst

Copyright 2021 Jeremy Stanley

This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode

Mailman 3 Migration

https://storyboard.openstack.org/#!/story/2009249

This specification covers the implementation of a new Mailman 3 based listserv in OpenDev, and subsequent migration of the existing Mailman 2 based mailing lists we provide to the new server, concluding with taking down the old server once all lists have been migrated. Care will be taken to preserve existing list configurations as much as the existing migration tooling allows, as well as import all historical list archives and ensure old archive hyperlinks still take people to the same content.

Problem Description

For years now, the Mailman community has been producing two series of mailing list software:

• Mailman 2, which we currently use and have since circa 2010, offers a stable and battle-hardened mailing list service focused on exchanging information via E-mail, with an optional public Web archive called Pipermail.
• Mailman 3, which was experimental for a very long time but has become more stable and essentially reached feature parity with its predecessor some years ago now, is functionally the same internally as far as exchanging E-mail messages is concerned, but has expanded substantially on its Web interface with a new Postorius list management interface and Hyperkitty interactive Web archive.

In the decade since we started running Mailman, significant shifts have occurred in the ways people interface with E-mail and hold group discussions. The mass freemail providers have become entrenched, ubiquitous, and emboldened to begin breaking long-established protocol standards. Portable handheld devices are becoming the primary way many people interact. Web forums have morphed into social media platforms, and their popularity has further altered expectations of the next generation of computer users. At the same time, these shifts in behavior have driven changes in how Web applications are developed and delivered.

Mailman 3's Hyperkitty archive interface is essentially a bolt-on Web forum frontend, with more advanced authentication and some (mis?)features borrowed from popular social media. Its Postorius list management interface uses the same new authentication subsystems, having distinct per-user accounts and role-based access controls instead of the shared password model used by MM2. Both of these are now Django based applications, so can take advantage of the Django ecosystem of additional features for further customization such as social authentication and integration with external identity providers.

Among relevant improvements in Mailman 3 is proper virtual domain support, which should allow us to greatly simplify our current implementation by hosting all our various list sites in a single deployment without fragile workarounds, and more easily add new sites in the future.

Proposed Change

Since there are semi-official Mailman 3 containers already available on GHCR/Quay/DockerHub, this seems like the easiest path forward and most compatible with our present model of configuration management and orchestration. They publish three applications as individual containers: mailman-core (the listserv), mailman-web (a.k.a. Hyperkitty), and postorius. In addition, it's assumed that the system(s) where these run will provide a Web server to act as a proxying front-end, an SMTP server, and an RDBMS.

We'll write Ansible playbooks and Zuul jobs to deploy the three Mailman service containers along with an accompanying MariaDB container, and then configure system packages of Apache and Exim to support these (similar to how we do with our existing services, e.g. Gerrit). The Mailman container documentation provides examples for configuring Exim and Nginx but the latter can be easily adapted to Apache for increased consistency with our other Web-based services, and the Mailman documentation also has some tips for this.

Following our usual test-driven development methodology, once basic functionality is working and tested successfully, we'll hold one or more test nodes and perform direct interactive testing, first with sample mailing lists, but later with imported production configuration and archives (taking sufficient measures to avoid annoying people with unwanted tests or duplicate messages). The Mailman developers provide a convenient migration tool which we can use for this purpose. We should also be sure to note any significant changes in behavior for things like bounce processing or which mailing-list-specific headers get included in messages, so that we can provide users with better guidance around the migration.

Once we're confident the deployment and config/archive imports are working as expected, we'll boot a new production mailing list server, then schedule and announce a migration window when we'll be moving all the existing sites and their lists to the new server. A full copy of the old pre-migration Pipermail archives will also be served from the new server in order to preserve the functionality of any existing bookmarks or historical hyperlinks. The lists.opendev.org virtual domain will migrate first in order to test the waters, and then the remaining virtual domains will be migrated individually as is convenient for their communities.

After all migrations are completed and any remaining issues sorted, the old mailing list servers will be imaged for posterity and then deleted.

Alternatives

We could consider using distro packaged Mailman 3 like we presently do for Mailman 2, the packages for it in Ubuntu 20.04 LTS are quite usable at this point, though slightly outdated compared to what we get from the Mailman 3 container images. The down-side here is that we only consume new features when upgrading the operating system itself to a new major release, and this tight coupling tends to complicate things as MM3 is still a bit of a moving target and two years of deferred upgrades could mean many significant changes in behavior to juggle and untangle at the same time. On the other hand, relying on distribution packages in theory shields us from unexpected changes outside operating system upgrades, and (if the packages are well maintained) means we get to skip bugs impacting upstream releases which may be ironed out in packaging before they ever trickle down to us.

We could also of course consider switching to a different mailing list software entirely, or sticking with Mailman 2 (it's still maintained), or migrate everyone to a SaaS list provider, or decide to stop running mailing lists entirely. Any of these would require a much different spec however.

Implementation

Assignee(s)

• fungi
• clarkb

Gerrit Topic

Use Gerrit topic "mailman3" for all patches related to this spec.

git-review -t mailman3

Work Items

1. Create deployment playbooks and related testing.
2. Test config migration and archive import on a held test node.
3. Create new production list server.
4. Schedule and announce virtual domain migrations.
5. Migrate all Mailman virtual domains to the new server.
6. Stop the old servers, image them for posterity, and then delete.

Repositories

No new git repositories need to be created.

Servers

A new lists01.opendev.org server will be created using our default image (currently Ubuntu 20.04 LTS), and the old lists.katacontainers.io and lists.openstack.org server will be removed.

DNS Entries

The lists.airshipit.org, lists.katacontainers.io, lists.opendev.org, lists.openstack.org, lists.starlingx.io, and lists.zuul-ci.org DNS records will need updating to the addresses of the new server during migration, as will the DNS records for any other virtual domains added to the old servers prior to migration. New lists01.opendev.org records will be added, as will ACME records to support HTTPS for all the virtual domains.

Backups

Borg backups will be configured for the new lists01.opendev.org server prior to migration of any virtual domains, and backups of the old server will be retained after migration for as long as is reasonable.

Documentation

The system-config mailing lists document will be updated to reflect the details of the new deployment. Additionally, user-facing documentation in the Infrastructure Manual or maintained by individual project communities may need updating if it mentions details of list subscription or includes screenshots.

Security

This is overall a net improvement in security of the service, since Mailman 3 uses stronger (effectively non-brute-forcible) password confirmation tokens, a proper password reset option rather than simple reminders, stores password hashes internally instead of plaintext copies, provides role-based access control rather than relying on shared passwords, can integrate with other authentication and identity providers if we want, and will be deployed with HTTPS in order to reinforce the stronger security model.

The only significant security risk is that we operate a handful of private mailing lists, and so extra care should be taken during migration to avoid exposing or otherwise leaking the archives for these. We should also make sure to double-check lists with moderated subscription settings to be sure unauthorized users can't subscribe after the list has been moved to the new system.

Testing

Basic functional testing of deployment and Web interfaces (including some screen capture artifacts) will be added as part of creating the configuration management and orchestration for the service. Additional on-demand config and archive migration testing will be performed with held job nodes. In future, end-to-end testing of message delivery and archiving can easily be added to help prevent regressions when evaluating new versions of the container images.

Dependencies

There are no dependencies outside of the spec itself.