January 2025 - Mailman-users - lists.mailman3.org

Upgrade docker-mailman to v0.5.2
by Christopher April 21, 2025

April 21, 2025

Hi Abhilash, I checkout the v0.5.2 for testing my upgrade from 0.4.6 and noticed, that the docker-image-version is still set to 0.4 in docker-compose.yaml. Is there any reason for it or should it be set to 0.5? I´m still using my good old mariadb. I did not find any upgrade-informations about breaking-changes. Do you have any experience with migrating from 0.4.6 to 0.5.2 on mariadb? Best regards Chrclaus

1 1

makemigrations --merge error after Debian upgrade
by Chupin Maxime April 16, 2025

April 16, 2025

Hi, I upgraded my debian version, and since, it seems that I have a problem. I tried `mailman-web migrate` (and `mailman-web collectstatic` etc.) and I get the error message: ``` CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph: (0010_auto_20220618_2315, 0014_schedule_cluster in django_q; 0014_auto_20220618_2315, 0018_alter_emailtemplate_language in postorius). To fix them run 'python manage.py makemigrations --merge' ``` So I tried `mailman-web makemigrations --merge` that ended with ``` PermissionError: [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/postorius/migrations/0019_merge_20240930_1159.py' ``` There is no such a file in the migration directory. Any idea ? Thanks in advance, All the best Maxime Chupin

7 28

Find a smtp server to send out emails
by Leon March 12, 2025

March 12, 2025

Hi, This problem is not caused by mailman, but I still want to give it a shot here. I'm hosting a mailing list on top of mailman3. Emails are supposed to be sent out by AWS ses. However, ses requires that sender address must be verified, which leads to a problem that emails sent by subscribers to mailing list cannot be sent to other subcirbers, since their addresses are not verified. And it is impossible to verified every subscriber. Are there smtp service providers allowing unverified email address to send out emails, or do I have other solutions? Thanks. -- Best wishes, Leon

4 3

Moderation of held emails vs. spam
by David Krantz March 11, 2025

March 11, 2025

Hi, I now run Mailman 3.3.1, installed from pip. Is there now any way to prevent Mailman from forwarding the contents of the held message when notifying moderators of held messages? In my case these messages often are spam messages and if the entire contents are forwarded it triggers spam handling in the recipient address(es) which over time risks classifying all moderation emails as spam or even classifying the list domain or server as a spam source. Even my own outgoing filter classifies it as spam, which I will fix. I currently run rspamd for spam filtering, graylisting and DKIM and ARC verification/signing. Someone on this recommended it, thank you for that as it removes the need for other milters. I guess that this would be a simple change to make as a kludge but a proper implementation as a setting would be a lot better. I'd like it as a global setting but that might be too coarse. Cheers // David

3 4

Issues with a DMARC record leading to message being shunted
by Ralf Hildebrandt March 4, 2025

March 4, 2025

Hi! Recently a mail was not being distributed via our MM3 instance; I searched the logs and found: Dec 22 04:54:27 2023 (24) ACCEPT: <1d431542-b386-467b-8bf2-305f19ee7eb4(a)somenet.org> Dec 22 04:54:28 2023 (28) Uncaught runner exception: list index out of range Dec 22 04:54:28 2023 (28) Traceback (most recent call last): File "/usr/lib/python3.10/site-packages/mailman/core/runner.py", line 179, in _one_iteration self._process_one_file(msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/core/runner.py", line 272, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/usr/lib/python3.10/site-packages/mailman/core/pipelines.py", line 53, in process handler.process(mlist, msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/handlers/validate_authenticity.py", line 125, in process authenticate(msg, msgdata) File "/usr/lib/python3.10/site-packages/mailman/utilities/retry.py", line 44, in f_retry return f(*args, **kwargs) File "/usr/lib/python3.10/site-packages/mailman/handlers/validate_authenticity.py", line 93, in authenticate auth_result = authenticate_message( File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 395, in authenticate_message dmarc_result = check_dmarc(msg, spf_result, dkim_result, dnsfunc=dnsfunc, psddmarc=psddmarc) File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 343, in check_dmarc result, result_comment, from_domain, policy = dmarc_per_from(from_domain, spf_result, dkim_result, dnsfunc, psddmarc) File "/usr/lib/python3.10/site-packages/authheaders/__init__.py", line 90, in dmarc_per_from record, orgdomain = receiver_record(from_domain) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 117, in receiver_record retval = lookup_receiver_record(hostSansDmarc, dnsfunc) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 92, in lookup_receiver_record tags = answer_to_dict(str(result)) File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 42, in answer_to_dict retval = {t[0].strip().lower(): t[1].strip().lower() for t in rawTags} File "/usr/lib/python3.10/site-packages/authheaders/dmarc_lookup.py", line 42, in <dictcomp> retval = {t[0].strip().lower(): t[1].strip().lower() for t in rawTags} IndexError: list index out of range Dec 22 04:54:28 2023 (28) SHUNTING: 1703220868.1418982+b27f5db154375fcd2c418d467172e4aad0d3d57a So the message has been shunted, due to this error. Something with DMARC and tags. So I checked the DMARC record for the sending domain somenet.org. It is "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@somenet.org;mailto:postmaster@somenet.org;ri=3600;fo=1;" which is syntactically incorrect. The extra ";mailto:postmaster@somenet.org" is wrong, I guess the tag "ruf=" is missing here! It should probably read "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@somenet.org;ruf=mailto:postmaster@somenet.org;ri=3600;fo=1;" So, should mm3 somehow "catch" this error somehow? According to the DMARC specs, an syntactically incorrect DMARC record should be ignored / handled as if none were present. -- Ralf Hildebrandt Charité - Universitätsmedizin Berlin Geschäftsbereich IT | Abteilung Netz | Netzwerk-Administration Invalidenstraße 120/121 | D-10115 Berlin Tel. +49 30 450 570 155 ralf.hildebrandt(a)charite.de https://www.charite.de

5 12

Easy way to integrate captcha/reduce subscription spam?
by Johannes Rohr Feb. 25, 2025

Feb. 25, 2025

Dear all, my mailman3 instance is flooded with subscription spam, yet I haven't found any canonical method of integrating a captcha in to the subscription form. Is there really none at all? Perplexity.ai says I should install the python package django-simple-captcha but is unclear about the steps needed to actually make the captcha appear. Are there other ways? Is there any method that is robust and survives upgrades? Also, would there be a way of batch-deleting all those thousands of spam registrations that have accumulated (what they have in common is that their email is unconfirmed) Thanks in advance for any suggestion! Johannes

2 3

How To Install Mailman 3 on Debian 10 (Complete Guide)
by Brian Carpenter Feb. 25, 2025

Feb. 25, 2025

I have finished my how-to install Mailman 3 on Debian 10 completed to the point where you can use it to get a fully functional Mailman 3 environment up and running. This is a comprehensive guide that walks you through installing EVERYTHING (currently!) that Mailman 3 needs to run. It also makes updating Mailman 3/Postorius/Hyperkitty very easy. I have not added the update directions yet but will very soon. I also plan on adding a section on setting up DKIM and Xapian with Mailman 3. I hope this helps those who wish to get a Mailman 3 server up and running quickly and brings a sense of sanity to the confusion regarding the various documentation out there concerning the installation of Mailman 3/Postorius/Hyperkitty. I intend to add a How-to for Ubuntu 20 at some point. Without further ado: https://wiki.list.org/DOC/Howto_Install_Mailman3_On_Debian10 I am sure the above guide can use improvements. Feel free to contact me off-list to point out mistakes/improvements, etc. -- Brian Carpenter Harmonylists.com Emwd.com

13 38

Hyperkitty performance problem
by Tobias Diekershoff Feb. 20, 2025

Feb. 20, 2025

Hey all, last week we caught some AI bots causing high LOAD on our mailman3 servers archive, that we blocked with nginx rules. I'm not sure if it is related to this incident, but we noticed this when analyzing the log files at the time. In the mailman.log hyperkitty is listed 10k times and more to archive a single mail over the time of 2h, seemingly all with the same process ID > (1186) HyperKitty archived message [...] The mailman3-web.log has matching log entries that the mail was archived > hyperkitty.views.mailman Archived message every 0.5 seconds. In the web interface to the archive, the mail is only listed once though. My suspicion is, that these archiving processes are now overloading the system, preventing a timely catching up work of the queued messages. I could not find anything in the Mailman3 nor Hyperkitty documentation to better distribute the system resources to the Mailman runner or prevent Hyperkitty to restart archiving a mail over and over again. Any hint about where to look for the reason of this is highly appreciated. Best, Tobias -- Tobias Diekershoff >>> System Hacker Free Software Foundation Europe Schönhauser Allee 6/7, 10119 Berlin, Germany | t +49-30-27595290 Registered at Amtsgericht Hamburg, VR 17030 | fsfe.org/support OpenPGP-Key ID ... 0x25FE376FF17694A1 Fingerprint ...... 23EE F484 FDF8 291C BA09 A406 25FE 376F F176 94A1

6 16

Mailman shutting down Hyperkitty: Request too large
by Arte Chambers Feb. 2, 2025

Feb. 2, 2025

Hello I need help troubleshooting a problem with MM. The service is stopping and throwing this error in the mailman log "Exception in the HyperKitty archiver: 413 Request Entity Too Large" I'm not sure what my next steps should be to troubleshoot this issue. System CTL has this message and i'm not sure if it's relevant to the issue. ``` >sudo systemctl status mailman3.service systemd[1]: mailman3.service: Can't open PID file /opt/mailman/mm/var/master.pid (yet?) after start: No such file or directory ``` The MM Service will run for a while then shuts down. /mailman.log ``` Jan 30 05:22:20 2025 (1547) HyperKitty failure on https://list.louisvillecommunitygrocery.com/archives/api/mailman/archive: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> (413) Jan 30 05:22:20 2025 (1547) Could not archive the message with id < CAPesOD3EWaFzfVZAbUzFm3Vk3Xnqcf9KodVic6agaT27Kmhy0w(a)mail.gmail.com> Jan 30 05:22:20 2025 (1547) archiving failed, re-queuing (mailing-list lcg-board-private.list.louisvillecommunitygrocery.com, message < CAPesOD3EWaFzfVZAbUzFm3Vk3Xnqcf9KodVic6agaT27Kmhy0w(a)mail.gmail.com>) Jan 30 05:22:20 2025 (1547) Exception in the HyperKitty archiver: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> Jan 30 05:22:20 2025 (1547) Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.12/site-packages/mailman_hyperkitty/__init__.py", line 158, in _archive_message url = self._send_message(mlist, msg) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman_hyperkitty/__init__.py", line 228, in _send_message raise ValueError(result.text) ValueError: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> Jan 30 05:22:21 2025 (1547) HyperKitty archived message <CAPesOD25frujqYsAXU3kk8MUOY=h3ZV4cgzUXSi9L0c3K2W0NQ(a)mail.gmail.com> to https://list.louisvillecommunitygrocery.com/archives/list/testing@list.loui… [30/Jan/2025:05:22:43 +0000] "GET /3.1/users/lcgcoop(a)gmail.com HTTP/1.1" 200 428 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "GET /3.1/users/a6af988d603c49c9ac61a3a41562d132/addresses HTTP/1.1" 200 927 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "POST /3.1/members/find HTTP/1.1" 200 3407 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "POST /3.1/members/find HTTP/1.1" 200 4705 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "GET /3.1/addresses/lcgcoop(a)gmail.com HTTP/1.1" 200 401 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "GET /3.1/lists/ testing(a)list.louisvillecommunitygrocery.com HTTP/1.1" 200 482 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:43 +0000] "GET /3.1/lists/ testing.list.louisvillecommunitygrocery.com/roster/owner HTTP/1.1" 200 763 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:22:44 +0000] "GET /3.1/addresses/lcgcoop(a)gmail.com HTTP/1.1" 200 401 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:23:53 +0000] "GET /3.1/lists/ testing.list.louisvillecommunitygrocery.com HTTP/1.1" 200 482 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:23:53 +0000] "GET /3.1/lists/ testing.list.louisvillecommunitygrocery.com HTTP/1.1" 200 482 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:23:53 +0000] "GET /3.1/lists/ testing@list.louisvillecommunitygrocery.com/requests/count?token_owner=moderator HTTP/1.1" 200 73 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:23:53 +0000] "GET /3.1/lists/ testing(a)list.louisvillecommunitygrocery.com/held/count HTTP/1.1" 200 73 "-" "GNU Mailman REST client v3.3.5" [30/Jan/2025:05:24:15 +0000] "GET /3.1/lists/ testing(a)list.louisvillecommunitygrocery.com HTTP/1.1" 200 482 "-" "GNU Mailman REST client v3.3.5" Jan 30 05:42:54 2025 (1550) ACCEPT: <008401db72ce$cb6d50d0$6247f270$@ iglou.com> Jan 30 05:42:55 2025 (1547) HyperKitty failure on https://list.louisvillecommunitygrocery.com/archives/api/mailman/archive: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> (413) Jan 30 05:42:55 2025 (1547) Could not archive the message with id < CAPesOD3EWaFzfVZAbUzFm3Vk3Xnqcf9KodVic6agaT27Kmhy0w(a)mail.gmail.com> Jan 30 05:42:55 2025 (1547) archiving failed, re-queuing (mailing-list lcg-board-private.list.louisvillecommunitygrocery.com, message < CAPesOD3EWaFzfVZAbUzFm3Vk3Xnqcf9KodVic6agaT27Kmhy0w(a)mail.gmail.com>) Jan 30 05:42:55 2025 (1547) Exception in the HyperKitty archiver: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> Jan 30 05:42:55 2025 (1547) Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.12/site-packages/mailman_hyperkitty/__init__.py", line 158, in _archive_message url = self._send_message(mlist, msg) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib/python3.12/site-packages/mailman_hyperkitty/__init__.py", line 228, in _send_message raise ValueError(result.text) ValueError: <html> <head><title>413 Request Entity Too Large</title></head> <body> <center><h1>413 Request Entity Too Large</h1></center> <hr><center>nginx/1.24.0 (Ubuntu)</center> </body> </html> Jan 30 05:42:57 2025 (1547) HyperKitty archived message <008401db72ce$cb6d50d0$6247f270$(a)iglou.com> to https://list.louisvillecommunitygrocery.com/archives/list/lcg-board-private… [2025-01-30 05:46:02 +0000] [1554] [ERROR] Worker (pid:1586) was sent SIGKILL! Perhaps out of memory? [2025-01-30 05:46:02 +0000] [2227] [INFO] Booting worker with pid: 2227 Jan 30 05:46:09 2025 (1553) pipeline runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1553) pipeline runner exiting. Jan 30 05:46:09 2025 (1551) lmtp runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1551) lmtp runner exiting. Jan 30 05:46:09 2025 (1558) digest runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1558) digest runner exiting. Jan 30 05:46:09 2025 (1557) virgin runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1557) virgin runner exiting. Jan 30 05:46:09 2025 (1556) task runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1556) task runner exiting. Jan 30 05:46:09 2025 (1541) Master watcher caught SIGTERM. Exiting. Jan 30 05:46:09 2025 (1555) retry runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1555) retry runner exiting. Jan 30 05:46:09 2025 (1552) out runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1552) out runner exiting. Jan 30 05:46:09 2025 (1549) command runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1549) command runner exiting. Jan 30 05:46:09 2025 (1548) bounces runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1548) bounces runner exiting. Jan 30 05:46:09 2025 (1547) archive runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1547) archive runner exiting. Jan 30 05:46:09 2025 (1550) in runner caught SIGTERM. Stopping. Jan 30 05:46:09 2025 (1550) in runner exiting. [2025-01-30 05:46:09 +0000] [1554] [INFO] Handling signal: term [2025-01-30 05:46:09 +0000] [2227] [INFO] Worker exiting (pid: 2227) [2025-01-30 05:46:09 +0000] [1585] [INFO] Worker exiting (pid: 1585) Jan 30 05:46:10 2025 (1541) Master watcher caught SIGTERM. Exiting. [2025-01-30 05:46:10 +0000] [1554] [ERROR] Worker (pid:1585) was sent SIGKILL! Perhaps out of memory? [2025-01-30 05:46:10 +0000] [1554] [ERROR] Worker (pid:2227) was sent SIGTERM! [2025-01-30 05:46:10 +0000] [1554] [INFO] Shutting down: Master Jan 30 05:46:11 2025 (1541) Master stopped ``` Thank you, Paul 'Arte Chambers' Robey 502-408-6922

3 13

The paradox of VERP probe messages
by Jeremy Stanley Feb. 2, 2025

Feb. 2, 2025

tl;dr: Is there a way to have VERP probe messages not include copies of the bounce message that triggered them? Longer form... Some months ago I turned on bounce processing for a fairly large list with a long history, because I could see from the MTA logs that there were easily hundreds of old subscribers whose addresses had become defunct and this seemed like a great way to clean that up and make the server more conscientious at not repeatedly trying to deliver messages to sites that were just going to reject them. This worked really well. Until legitimate subscribers also started to get removed, that is, due primarily to false positives from overzealous spam filters. I had previously read about MM3's VERP probe message feature, which seemed to solve the main reason I had kept bounce processing disabled in the MM2 days, and thought it was on by default but clearly it wasn't. So I found the necessary config option and switched that on. This worked really well. Until legitimate subscribers started to get removed again, that is, due primarily to false positives from overzealous spam filters. See, I realized that the VERP probes were helpfully including copies of the most recent bounce for that user, and those NDRs quite often helpfully included their own copies of the messages which had been bounced. So if a subscriber's MTA was rejecting a message because the content looked like spam, then there was a good chance it would also reject the VERP probe message because it included the same content which had caused the earlier rejection. The VERP probe feature was essentially defeating its own purpose as a neutral confirmation that the subscriber hadn't been rejecting messages due to their content. This behavior raises a more significant concern as well. The MTA on the Mailman server in this case (Exim if it matters) wants to perform batch deliveries when there are multiple subscribers who all share the same remote MTAs. If multiples of those reject a list post, then the NDR back to Mailman aggregates those multiple rejections into a single message. If that incident triggers a VERP probe, the bulk NDR enumerating the addresses of all the subscribers who were rejected gets forwarded along as part of the VERP probe, disclosing a potentially sensitive litany of random subscriber addresses to other subscribers within that batch. I don't know whether this is a misconfiguration, or an unintended consequence of the intersection of Mailman and MTA features. While I understand that the inclusion of a bounce sample within a VERP probe may be a way to take some workload off list admins in that users can possibly see a message/rejection to explain the reason for the probe, the downsides of that choice make the feature unfortunately unusable for some lists that could otherwise have benefited from it. Am I possibly missing an existing switch that could make VERP probe messages *not* include copies of the bounces that triggered them? This is in some ways similar to the problems I've noticed with message hold notifications to moderators including copies of the held messages, since these are (at least in the case of the lists I manage) often spam, so has a tendency to cause the mail providers for the list moderators to start considering the mailing list server a spam source. If there were a switch to make message hold notifications to moderators *not* include copies of the messages themselves, I'd start allowing moderator notifications again. Suggestions welcome! Also willing to try to submit patches to enable some of these solutions if they don't already exist. And thanks to anyone who read this far! -- Jeremy Stanley

2 4