Paul Tomblin via Mailman-users writes:
I backup the full server at a time I’ve been running Mailman 3 on it since January. The current disk use of my entire server is around 35 Gb. The borg backup is somewhere around 600 Gb.
OK, if you just did tar backups (uncompressed!) you should see (7 + 4 + 6) x 35GB = 595 GB, no? So this is just nuts.
BLUF: I think you have a borg vs. DB problem, not a borg vs. Mailman problem. You should ask them, because they're the ones who understand their algorithms and software. I would imagine that (1) it's possible to tune borg to keep its own overhead reasonable even if the use case is outside of its primary desigh, and (2) the borg devs have very likely had experience with "database backups bloat quickly". They can tell you how to configure PostgreSQL to be more borg-friendly, or vice-versa, or to configure borg around some other backup method for PostgreSQL.
A few thoughts since I already thought them:
By "my entire server" do you mean the /var partition, or everything the OS can access? If the latter, about 20GB of that is just operating system and server software, no? Which presumably doesn't change very often in a 6-month period? I'd guess you're looking at most 15GB of potentially variable data, so 255GB in uncompressed tarballs. Even if the 35GB is just /var, I just don't see how you could have a Mailman installation generating 500GB of new backup in 5 months unless the traffic is truly gargantuan.
One possibility for a fraction of that bloat is that you have the prototype ("write-only to maildir") archiver enabled. In that case $var_dir/archives/prototype could be collecting GB fairly quickly. Since it's write-only, there's no loss to disabling it and deleting it. But it's append-only so should deduplicate well, and unless mail traffic is exploding it should pretty quickly stabilize to contribute a constant amount to a stable archive size -- it wouldn't explain your 600-lb gorilla. I could be wrong if you have enough traffic, so check that.
If you want to continue the conversation here I'd need du data on the mailman installation (especially Mailman's $var_dir), on Mailman's $log_dir (which is $var_dir/logs by default but can be configured elsewhere, often /var/log/mailman3), and on the PostgreSQL database (usually /var/lib/postgresql/$version/main) Mailman is using, but I really don't think I can help much. Also how big your Mailman 2 archives were, if you kept them.
The old borg backup when I was using Mailman 2 was less than 100 Gb
That sounds right given the schedule of "keeps".
Steve
-- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan