Errors while importing mm2 list archives
MM3-3.3.7, Hyperkitty-1.3.6
While importing, I encountered several of this error:
<CUT> MySQLdb.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x8A, ...' for column 'content' at row 1") The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_import.py", line 232, in from_mbox add_to_list(self.list_address, message) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/lib/incoming.py", line 153, in add_to_list email.save() File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/base.py", line 812, in save self.save_base( File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/base.py", line 863, in save_base updated = self._save_table( File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/base.py", line 1006, in _save_table results = self._do_insert( File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/base.py", line 1047, in _do_insert return manager._insert( File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/query.py", line 1791, in _insert return query.get_compiler(using=using).execute_sql(returning_fields) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1660, in execute_sql cursor.execute(sql, params) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers( File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers return executor(sql, params, many, context) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/utils.py", line 91, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/backends/mysql/base.py", line 75, in execute return self.cursor.execute(query, args) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/MySQLdb/cursors.py", line 206, in execute res = self._query(query) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/MySQLdb/cursors.py", line 319, in _query db.query(q) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/MySQLdb/connections.py", line 254, in query _mysql.connection.query(self, query) django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x8A, ...' for column 'content' at row 1") </CUT>
That occurred several times, but the import process went all the way, to 99% then came to the point below.
Message 253d9d44-cc3e-fed1-144b-34da88c52b39@emailplus.org failed to import, skipping Computing thread structure Synchronizing properties with Mailman 904 emails left to refresh, checked 0 764 emails left to refresh, checked 0 Warming up cache Traceback (most recent call last): File "/opt/mailman/mm/venv/bin/django-admin", line 8, in <module> sys.exit(execute_from_command_line()) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line utility.execute() File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv self.execute(*args, **cmd_options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_import.py", line 411, in handle call_command("hyperkitty_warm_up_cache", list_address) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 198, in call_command return command.execute(*args, **defaults) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_warm_up_cache.py", line 53, in handle self.warm_up_mlist(mlist, options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_warm_up_cache.py", line 61, in warm_up_mlist for thread in mlist.recent_threads: File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 162, in recent_threads return self.cached_values"recent_threads" File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/common.py", line 58, in __call__ return self.get_or_set(*args, **kwargs) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 317, in get_or_set return [Thread.objects.get(pk=pk) for pk in thread_ids] File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 317, in <listcomp> return [Thread.objects.get(pk=pk) for pk in thread_ids] File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/query.py", line 650, in get raise self.model.DoesNotExist( hyperkitty.models.thread.Thread.DoesNotExist: Thread matching query does not exist. (venv) [mailman@gw ~/mm]$
Should I take it that the import was successful, or what should I do next?
PS: This is the 1st run after installing MM3 on this server and the MM3 list hasn't gone live. So if anything, I am okay with doing what needs to be done and rerun the import.
TIA
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
On 12/8/22 07:23, Odhiambo Washington wrote:
MM3-3.3.7, Hyperkitty-1.3.6
While importing, I encountered several of this error:
<CUT> MySQLdb.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x8A, ...' for column 'content' at row 1") The above exception was the direct cause of the following exception: ... File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/backends/mysql/base.py",
That's a 4-byte utf-8 encoding for an emoji. MySQL and MariaDB have to be told to accept 4-byte encodings.
In the DATABASES['default'] definition, you need to set 'OPTIONS': {'charset': 'utf8mb4'}
Also, although it doesn't affect this, see https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/databa...
You might also need to modify the tables, particularly hyperkitty_email with something like
ALTER TABLE hyperkitty_email
DEFAULT CHARACTER SET utf8mb4,
MODIFY subject varchar(512)
CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci,
MODIFY content
CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Thu, Dec 8, 2022 at 10:41 PM Mark Sapiro <mark@msapiro.net> wrote:
On 12/8/22 07:23, Odhiambo Washington wrote:
MM3-3.3.7, Hyperkitty-1.3.6
While importing, I encountered several of this error:
<CUT> MySQLdb.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x8A, ...' for column 'content' at row 1") The above exception was the direct cause of the following exception: ... File
"/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/backends/mysql/base.py",
That's a 4-byte utf-8 encoding for an emoji. MySQL and MariaDB have to be told to accept 4-byte encodings.
In the DATABASES['default'] definition, you need to set
'OPTIONS': {'charset': 'utf8mb4'}
I added that:
[database] class: mailman.database.mysql.MySQLDatabase url: mysql+pymysql://mailman_user:XXXXXXXXX@localhost /mailmansuite?charset=utf8mb4&use_unicode=1
Also, although it doesn't affect this, see
https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/databa...
I read this too. Did not pick much from there.
You might also need to modify the tables, particularly hyperkitty_email with something like
ALTER TABLE hyperkitty_email DEFAULT CHARACTER SET utf8mb4, MODIFY subject varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci, MODIFY content CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
So I removed my list, dropped the database, and proceeded as follows:
create database mailmansuite CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci; ALTER TABLE hyperkitty_email DEFAULT CHARACTER SET utf8mb4; ALTER TABLE hyperkitty_email MODIFY subject varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY content longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY sender_name varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY in_reply_to varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
I did all those (although looks overzealous) just to be on the safe side. I have seen people putting emojis in their names.
After that, I did the mailman-web migrate, created the list, imported the config from MM2 and then proceeded to import the archives. However, the errors still are there, and several messages were not imported
- I have several "failed to import" warnings.
<CUT> django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x8A, ...' for column 'content' at row 1")
Message 253d9d44-cc3e-fed1-144b-34da88c52b39@emailplus.org failed to import, skipping *<================== several of these "failed to import" messages* Computing thread structure Synchronizing properties with Mailman 905 emails left to refresh, checked 0 765 emails left to refresh, checked 0 Warming up cache Traceback (most recent call last): File "/opt/mailman/mm/venv/bin/django-admin", line 8, in <module> sys.exit(execute_from_command_line()) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line utility.execute() File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv self.execute(*args, **cmd_options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_import.py", line 411, in handle call_command("hyperkitty_warm_up_cache", list_address) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 198, in call_command return command.execute(*args, **defaults) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_warm_up_cache.py", line 53, in handle self.warm_up_mlist(mlist, options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_warm_up_cache.py", line 61, in warm_up_mlist for thread in mlist.recent_threads: File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 162, in recent_threads return self.cached_values"recent_threads" File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/common.py", line 58, in __call__ return self.get_or_set(*args, **kwargs) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 317, in get_or_set return [Thread.objects.get(pk=pk) for pk in thread_ids] File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 317, in <listcomp> return [Thread.objects.get(pk=pk) for pk in thread_ids] File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/query.py", line 650, in get raise self.model.DoesNotExist( hyperkitty.models.thread.Thread.DoesNotExist: Thread matching query does not exist. (venv) [mailman@gw ~/mm]$
It therefore means I have several broken threads in the archives. I am still willing to try other solutions if they are suggested.
And what do I make of the final error - *hyperkitty.models.thread.Thread.DoesNotExist: Thread matching query does not exist*. ??
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
On 12/10/22 01:13, Odhiambo Washington wrote:
On Thu, Dec 8, 2022 at 10:41 PM Mark Sapiro <mark@msapiro.net> wrote:
In the DATABASES['default'] definition, you need to set
'OPTIONS': {'charset': 'utf8mb4'}I added that:
[database] class: mailman.database.mysql.MySQLDatabase url: mysql+pymysql://mailman_user:XXXXXXXXX@localhost /mailmansuite?charset=utf8mb4&use_unicode=1
The above is for Mailman core. It doesn't affect HyperKitty. For that
you need to look at the DATABASES definition in
/etc/mailman3/settings.py
<https://docs.mailman3.org/projects/mailman-web/en/latest/settings.html#mailm...>
and ensure that in addition to the other settings under default { you have
'OPTIONS': {'charset': 'utf8mb4'},
...
You might also need to modify the tables, particularly hyperkitty_email with something like
ALTER TABLE hyperkitty_email DEFAULT CHARACTER SET utf8mb4, MODIFY subject varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci, MODIFY content CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;So I removed my list, dropped the database, and proceeded as follows:
create database mailmansuite CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci; ALTER TABLE hyperkitty_email DEFAULT CHARACTER SET utf8mb4; ALTER TABLE hyperkitty_email MODIFY subject varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY content longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY sender_name varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY in_reply_to varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
I did all those (although looks overzealous) just to be on the safe side. I have seen people putting emojis in their names.
None of that should be necessary. mailman-web migrate should create
the database and tables as needed.
After that, I did the mailman-web migrate, created the list, imported the config from MM2 and then proceeded to import the archives. However, the errors still are there, and several messages were not imported
Do you have the OPTIONS setting in your DATABASES definition in settings.py?
...
Warming up cache Traceback (most recent call last): File "/opt/mailman/mm/venv/bin/django-admin", line 8, in <module> sys.exit(execute_from_command_line()) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line utility.execute() File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv self.execute(*args, **cmd_options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_import.py", line 411, in handle call_command("hyperkitty_warm_up_cache", list_address) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 198, in call_command return command.execute(*args, **defaults) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_warm_up_cache.py", line 53, in handle self.warm_up_mlist(mlist, options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_warm_up_cache.py", line 61, in warm_up_mlist for thread in mlist.recent_threads: File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 162, in recent_threads return self.cached_values"recent_threads" File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/common.py", line 58, in __call__ return self.get_or_set(*args, **kwargs) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 317, in get_or_set return [Thread.objects.get(pk=pk) for pk in thread_ids] File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 317, in <listcomp> return [Thread.objects.get(pk=pk) for pk in thread_ids] File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/query.py", line 650, in get raise self.model.DoesNotExist( hyperkitty.models.thread.Thread.DoesNotExist: Thread matching query does not exist. (venv) [mailman@gw ~/mm]$
I have no idea what would cause this in a newly created database with no archives before running hyperkitty_import.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Sat, Dec 10, 2022 at 9:45 PM Mark Sapiro <mark@msapiro.net> wrote:
On 12/10/22 01:13, Odhiambo Washington wrote:
On Thu, Dec 8, 2022 at 10:41 PM Mark Sapiro <mark@msapiro.net> wrote:
In the DATABASES['default'] definition, you need to set
'OPTIONS': {'charset': 'utf8mb4'}I added that:
[database] class: mailman.database.mysql.MySQLDatabase url: mysql+pymysql://mailman_user:XXXXXXXXX@localhost /mailmansuite?charset=utf8mb4&use_unicode=1
The above is for Mailman core. It doesn't affect HyperKitty. For that you need to look at the DATABASES definition in /etc/mailman3/settings.py < https://docs.mailman3.org/projects/mailman-web/en/latest/settings.html#mailm...>
and ensure that in addition to the other settings under
default {you have'OPTIONS': {'charset': 'utf8mb4'},...
You might also need to modify the tables, particularly hyperkitty_email with something like
ALTER TABLE hyperkitty_email DEFAULT CHARACTER SET utf8mb4, MODIFY subject varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci, MODIFY content CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;So I removed my list, dropped the database, and proceeded as follows:
create database mailmansuite CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci; ALTER TABLE hyperkitty_email DEFAULT CHARACTER SET utf8mb4; ALTER TABLE hyperkitty_email MODIFY subject varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY content longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY sender_name varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY in_reply_to varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
I did all those (although looks overzealous) just to be on the safe side. I have seen people putting emojis in their names.
None of that should be necessary.
mailman-web migrateshould create the database and tables as needed.After that, I did the mailman-web migrate, created the list, imported the config from MM2 and then proceeded to import the archives. However, the errors still are there, and several messages were not imported
Do you have the OPTIONS setting in your DATABASES definition in settings.py?
Here is what I have tested lately in settings.py:
# MySQL Database setup DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'mailmansuite', 'USER': 'mm3', 'PASSWORD': 'XXXXXXXXX', 'HOST': 'localhost', 'PORT': '', 'OPTIONS': { 'init_command': "SET sql_mode='STRICT_TRANS_TABLES'", 'charset': 'utf8mb4', }, } }
I have dropped the DB, recreated, run the 'migrate', created the list, imported config from MM2. However, nothing seems to be able to address the errors with importing the archives.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
On 12/17/22 23:22, Odhiambo Washington wrote:
I have dropped the DB, recreated, ...
When you created the DB did you specify CHARACTER SET utf8mb4, e.g.
CREATE DATABASE mailmansuite CHARACTER SET utf8mb4
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Tue, Dec 20, 2022 at 3:52 AM Mark Sapiro <mark@msapiro.net> wrote:
On 12/17/22 23:22, Odhiambo Washington wrote:
I have dropped the DB, recreated, ...
When you created the DB did you specify
CHARACTER SET utf8mb4, e.g.CREATE DATABASE mailmansuite CHARACTER SET utf8mb4
Yes. That is what I do at the minimum for my databases unless specific instructions exist against it. So it seems like I have done everything that is commonly expected, but the import still fails on several messages.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
On Tue, Dec 20, 2022 at 9:50 AM Odhiambo Washington <odhiambo@gmail.com> wrote:
On Tue, Dec 20, 2022 at 3:52 AM Mark Sapiro <mark@msapiro.net> wrote:
On 12/17/22 23:22, Odhiambo Washington wrote:
I have dropped the DB, recreated, ...
When you created the DB did you specify
CHARACTER SET utf8mb4, e.g.CREATE DATABASE mailmansuite CHARACTER SET utf8mb4
Yes. That is what I do at the minimum for my databases unless specific instructions exist against it. So it seems like I have done everything that is commonly expected, but the import still fails on several messages.
It turns out that all this time I was barking up the wrong tree! I was having all my modifications in /etc/mailman3/*. When I did run 'mailman info', the information that came out is that mailman was actually reading /opt/mailman/mm/mailman.cfg I then just moved all the files I had put in /etc/mailman3/ to /opt/mailman/mm/.
There is a WARNING on https://docs.list.org/en/latest/install/virtualenv.html#virtualenv-install which says one must create the config file as /etc/mailman3/mailman.cfg How to force mailman to read that file is something I haven't understood, but turns out to be what has taken me days trying to get things right.
Ultimately, when I did the import, there was only one email from the archives that failed to be imported - which I honestly don't mind, but maybe there is a solution for this "'utf-8' codec can't encode character '\udcae' in position 331: surrogates not allowed" ??
(venv) [mailman@gw ~/mm]$ /opt/mailman/mm/bin/django-admin hyperkitty_import -l kictanet@lists.kictanet.or.ke /usr/local/mailman2/archives/private/kictanet.mbox/kictanet.mbox Importing from mbox file /usr/local/mailman2/archives/private/kictanet.mbox/kictanet.mbox to kictanet@lists.kictanet.or.ke 21%*Failed adding message <1260802782.6596.1350003565@webmail.messagingengine.com <1260802782.6596.1350003565@webmail.messagingengine.com>>: 'utf-8' codec can't encode character '\udcae' in position 331: surrogates not allowed* Computing thread structure Synchronizing properties with Mailman 915 emails left to refresh, checked 0 767 emails left to refresh, checked 0 Warming up cache The full-text search index is not updated for this list. It will not be updated by the 'minutely' incremental update job. To update the index for this list, run the Django admin command with arguments 'update_index_one_list kictanet@lists.kictanet.or.ke'. (venv) [mailman@gw ~/mm]$
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
On 12/20/22 06:48, Odhiambo Washington wrote:
There is a WARNING on https://docs.list.org/en/latest/install/virtualenv.html#virtualenv-install which says one must create the config file as /etc/mailman3/mailman.cfg How to force mailman to read that file is something I haven't understood, but turns out to be what has taken me days trying to get things right.
If you run the mailman command without any -C option, it will look for
a config in /etc/mailman3/mailman.cfg
Ultimately, when I did the import, there was only one email from the archives that failed to be imported - which I honestly don't mind, but maybe there is a solution for this "'utf-8' codec can't encode character '\udcae' in position 331: surrogates not allowed" ??
If you provide a copy of that message from the mbox, either on the list or to me directly, I'll investigate.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Tue, Dec 20, 2022 at 8:32 PM Mark Sapiro <mark@msapiro.net> wrote:
On 12/20/22 06:48, Odhiambo Washington wrote:
There is a WARNING on
https://docs.list.org/en/latest/install/virtualenv.html#virtualenv-install
which says one must create the config file as /etc/mailman3/mailman.cfg How to force mailman to read that file is something I haven't understood, but turns out to be what has taken me days trying to get things right.
If you run the
mailmancommand without any -C option, it will look for a config in /etc/mailman3/mailman.cfg
In my case, running mailman without any arguments just makes it grok :)
(venv) [mailman@gw ~/mm]$ mailman info GNU Mailman 3.3.7 (Tom Sawyer) Python 3.9.15 (main, Dec 6 2022, 09:25:59) [Clang 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a *config file: /opt/mailman/mm/mailman.cfg* db url: mysql+pymysql://mailman_user:XXXXXXXX@localhost /mailman3db?charset=utf8mb4&use_unicode=1 devmode: DISABLED REST root url: https://localhost:8001/3.1/ REST credentials: restadmin:restpass
(venv) [mailman@gw ~/mm]$ mailman Usage: mailman [OPTIONS] COMMAND [ARGS]...
The GNU Mailman mailing list management system Copyright 1998-2018 by the Free Software Foundation, Inc. http://www.list.org
Options: --version Show the version and exit. --run-as-root Running mailman commands as root is not recommended and mailman will refuse to run as root unless this option is specified. -C, --config FILE Configuration file to use. If not given, the environment variable MAILMAN_CONFIG_FILE is consulted and used if set. If neither are given, a default configuration file is loaded. -h, --help Show this message and exit. ....
Ultimately, when I did the import, there was only one email from the
archives that failed to be imported - which I honestly don't mind, but maybe there is a solution for this "'utf-8' codec can't encode character '\udcae' in position 331: surrogates not allowed" ??
If you provide a copy of that message from the mbox, either on the list or to me directly, I'll investigate.
A mbox file containing that email and the associated thread can be downloaded from here: https://webmail.kictanet.or.ke/~wash/chunk_7.txt.gz
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
On 12/20/22 22:25, Odhiambo Washington wrote:
In my case, running
mailmanwithout any arguments just makes it grok :)(venv) [mailman@gw ~/mm]$ mailman info
...
*config file: /opt/mailman/mm/mailman.cfg*
That's because it first looks in the current directory for mailman.cfg
and only if it doesn't find it there does it look in /etc/mailman3.
Ultimately, when I did the import, there was only one email from the
archives that failed to be imported - which I honestly don't mind, but maybe there is a solution for this "'utf-8' codec can't encode character '\udcae' in position 331: surrogates not allowed" ??
If you provide a copy of that message from the mbox, either on the list or to me directly, I'll investigate.
A mbox file containing that email and the associated thread can be downloaded from here: https://webmail.kictanet.or.ke/~wash/chunk_7.txt.gz
The offending message is multipart with an attached message/rfc822 part and that part has a preamble which in bytes is
b'This document is a Single File Web Page, also known as a Web Archive file. If you are seeing this message, your browser or editor doesn't support Web Archive files. Please download a browser that supports Web Archive, such as Windows\xae Internet Explorer\xae.'
The '\xae' is an iso-8859-1 encoding for '®', but no charset is
specified for this preamble so email message parsing ends up decoding
those as surrogates '\udcae' which throws the exception when
hyperkitty.lib.incoming.add_to_list() in trying to save the attachment
does att.set_content(content).
Normally, when messages to be archived arrive from core via mailman_hyperkitty, the surrogates are replaced by email.utils._sanitize(), but this doesn't happen with messages from hyperkitty_import.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Thu, Dec 22, 2022 at 9:36 AM Mark Sapiro <mark@msapiro.net> wrote:
On 12/20/22 22:25, Odhiambo Washington wrote:
In my case, running
mailmanwithout any arguments just makes it grok :)(venv) [mailman@gw ~/mm]$ mailman info
...
*config file: /opt/mailman/mm/mailman.cfg*
That's because it first looks in the current directory for
mailman.cfgand only if it doesn't find it there does it look in /etc/mailman3.
And that happens because I have all my config files in /opt/mailman/mm/. So if I moved all of them to /etc/mailman3/ it would settle for that? I will test it.
Ultimately, when I did the import, there was only one email from the
archives that failed to be imported - which I honestly don't mind, but maybe there is a solution for this "'utf-8' codec can't encode character '\udcae' in position 331: surrogates not allowed" ??
If you provide a copy of that message from the mbox, either on the list or to me directly, I'll investigate.
A mbox file containing that email and the associated thread can be downloaded from here: https://webmail.kictanet.or.ke/~wash/chunk_7.txt.gz
The offending message is multipart with an attached message/rfc822 part and that part has a preamble which in bytes is
b'This document is a Single File Web Page, also known as a Web Archive file. If you are seeing this message, your browser or editor doesn't support Web Archive files. Please download a browser that supports Web Archive, such as Windows\xae Internet Explorer\xae.'
The '\xae' is an iso-8859-1 encoding for '®', but no charset is specified for this preamble so email message parsing ends up decoding those as surrogates '\udcae' which throws the exception when hyperkitty.lib.incoming.add_to_list() in trying to save the attachment does
att.set_content(content).Normally, when messages to be archived arrive from core via mailman_hyperkitty, the surrogates are replaced by email.utils._sanitize(), but this doesn't happen with messages from hyperkitty_import.
I am glad it's an email that is probably overtaken by events. However, the good thing is that I will have the archives from MM2 still accessible once I move to MM3.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
Mark Sapiro writes:
If you run the
mailmancommand without any -C option, it will look for a config in /etc/mailman3/mailman.cfg
"mailman start" (at least) ALSO will create an empty config in var/etc/mailman.cfg. This cause me a lot of pain until I linked the mailman user's $HOME/var/etc/mailman.cfg to /etc/mailman3/mailman.cfg.
I will report this in a few hours if nobody else gets to it first.
Steve
the third part
alter table hyperkitty_email modify content character set utf8mb4 collate utf8mb4_unicode_ci;
throws ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'set utf8mb4 collate utf8mb4_unicode_ci' at line 1
The correct syntax is:
alter table hyperkitty_email modify content longtext character set utf8mb4 collate utf8mb4_unicode_ci;
participants (4)
-
Mark Sapiro -
Odhiambo Washington -
Stephen J. Turnbull -
Wikinaut