offlineimap3 produces error due to defect in Spam email

[dominique-unruh]

General informations

• EndeavourOS
• offlineimap version (offlineimap -V): v8.0.0 (but probably a very recent git master because I installed using AUR package offlineimap3-git).
• Python version: 3.11.3
• server name or domain: GMail
• CLI options: offlineimap -c config/offlineimap.conf -a gmail-work

Configuration file offlineimaprc

[DEFAULT]
emailrepo = ~/r/emails
configdir = %(emailrepo)s/config
default_sslcacertfile = /etc/ssl/certs/ca-certificates.crt

[general]
metadata = %(configdir)s/metadata
accounts = gmail-private, gmail-work, XXXXXXX
ui = basic
# In case of crash, we simply reset the git repo:
fsync = False



[Account gmail-private]
localrepository = gmail-private-local
remoterepository = gmail-private-remote
#maildir-windows-compatible = no
#synclabels = no
ignorelabels = \Starred, \Spam, \Trash, \Important

[Repository gmail-private-local]
type = GmailMaildir
localfolders = %(emailrepo)s/gmail-private
sep = /
sync_deletes = no
filename_use_mail_timestamp = yes
synclabels = yes

[Repository gmail-private-remote]
type = Gmail
remoteuser = [email protected]
sslcacertfile = %(default_sslcacertfile)s
ssl = yes
# If this file doesn't exist, create an "application password", and put it in this file:
remotepassfile = ~/.offlineimap_pwd_3XXXXXXXXX
auth_mechanisms = PLAIN
maxconnections = 1
expunge = no
subscribedonly = no
sync_deletes = yes
readonly = True
folderfilter = lambda foldername: foldername not in ['[Google Mail]/Important', '[Google Mail]/Trash', '[Google Mail]/Spam', 'Trash']



[Account gmail-work]
localrepository = gmail-work-local
remoterepository = gmail-work-remote
#maildir-windows-compatible = no
#synclabels = no
ignorelabels = \Starred, \Spam, \Trash, \Important

[Repository gmail-work-local]
type = GmailMaildir
localfolders = %(emailrepo)s/gmail-work
sep = /
sync_deletes = no
filename_use_mail_timestamp = yes
synclabels = yes

[Repository gmail-work-remote]
type = Gmail
remoteuser = [email protected]
sslcacertfile = %(default_sslcacertfile)s
ssl = yes
# If this file doesn't exist, create an "application password", and put it in this file:
remotepassfile = ~/.offlineimap_pwd_dXXXXXXXXX
auth_mechanisms = PLAIN
maxconnections = 1
expunge = no
subscribedonly = no
sync_deletes = yes
readonly = True
folderfilter = lambda foldername: foldername not in ['[Google Mail]/Important', '[Google Mail]/Trash', '[Google Mail]/Spam', 'Trash']



[Account XXXXXXX]
localrepository = XXXXXXX-local
remoterepository = XXXXXXX-remote

[Repository XXXXXXX-local]
type = Maildir
localfolders = %(emailrepo)s/XXXXXXX
sep = /
sync_deletes = no
filename_use_mail_timestamp = yes
synclabels = yes

[Repository XXXXXXX-remote]
type = IMAP
remotehost = outlook.office365.com
remoteuser = XXXXXXXXX@XXXXXX
sslcacertfile = %(default_sslcacertfile)s
auth_mechanisms = XOAUTH2
ssl = yes
starttls = yes
# If this file doesn't exist, create an "application password", and put it in this file:
#remotepassfile = ~/.offlineimap_uni_tartu
maxconnections = 1
expunge = no
subscribedonly = no
sync_deletes = yes
readonly = True

oauth2_request_url = https://login.microsoftonline.com/common/oauth2/v2.0/token
# From https://hg.mozilla.org/comm-central/file/tip/mailnews/base/src/OAuth2Providers.jsm
oauth2_client_id = 9e5f94bc-e8a4-4e73-b8be-63364c29d753
oauth2_client_secret = 
# Extracted from configured Thunderbird, in Settings, Saved Logins:
oauth2_refresh_token_eval = open("/home/XXXXXXXXX/.offlineimap_uni_tartu").read()
#folderfilter = lambda foldername: not exists(foldername not in ['Junk Email', 'Deleted Items', 'Deleted Items/tmp-cerqus-report']
folderfilter = lambda foldername: not any(re.fullmatch(pattern,foldername) for pattern in ['Junk Email', 'Deleted Items.*', 'Calendar.*'])

pythonfile (if any)

none

Logs, error

OfflineIMAP 8.0.0
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v3.06, Python v3.11.3, OpenSSL 3.1.2 1 Aug 2023
*** Processing account gmail-work
Establishing connection to imap.gmail.com:993 (gmail-work-remote)
Syncing [Google Mail]/All Mail: Gmail -> GmailMaildir
Copy message UID 347417 (1/1) gmail-work-remote:[Google Mail]/All Mail -> gmail-work-local:[Google Mail]/All Mail
UID 347417 has defects: [CloseBoundaryNotFoundDefect()]
ERROR: UID 347417 (<[email protected]>) has defects preventing it from being processed!
  UnicodeEncodeError: 'ascii' codec can't encode characters in position 830-832: ordinal not in range(128)
[... more Syncing lines ...]
*** Finished account 'gmail-work' in 1:08
ERROR: Exceptions occurred during the run!
ERROR: offlineimap.error.OfflineImapError: UID 347417 (<[email protected]>) has defects preventing it from being processed!
  UnicodeEncodeError: 'ascii' codec can't encode characters in position 830-832: ordinal not in range(128)


Traceback:
  File "/usr/lib/python3.11/site-packages/offlineimap/folder/Base.py", line 810, in copymessageto
    message = self.getmessage(uid)
              ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/offlineimap/folder/Gmail.py", line 69, in getmessage
    data = self._fetch_from_imap(str(uid), self.retrycount)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/offlineimap/folder/IMAP.py", line 930, in _fetch_from_imap
    raise OfflineImapError(

Steps to reproduce the error

• There is one specific email in my Gmail account that leads to the above encoding issue.
• Doing a sync (GMail -> local) triggers the error above every time.
• Also happens when I install the arch package offlineimap (which installs v8.0.0).
• My expectation would be that the email is backed up anyway, including any bad encodings or similar.
• This seems similar to offlineimap3 crashes with encoding error #62, but since the latter was fixed according to the issue I assume it is a different problem.

The source code of the email is as follows. (Copy and pasted from Thunderbird's show source, I don't know if that changes some encoding issues.) Replaced my username by XXXXXXX. If needed, I can try other specific ways to get the source (I won't delete the email for now).

Delivered-To: [email protected]
Received: by 2002:a1c:1f02:0:0:0:0:0 with SMTP id f2csp7332194wmf;
        Fri, 6 Mar 2020 04:21:11 -0800 (PST)
X-Google-Smtp-Source: ADFU+vstJhHk+gwiEbvjgRHq3mzncylI2NARWCrurf34SP18KBRqe1JTbPNYhf9lfHmMNxmuyK9K
X-Received: by 2002:ac2:4c29:: with SMTP id u9mr1757314lfq.141.1583497271289;
        Fri, 06 Mar 2020 04:21:11 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1583497271; cv=none;
        d=google.com; s=arc-20160816;
        b=LnhCjDSzk0fhST+6gGvc0Np4Zx7zFQj0zFZYpeAEOEfe8ii+ZkdXkhN5JC/3Axpeme
         lpzvNgv8rbuVR8d8a8P03GisrCeATFOOMC5KD37Nl3u7iQqkvaw7dtpXWafQl4olPIKi
         TYCaXgkg5A+MmVMh7c51YAGZL8NyKQCz6GBSLwuom4Aeh7Bv7H6W+8OtA8L6hjlpqb5y
         4D3MxmCO3jdqxgqVnBpus+qguCfdip9wgKwce2AC1wjS4ofDwCOjvc8dAj0bA+tBaHkM
         PO74QXGqKqgoB5Y+k2rfWtfrjxjOyKyXLsRe7GBPD4ds9SsmKnHk53IFlVT+JhxubPL3
         YP3g==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=date:message-id:from:references:in-reply-to:subject:mime-version:to
         :cc:sender;
        bh=1J5Pi5p/aaEkZCbcX0tGNE9cbqvr7PAcZa8b9z1mNaw=;
        b=zAhQ0FfnFKuWp3w83TKkAmMtQ10aVS3V6wguRrrpOlzZkynLhvKhJnT8/dHcGJRtvS
         Tany4A36UcVZVHw87ldK9p2iDD4cNT1QykwhFeZStugMkATk0/nvTXGO0G2SZAULOCbt
         K3Te/5tP4U5sU9cnNj52V4kcPYPmg4x6lnwxc/pymuJG7zGDFhXo3g+1dqPHo+I5w+eR
         kpaOlky0mUiYzY/qAYALd715v8f6K67h4JRbE2znEquoagezqITT6mF+r5xfw1PbN/d7
         Yuz+Br8AAP3LNJ3YzI1XgxqfgXlSTv0LiH/zRZmtt5hNv+6mi+HIEinXflAAmAXQqDrk
         bZsw==
ARC-Authentication-Results: i=1; mx.google.com;
       spf=neutral (google.com: 45.35.40.198 is neither permitted nor denied by best guess record for domain of [email protected]) [email protected]
Return-Path: <[email protected]>
Received: from phase-select.peaceevil.com (phase-select.peaceevil.com. [45.35.40.198])
        by mx.google.com with ESMTPS id c26si2235544ljk.8.2020.03.06.04.21.10
        for <[email protected]>
        (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
        Fri, 06 Mar 2020 04:21:11 -0800 (PST)
Received-SPF: neutral (google.com: 45.35.40.198 is neither permitted nor denied by best guess record for domain of [email protected]) client-ip=45.35.40.198;
Authentication-Results: mx.google.com;
       spf=neutral (google.com: 45.35.40.198 is neither permitted nor denied by best guess record for domain of [email protected]) [email protected]
Received: from rvirapvdc00.net.ravensburger.ag ()
          by mx01.ravensburger.ag (SMTP DAEMON FROM RAVENSBURGER AG) with SMTP ID 555;
          Mon, 15 Jan 2018 03:52:48 +0100 (CET)
Received: from app19.muc.ec-messenger.com (app19.muc.ec-messenger.com )
  (envelope-from <[email protected]>)
  by gp13mtaq123 (mtaq-receiver/2.20190311.1) with ESMTP id yA3jJ-_S5g8Z
  for <[email protected]>; Thu, 30 May 2019 19:00:22 +0200
Content-Type: multipart/digest; boundary=Nfqsl1jk1465dqsd10sdqsdq5s1dzaer84er0zeil8jhkl15
Sender: XXXXXXX-6cYXAQ3UxZO@
Cc: <[email protected]>
To:  <[email protected]>
X-Gm-Message-State: APjAAAVK0NYMmxVlEDhN8/Aw2l5boCBwoS34CKHdu+8YCQpHxKLz4tlK Rsv01B4kaJNWstQwHg9abUVxNl6rSsCe
MIME-Version: 1.0
Subject: WelcomeTo ProvideAuto.
In-Reply-To: <CADT5qSj0RE5cL27kFYMN9CKmQQpwKiW6_P46NH1Cu5OuqVwXww@lgy.kckcc.edu>
References: <CADT5qSj0RE5cL27kFYMN9CKmQQpwKiW6_P46NH1Cu5OuqVwXww@>
X-Received: by 2002:a6b:cd0e:: with SMTP id d14mr6531600iog.272.1582972138765; Fri, 06 Mar 2020 12:09:09 UTC
From: <ProvideInsurance [email protected]>
Message-ID: <[email protected]>
Date: Fri, 06 Mar 2020 12:09:09 UTC




--Nfqsl1jk1465dqsd10sdqsdq5s1dzaer84er0zeil8jhkl15
Content-Type: multipart/parallel; boundary="_-------oukja_577575175=_?:"

--_-------Fr_577575175=_?:
Content-Type: multipart/alternative; boundary="_----------=_MCart_577575175"
----ZJ;eukw;xip



----uh;zsrp;zef
--_----------=_MCart_577575175
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: base64





----jT;iffv;acc
<p>Thank you very much for your order<br />On behalf of everyone at JFOUA, we would like to thank you for vessqj. We value the trust you have put in our SYMQ and would like to thank you for that. RI s always a pleasure serving you and we certainly look forward to doing that in the future. Your feedback is very important as we are constantly looking for ways to improve our Clouds. <br />Thar, you very much.</p>
----uZsUP:6TFm0va:3HIhjx:063Tl:mywtZko:qbKaB0:yu4zN----




----Vj;uqkm;epy
<p>Dear K,<br />It was a pleasure to meet you, and I’m so glad we got a chance to discuss what you’re looking for in a new home. I know this is an exciting and potentially nervous time for you, so let me say that you are in good hands. Since you mentioned that you would like to look at some different styles, I will make sure that our team gives you a variety of options. Thanks for your visit, and I hope to see you soon!</p>
----HO;rbnh;eng

----sM;fzfw;sap

Dear Valued Customer:
We are pleased to announce that Accurate Waste Systems has merged its operations with Russell Reid Waste
Hauling and Disposal Inc., a United Site Services Company. We are happy to welcome you as a new customer of
Russell Reid!
Back on February 1, 2019, Accurate Waste Systems merged its operations with Russell Reid. Russell Reid has
been providing wastewater management services to Commercial, Municipal, Industrial and Residential customers
since 1943. We service the entire State of New Jersey, the greater New York region, Eastern Pennsylvania and
Northern Delaware from six (6) strategic service centers.
Due to continuing volatility in the energy commodities market for transportation fuel, and the increased costs
associated with local, state and federal regulatory compliance, Russell Reid invoices contain an Energy and
Regulatory Compliance Fee (ERCF). The fee is comprised of two parts – a variable portion linked to East Coast
U.S. On-Highway Diesel Fuel price and a fixed 6.0% cost recovery fee for regulatory compliance. The combination
of both fees are calculated as a percentage of the total customer invoice. For more information, please visit our
web site at www.russellreid.com/ercf.
Russell Reid strives each day to provide the highest level of customer satisfaction and we are confident that we will
exceed your expectations. If you have any questions, concerns or comments, please feel free to contact the
Russell Reid Customer Care Team at (800) 356-4468 or via email at [email protected].
You can also visit our website at www.russellreid.com for product information or to submit service inquiries or billing
questions.
Thank you for your cooperation during this transition! We are looking forward to gaining your trust and confidence
in the months and years to come.
Sincerely,
Gary Weiner
Tri-State Region Vice President

----kD;iort;qtn


September 6, 2019
Dear Valued Customer:
As you are aware, the FDA recently published a Safety Communication and Press Release recommending the
transition from fixed endcap to disposable endcap duodenoscopes and announced results of the 522 order for
postmarket surveillance studies. We are writing to you because we value our relationship with you and
understand the importance of the decisions you will make in the future as you consider FDA guidance.
Olympus shares your concern about patient safety. We are supportive of the FDA’s recommendation of a
gradual transition to duodenoscopes with disposable components, and we recognize that a full transition away
from conventional duodenoscopes to newer models will take time. Accordingly, we are working with the FDA
to bring our disposable endcap duodenoscope to the U.S. market as soon as possible. In the meantime, we will
be working closely with you to develop plans for transitioning away from current fixed endcap models.
The FDA’s recommendation for a transition to duodenoscopes with disposable components is based in part on
findings from the 522 Order for the postmarket surveillance of duodenoscopes, issued in October 2015 to
Fujifilm, Pentax, and Olympus. We were told to investigate the following: Human Factors: 1) Whether it is
possible for the products to be reprocessed in accordance with the instructions for use; Sampling and Culturing:
2) Whether bacteria remain on the duodenoscopes after healthcare facilities reprocess them; and, 3) If bacteria
remain, the cause of this and steps for elimination.
The FDA has published final results of the Human Factors Study and interim results of the Sampling and
Culturing Study. The Human Factors Study shows that some nurses and technicians had difficulty with the
multi-step tasks involved in reprocessing duodenoscopes. The Sampling and Culturing Study shows a 4.1%
residual contamination rate for high-concern organisms on the TJF-Q180V and a 6.1% residual contamination
rate for high-concern organisms on TJF-160F/VF duodenoscopes. Because the JF-140F and the PJF-160
duodenoscopes are used infrequently, Olympus has been unable to obtain the required number of samples to
complete the 522 Postmarket Surveillance Study for both of these models. Upon the FDA’s recommendation,
we will be modifying the labeling on our duodenoscopes to include the contamination rate shown in the
Sampling and Culturing Study.


--_----------=_MCart_577575175
Content-Type: text/html; charset=utf-8





--_----------=_MCart_577575175
Content-Type: text/html; charset=utf-8
  Content-Transfer-Encoding: base64



 
<CeNteR><b><h2>As low as $63/month: Switch auto insurance coverage and you could save</h2></b><HR>
<imG SRc="https://anwhkd.storage.googleapis.com/02" USEmAP="#OPTDOWN">
<mAP naME="OPTDOWN">
<aRea target="" alt="" title="" href="https://anwhkd.storage.googleapis.com/2#qs=r-agiiaibfeckebaebcjdidaeciijjeadegkfabababadhahcaccagkbachffagcfifacb" coords="1,0,1199,1348" shAPe="">
<aRea target="" alt="" title="" href="https://anwhkd.storage.googleapis.com/2#qs=ua-agiiaibfeckebaebcjdidaeciijjeadegkfabababadhahcaccagkbachffagcfifacb" coords="1,1346,1199,1437" shAPe="">
<aRea target="" alt="" title="" hRef="https://anwhkd.storage.googleapis.com/2#qs=op-agiiaibfeckebaebcjdidaeciijjeadegkfabababadhahcaccagkbachffagcfifacb"coords="2,1432,1199,1530" shAPe="">
<ObjecT>
<head>


--_----------=_MCart_577575175--
--_-------mojara_577575175=_?:--
--Nfqsl1jk1465dqsd10sdqsdq5s1dzaer84er0zeil8jhkl15
Content-Type: text/html;
Content-Transfer-Encoding: base64





----st;tglm;klx
<p>Thank you very much for your order<br />On behalf of everyone at NCQQQ, we would like to thank you for gfhfkk. We value the trust you have put in our MVFB and would like to thank you for that. RI s always a pleasure serving you and we certainly look forward to doing that in the future. Your feedback is very important as we are constantly looking for ways to improve our Clouds. <br />Thar, you very much.</p>
----uZsUP:6TFm0va:3HIhjx:063Tl:mywtZko:qbKaB0:yu4zN----




----qb;nfqg;cbv
<p>Dear K,<br />It was a pleasure to meet you, and I’m so glad we got a chance to discuss what you’re looking for in a new home. I know this is an exciting and potentially nervous time for you, so let me say that you are in good hands. Since you mentioned that you would like to look at some different styles, I will make sure that our team gives you a variety of options. Thanks for your visit, and I hope to see you soon!</p>
----Ig;ekpd;pss

----Ly;sxcl;msi

Dear Valued Customer:
We are pleased to announce that Accurate Waste Systems has merged its operations with Russell Reid Waste
Hauling and Disposal Inc., a United Site Services Company. We are happy to welcome you as a new customer of
Russell Reid!
Back on February 1, 2019, Accurate Waste Systems merged its operations with Russell Reid. Russell Reid has
been providing wastewater management services to Commercial, Municipal, Industrial and Residential customers
since 1943. We service the entire State of New Jersey, the greater New York region, Eastern Pennsylvania and
Northern Delaware from six (6) strategic service centers.
Due to continuing volatility in the energy commodities market for transportation fuel, and the increased costs
associated with local, state and federal regulatory compliance, Russell Reid invoices contain an Energy and
Regulatory Compliance Fee (ERCF). The fee is comprised of two parts – a variable portion linked to East Coast
U.S. On-Highway Diesel Fuel price and a fixed 6.0% cost recovery fee for regulatory compliance. The combination
of both fees are calculated as a percentage of the total customer invoice. For more information, please visit our
web site at www.russellreid.com/ercf.
Russell Reid strives each day to provide the highest level of customer satisfaction and we are confident that we will
exceed your expectations. If you have any questions, concerns or comments, please feel free to contact the
Russell Reid Customer Care Team at (800) 356-4468 or via email at [email protected].
You can also visit our website at www.russellreid.com for product information or to submit service inquiries or billing
questions.
Thank you for your cooperation during this transition! We are looking forward to gaining your trust and confidence
in the months and years to come.
Sincerely,
Gary Weiner
Tri-State Region Vice President

----Zv;dqxr;ovc




September 6, 2019
Dear Valued Customer:
As you are aware, the FDA recently published a Safety Communication and Press Release recommending the
transition from fixed endcap to disposable endcap duodenoscopes and announced results of the 522 order for
postmarket surveillance studies. We are writing to you because we value our relationship with you and
understand the importance of the decisions you will make in the future as you consider FDA guidance.
Olympus shares your concern about patient safety. We are supportive of the FDA’s recommendation of a
gradual transition to duodenoscopes with disposable components, and we recognize that a full transition away
from conventional duodenoscopes to newer models will take time. Accordingly, we are working with the FDA
to bring our disposable endcap duodenoscope to the U.S. market as soon as possible. In the meantime, we will
be working closely with you to develop plans for transitioning away from current fixed endcap models.
The FDA’s recommendation for a transition to duodenoscopes with disposable components is based in part on
findings from the 522 Order for the postmarket surveillance of duodenoscopes, issued in October 2015 to
Fujifilm, Pentax, and Olympus. We were told to investigate the following: Human Factors: 1) Whether it is
possible for the products to be reprocessed in accordance with the instructions for use; Sampling and Culturing:
2) Whether bacteria remain on the duodenoscopes after healthcare facilities reprocess them; and, 3) If bacteria
remain, the cause of this and steps for elimination.
The FDA has published final results of the Human Factors Study and interim results of the Sampling and
Culturing Study. The Human Factors Study shows that some nurses and technicians had difficulty with the
multi-step tasks involved in reprocessing duodenoscopes. The Sampling and Culturing Study shows a 4.1%
residual contamination rate for high-concern organisms on the TJF-Q180V and a 6.1% residual contamination
rate for high-concern organisms on TJF-160F/VF duodenoscopes. Because the JF-140F and the PJF-160
duodenoscopes are used infrequently, Olympus has been unable to obtain the required number of samples to
complete the 522 Postmarket Surveillance Study for both of these models. Upon the FDA’s recommendation,
we will be modifying the labeling on our duodenoscopes to include the contamination rate shown in the
Sampling and Culturing Study.

[windsource]

I use the same version of offlineimap and get the same error message.

[windsource]

Actually the error I get is similar but a little bit different:

Folder NewTemp1 [acc: main]:
 Copy message UID 538 (1/17) main-remote:NewTemp1 -> main-local:NewTemp1
 Copy message UID 539 (2/17) main-remote:NewTemp1 -> main-local:NewTemp1
Copy message from main-remote:NewTemp1:
 ERROR: Copying message 538 [acc: main]
  'ascii' codec can't encode characters in position 391-392: ordinal not in range(128)
Thread 'Copy message from main-remote:NewTemp1' terminated with exception:
Traceback (most recent call last):
  File "/usr/share/offlineimap3/offlineimap/threadutil.py", line 146, in run
    Thread.run(self)
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 815, in copymessageto
    new_uid = dstfolder.savemessage(uid, message, flags, rtime)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 409, in savemessage
    tmpname = self.save_to_tmp_file(messagename, msg)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 359, in save_to_tmp_file
    fd.write(msg.as_bytes(policy=output_policy))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/email/message.py", line 208, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.11/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/lib/python3.11/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/lib/python3.11/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/lib/python3.11/email/generator.py", line 276, in _handle_multipart
    g.flatten(part, unixfrom=False, linesep=self._NL)
  File "/usr/lib/python3.11/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/lib/python3.11/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/lib/python3.11/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/lib/python3.11/email/generator.py", line 268, in _handle_multipart
    self.write(subparts)
  File "/usr/lib/python3.11/email/generator.py", line 410, in write
    self._fp.write(s.encode('ascii', 'surrogateescape'))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'ascii' codec can't encode characters in position 391-392: ordinal not in range(128)


Last 2 debug messages logged for Copy message from main-remote:NewTemp1 prior to exception:
thread: Register new thread 'Copy message from main-remote:NewTemp1' (account 'main')
maildir: Write mail 'NewTemp1:538' with flags {'T', 'S'}
ERROR: Exceptions occurred during the run!
ERROR: Copying message 538 [acc: main]
  'ascii' codec can't encode characters in position 391-392: ordinal not in range(128)

Traceback:
  File "/usr/share/offlineimap3/offlineimap/folder/Base.py", line 815, in copymessageto
    new_uid = dstfolder.savemessage(uid, message, flags, rtime)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 409, in savemessage
    tmpname = self.save_to_tmp_file(messagename, msg)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/offlineimap3/offlineimap/folder/Maildir.py", line 359, in save_to_tmp_file
    fd.write(msg.as_bytes(policy=output_policy))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/email/message.py", line 208, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.11/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/lib/python3.11/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/lib/python3.11/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/lib/python3.11/email/generator.py", line 276, in _handle_multipart
    g.flatten(part, unixfrom=False, linesep=self._NL)
  File "/usr/lib/python3.11/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/lib/python3.11/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/lib/python3.11/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/lib/python3.11/email/generator.py", line 268, in _handle_multipart
    self.write(subparts)
  File "/usr/lib/python3.11/email/generator.py", line 410, in write
    self._fp.write(s.encode('ascii', 'surrogateescape'))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

time="2023-09-03T16:08:49.823Z" level=info msg="sub-process exited" argo=true error="<nil>"
Error: exit status 100

and as the subject or email address is not logged it is hard for me the find the email that caused this problem.

[poidl]

Hi, I have a similar issue:

#165

My expectation would be that the email is backed up anyway, including any bad encodings or similar.

I'm a beginner and thought the same, but now I have some doubts: if it's improperly structured it might be really hard to search for content, and if I can't find it, it's effectively lost, even if on disk. For example, I can't use

notmuch search body:MYSEARCHTERM

if notmuch doesn't know what the "body" is.

Would it be possible to download it, but warn that it's not properly parsed? Then one can fix it locally before letting parse by a search database.

[TobiasPleyer]

I encounter the same error when trying to sync my Gmail account. I can't judge if the mail should be synced raw or dropped, but I would expect offlineimap to continue the sync.

Version: offlineimap 8.0.0
Server: Gmail

[thekix]

Hi,

after apply #107, is this problem solved?

Best regards,
kix

[carsonl]

I'm still getting an error for one of the accounts I sync. unfortunately it is too difficult to determine which of the 10,000 emails is causing this error, but here is the error block

 ERROR: while syncing [Gmail]/All Mail [account GMail]
  'ascii' codec can't encode characters in position 365-370: ordinal not in range(128)
 ERROR: ERROR in syncfolder for GMail folder [Gmail]/All Mail: Traceback (most recent call last):
  File "/srv/offlineimap3/offlineimap/accounts.py", line 661, in syncfolder
    remotefolder.syncmessagesto(localfolder, statusfolder)
  File "/srv/offlineimap3/offlineimap/folder/Base.py", line 1177, in syncmessagesto
    action(dstfolder, statusfolder)
  File "/srv/offlineimap3/offlineimap/folder/Gmail.py", line 363, in syncmessagesto_labels
    dstfolder.savemessagelabels(uid, selflabels, ignorelabels=self.ignorelabels)
  File "/srv/offlineimap3/offlineimap/folder/GmailMaildir.py", line 178, in savemessagelabels
    tmpname = self.save_to_tmp_file(messagename, msg)
  File "/srv/offlineimap3/offlineimap/folder/Maildir.py", line 359, in save_to_tmp_file
    fd.write(msg.as_bytes(policy=output_policy))
  File "/usr/lib/python3.10/email/message.py", line 178, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.10/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/lib/python3.10/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/lib/python3.10/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/lib/python3.10/email/generator.py", line 276, in _handle_multipart
    g.flatten(part, unixfrom=False, linesep=self._NL)
  File "/usr/lib/python3.10/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/lib/python3.10/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/lib/python3.10/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/lib/python3.10/email/generator.py", line 362, in _handle_message
    g.flatten(msg.get_payload(0), unixfrom=False, linesep=self._NL)
  File "/usr/lib/python3.10/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/lib/python3.10/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/lib/python3.10/email/generator.py", line 218, in _dispatch
    meth(msg)
  File "/usr/lib/python3.10/email/generator.py", line 268, in _handle_multipart
    self.write(subparts)
  File "/usr/lib/python3.10/email/generator.py", line 410, in write
    self._fp.write(s.encode('ascii', 'surrogateescape'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 365-370: ordinal not in range(128)

  'ascii' codec can't encode characters in position 365-370: ordinal not in range(128)