Thunderbird splits email addresses with non-ascii characters and a comma in From: field

Oops, sorry, I'm a newbie... I should add a few details...

The problem is only apparent because of the issues with commas (",") but it is
believed that it might be caused by a slight interpretation problem as to how to
deal with email address which contain things which under RFC (2)822 should be
quoted but because they contain non-ASCII characters must be encoded using RFC 2047.

In that case, it is believed that quoting should not be added before encoding
the email address using RFC 2047 (which unfortunatly Mozilla does).

It also expects to see that quoting when it receives emails and if it is missing
 and the "From:" line contains a comma (",") it think that the "From:" line
contains two email address (which cause problem when replying and when
displaying the list of messages (only one email adddress is shown)).

This is an example of an email address Mozilla has trouble with:

From: =?iso-8859-1?Q?Martin=2C_Andr=E9?= <email address hidden>

which get converted into this

From: Martin, André <email address hidden>

Due to the presence of an unquoted "," (comma) when we try to reply it thinks
we're trying to reply to two addresses...

This is what Mozilla generates when sending an email:

From: =?ISO-8859-1?Q?=22Martin=2C_Andr=E9=22?= <email address hidden>

Appears to be a dupe of bug 231732 / bug 252240.

I found it somewhat related to bug Bugzilla Bug #249626 (as I had mentionned
below) though it wasn't logged under the same product (in that case it was
Thunderbird) and the problem was concerning appearance of the "To:" line.

It is also similar to Bugzilla Bug 122972.

It does indeed have similarities with Bugzilla bug 231732 / bug 252240 though I
cannot say if it is for the same reason (in my case it is because of an encoded
comma (",").

There might be, indeed, a part of this bug report that could be considered a
dupe if the reasons are the same but the fact that the "From:" line, as
generated by Mozilla is not propoerly encoded is not.

Keith Moore, RFC 2047 author's comment on this was:

<<Adding quotes around the name before encoding doesn't solve the problem, and
actually causes other problems.>>

I don't know what Keith Moore could mean by that. RFC2047 (section 5, item 3)
states:
  + An 'encoded-word' MUST NOT appear within a 'quoted-string'.

Quotes are required by RFC822, since without quoting, the comma is a delimiter
between addresses -- as you see. Since we cannot include the encoded-word
*within* the quoted-string, the quotes must be part of the encoded-word.

This is indeed the same issue as bug 122972 and bug 249626 (Thunderbird and the
suite share this code). Those bugs are also invalid.

Bug 231732 is unrelated.

These are quotes from the emails we exchanged (used with permission):

Keith Moore says:

<<Any_ MUA that tries to "convert" something encoded in RFC 2047 to a form
without encoding is broken. There's no way to convert every 2047-encoded
message to an unencoded message with valid syntax. 2047 is only intended to be
used for _display_ purposes, not to convert messages into a form which can be
parsed by a normal mail parser.

Adding quotes around the name before encoding doesn't solve the problem, and
actually causes other problems. >>

In the next part, the questions are by me, the answer is given by him:

<<
>So in the case of the "From:" line, what should MUA authors do?
>>
>> - Convert RFC 2047 encoded messages to an unencoded message, quoting it as
>> necessary to make it look like RFC (2)822 compliant (except for the presence of
>> non-ASCII characters).

no. (or maybe, this is a last-resort option for patching legacy code where
it's infeasible to do it right)

>>
>> [otherwise it causes problem with the MUA email address parsers]
>>
>> - Convert RFC 2047 encoded messages to an unencoded message without bothering
>> about syntax for display but quoting it, as necessary, when using it to reply.
>> That quoting would actually be temporary as it would have to be removed before
>> encoding it using RFC 2047 (once again, this is to please these MUA address
>> parsers).

no.

>>
>> - Use the RFC 2047 encoded message for display but remove the RFC 2047 encoded
>> part when replying (keeping only what's between the "<" ">").

definitely not.
>>

This part is interesting:

<<
here's what should be done in MUAs:

always maintain the message in original format. the 2047-decoding routines
should be used only by the code that displays messages.

replies should be based on the unencoded message headers. that way, the
exact same 2047 encoding is used on a reply as was used on the message
being replied to.

the 2047-encoding routines should only be called from the code that handles
message composition.

message stores that do searching (as in, those that support IMAP) probably
need to do it differently. IMHO, they need to store messages in original
format but build indices based on text extracted from decoding headers
(and for that matter, body parts).
>>

His ok for quoting him:

<<
>> Do I have your permission to quote your reply when contacting the companies in
>> question?

yes.
>>

There was a small typo in what he said above so I asked for precisions:

<<
>Your reply (or at least my interpretation of it) suggests that you might
>> mean "encoded message headers" (that would be the only way to be
>> *entirely* sure that we use the exact same 2047 encoding.

yes, I meant to say "undecoded message headers".
>>

He was a very nice person to deal with:

<<
>> Sorry for taking so much of your time...

it's really no problem. I'm very familiar with this topic and I don't
have to think much about it ... and I type reasonably fast :)

Keith
>>

There seems to be a small interpretation problem of that RFC. If what I copied
above doesn't provided you with the answers you seek I can forward your question
to him...

Let me...

Thanks for your insistence. I have read RFC2047 in more depth; section 6.2, in
particular the NOTE there, supports your argument.

For now, I'll leave this bug as it stands, but it's possible the encoding
portion (Mozilla should not place quotes within the encoded string) is a
separate bug from the decoding portion (Mozilla should treat encoded-words as
atoms).

*** Bug 122972 has been marked as a duplicate of this bug. ***

*** Bug 249626 has been marked as a duplicate of this bug. ***

I agree that adding quotes to a name before encoding is a separate issue than
how an MUA handles decoding and presentation of 2047 encoded-words.

Adding quotes to a name before encoding that name using 2047 is not strictly
wrong, but it can produce annoying cruft on a recipient's display. (there are
lots of annoying things that MUAs do to people's names - this is just another
one)

On the other hand, if a sender's MUA adds double quotes before encoding it might
help to work around bugs in some recipients' MUAs that try to decode 2047
encoded-words and then parse the resulting text.

-- Keith Moore <email address hidden>

I will begin by saying that I should indeed have opened two different bug
reports as, even though the issues are are related, this surely affects two very
different part of the code (I do believe that they should be corrected at the
same time however).

Mike Cowperthwaite said:

> Thanks for your insistence

No problem... I use Mozilla, I like it and I wanted it to get it right...

Keith Moore said (Hi!):

> Adding quotes to a name before encoding that name using 2047 is not strictly
> wrong, but it can produce annoying cruft on a recipient's display. (there are
> lots of annoying things that MUAs do to people's names - this is just another
> one)
>
> On the other hand, if a sender's MUA adds double quotes before encoding it
> might help to work around bugs in some recipients' MUAs that try to decode
> 2047 encoded-words and then parse the resulting text.

I hope you won't mind if I have a different opinion on this (especially after
harassing you so much... (-; or )-; ? ).

If the only reason for these quotes is to work around bugs in other MUA than I
think that they *should not* be added. If we use workarounds instead of
correcting the problem then they *will never* correct it and eventually assume
that the ones who got it right were wrong.

What I believe is (unfortunatly) the most popular MUA already got it right,
Mozilla will (eventually) get it right and other MUA are already getting it
right when they *send* their messages.

If one wants to have quote to workaround another MUA's bug then they could type
them in Mozilla's "Your name:" field.

What do you think?

Thanks!

Nick

PS: Please forgive my English as it is not my mother tongue. Thanks!

That should have been MUAs and quotes...

I either neeed to augment my caffeine intake or take a few days off... (-;

*** Bug 256958 has been marked as a duplicate of this bug. ***

*** Bug 191102 has been marked as a duplicate of this bug. ***

*** Bug 263976 has been marked as a duplicate of this bug. ***

*** Bug 264151 has been marked as a duplicate of this bug. ***

*** Bug 264665 has been marked as a duplicate of this bug. ***

*** Bug 272093 has been marked as a duplicate of this bug. ***

*** Bug 274603 has been marked as a duplicate of this bug. ***

*** Bug 274561 has been marked as a duplicate of this bug. ***

From Bug 274561.
I tested Bug 274561 case(same problem when To: header)
> To: =?iso-8859-1?Q?Fa=2E_St=FCrtz=2C_Engel?= <email address hidden>
And problem was re-created.
Then I tested escaped "," case.
> To: =?iso-8859-1?Q?Fa=2E_St=FCrtz=5C=2C_Engel?= <email address hidden>
  - "\"(=5C,escape char) is inserted before ","(=2C)
Although another problem occured in this test,
  - Display becomes Fa. Stürtz\\, Engel
    Special data was displayed(escape char in this case).
    This is already reported problem.
problem of this bug was resolved.
Mozilla handled "," correctly when "," was escaped before encoding.

Sorry but I'm not familiar with Quoted Printable, so I don't know whether result
of inserting "=5C" only is valid QP or not.
And I can not say RFC 2822's spec is which, "," in atom should be escaped before
encoding, or no need to escape.

*** Bug 277606 has been marked as a duplicate of this bug. ***

*** Bug 279088 has been marked as a duplicate of this bug. ***

*** Bug 280624 has been marked as a duplicate of this bug. ***

As I said comment #20, if "," in encoded-word is escaped, Mozilla/Thunderbird
doesn't split email addresses.
(My understanding of RFC is now : "," in encoded-word needn't be escaped.)
This indicates that parsing is done after decoding of encoded-word.
I think this is the main cause of RFC violation, encoded-word is not treated as
atom.
Order of parsing and decoding of ecoded-word should be reverted.

*** Bug 206491 has been marked as a duplicate of this bug. ***

More examples: Thunderbird Version 1.0 (20041206)

Working:

=?ISO-8859-1?Q?Andreas_Niederl=E4nder?= <email address hidden>

Not working:

"Biehl, Alexander" <email address hidden>

=?iso-8859-1?Q?Altpeter-R=FCb_Dr=2E_Herbert=2C_ITS-IT?=
<email address hidden>

(In reply to comment #26)
> More examples: Thunderbird Version 1.0 (20041206)
>
> Working:
>
> =?ISO-8859-1?Q?Andreas_Niederl=E4nder?= <email address hidden>

The problem is not with embedded spaces in the encoded string, but with commas.

> Not working:
>
> "Biehl, Alexander" <email address hidden>

Testing this, it is "working" for me -- that is, replying to a from header
containing that string, the To: in the reply matches that string.

> =?iso-8859-1?Q?Altpeter-R=FCb_Dr=2E_Herbert=2C_ITS-IT?=
> <email address hidden>

This last example is basically the same as that described in the original
report.

*** Bug 292603 has been marked as a duplicate of this bug. ***

*** Bug 292826 has been marked as a duplicate of this bug. ***

*** Bug 295379 has been marked as a duplicate of this bug. ***

*** Bug 292008 has been marked as a duplicate of this bug. ***

*** Bug 299640 has been marked as a duplicate of this bug. ***

*** Bug 301136 has been marked as a duplicate of this bug. ***

is there any activity to this bug? I still have the same problem with the new
Thunderbird 1.5beta version...

*** Bug 312977 has been marked as a duplicate of this bug. ***

I believed this first was a bug in Outlook 2003, and managed to report it to them (not as easy as here...). Their answer when it comes to the decoding part is that tokenization should take place before decoding. This is according to RFC 2047 section 6.2.

My own screen dumps and Microsofts internal bu report on this can be found on Mozillazine: http://forums.mozillazine.org/viewtopic.php?t=318281

Is there anything I can do to help fix this problem? (Testing, QA, ...)

*** Bug 322449 has been marked as a duplicate of this bug. ***

*** Bug 310547 has been marked as a duplicate of this bug. ***

*** Bug 319422 has been marked as a duplicate of this bug. ***

Is anyone working on the bug? I found this bug to be very anoying. Unfortunately, I'm not familiar with Mozilla internals so I can't fix it myself.

(In reply to comment #0)
> When an email contains a "From:" line encoded with RFC 2047 which contains
> characters which should, per RFC (2)822, be quoted (because it contains commas
> (",") or similar characters), Mozilla doesn't decode it properly and thinks
> these are two addresses.

Verifying this has been fixed. I am seeing correct behavior in
 Gecko/20081207 Shredder/3.0b2pre
(Find it here:
  ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-central/
)

The name is correctly shown in the Recipient column, and replying to the message constructs the name as expected in the addressing widget of the compose window -- except, the name is quoted, as in...

> The reverse is also true. It quotes an email address which contains a comma
> and then apply RFC 2047 (which it shouldn't do).

The quotes aren't necessary for the MIME encoding, but I think they may be necessary for the addressing widget to parse correctly. If you simply type a comma-separated name into the field without the quotes, the widget automatically splits it into two separate addresses. If the addition of quotes is still a problem, that should go into a new bug -- if not every dupe to this bug is about the first problem, I bet 98% of them are.

Thanks very much, Martin and David.

Tested with Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20081207 Shredder/3.0b2pre and reply works.
But if i click on the sender in mail view and choose "compose mail to" in the dropdown menu, the address is still split into two addresses.

the "clicking on the sender" issue should be a new, spin-off bug. Do you mind filing that as a thunderbird front end bug?

(In reply to comment #162)

In my opinion as an end user this is the same bug.
But I don't know the code of thunderbird and therefore can't estimate in which part of thunderbird the fault is.

This FIXED bug is flagged with in‑testsuite? It would be great if assignee or someone else can clear the flag if a test is not appropriate. And if appropriate, create a test and plus the flag to finish off the bug.

*** Bug 490436 has been marked as a duplicate of this bug. ***

(In reply to comment #164)
> This FIXED bug is flagged with in‑testsuite? It would be great if assignee or
> someone else can clear the flag if a test is not appropriate. And if
> appropriate, create a test and plus the flag to finish off the bug.

ping ?

*** Bug 490689 has been marked as a duplicate of this bug. ***

Setting blocking‑seamonkey2.0b1?

IIRC, bug fixes from TB were supposed to be migrated to SeaMonkey before beta, but as of 2.0a3, this has not happened yet. This bug is

 (a) a blocker for enterprise use in non-English-language countries
     (cf. the number of votes, number of dupes and the bug spam
     by desperate victims), also blocking‑thunderbird3+

 (b) committed to TB fairly recently (2008-12-04 to Hg-trunk), so that
     it can serve as an indicator whether the bug fix migration
     has actually happened

There's no "migration" - tb and sm use the same code.

(In reply to comment #168)
> Setting blocking‑seamonkey2.0b1?
>
> IIRC, bug fixes from TB were supposed to be migrated to SeaMonkey before beta,
> but as of 2.0a3, this has not happened yet.

All the code here is shared between TB + SM. This bug also seems to have a unit test which we know hasn't regressed. Therefore not blocking. If you think you have a specific problem in SM I suggest you file a new bug for it and it can be covered there.

(In reply to comment #161)
> Tested with Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US;
> rv:1.9.1b3pre) Gecko/20081207 Shredder/3.0b2pre and reply works.
> But if i click on the sender in mail view and choose "compose mail to" in the
> dropdown menu, the address is still split into two addresses.

Just for the reference: I cannot confirm this with
Mozilla/5.0 (Windows; U; Windows NT 6.0; lt; rv:1.9.1b4pre) Gecko/20090414 Lightning/1.0pre Shredder/3.0b3pre

(In reply to comment #171)
> (In reply to comment #161)
> > Tested with Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US;
> > rv:1.9.1b3pre) Gecko/20081207 Shredder/3.0b2pre and reply works.
> > But if i click on the sender in mail view and choose "compose mail to" in the
> > dropdown menu, the address is still split into two addresses.
>
> Just for the reference: I cannot confirm this with
> Mozilla/5.0 (Windows; U; Windows NT 6.0; lt; rv:1.9.1b4pre) Gecko/20090414
> Lightning/1.0pre Shredder/3.0b3pre

Please clarify: Are you saying this bug is fixed or not?

(In reply to comment #172)
> (In reply to comment #171)
> > (In reply to comment #161)
> > > Tested with Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US;
> > > rv:1.9.1b3pre) Gecko/20081207 Shredder/3.0b2pre and reply works.
> > > But if i click on the sender in mail view and choose "compose mail to" in the
> > > dropdown menu, the address is still split into two addresses.
> >
> > Just for the reference: I cannot confirm this with
> > Mozilla/5.0 (Windows; U; Windows NT 6.0; lt; rv:1.9.1b4pre) Gecko/20090414
> > Lightning/1.0pre Shredder/3.0b3pre
>
> Please clarify: Are you saying this bug is fixed or not?
Reply to an RFC2047 encoded sender is fixed.

Right-click on sender address and choosing "Compose Mail To" is still broken, the new mail has two "to:"-addresses in it.

Tested on MacOSX:
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b5pre) Gecko/20090506 Shredder/3.0b3pre

(In reply to comment #173)
> Right-click on sender address and choosing "Compose Mail To" is still broken,
> the new mail has two "to:"-addresses in it.

Ok, so please raise that in a separate bug as previously requested.

(In reply to comment #172)
Yes, it works fine for me. Including the scenario described in comment #173.
Could that be OS-specific? Or maybe it has regressed between the dates?..

(In reply to comment #174)
> (In reply to comment #173)
> > Right-click on sender address and choosing "Compose Mail To" is still broken,
> > the new mail has two "to:"-addresses in it.
>
> Ok, so please raise that in a separate bug as previously requested.
Ok, I raised a new bug: Bug 491832

marking in testsuite +

*** Bug 501921 has been marked as a duplicate of this bug. ***

Please backport this patch to Thunderbird 2.0.0.x. Thunderbird 3 is still beta, and there already has been a testing period for this patch.

Conformance to an RFC should be enough of a reason to introduce this change into the stable branch.

Thanks alot for the work on this issue, it is crucial for acceptance in corporate environments.

(In reply to comment #179)
> Please backport this patch to Thunderbird 2.0.0.x. Thunderbird 3 is still beta,
> and there already has been a testing period for this patch.
>

3.0RC will come out next week. We just don't have the resources to backport and test for the next release of the 2.X branch.

*** Bug 474587 has been marked as a duplicate of this bug. ***

*** Bug 542194 has been marked as a duplicate of this bug. ***

The actual status is correct?
"VERIFIED FIXED"?
Still failing under 3.0.1

(In reply to comment #183)
> The actual status is correct?
> "VERIFIED FIXED"?
> Still failing under 3.0.1

Could you attach an example message please?

Hi!

Address at header:
=?iso-8859-1?Q?L=E1zaro=2C_Nuria?= <email address hidden>

Thunderbird decode this as:

Lázaro, Nuria <email address hidden>

The parsing of this string fails, identify two address "Lázaro", and "Nuria
<email address hidden>"

So sorry.

The computer used to test my software has a old version of Thunderbird. Updated to 3.0.1 like the other computers and works Ok, please ignore comments 183 and 185

*** Bug 563298 has been marked as a duplicate of this bug. ***

*** Bug 560886 has been marked as a duplicate of this bug. ***

Hi,

I'm using the (german) version 3.1.6 of Thunderbird on Windows XP.
The bug seems to be fixed when replying to a message by using the "Reply" button, the menue "Message/Reply" or the shortcut Ctrl+A.

BUUUT ....
Writing a new email with following procedure still fails:
1. Use the right mouse button at the sender's adress of a displayed email
2. Select the entry "Write email to" (resp. in German "Nachricht verfassen an")

I hope my comment helps to improve Thunderbird.

Just to confirm Ralf previous comment.

I have the same problem on Thunderbird 3.1.10 over Ubuntu 10.10 (64 bits).

This bug should be reopened.

Problem still exists in Thunderbird 3.1.14 on openSUSE 11.4 64 bit
Mozilla/5.0 (X11; U; Linux x86_64; de; rv:1.9.2.22) Gecko/20110907 SUSE/3.1.14 Thunderbird/3.1.14

Right-clicking the "From" field of a displayed mail with the "From" header:

From: =?ISO-8859-1?Q?Dr=2E_J=F6rg_Schr=F6per=2C_LANline?= <email address hidden>

results in a message composition window with two recipients:
"Dr. Jörg Schröper" and "LANline <email address hidden>".

(That's the sender address of a public newsletter so I think posting the unsanitized address is ok.)

*** Bug 444771 has been marked as a duplicate of this bug. ***

The problem still exists in 9.0.1 (Linux and Windows). Adresses with german umlauts like ä,ö,ü are displayed incorrectly in the main screen. In the message view

I attached a screenshot to show the issue in detail.

To complete my comment:
In the message view the address is displayed correctly.

Created attachment 589494
Screenshot

(In reply to Christian Reischl from comment #193)
> The problem still exists in 9.0.1 (Linux and Windows). Adresses with german
> umlauts like ä,ö,ü are displayed incorrectly in the main screen. In the
> message view
>
> I attached a screenshot to show the issue in detail.

I think that's likely to have been fixed by bug 669925 (although a slightly different issue, the fix would probably cover this as well).

If you can reproduce this in Thunderbird 10 beta, please file a new bug for it. Thunderbird 10 Beta is available here:

http://www.mozilla.org/en-US/thunderbird/channel/

Also generally note that it is not a good idea to comment on fixed/resolved bugs, comments will likely get lost.