BETA: Outlook 2003 doesn't generate Message-Id headers

[Malte S. Stretz]

Hi,

I'm one of the developer of SpamAssassin [1], a free Spam filter for Unix
and -- with SAproxy [2] -- Windows. SpamAssassin tries to divide mail into
spam and ham (aka non-spam ) by checking the mails structure and contents
against a set of typical pattern.

Till now did we have some pretty reliable rules to check for faked Outlook
mail. Part of those is a check against the typical Message-Id format.
Lately do we get more and more reports telling us that legit mail sent with
the Outlook 2003 beta is flagged as spam because it seems like Outlook
doesn't generate a Message-Id header anymore but relies on the server to do
so. The problem is filed as bug 1970 in our bug tracker [3].

I guess some of our users formulated the real problem (not only for us, also
for Microsoft) best:

16:45 2003, Tony Finch wrote:
| > Lots of headers that SHOULD be present in normal SMTP don't have to be
| > present at message submission time. I think it's fine for the MUA to
| > rely on the MTA to add the message-ID; in fact the MTA is probably
| > better at choosing a good one than the MUA.
|
| I understand that point.
|
| But Outlook has been generating its own Message-ID for years, so to
| remove it looks unusual. So either someone at MS has made a
| deliberate decision to remove it (in which case their answer will be
| "it's a design decision"), and that's fine. Or for some reason, it
| has been unintentionally removed ("Oops, we'll put that back"). But
| if I were on the beta program, I'd want to report it based on the
| possibility that it is a bug. To me, it's certainly "an unexpected
| change in behaviour" that merits being queried.

On Saturday 25 July 2003 15:53 CET Brian White wrote:
| Has anybody written to Microsoft telling them that the change they are
| making will cause Outlook mail to be "rejected as spam by X% of the
| world's email"? Perhaps they might consider fixing it if it will cause
| there users some grief. One can always hope, anyway.

I hope somebody with knowledge reads this group. Or can somebody give me an
official response address for problems with the beta? Somebody from the
devteam with a clue would be the best ;-)

Cheers,
Malte

[1]http://spamassassin.org
[2]http://saproxy.bloomba.com/
[3]http://bugzilla.spamassassin.org/show_bug.cgi?id=1970

 

[Malte S. Stretz]

A number of people have objected to this for two major reasons:

1) Revealing internal machine names provides information that hackers can
potentially use to compromise the network.
2) They don't want to reveal their employer when sending mail via their
ISP from work, and a message id generated by Outlook would contain the
domain name of their employer.

We felt that the requests to change this were very valid, and thus changed
Outlook so that it relies on the SMTP server to generate the message id.

Hmmm... these definitely sound like valid reasons not to add the Message-Id
at the client side but I don't think they will really fix the "problem" of
internal information about the network "leaking" to the outside world.

Because as you both pointed out, if there's no Message-Id added by the MUA,
the first MTA (I don't know how Exchange handles these messages internally)
has to add it, thus generating a Message-Id of <[email protected]>
instead of <[email protected]>. Additionally will most server add a
Received header which (of course optionally) contains the name of the
machine the mail was received "from" (paul) and the server itself, probably
including IP-addresses.

And not forget to point out that this might backfire to the Outlook users
because there are quite many spam filters out there which aren't as
sophisticated as SpamAssassin and do reject all mail containing a
Message-Id generated by a server at SMTP level. (Yes, I know that's on one
level as rejecting every mail containing the word "VIAGRA" or everything
sent from Dial-Up IPs, but tell that those sysadmins from whose system I
receive loads of rejected mail because some buffoon of a spammer sent some
spam with my address in the From line.)

And what's even worse, if you look at one of those header samples posted at
the bug ticket I referred to in the first posting, you will see that the
Yahoo! servers obviously *don't* add any Message-Id if they receive a mail
lacking that header thus "generating" invalid mails. I know, this is not
the failure of the Outlook developers, but I don't know how many other
servers with this Bad Behaviour(TM) are out there which might in the end
hit back on the Outlook users when some other server down the line might
reject those mails as invalid as they contain at least one Received header
but no Message-Id at all.

So I don't know if it's really such a good idea not to generate the header
on the client side. In the end is Outlook the most used MUA worldwide and
some (too many) sysadmins expect the mail world to work like Outlook does.
So the next Outlook not behaving like Outlook anymore is quite unexpected
and might for the first few months/years result in users who pester their
MSCEs because their "Mail doesn't work anymore" (ie. some server rejected
their stuff).

I think the better solution would be to give the sysadmins concerned about
privacy an option in Exchange to re-generate an "anonymous" Message-Id at
server level. That would be a much cleaner solution as it would also stop
information leaking from users who use some different/older MUA which still
generates that header.

Cheers,
Malte

 

[Jeff Stephenson [MSFT]]

Not generating the Message-ID does protect the internal network - servers
are much more likely to be hardened than client workstations, and as you say
they're already revealing themselves in Received lines. What is not
revealed by Outlook not writing the Message-ID is the typically more
vulnerable Joe User workstation. Many corporate server environments will
either not write the original client submission information in the Received
line or will rewrite the entire Received history when the message reaches
the boundary between the corporate intranet and the Internet so as to remove
that internal machine information.

As to the argument of not changing the Message-ID format because there are
anti-spam programs that verify that against, say, X-Mailer, how long do you
think it would be before the majority of spammers caught on to that anyway?
Used to be that most spam was incredibly ill-formed, but clearly they're
getting more and more sophisticated and anti-spam programs clearly have to
deal with that. Saying that Outlook should no longer change to meet
customer needs because it might break an anti-spam program somewhere just
isn't going to fly, IMHO.

If Yahoo's servers aren't doing the right thing and adding a Message-ID,
they need to be reconfigured to do so. As has been pointed out, RFC 2822
applies to server-server interactions, not client-server interactions, so
it's necessary for servers to ensure the validity of the messages they send.
Yahoo is apparently not doing so.

 

[Johnny Walker]

Neo's right (I'm the developer he refers to). We made this change because
we've had a number of complaints about revealing internal machine names in
the Message-IDs we generated.

---- snip ----

We felt that the requests to change this were very valid, and thus changed
Outlook so that it relies on the SMTP server to generate the message id.

Thanks, Jeff. This is good information.

1) Is there any ability to implement this behavior now in Outlook 2002,
via registry key entry or modification? (For example, An internet search
reveals info on an old registry switch for suppressing locally generated
Message-IDs in MS Internet News...)

2) Additionally, consider providing for the ability to turn on, and
optionally influence the structure of, Outlook generated Message-IDs in
Outlook 2003 to form a message ID consisting of a string which includes
the domain shown in the From field settings of the account currently
sending the message (TheBat! does this).


Thanks,
JW

 

[Jeff Stephenson [MSFT]]

1) No, Outlook 2002 will always generate a Message-ID.
2) I've got some ideas for alternatives that do generate a Message-ID while
addressing the privacy concerns, but those won't be in Outlook 2003 (we
wouldn't take a change of that nature at this stage). And whatever we do is
bound to bother somebody, unfortunately. Adding options is always a
problem, as every option is another node at which some weird bug can crop
up...