SmtpClient and Outlook junk mail

[kleinma]

I have an issue where emails sent out via the SmtpClient class in .NET go directly into the outlook junk mail folder (Outlook 2003 and 2007).

If I send the exact same email (same body, same subject, to and from the same accounts as I use in .NET) from Outlook itself, it will come in fine and not be marked as junk mail.

The only difference is the extra headers that Outlook appends to the email when it is sent directly from Outlook.

I have sort of exhausted my searches in trying to find out how or why this happens and what I can do about it, so I figured I would throw the question out here just to see if anyone may have come across something like this previously.

[chris128]

Yep, those extra headers do affect spam filtering. This is supposed to get a bit better in .NET 4.0 (see http://blogs.msdn.com/ncl/archive/20...-net-mail.aspx)

[RobDog888]

Headers can be customized as you wish but I would just try to duplicate the headers Outlook sends and you should be good.

[kleinma]

That doesn't work unfortunatly. I can get some example headers/code to show you what is going on.

[kleinma]

Yep, those extra headers do affect spam filtering. This is supposed to get a bit better in .NET 4.0 (see http://blogs.msdn.com/ncl/archive/20...-net-mail.aspx)

interesting.. off to test on 2010 beta

[kleinma]

Running the code against .NET 4.0 actually resulted in it being marked as spam even worse.

When running against .NET 2.0 my host mail server (which has spam checking too) doesn't flag it as spam at all, only outlook does.

When running against .NET 4.0 my host marks it as spam (with a low spam rating), and so does outlook.

[RobDog888]

Good read on that blog.

"The ReplyTo property has been obsoleted."

Sounds incorrect grammar. I think they need my SpellChecker™.NET

Sure post up what you can.

[RobDog888]

Could it possibly also be the content you are sending?

[chris128]

haha I thought the exact same thing about that sentence rob

Klienma - I dont know if that blog relates to the BETA or the final release of .NET 4.0

[RobDog888]

The blog stated it would be added to the final version so RTM would be where we first get to see it.

It would have been better as "depreciated" instead of "obsoleted" as that is not even a word

[kleinma]

Actually it was not worse. I forgot to add a credentials property to the SMTP server. Which scared the crap out of me because then I was wondering how the mail could even be sent my the server when I specified only an SMTP address and no credentials, but it looks like if I send to and from addresses that are the same domain as the SMTP server, it works even with no credentials (but marks it as likely spam on the server) but sending to an outside domain email throws an exception when send() is called.

So anyway, adding a credentials property made it at least the same as .NET 2.0, however they both still send everything to the junkmail folder.

[kleinma]

The blog stated it would be added to the final version so RTM would be where we first get to see it.

It would have been better as "depreciated" instead of "obsoleted" as that is not even a word

Where did you see that it said it was not in the beta?

[RobDog888]

Did you try sending a blank or non-spam content email?

[kleinma]

The content is not spam. The subject is "this is a test" and the message is something similar.

Sending the exact same message from outlook works fine and it is NOT marked as spam. It has to be with the headers, not the content.

[RobDog888]

Where did you see that it said it was not in the beta?

I read it here...

Chris, you won't see that update in the Beta, but we have plans to address it before release.

Just noticed the date of the blog as Aug 6, 2009 so it could be in Beta 2

[kleinma]

That comment was specific to Chris asking about the QUIT command. I have to imagine the beta version I have does contain these new SMTP features, since the class does contain the ReplyToList collection property, and ReplyTo is marked as obsolete in what I have.

[RobDog888]

Bah your right. Multi-taking fail and reading the blog from the bottom up didnt help lol.

Time for lunch then

[chris128]

Actually it was not worse. I forgot to add a credentials property to the SMTP server. Which scared the crap out of me because then I was wondering how the mail could even be sent my the server when I specified only an SMTP address and no credentials, but it looks like if I send to and from addresses that are the same domain as the SMTP server, it works even with no credentials (but marks it as likely spam on the server) but sending to an outside domain email throws an exception when send() is called.

Yeah credentials only come into it when you attempt to 'relay' for a domain that the SMTP server does not handle mail for. The actually sending address can still be anything at all even without credentials, it is just the address that it is being sent to that must be one of the domains that the server is responsible for

[chris128]

Perhaps I should add the ability to specify whatever headers you want in my SMTP server class (http://www.vbforums.com/showthread.php?t=590429)

</shameless plug>

can you post the example headers that you were talking about? ie the headers when you send it through Outlook and the headers when you send from SmtpClient

[kleinma]

I can add headers no problem with the SMTP class. However the headers that are added by outlook contain a bunch of encrypted information. They don't do me any good as far as how I could simulate the same thing via SMTP.

[chris128]

Ahh I see, can you still post both versions anyway so I can see exactly what you mean?

[kleinma]

Yeah, I need to edit some info out, but hang on.

[kleinma]

Here is the header for when it is sent via SMTPClient:

Return-Path: <[email protected]>
Received: from MY-LOCAL-IP-HERE by HOST-MAIL-SERVER-HERE with SMTP;
   Thu, 5 Nov 2009 11:45:20 -0800
MIME-Version: 1.0
From: <[email protected]>
To: <[email protected]>
Date: 5 Nov 2009 14:44:47 -0500
Subject: Software renewal notice
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
X-SmarterMail-TotalSpamWeight: 0 (Authenticated)

and here is when it is sent via outlook:

Return-Path: <[email protected]>
Received: from MY-LOCAL-IP-HERE by HOST-MAIL-SERVER-HERE with SMTP;
   Thu, 5 Nov 2009 09:28:35 -0800
From: <[email protected]>
To: <[email protected]>
Subject: Software renewal notice
Date: Thu, 5 Nov 2009 12:28:01 -0500
Message-ID: <016001ca5e3d$53762c70$fa628550$@com>
MIME-Version: 1.0
Content-Type: multipart/related;
	boundary="----=_NextPart_000_0161_01CA5E13.6AA02470"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcpePVISNeiAFju8TzybuHYww6Djjw==
Content-Language: en-us
x-cr-hashedpuzzle: AEK9 Ab8G AhEJ B1ve DbY5 EKUE E684 FJjB Fz/v GNl2 GROA G1LM G7lD Hml9 Hv4G HyKj;1;bQBhAHQAdABAAGEAYgBiAHkAcgBhAHQAaQBuAGcALgBjAG8AbQA=;Sosha1_v1;7;{39FD103E-7A83-40B7-8D62-9739D0F188BC};bgBvAHQAaQBmAHkAQABhAGIAYgB5AHIAYQB0AGkAbgBnAC4AYwBvAG0A;Thu, 05 Nov 2009 17:27:59 GMT;QQBiAGIAeQAgAFIAYQB0AGkAbgBnACAAUwB5AHMAdABlAG0AcwAsACAASQBuAGMAIAAtACAAUwBwAGUAYwBpAGEAbAAgAE8AZgBmAGUAcgA=
x-cr-puzzleid: {39FD103E-7A83-40B7-8D62-9739D0F188BC}
X-SmarterMail-TotalSpamWeight: 0 (Authenticated)

and no you can't just add the X-Mailer header and try to claim your SMTPClient is outlook

[kleinma]

I just also turned off postmarking in outlook which omits those x-cr headers, and it still comes through as NOT junkmail when sent from outlook, so those headers don't seem to matter.

Here are the headers from the email when sent from outlook without postmarking:

Return-Path: <[email protected]>
Received: from MY-LOCAL-IP-HERE by HOST-MAIL-SERVER-HERE with SMTP;
   Thu, 5 Nov 2009 11:57:24 -0800
From: <[email protected]>
To: <[email protected]>
Subject: Software renewal notice
Date: Thu, 5 Nov 2009 14:56:49 -0500
Message-ID: <01b401ca5e52$1dd1fa30$5975ee90$@com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_01B5_01CA5E28.34FBF230"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcpeUhyFV2GsszGPQ+Cx2aZjWxUtoQ==
Content-Language: en-us
X-SmarterMail-TotalSpamWeight: 0 (Authenticated)

[kleinma]

Here are the specific differences:

SMTPClient:

Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: quoted-printable

Outlook:

Message-ID: <01b401ca5e52$1dd1fa30$5975ee90$@com>
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_01B5_01CA5E28.34FBF230"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcpeUhyFV2GsszGPQ+Cx2aZjWxUtoQ==
Content-Language: en-us

[chris128]

I've never seen those x-cr headers before, so I would have been surprised if it was anything to do with them.

So how is this email being sent by Outlook, is it connected to an Exchange server or does it just pass the emails straight to the same SMTP server that you are using with your SmtpClient? Is the IP address or name behind "HOST-MAIL-SERVER-HERE" the same in both the Outlook and SmtpClient version? EDIT: Oops, I meant the external IP that you have labelled "MY-LOCAL-IP-HERE" not the host mail server bit

[chris128]

Yeah I just found the same differences, probably the first time I have actually found the new Windows 7 'snap a window to each side of the screen' feature useful

Thread-Index as I understand it is just to help keep related emails (ie replies) grouped together, I wouldn't have thought that would have any effect.
Message-ID I guess could have an effect as that helps to show that it has come from a 'proper' email system rather than just been sent straight through an email server.

What happens if you just try and completely mimic the Outlook headers that you just posted there? (I'm talking literally copy and paste..)

[kleinma]

Well you can't for all of them because some get implied and set by the .NET class itself so you end up with duplicates of some headers. For example, setting a content type header manually will result in 2 content-type headers being in the email header list.

[chris128]

Ah right, so this response about my smtp class letting you specify whatever headers you want wasnt strictly true then :P

I can add headers no problem with the SMTP class.

Have you tried just copying the ones that dont get set by the SmtpClient class? Also what about the question I asked earlier about how Outlook is sending the email - if the email sent by Outlook is actually using a different SMTP server to send the message then that could be something to do with it

[kleinma]

Well you can change the encoding property for the body in a mailmessage which will change that header, I just don't know a specific encoding to use to match what outlook uses.

Outlook is sending mail via POP3, and is connecting to the same exact SMTP server, over the same exact ISP connection (as a side note, the ISP and the SMTP server are not the same company, but this does not matter, as I am not relaying mail, as in I am not using my ISP's outgoing SMTP server to send via outlook, I use my mail hosts).

[chris128]

Outlook is sending mail via POP3

No it isnt POP3 cannot send emails, it just retrieves them.

I'm a bit confused now. Are you sending an email to the same email address that you are sending from then?

[kleinma]

You know what I mean... when you setup an account in outlook "SMTP" isn't an option

[chris128]

no but it does ask you for an SMTP server at some point, otherwise you wouldnt be able to send email (unless you are connecting it to Exchange or something). Can you just double check that the SMTP server that you have got set in Outlook is the same thing as the SMTP server you are using with your SmtpClient?

[kleinma]

I don't even need to look. Yes it is all the same. I only have 1 SMTP server.

[chris128]

ok cool as long as you are sure. I'm just looking for other reasons as I'm surprised that one of those headers that is different is causing this - I wouldnt have expected any of those particular headers to have made any real difference. Did you try adding your own fake Thread-Index header and Message-ID etc?

[kleinma]

Yeah... without any better result.

I mean part of it could be outlook has some logic that if the email was sent FROM outlook, then it is less likely to be spam than if it was sent from just somewhere else.

Remember that my mail servers spam filter tools mark this with a spam weight of 0 meaning it doesn't think it is spam at all. It is only outlooks junk mail filter (which is set to low of all things).

What annoys me most is the junkmail filter isn't even that good. I still get crap spam in my inbox, and of all things, the email to tell me I was reawarded as an MVP this past january ended up in the junkmail folder...

[chris128]

haha yeah that is typical...

I guess there is one way you could confirm which of the headers is the primary cause of it - use telnet to send the email and type the entire email in, including the headers, as it appears in Outlook (well, a bit of copying and pasting obviously). Then one by one, change the headers to be more like your SmtpClient version and see when it starts getting marked as spam.

[kleinma]

One thing I noticed is that if I make the subject line say

This is a test

it does in fact go through and not get flagged.

If I change the subject line to

Software renewal notice

it goes as junk.

So there is more to it than headers, however the headers are the only difference between when identical emails get flagged or not when sending from SMTPClient or Outlook.

[chris128]

well yeah that does sound more 'spammy' but even so it must be a combination of that and one of the headers if an email with the same subject gets through when it is sent from Outlook

[kleinma]

I agree, and that isn't exactly the subject that gets sent out in the real emails, just something similar to our customers when they are up for renewal.