Email is Hard

By Aaron Stockdill - 12 Jun 2014

We have just started to get our email system up and running here at Potato Softworks. We've had the proper mail server running for a while now, but the automated emails have proven a little more complicated.

Some Background

When we first looked at automated emails, it didn't look too tricky. PHP even has a really nice mail function to do it for us. Fill in some headers, write the message, done. Well, almost done. Sure, it sent, but did it arrive?

No. Sort of.

It got through to some email addresses, sometimes is got sent to junk, and other times it just didn't make it at all. The most depressing thing was, that last one was our own server. We were classing ourselves as spam.

Sorting it Out

First things first: were our headers ok? For the most part, yes. Sure, we'd skipped a few that were unnecessary, but we filled them in now to make sure. Surely a 100% RFC 5322 compliant message would make it?

No.

Hmm. Well, how do email providers decide what spam is? Sure, there are statistical measures using the words in the messages. But I didn't exactly sign it 'Nigerian Price'. So what made them think we were the bad guys?

Servers are Hard

Hands up if you know what PTR is? Congrats to that guy. For the rest of us, it is a reverse DNS record: a DNS takes a name -- for example, potatosoftworks.com -- and tells us an address. The PTR tells us the name at a given address. Many eons ago at the dawn if the internet, a PTR was a nice-to-have, not an absolutely-necessity.

So why does this matter now? Because servers check this. If the email comes from a server that does not have a PTR, the mail reviewer cannot be sure that this is actually the name of the server. Of course, this is not a failsafe way to check an identity. Take as an equivalent conversation:

Hey, what's your name?

Steve.

Ok. Just to make sure, let's do a security clearance. Is your name Steve?

Yes.

Well I'm convinced.

Unsurprisingly, this still didn't convince the server that we're who we say we are. Which leads us to...

Security is Really Hard

Given the conversation we had above, can we really blame the server for not trusting us? That's why we need DKIM. DKIM is one of those clever little tricks that are really simple, but utterly brilliant. Like the Diffie-Hellman Key exchange. (If you don't know what it is, look it up, because it's really cool! Even if not particularly relevant.)

The general idea is that you send your message with a special key, and a note saying where the message has come from. When the server recieves your message, it goes back to where the message claims it has come from, and check that your key matches up. So we can change our converstation to be this:

Hey, what's your name?

Steve.

Cool, can I see your ID?

Sure, here.

Great. I'll just call the boss of the company you say you're from, and make sure you actually are who you say you are.

Uhh...

Making Things less Hard

Lets be honest: if it wasn't this way, we'd be overrun with spam from companies who actually had nothing to do with it. Without this security, it wouldn't be hard for someone to send an email as apple.com saying you've won an iPad, to claim just send your bank account details to blah blah blah. We need to do it so you don't have to worry. If you couldn't be sure whether your next message was going to reach you, things would quickly spiral out of control. So as much as it can be a real pain in the arse, we have to do this.

Just know how hard it was.


Related Posts

Comments