mildbyte's notes tagged 'kimonote'

MTA, SPF, DKIM, PTR, WTF: a quick checklist on how to send e-mail from your domain

@mildbyte 5 years, 5 months ago | programming | kimonote | email |

Introduction

While I'm writer's-blocked on continuing with project Betfair (well, more like blocked on not being able to reproduce my old research results in order to fit the narrative I thought I had), here's a quick story from the development of Kimonote.

I got stuck following this guide on how to startup:

How to startup:

1. Set up a landing page to collect e-mail addresses
2. ??????
3. ??????

and had figured out that the next step after collecting e-mail addresses was sending something to them. Except the issue was that I also wanted to make sure most e-mail providers didn't mark my mail as spam and it actually got delivered.

So here's a quick checklist on what to do in order to achieve that. This guide assumes that you have a dedicated IP address and a domain name whose records you can edit.

Checklist

Mail Transfer Agent: exim

Or Postfix. This is the server that will run on your machine and, when a client requests it to send mail, will do so.

The following commands are all executed as root, assuming a Debian-like system (say, Ubuntu). Install exim:

apt-get install exim4

I used the "Single configuration file" option during the installation: the only change I needed to make to it was disabling IPv6. To do that, add disable_ipv6=true under "Main configuration settings" in /etc/exim4/exim4.conf.template and reload the config with /etc/init.d/exim4 reload.

Try sending a test message:

echo "testing" | mail -s Testing (your personal email address)

You should get a message in your spam folder from root@(hostname, which is probably not your domain name).

Try again, this time setting the from address:

echo "testing" | mail -s Testing -aFrom:test@(your domain name) (your personal email address)

You should get an email from test@(domain name), still in your spam folder (if you're lucky). If you don't, check the exim logs at /var/log/exim4/mainlog.

SPF record

An SPF record is the first step on the route towards not getting your emails sent to the spam folder. In the world of email, anyone can pretend to be anything. I could right now use the exim instance running on my server to send an email from support@microsoft.com to (my best friend)@gmail.com claiming to be Microsoft. The only thing that's stopping me is that the recipient's email service will perform a DNS query to check the microsoft.com's SPF records in order to see who's authorised to send emails on behalf of microsoft.com, thus rejecting my email.

The simplest SPF record is:

v=spf1 a ~all

This record is read left-to-right and says: "If the IP you're getting mail from is in my A record, then it's OK, otherwise, reject the email".

Add that as a TXT record in your domain management panel with Host=@. In the case of Namecheap with e-mail forwarding on, I had to instead edit the TXT record under Mail Settings and add a between v=spf1 and include:spf.efwd.registrar-servers.com ~all (the latter says "any IPs that are in the SPF record of efwd.registrar-servers.com" are fine too)

DKIM record

If the SPF record says that the email is indeed from your domain, then the DKIM record verifies that the email wasn't altered in transit. In essence, you publish a public key to another DNS record and then sign your messages with the private key. Strictly speaking, this wasn't required for Gmail to stop classifying my emails to myself as spam, but it's worth doing anyway.

First, generate a key pair:

cd /etc/exim4 && mkdir dkim && cd dkim
openssl genrsa -out dkim-private.pem 1024 -outform PEM
openssl rsa -in dkim-private.pem -out dkim.pem -pubout -outform PEM

Go to your domain control panel and add a TXT record with host (selector)._domainkey (the selector could be anything: I used a timestamp 20171206) and value

k=rsa; p=(your public key from dkim.pem, all in one line)

Now, set up exim to actually sign outgoing emails with the private key. If you are using the single Exim configuration file option, create (or edit) the file /etc/exim4/exim4.conf.localmacros and add:

DKIM_CANON = relaxed
DKIM_SELECTOR = (selector, e.g. 20171206)
DKIM_DOMAIN = (domain name, like example.com)
DKIM_PRIVATE_KEY = /etc/exim4/dkim/dkim-private.pem

Reload the config:

/etc/init.d/exim4 reload

And check that exim picked up the settings:

exim -bP transports | grep dkim

dkim_canon = relaxed
dkim_domain = example.com
dkim_private_key = /etc/exim4/dkim/dkim-private.pem
dkim_selector = (selector, e.g. 20171206)

You should wait for both records to propagate around. To test whether this has worked, you can use any of the online services that do some DNS lookups and tell you whether they have seen your records (e.g. https://www.mail-tester.com/spf-dkim-check). You can also send an email to your personal address. If it arrives (or arrives into your spam folder), you can inspect its headers:

Received-SPF: pass (google.com: domain of hello@kimonote.com designates 91.220.127.149 as permitted sender) client-ip=91.220.127.149;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@kimonote.com header.s=20171206 header.b=K0nBKUQ7;
       spf=pass (google.com: domain of hello@kimonote.com designates 91.220.127.149 as permitted sender) smtp.mailfrom=hello@kimonote.com

PTR record

Are we done? Not completely. Some ISPs do what's called a reverse DNS lookup on the IP that's sending them emails by contacting the (via several levels of referrals) DNS servers of the organization that IP belongs to in order to find out what domain it corresponds to. Then they compare that to the domain the email is claimed to be from. This is similar to having an SPF record and I thought it wasn't necessary until I had some emails from users saying that they weren't receiving their registration confirmations.

First, check that it's actually the case:

 dig -x (your IP address)

If the Answer section doesn't contain your domain name (yes, it is supposed to end with a dot), then you do need to add a PTR record. The PTR record has to be added on your hosting service's side: there usually is a control panel (in my case I had to ask them nicely). This will probably take up to 24 hours to propagate around the world.

What have I been up to

@mildbyte 5 years, 6 months ago | admin | programming | kimonote | betfair | 1 comment

It's been more than 3 months since my last post. Have I finally sunk into the depths of depression and debauchery since leaving my job? Do I start my day by waking up at 11am and having a mimosa? Do I end it with a can of Pringles at 3am, going through a box set of The Wire?

As if.

project Betfair

I tried my hand at automated trading on Betfair, which is a sports betting exchange. Automated trading on Betfair is similar to automated trading in real life, except the stakes are lower, the taxes are none and the APIs are more pleasant.

In the process of getting this whole system to work, I managed to do many interesting things:

• wrote a collector/reconstructor of order book events from the Betfair Stream API
• wrote a harness that backtests high-frequency trading strategies by replaying order book events into them (thus simulating market impact) and can collect various statistics
• learned about market making (fun things like adverse selection and inventory risk)
• wrote a harness in Scala for live trading against the Betfair Stream API
• let an extremely simple (non-market-making) fully automated strategy run for a week against ~150 horse races -- it made about £1500 worth of bets and lost £9. Since it didn't lose too much money, looks like the basic idea worked. Since it didn't make money, I can now blog about it.

In fact, even if it did make money, I think there's a lot of interesting stuff and pretty pictures to look at and write about. Perhaps one day I even will.

project Bitcoin

I read some Bitcoin and Ethereum whitepapers. It's a pretty cool idea.

project Kimonote

You may have noticed that it looks like blog isn't statically generated by Hakyll any more. That's because it isn't.

One day when I was taking a break from my Betfair adventures, I decided to reorganise all the stuff in my private diary (that I use DokuWiki for) and realised that it would be cool to:

• run LDA on my entries to see how they classify into topics
• hmm, topics are just like tags, what if I could auto-tag my posts?
• what if I used this for my blog as well?
• what if tags didn't suck and were a valid navigational tool?
• what if I could filter posts by not one, but multiple tags?
• what if I could subscribe to posts with some specific tags from a user? Like RSS, but with extra filters, or like Facebook, but without influencing elections?
• what if text-based websites didn't exceed in size the major works of Russian literature?

At least that's what I think my thought process was now, back then it was more like "what if I learned Django?".

So I made Kimonote, which I guess is a note-organizer-slash-blogging-platform that focuses on minimalism and ease of navigation. Wanna see my previous post tagged 'programming'? It's there, on the left, in the sidebar. Previous post on the blog? It's also there. News from Paddington? Here. News from Paddington, but in a format that allows you to consume the whole history without clicking "next"? Sure. Project Morrowind and my posts on Raspberry Pi from before I went to university? No problem.

It's Javascript-free as well and uses basscss for styling. This means this whole page takes up 20KB, about 100 times less than a small Medium blog post.

Interested? Head on to the landing page: there's a sign-up link there that lets you... no, not sign up, but leave your email so I can see if it's worth making it more than a place where my blog lives.