While I'm writer's-blocked on continuing with project Betfair (well, more like blocked on not being able to reproduce my old research results in order to fit the narrative I thought I had), here's a quick story from the development of Kimonote.
I got stuck following this guide on how to startup:
How to startup:
- Set up a landing page to collect e-mail addresses
and had figured out that the next step after collecting e-mail addresses was sending something to them. Except the issue was that I also wanted to make sure most e-mail providers didn't mark my mail as spam and it actually got delivered.
So here's a quick checklist on what to do in order to achieve that. This guide assumes that you have a dedicated IP address and a domain name whose records you can edit.
Or Postfix. This is the server that will run on your machine and, when a client requests it to send mail, will do so.
The following commands are all executed as root, assuming a Debian-like system (say, Ubuntu). Install exim:
apt-get install exim4
I used the "Single configuration file" option during the installation: the only change I needed to make to it was disabling IPv6. To do that, add
disable_ipv6=true under "Main configuration settings" in
/etc/exim4/exim4.conf.template and reload the config with
Try sending a test message:
echo "testing" | mail -s Testing (your personal email address)
You should get a message in your spam folder from
root@(hostname, which is probably not your domain name).
Try again, this time setting the from address:
echo "testing" | mail -s Testing -aFrom:test@(your domain name) (your personal email address)
You should get an email from
test@(domain name), still in your spam folder (if you're lucky). If you don't, check the exim logs at
An SPF record is the first step on the route towards not getting your emails sent to the spam folder. In the world of email, anyone can pretend to be anything. I could right now use the exim instance running on my server to send an email from
(my best friend)@gmail.com claiming to be Microsoft. The only thing that's stopping me is that the recipient's email service will perform a DNS query to check the microsoft.com's SPF records in order to see who's authorised to send emails on behalf of microsoft.com, thus rejecting my email.
The simplest SPF record is:
v=spf1 a ~all
This record is read left-to-right and says: "If the IP you're getting mail from is in my A record, then it's OK, otherwise, reject the email".
Add that as a TXT record in your domain management panel with Host=@. In the case of Namecheap with e-mail forwarding on, I had to instead edit the TXT record under Mail Settings and add
include:spf.efwd.registrar-servers.com ~all (the latter says "any IPs that are in the SPF record of efwd.registrar-servers.com" are fine too)
If the SPF record says that the email is indeed from your domain, then the DKIM record verifies that the email wasn't altered in transit. In essence, you publish a public key to another DNS record and then sign your messages with the private key. Strictly speaking, this wasn't required for Gmail to stop classifying my emails to myself as spam, but it's worth doing anyway.
First, generate a key pair:
cd /etc/exim4 && mkdir dkim && cd dkim openssl genrsa -out dkim-private.pem 1024 -outform PEM openssl rsa -in dkim-private.pem -out dkim.pem -pubout -outform PEM
Go to your domain control panel and add a TXT record with host
(selector)._domainkey (the selector could be anything: I used a timestamp 20171206) and value
k=rsa; p=(your public key from dkim.pem, all in one line)
Now, set up exim to actually sign outgoing emails with the private key. If you are using the single Exim configuration file option, create (or edit) the file
/etc/exim4/exim4.conf.localmacros and add:
DKIM_CANON = relaxed DKIM_SELECTOR = (selector, e.g. 20171206) DKIM_DOMAIN = (domain name, like example.com) DKIM_PRIVATE_KEY = /etc/exim4/dkim/dkim-private.pem
Reload the config:
And check that exim picked up the settings:
exim -bP transports | grep dkim dkim_canon = relaxed dkim_domain = example.com dkim_private_key = /etc/exim4/dkim/dkim-private.pem dkim_selector = (selector, e.g. 20171206)
You should wait for both records to propagate around. To test whether this has worked, you can use any of the online services that do some DNS lookups and tell you whether they have seen your records (e.g. https://www.mail-tester.com/spf-dkim-check). You can also send an email to your personal address. If it arrives (or arrives into your spam folder), you can inspect its headers:
Received-SPF: pass (google.com: domain of email@example.com designates 18.104.22.168 as permitted sender) client-ip=22.214.171.124; Authentication-Results: mx.google.com; dkim=pass firstname.lastname@example.org header.s=20171206 header.b=K0nBKUQ7; spf=pass (google.com: domain of email@example.com designates 126.96.36.199 as permitted sender) firstname.lastname@example.org
Are we done? Not completely. Some ISPs do what's called a reverse DNS lookup on the IP that's sending them emails by contacting the (via several levels of referrals) DNS servers of the organization that IP belongs to in order to find out what domain it corresponds to. Then they compare that to the domain the email is claimed to be from. This is similar to having an SPF record and I thought it wasn't necessary until I had some emails from users saying that they weren't receiving their registration confirmations.
First, check that it's actually the case:
dig -x (your IP address)
If the Answer section doesn't contain your domain name (yes, it is supposed to end with a dot), then you do need to add a PTR record. The PTR record has to be added on your hosting service's side: there usually is a control panel (in my case I had to ask them nicely). This will probably take up to 24 hours to propagate around the world.
It's been more than 3 months since my last post. Have I finally sunk into the depths of depression and debauchery since leaving my job? Do I start my day by waking up at 11am and having a mimosa? Do I end it with a can of Pringles at 3am, going through a box set of The Wire?
I tried my hand at automated trading on Betfair, which is a sports betting exchange. Automated trading on Betfair is similar to automated trading in real life, except the stakes are lower, the taxes are none and the APIs are more pleasant.
In the process of getting this whole system to work, I managed to do many interesting things:
In fact, even if it did make money, I think there's a lot of interesting stuff and pretty pictures to look at and write about. Perhaps one day I even will.
I read some Bitcoin and Ethereum whitepapers. It's a pretty cool idea.
You may have noticed that it looks like blog isn't statically generated by Hakyll any more. That's because it isn't.
One day when I was taking a break from my Betfair adventures, I decided to reorganise all the stuff in my private diary (that I use DokuWiki for) and realised that it would be cool to:
At least that's what I think my thought process was now, back then it was more like "what if I learned Django?".
So I made Kimonote, which I guess is a note-organizer-slash-blogging-platform that focuses on minimalism and ease of navigation. Wanna see my previous post tagged 'programming'? It's there, on the left, in the sidebar. Previous post on the blog? It's also there. News from Paddington? Here. News from Paddington, but in a format that allows you to consume the whole history without clicking "next"? Sure. Project Morrowind and my posts on Raspberry Pi from before I went to university? No problem.
Interested? Head on to the landing page: there's a sign-up link there that lets you... no, not sign up, but leave your email so I can see if it's worth making it more than a place where my blog lives.