Celebrating 8 years of self-hosting Mail-in-a-Box

One fateful weekend I decided to try and self-host my e-mail. Now over eight years later I have to commend the Mail-in-a-Box (MiaB) project, its main author @JoshData and the community around it. Thanks for the eight years so far and here’s to eight years more, and beyond!

In short, Mail-in-a-Box is a software project that aims to provide a well-configured self-hosted email server. It also configures nginx for static web site support (like hosting this blog), and includes Nextcloud to provide support for storing your cellphone contacts.

With this post I want to share some of the challenges I’ve faced and the things I’ve learned over the years.

Things have been great, except Hotmail #

There is but one snag to MB. I’ve had trouble sending email to Hotmail accounts. Microsoft apparently has a quite strict setup. I’ll need to try and debug that. But that’s a sys(adventure) for another time. :) (me, ca. 2016)

Little did I know that this Hotmail-issue would plague me all the way until august 2023, haha. Thankfully I never needed to send many emails to Hotmail-users, but there was definitely some key moments where I wanted to just send a quick email with attachment only to be rebuffed by Microsoft’s overeager spam detection.

This is a known issue when it comes to self-hosting email (see this and also this). When you decide to run your own mail server you need to be careful about your IP-address’s reputation. At the outset your IP-address will not have built up any trustworthiness with other email vendors. However, by receiving and sending email that’s not flagged as spam you’ll be able to build up your IP’s reputation. That’s the theory at least.

Despite years of honorable service your mail server might still get flagged as a spammer by simply being on the same IP-block (group of IPs) as other spammers. So, while your IP has never sent spam, other servers in the same data center has been carpet bombing people with the latest in bitcoin spammage. Consequently large email providers such as Gmail and Microsoft choose to mark large IP-blocks as spammers. I suspect this is what happened to me because I happened to use Digital Ocean to rent a virtual private server. Probably there has been enough spammers using DO to force Gmail and Microsoft to block DO’s IPs.

I solved this issue in the end by (… inserts drumroll), jumping ship for a different host provider.

My 2023 MiaB migration #

MiaB has always been purpose-built to only be installed on servers running Ubuntu LTS (long-term-support). These LTS versions promise software support including security upgrades for five years (release cycle).

When I first installed MiaB it was on top of Ubuntu 14.04 codename “Trusty Tahr”. Some years later I double-checked my backups and carefully upgraded to Ubuntu 18.04 “Bionic Beaver”. Time flew again and suddenly this year all the MiaB users had to contend with 18.04 nearing end-of-life and start migrating to Ubuntu 22.04 “Jammy Jellyfish”.

This migration marinated in my list of TODOs for many months before I got to it last month of august. I figured I’d try a different server provider (1984.hosting) besides Digital Ocean. So, without deleting my present mail server I downloaded its backups and tried installing it on the new host. It went almost without a hitch.

This migration included several changes. It meant upgrading Ubuntu from 18.04 to 22.04, and MiaB from v57 to v63 which in turn would update many of MiaB’s underlying dependencies.

Upgrading Nextcloud threw errors because they had updated some of their downloadable .zip files which caused the install script to suspect that the files had been tampered with. It later turned out to be caused by some minor mishap in Nextcloud’s release process, not viruses being inserted. It’s nice to know that MiaB correctly throws errors on these things so that the community can investigate. These updates issues are now fixed, but I had to locally patch the nexcloud.sh install script to proceed.

After proceeding some more Nextcloud’s upgrade process started throwing new errors due to duplicate user entries in its database. Some searching revealed that other MiaB users had encountered the same issue, and I learned that a user had already published a guide on how to fix this issue manually.

I’m not quite sure how I ended up getting duplicate user entries in that database table. This installation has lived for over eight years undergoing various upgrades, so maybe the duplicates snuck in early on and was only discovered now because a newer version of Nextcloud enforced stricter duplicate checks on that table. Just speculation on my part. Anyhow, after resolving this issue I was able to complete the MiaB upgrade process.

With MiaB fully restored on a new server, I still hadn’t touched the old server. The old server was still receiving email and I would not touch it until I had validated that the new server was working well.

By editing the /etc/hosts file on my laptop I could then test out my newly setup MiaB server, without disrupting email service on the old server. With this I was able to confirm that restoring the backup and upgrading MiaB on the new server had worked properly.

Now it was time for the real deal. Following MiaB’s migration guide I blocked off all ports on the old mail server except port 22. The nifty thing about the email protocol is that it specifies that all email senders should retry sending email for several days before giving up (RFC 5321). This meant that I had some days to complete the migration.

I then made the final backup of the old server that was no longer receiving any new email. On the new server, I went and wiped it clean so that I could run through the upgrade just like before only now with updated data. It wasn’t long until I again had a fully functioning mail server. All that was missing now was to update the DNS.

Oh, DNS we meet again #

Interestingly MiaB is built to run its own DNS using the server program nsd. This setup allows the mail server to correctly configure and maintain a strict regime of DNS entries that modern email servers require. And for this to work I had to log into my domain name registrar and update the glue records for my email server’s domain address. And then I had to wait, and wait and wait. 24 hours to be exact.

But it didn’t seem work.

This is the tricky thing about DNS, it’s a network of servers that uses a lot of caching and any changes takes time to propagate. If you do several changes at once you might be breaking and fixing things at once. What helped me solve this was Norid’s DNS troubleshooting tool, which is built to debug DNS changes for any .no domains. With this tool in hand I could ask it to resolve my email domain using the IP of my new server, I then learned that something was wrong with its DNS response.

I revisited my mail server dashboard and re-configured the secondary name server. All glue records mandate you to have at least two DNS servers for redundancy. And my domain name provider offers a secondary DNS server to be used when configuring glue records. I wasn’t sure that simply re-configuring this would solve it but upon retrying the DNS debugging tool I got no errors anymore. Hurrah! I guess that my new DNS server had to somehow instruct the secondary DNS server that it was allowed to interact with it.

At this point I still wasn’t sure that I was in the clear. Some days had passed and I had been emailing my domain name provider in the mean time to see if something was wrong on their end. When they finally responded they noted that the configuration looked to be fixed.

I went into my email app on my phone, hit refresh and saw new unread emails pour in. All this time the other mail servers had been nicely stockpiling my undelivered email to try again at a later time. Nice! The folks who developed the email protocol designed it to nicely fault tolerant.

Self-hosting is a stubborn thing to do #

Now you might wonder, how could have been emailing with my domain name provider when my mail server was not working? Well, I also pay a small monthly fee to tutanota.com to host a backup email that I use for interacting with the providers of my hosting and domains. All the rest of my emailing goes through MiaB.

When you work with digital stuff you need to have backups and contingency plans. I guess I could have ran two separate instances of MiaB for two separate email addresses and only upgrade one at a time. However, that seems a little tedious even for me. You need to depend on things. But I’d argue that being independent on the web is not a binary thing, it’s a spectrum, where you can decide who to trust and entrust them with tasks of varying importance.

Honestly I could have configured Tutanota with my domain name in front and skipped running my own mail server. For many people I think it’s totally awesome to pay an ethical provider to host their email.

But for me I really do like the fact that I can run my own mail server thanks to MiaB and its community. I realize it’s a stubborn thing to do in this day and age, and yet I’m learning a lot and for me it feels great to exercise software freedom on the web.