Email is the focal point of our professional online identities. Whether you like it or not, that is the most reliable and consistent way for people to get in touch with you, and it’s not going away anytime soon (sorry, Slack). That’s why it’s imperative for your email provider to be reliable, resilient, and especially in today’s world, trust-worthy.

Recently I realized that my email host is not great. I registered my domain and set up email accounts there long before I was thinking about privacy concerns or shopping for competitors. They were ok at what they did, but they specialized in assisted hosting services and email was clearly an afterthought, let alone any privacy or scalability concerns.

I knew I needed to migrate email providers. And I knew that this is going to be a tricky and intricate operation. What I was looking for was:

• minimal downtime,
• end up on a provider that I trust long term and,
• migrate all of my current emails and folders.

It also happens that my domain was registered through the same email provider host and I wanted to transfer that over to my everyday registrar, Namecheap. Might as well knock everything out at once and be in good shape for the next while so that I can sleep well at night without worry.

Decisions, decisions, decisions

First thing’s first is you have to find the new registrar and email provider you want to migrate to. This involved some heavy research on my end, because you don’t want to have to do this migration more than once. Everyone has their own priorities in what they’re looking for; for me it was privacy, management, and support. That One Privacy Site has a great article on what to look for in email providers, along with a Simple Email Comparison chart. I spent quite a bit of time looking through these options, and ironically ended up picking a host that wasn’t even on the list, Migadu. As for a registrar, I default to Namecheap as I already have quite a few domains registered there and like their service.

Inform Your Users

I am a big believer in transparency, and especially with today’s internet climate, more transparency is better. I sent a letter to all of the active users on the domain stating that 1) I plan to migrate our email provider and why, 2) to which provider and why and any differences in services they should expect, 3) estimated timeline, and 4) I need their email passwords.

That last point is a point of contention. In order to back up and migrate my users’ email accounts to a new host, I need to ask them to give up their privacy temporarily. Since I am my users’ everyday ‘IT guy’, they trust me and didn’t worry about this point. As a good measure, I linked them to the page where they can change their passwords before they share it with me, so if their password is one they use a lot, they can change it to something they’re more comfortable sharing.

That being said, even with the proper transparency, be sure to listen to your users and address their concerns. If you did the proper research ahead of time, they should trust you with this transition, but it is up to you to keep them at ease and make sure that they are ok with your plan moving forward.

Start Back Up & Migration (Emails)

I wanted to not only migrate my current emails, but back them all up as well, in case something went horribly wrong. Here’s the thing, there were a lot of emails on our server. I wanted this back up and migration to run continuously on a stable connection. AWS to the rescue! I simply spun up a t2.micro EC2 instance running Ubuntu and attached a 100GB Block Store to it. This was going to be my ‘command center’ for all things IMAP (and SMTP) during this transition.

Behind every software success story is a powerful open source tool behind it. This story is no different.

As for the tools necessary, I searched far and wide for reliable, stable, and open source solutions capable of iterative processes. I came across imap-backup for backing up emails and imapsync for migrating. Once you have these 2 tools installed, follow their respective set up processes and plug in your user’s credentials.

imap-backup

For imap-backup, you want your config.json file to look something like this:

{
  "accounts": [
    {
      "username": "username@domain.com",
      "password": "",
      "local_path": "/ext_vol_data/username_domain",
      "folders": [

      ],
      "server": "imap.server.com"
    }
  ],
  "debug": true
}

Of course, be sure to fill in the values properly. You can leave the folders array empty to move over all of the folders. The local_path here is currently pointing to an EBS store that I mounted onto the EC2 instance. And I have debug set to true so that I can monitor the progress of the script through its output. Then, simply create 1 account object per user on your email domain.

Since this is simply backing up our accounts, you can start running this immediately! This script works iteratively so you only need to do a ‘full’ backup once, and every time after that it will only download the difference.

imapsync

Once imapsync was installed properly, I found this Github Gist to help sync multiple accounts at once. For multiple accounts, simply duplicate the line in the accounts.list file with the proper credentials. Make sure you also properly set the CONFIGURATION variables in the imapsync.sh file. This script is now ready to run, but it won’t work until you have both hosts set up and ready to receive requests. More on this to come.

Start Back Up & Migration (Hosting)

The domain I was migrating also happened to have 2 different sites hosted on it on 2 subdomains that I wanted to bring along with me. Be sure you’re prepared for this by picking a new host and making sure that your environments, databases, and source code are all backed up and can be spun up in a new hosting environment.

Take any action you need to try and spin it up before the actual migration. This is easier to do than an email migration since you can have the same site set up on multiple servers, but you can only have 1 email host at a time! Ideally, your sites should be migrated and simply waiting for the DNS change to take effect.

Configure New Email Host

Because I wasn’t only migrating email hosts, but also registrars, I couldn’t pull the trigger on the domain transfer until I was sure that things looked right on the new email host’s side. I got in touch with Migadu’s support and they worked with me to ‘enable’ my domain name even though it wasn’t transferred yet. They also gave me a 1 month free trial ‘Basic’ account so I can see the full extent of the service.

Using this, I was able to make sure that the new email host was up and running.

I created email accounts for all our users on Migadu with template passwords that I could provide them for their first login. I also set up things like alias email addresses (so one can use emails like name+service@domain.com) for everyone.

Regex for ‘+’ email aliases: ^username\+(.+)@domain.com

Since I know had the original email host up and the new email host up, I was able to use imapsync to start transferring all the data to the new email host. With the proper configuration, the previously mentioned imapsync.sh script made the transfer smooth and seamless, migrating over all emails within their respective folders, and incrementally.

Admin Stuff

If you also plan on transferring your domain to a new registrar, be sure that the ‘webmaster’ email for that domain is an email you have definite access to and that is not tied to the domain that you are migrating.

Pulling the Trigger

Ok, now everything is in place and backed up. What’s left is actually pulling the trigger and starting the migration process. Pick a Friday or Saturday night, a relatively quiet night for emails in order to minimize the ‘blast radius’ in case anything does go wrong. Again, for transparency purposes, be sure to inform your users when you plan on doing this, so that if they see anything fishy, they know that you’re be behind the scenes working on it.

Before officially starting the transfer process, be sure to run the imapsync and imap-backup commands one last time to ensure the mailboxes on each host and backups are as up to date as can be.

Now, begin by transferring the domain to your new registrar. You can do this by going to the ‘transfer domain’ page at your new registrar. You will need to ‘unlock’ your domain from your previous registrar and make a note of the domain code for the transfer.

Once that is triggered, be on the look out for any emails coming through to the webmaster email of the domain from your previous registrar. A lot of times they will confirm it with you before enabling the transfer to go through.

Now you should start configuring any MX and DNS settings for your domain on the new registrar. The MX records are the most important to not have any downtime for your email accounts. Once those are complete, be sure to set up any other DNS records you may need for the sites you had hosted on the domain and point the nameservers over to the new web host.

Time to make sure everything is working properly!

In your mail client of choice, update your email settings to point to the new email host with updated credentials. The first thing you should notice is that your Inbox and all other folders should be present just as you expected, thanks imapsync! Now compose an email from any other email address of yours to the email on the domain you just transferred and make sure it comes through – IMAP is working properly. And compose an email from your newly transferred domain to your other email account – SMTP is working properly.

Inform your users that the migration has taken place and provide them with the new host’s settings (IMAP and SMTP servers, ports, etc) and their new passwords. Of course, you’ll have to do this through means other than their email Be sure to point them towards the settings page they need in order to update their passwords to something personal and unique.

Once you’re sure your email is working properly, be sure to also test any hosted sites you migrated and all of their functionality as they now live in a new environment.

It’s All About the Customer

Follow up with your users to make sure that everything is working as expected for them. This is also a good opportunity to talk to them and see how they felt about the whole process: was it smooth? Painful? What could have been better? Hopefully you won’t need to do this again, but it’s good to get this type of feedback in general.

Clean Up

So now you should be sure that all of your emails and sites are migrated properly and working as expected.

What’s left here is cleanup, which is just as important as set up! Do not cross this off your list until you finish properly cleaning up any set up you may have done. I would still wait a solid few weeks or so before starting this to make sure that everything is in place before you can’t go back, but be sure to set yourself a reminder so you don’t forget.

Our main cleanup consists of running rm -rf on any and all email backups in the EC2 and Block Store. Be sure to do this for all the settings and config files as well, as your user’s (as well as your own) passwords are present there! You can now tear down these resources confidently knowing that your information lives only on your new host’s servers.

Aftermath

Phew! Give yourself a pat on the back for a job well done!

Be sure you have a real, paid account with your new host to stay on their good side and not let a trial expire without realizing.

That’s it! Congratulations on a completed migration performed well with minimal downtime!

Photo by Web Hosting on Unsplash