Mail migration: pumping mails into IMAP account

Question

I'm having troubles migrating the email system of our company from an old provider to a new one, call this one FOO.

I already created, via control panel, the email addresses and the mailboxes on FOO to duplicate the existing structure. To check that everything works I added and configured a new account on each email client with FOO smtp and imap data and did a simple send and receive. So far so good, clients send and receive mails without any problem.

The problem is that new mailboxes on FOO are empty and I need to populate them with existing mails.

What I have: for each client of my local users I have a bunch of .eml files organized in folders (Inbox, Sent Items, etc.). Each .eml is a mail.

What I want: for each account I want to import these .eml files into FOO mailboxes, so I can configure the clients with FOO imap server/username/password and users have their mails available again.

Initially I did this way: on my personal local Thunderbird installation I created one account for each user account to migrate. I imported (via ImportExportTools TB extension) .eml files into the newly created account. Automatically Thunderbird syncs mails between local client and the imap server on WF.

It would be GREAT if it'd work… but after some time the task failed. Sometimes for timeouts, another times for network problems and so on... Basically the process stops for some reason and I can't get all my mail into the remote server.

I have to migrate a lot of mails (avg of 5/6 Gb for each user).

My solution:

1. for each existing client to migrate create a zip containing the corresponding .eml files.
2. upload each zip to my FOO account (using scp or ftp, I have to think about that)
3. install a command line mail client (pine? mutt? others?! WHICH ONE?) in my FOO account
4. create a mail account for each account to migrate
5. import .eml into the client and hope that IMAP sync happens

To me it seems fairly complex, and I'm not even sure it could work. Do you have a better solution for this? And which client does it support eml import and IMAP?

Any suggestion greatly appreciated.

Solution

If you still have access to the original service, the best method by far is to use the tool ìmapsync (or OfflineIMAP as an alternative).

This will allow you to temporarily sync from the old service to the new one. It will retain all of the flags as well so that unread markers are retained. It will also retain any folder structures.

The second most common way of achieving this would require some careful coordination of mail routing. That would be a file copy of the source data which would normally be either in maildir or mbox format. This may also require help from the previous mail provider unless you have shell access to the old service.

imapsync is certainly the preferred method. Trying to do a transfer using eml files is not recommended. For starters, you will have lost all of the flags and folders. In addition, trying to do this for 5-6GB per user is going to take a LONG time. You will have to do it in stages.

Additionally, I'm not really sure that Pine or MUTT is going to help doing it the way you've outlined although you may be able to write macro's to transfer the files a few at a time.

---

UPDATE:

As we now know that sync from the original is not possible. The best way to script input from EML files to a maildir based system (if that is what you have, it is the most common storage format for Linux IMAP servers) is to use

getmail_maildir ~/Maildir/ < email_file.eml

getmail_maildir is part of the getmail package. This only works if you have direct access to the mail folders though this is commonly true with the better hosts. Not so sure about doing this with the other mail storage format mbox but I think that getmail also has a getmail_mbox command. In addition, the Windows application "IMAPSize" has a command for converting from EML to mbox.

So again, it is much easier to migrate the emails to a physical mail store rather than trying to pass everything through IMAP. However, it may be that you have to do this because the new provider cannot provide suitable access (as would be the case if migrating to GMail for example).

If this is the case, what may be best is to migrate the EML files to maildir format using a "synthetic" local maildir (there is nothing really special about maildir except the naming conventions so you don't actually need an IMAP server to use them) and getmail_maildir. Then use IMAPSync or OfflineIMAP to push from that local maildir to the new service. That way you don't need to mess with trying to script MUTT.

Comments

[Paolo]

The original server was used as POP, so I don't have mails there anymore. And I can't even consider the kind of support you suggested from the actual mail provider. Though, I'm working out a possible solution to my question, I'll keep this thread in sync with my discoveries...

[Julian Knight]

OK, so what is the target mail system? Do you have direct access to that?

[Julian Knight]

@Guandalino, see my updated answer.