Migrating Outlook PSTs -> Maildir via IMAP

Question

Our small shop has been on Exchange since forever (currently on 2003). However, we use a 3rd party managed Exchange provider and set tight limits on how much server-side storage we allow each user. The result is that our users all have additional email in local PST files, which we make daily backups of.

We're trying to move everyone to dovecot backed by Maildir. I basically have the reverse of this guy's question: Recommended Dovecot / Thunderbird to Exchange / Outlook Migration Strategy , with the additional complication that Exchange doesn't currently hold all of the email, or even close to it.

For most of the users we can simply create the IMAP account in Outlook and copy/paste or drag the folder in Outlook, and Outlook pushes the messages up through IMAP to dovecot and they pop out the other side in the user's maildir.

For a couple of the users, though, we are stuck. They have multiple, large PST archives. Everyone is running Outlook 2003. Outlook 2003 supports two types of PST files: a backwards-compatible version that has a maximum size of 2GB, and a breaking-change one that supports several TB I believe.

We have users that have a mix of old format files that are at maximum size and new format files. We tried the same kludgy approach with them (its only two or three users that have this problem, and we only have a dozen email accounts or so total), but when you create a new IMAP-based account in Outlook 2003, it doesn't give you the choice of PST filetype, you get stuck with the 2GB max size.

The result is that Outlook eventually won't let you copy any more data into the PST, means you can't move any more of the data in the other PSTs up to the server via this method.

Creating a NEW IMAP account to the server with another 2GB PST won't work, because we can't seem to get Outlook to reproduce the IMAP-side folder structure without downloading all the messages, which just fills it back up to 2GB.

Tools like imapsync won't work because the email doesn't exist on the Exchange server, its only in the users' local PSTs.

I'm sorry if I'm missing something obvious here, I don't really have much experience with Outlook/Exchange.

Solution 1

Consider using Thunderbird as an intermediate conversion tool -- it in theory can read PST files for you into it's native format (mbox), then you can use it to connect to your Dovecot IMAP server to drag & drop the bits upstream from "Local Folders" to the "Server IMAP" account. See this KB:

http://kb.mozillazine.org/Import_.pst_files

Thunderbird works fantastically with Dovecot so you'll probably see really good performance copying the bits upstream in the last step...

Solution 2

Just for the records:

With Outlook 2003 you can upload to an IMAP server more than 2 GB, but it's a little bit tricky. You have to upload in max. 2 GB runs, you have to put each upload run into different folders and after each upload run you have to unsubscribe from the folder in Outlook. I was doing this quite some time ago, and if I remember correctly, I even did a restart of Outlook between the runs, but I'm not sure that this is a must.

But as I discovered copying with Outlook to IMAP can be really slow, so Thunderbird is the better approach (nowadays I'm doing the copy only with TB).

Comments

[Joel E Salas]

I gave this a try, and while I was super-impressed with TBird's ability to import huge outlook archives quickly, TBird crashes when I try to push the mail up. Im going to look in to whether TBird does/can use Maildir and then just copy the maildirs to the IMAP server.

[Joel E Salas]

Was crashing even on very small subsets, but I disabled the Global Indexer and its running further now without crashing. Note to future readers: TBird3.0 may have a crashing bug when moving between Local->IMAP folders while the Global Indexer is running. Disable it in Tools|Options|Advanced.

[Joel E Salas]

Even without that, it still crashes roughly every 10k messages in my config.

[Joel E Salas]

Are you trying to push 10k+ in one chunk upstream, or are you trying it in smaller <=10k batches? Thunderbird used to have a problem with performing large scale operations -- basically it churned so long internally to generate the actions locally before sending to the server that it lost the upstream connection and crashed. The author of Dovecot IMAP added a feature to his code that will sort of keep-alive Thunderbird while it's churning and keep it from crashing; this server patch solved a ton of problems at my company with huge ops like you're trying to do.