IMAP folders: how big is too big?

Question

I've switched from using Thunderbird to offlineimap and mutt. My mailboxes have over 50k messages. After tweaking some settings, mutt is very responsive with this many messages. notmuch is extremely fast for searches. But offlineimap can take several minutes to sync all the folders.

After a little research, it appears that IMAP was not intended for use with very large folders. If that's the case, then what kind of emal workflow should I be using to keep folders smaller?

Solution 1

Not all IMAP servers are fast, but as previously noted, it is possible to have an efficient IMAP implementation and setup. I've had reasonable luck with dovecot. However, offlineimap is a particularly slow program. It worked fine in small tests, so I switched over to it and redid my whole email setup to depend on it. But then the more mail I got the slower it got, until in the end I started losing my mind every time I checked email.

If you want a completely offline search experience and like notmuch, you might consider muchsync, which is kind of like the equivalent of offlineimap but specialized for notmuch to take advantage of notmuch's index. muchsync is much much faster than offlineimap once you'd done the initial sync, because it avoids all the network round trips that offlineimap has and takes advantage of notmuch's index data structures for efficiency. I switched over to it from offlineimap and have been very happy.

Yet another alternative to consider is isync. I have no personal experience with isync, but some people seem to like it. Having previously used and hated offlineimap, I'd be seriously investigating isync if I hadn't switched to muchsync.

Solution 2

There is the off-chance that you are referring to uwimapd, an implementation for an IMAP server, when you say "IMAP". In that case you need to make sure none of your mboxes becomes larger than 2GB, that is all emails in the same folder, not counting sub-folders.

If you are not referring to uwimapd, I believe your problem is the performance of your computer and internet uplink, not IMAP. Therefore:

Only keep emails no older than 3 months and those belonging to ongoing projects. Put the rest in an archive folder. Create the same folder structure under archive as you have under INBOX. That makes it easy to find stuff in archive, if you can't find it in INBOX.

IMAP, the clear text protocol your mail user agent employs to retrieve and manage the emails stored on the (remote) mail server, becomes problematic when it has to handle thousands of emails at once. The key aspect is "has to handle". The overall amount and size of emails is not a problem for IMAP, per se. The problem arises if it has to actualy perform work on a lot of emails.

This happens, for instance, if thousands of emails have their content and/or status (flagged, un-/read, priority, timestamps, ...) changed between two syncs. As you use offlineimap this may indeed happen, depending on your use case. However, in that case the only thing that will help you are more frequent syncs.

Solution 3

"After a little research, it appears that IMAP was not intended for use with very large folders."

My opinion (and experience) is exactly the opposite. Imap has very sophisticated server-side searching, indexing abilities - the only problem is, that most mailclient uses this practically as a remote file handling protocol.

For example, with IMAP is it possible to search for mails in a folder with a given SMTP header. Or you can handle the file attachments of a mail separately from its body. But this has the price: the protocol is much complexer as any other I've ever seen.

The real power of the imap will be only visible, if

• the actual IMAP server is able to index your mails

• and your mailclient is able to handle the imap folders intelligently (i.e. not using the imapd as a remote filesystem, but doing most of it with remote queries on the server side).

My only good experience was the thunderbird/dovecot pair with it. On the clientside, kmail is also relative okay, and also cyrus on the server side. Another systems aren't really okay.

In thunderbird you can make "search on the server" queries and see their results as a virtual folder. And the serverside search allows the imapd to make fast, keyword-based searches based on its internal index data.

I don't know offlineimap, but I know mutt. Clean, char console apps are very good and mostly very efficient, with a single exception and exactly this is what you found: they tend to avoid the indexed things and try to things linearly (although very fast).

Comments

[Kondybas]

Storage format is also significant. Mailbox and maildir are bit different in performance. Dovecot IMAP can easily deal with maildir 17GB large and 200k messages inside. Also dovecot have dsync utility that sync two maildirs very efficiently as far as used internal maildir indicies.

[Marco Ribolla]

I have 2 Google Email accounts. The other email uses dovecot. Since Google encourages not deleting anything, I've just let those accounts accumulate messages, mostly in the Inbox. I don't really want to adopt a new workflow (unless there are other benefits) just to deal with a large Inbox. Is there a configuration for offlineimap to move messages over 3 months old into an Archive folder?

[Marco Ribolla]

I want to have the ability to search offline, hence the use of offlineimap. Paired with notmuch, searches are nearly instantaneous. offlineimap itself seems to have some problems, but it's much faster than using mutt's built-in IMAP support.

[peterh]

@ReedG.Law Thank you the information. When I faced the same problem, I finally ended up using a local imap server (dovecot), and mirrored my gmail (and other accounts) into that. But it wasn't also perfect because of the spamfiltering and the mirror of the message deletion operations, but I could live with it (especially if the source mail accounts are used only for a simple polled mail mirror).

[Bananguin]

@ReedG.Law: Of course Google doesn't want you to delete anything, because then they would need to worry about searching their archives too. I'm not aware of such a backup-function for offlineimap. I also don't know about existing email backup, but IMAP does forsee searching for things like date. So such backup strategies are not too hard to write. I'm sure Google can help you. And last, if you don't want to adopt a new workflow, you shouldn't ask for one. :-)

[Jason h]

I agree, it looks like the problem is not that IMAP folders are too big but that offlineimap is too slow. Switching to isync (now mbsync) or muchsync is the way to go.