Evolution 2.24.1 really slow with big IMAP folders

thank you for your bug report, the issue is an upstream one though, could you open the bug on bugzilla.gnome.org where the people writting the software will read it? optimization work is rather upstream work than distribution one, especially that you are the first one to complain about the new version speed so some option you are using could trigger the issue

It seems that someone has already reported it upstream, this is the link to the upstream bug report:

http://bugzilla.gnome.org/show_bug.cgi?id=558883

Hope this helps.

As I can see, patches are already being developed upstream.

I hope they will be soon included in Ubuntu, and I hope that they will be pushed as updates as soon as possibile, because I can't afford to wait for Ubuntu 9.04 to have a working Evolution again.

did you try the upstream changes? can you confirm they fix your issue?

I have manually created indexes as stated in the upstream bug report, I have not recompiled Evolution, so I don't know if the patch actually works.

Anyway, even if I have created indexes, Evolution is still slow (not as much as before) because (as it seems from upstream discussion) it still handles lots of UIDS using a slow method. (see post #8)

I think that if this issue is not solved upstream properly and in a reasonable time, it could be nice to have the old version of Evolution packaged for Ubuntu 8.10, so people can install the old one if they need.

Switching to a new database format seems to have created a lot of efficiency issues, that will probably need months to solve upstream, so maybe packing the old Evolution for Intrepid is the way to go.

Thanks for your prompt answers and continued help.

the old version will not be distributed in intrepid no since that would be lot of extra work for almost no win, you can still use the lts version if you need to run the previous version

I tried the trick with creating indexes and for me evolution now is even faster than the previous version. I'm using it on reiserfs, a normal intrepid upgrade on x86_64.

Upstream analisys continues.

http://bugzilla.gnome.org/show_bug.cgi?id=558883

It seems that the bug is far more evident (Evolution is really slower) when you have more than one account. There is some sort of concurrency issue.

I applied both patches to e-d-s, the index patch really helps a lot.
If you want to test yourself you can get packages at my ppa http://ppa.launchpad.net/manno/ubuntu

I have tried the updated version, and it is still terrilbly slow when you have more than one IMAP account. If I keep only one account enabled and disable all the others, it becomes notably faster. I suppose that there is some sort of concurrency issue that forces continuous flushes to disk to the sqlite3 database files.

This issue has been reported by me at http://bugzilla.gnome.org/show_bug.cgi?id=558883#c23

Fabio, do you use filters in Evolution to sort the email? I found out that if I disable filtering on the INBOX (while still having two IMAP accounts) it's definitely better. Seems like it's filtering the whole INBOX (there are about 3k messages there) instead of just the new stuff.

I use server-side filtering, so I have completely disabled filtering in Evolution. But I have enabled checking for new mail in every folder, since I get mail delivered directly to the folders by the server.

If you look at the upstream bug tracking page, at http://bugzilla.gnome.org/show_bug.cgi?id=558883, you'll see that there are new patches addressing issues like the one you seem to have. Basically it seems that Evolution is rescanning every single email in every single folder when checking for new mail, and maybe it does the same when filtering.

I am really disapponted by the incredible number of crippling bugs that this update of Evolution has introduced.

It seems that patches have been commited to stable tree upstream.

http://bugzilla.gnome.org/show_bug.cgi?id=558883#c45

Maybe it's time to include these patches and post the new binary packages to proposed upates.

new GNOME tarballs are due today so there is no need to hurry backporting patches now rather than using those

Accepted evolution-exchange into intrepid-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Accepted evolution-data-server into intrepid-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

I have just installed the new evolution-data-server from proposed, it is considerably faster than before, but still not as fast as the old evolution that did not use sqlite. It seems to me that when it runs a lot of concurrent activities (like scanning different folders for different accounts for new messages) it slows down a lot.

If I enable 4 accounts, a full "send/receive" runs for one minute (with gigabit connection to the imap server). If I enable only one account, a full "send/receive" runs for 4 seconds with quite no disk activity. It is obvious that there is something wrong when Evolution runs imap folder update tasks concurrently.

It also uses a lot of memory, and grows indefinitely: after 10 minutes it's using more than 400 MB of memory.

Thanks Fabio for testing. So it's still not "good" yet, but since it is an improvement, I mark this as verified, so that this version can go to -updates.

I agree, it's an improvement.

This bug was fixed in the package evolution-data-server - 2.24.2-0ubuntu1

---------------
evolution-data-server (2.24.2-0ubuntu1) intrepid-proposed; urgency=low

  * New upstream version:
    Bug Fixes:
    - #546637: Now unread vfolder should be better (lp: #275952)
    - #557348: Clear summary when we reset query.
    - #558883: Improve performance in case of large IMAP folders. (lp: #292739)
    - #560076: Handle followup better
    - #561069: If nothing to expunge, don't crash. (lp: #298526)

evolution-data-server (2.24.1.1-0ubuntu1) intrepid-proposed; urgency=low

  * New upstream version
    Bug Fixes:
    - #209514: (Novell Bugzilla) Evolution Groupwise missing mails
    - #372382: (Novell Bugzilla) VUL-0: OTRS SOAP interface vulnerability
    - #434946: (Novell Bugzilla) Deleted Messages in Trash Show Bold and Behave
      Like New Messages
    - #434950: Newly Fetched Mail Doesn't Update Mailbox Count in
      QuickViewer Until Exit and Entry
    - #434958: 'Empty Trash' Sometimes Requires Multiple Attempts Before
      Initiating
    - #435725: Disable Right-Mouse "Delete" On GrouPWise System Email Folders
    - #435727: Deleted Shared Folder Crashes Evolution
    - #435964: Evo crashed after switching from offline to online
      in GW addressbook
    - #440502: Tracker Bug: GroupWise Magic Patch Tracker
    - #532136: Evolution crashed - modifying a local task and changed offline
      state
    - #555979: Evolution does not empty trash on exit (lp: #281093)
    - #556119: Takes a very long time to index newsgroup.
    - #558727: Format string bugs causing potential crashes
    - #558737: Evolution freezes on trying to store folder
    - #558883: Evolution 2.24 is terribly slow with large IMAP folders.

 -- Sebastien Bacher <email address hidden> Mon, 24 Nov 2008 15:09:05 +0100

Copied e-d-s intrepid-proposed to jaunty.

This bug was fixed in the package evolution-exchange - 2.24.2-0ubuntu1

---------------
evolution-exchange (2.24.2-0ubuntu1) intrepid-proposed; urgency=low

  * New upstream version:
    Bug Fixes:
    - #540346: Do not try to fetch contacts when not connected
    - #558883: Improve performance in case of large IMAP folders (lp: #292739)

 -- Sebastien Bacher <email address hidden> Mon, 24 Nov 2008 15:18:45 +0100

e-exchange intrepid-proposed copied to jaunty.

Please note the upstream bug is still open.
However the latest (4th December) patch does away with dbsort and this helps a lot. You can try my ppa if you want to test it.

Mario, I'm testing your patched Evolution now. It still has a bug regarding update of unread messages count in folders pane, which makes it hard to tell if I have got new mail or not. (see http://bugzilla.gnome.org/show_bug.cgi?id=559391)

It seems to be still slow compared to the version that used the old database backend, but it now seems at least acceptable.

I'll test it further, while at the office (with a GB connection to the imap server) and out (with a 512K adsl uplink).

Accepted into intrepid-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Using the version from proposed, it's definitely a bit better, but still with two reasonably big IMAP accounts my drive grinds for a few minutes until everything is updated. It's still worse than with hardy, but somewhat livable.

Okay, I finally used my head for a while, and found a work-around for this problem. In short,
cat ~/evolution/mai/imap/*/folders.db > /dev/null
before starting evolution seems to make it load at least as fast as the versions before sqlite.

Now, this seems to be an issue either with the block device scheduler, sqlite or both. It's visible that when you have two tasks that do a lot of disk IO (firefox tends to do that when first reading the location bar cache, or updatedb, or dpkg), then the whole system grinds down to a halt. Evolution does the same just by itself, by doing a lot of small reads on two files in parallel and most drives don't really like that.
(I might try testing this on a RAID1 array of faster drives, if that could be useful)

For the record, I'm using standard intrepid + proposed.

Dear Vasil,

There are discussions going on elsewhere about the poor performance of the block device scheduler in new kernels, so maybe you are right. Evolution does a lot of small i/o requests, and the whole problem gets really worse when you have more than one account, because the number of parallel requests grows with the number of accounts you have.

Maybe this performance problem is caused by a sum of causes: poor coding, poor sqlite performance, poor kernel i/o scheduler performance. We should test by recompiling the kernel (on the same hardware) and changing the i/o scheduler it uses.

Fabio,

Changing the IO scheduler is easy - it's in /sys/block/$devname/queue/scheduler ($devname = sda in my case), and ubuntu comes with all the normal in-kernel schedulers compiled in - deadline, anticipatory, noop and cfq (the last one being the default one).

I'll try this with the other schedulers and will see if that makes any difference. BTW, can you post some links to the discussions about the block scheduler performance, they might be useful to this bug too.

I don't think that SQLite is the problem, as when I preload it it's blindingly fast - and when you have a single account, it's again really fast. If someone can test evolution with postgresql or mysql, I think we'll see the same results (without the preloading).

Vasil,

I have tried testing every scheduler available. I have copied a 700 MB file between tests to flush the cache.

The results are as follows:

- Noop is the slowest scheduler, by maybe 5-10%, no more.

- All of the other schedulers seem to have the same performance.

- If I don't flush the cache, Evolution is *REALLY* fast (at least 5 times faster, maybe 10) with every scheduler.

- Just for information, copying the 700 MB file from and to the same disk takes approx 36 seconds with every scheduler but "noop", and 40 with "noop".

So the bottom line, at least on my system (Dell Precision M65 laptop, 1 GB RAM, 7200rpm sata hdd, 32 bit Core Duo T2500, seems to be that basically all schedulers show equal performance.

I have not tried to see how responsive is the system while running the tests, so my test does not focus on multitasking or desktop responsiveness, only on the timing of a single operation that basically can use all of the CPU time and disk I/O bandwiidth (copying a single file or opening Evolution).