[kdepim-users] IMAP and permanent folder syncing

[Sandro Knauß]

Hey,

Do you really complain about, that akonadi syncs multiple times with the imap?
I don't understand that, if akonnadi would not sync with imap server you won't
see new mails nor any changes from other imap clients.

The logic is, that if you change anything in kmail this is synced directly to
the server, but to not push outdated changes to the server, akonadi updates
the folder. That is independend for them the "check for new mail" round.
That's why it can happen that you sync two times in a row with the server. But
in good state Akonadi only sync the changes.

As I understand your mail, that is not happening and akonadi is download
everytime your whole inbox. The reason why akonadi does this is that it
detects an error in the local cache and downloads all mails again. But in your
case the error isn't solved via resync the inbox completly so it resyncs it
again the next time. Normally that means, that you have anything in the inbox
folder that wasn't synced to the server (you can see items without a remote
id). If you delete these akonadi should get in a good state. Or you delete the
comple local cache for the inbox folder. You can do that in akonadiconsole
Browser sleect inbox right click "Clear Akonadi Cache", or on the right search
for items with an empty Remote Id.

Regards,

sandro

--

I am using KDE PIM 15.04.3 w/ Qt 5.5 (recent git head). I run (for various
reasons) a local Dovecot installation (2.2.19) which gets fed by fetchmail
and is used by Kmail2 via IMAP (obviously :P).
My problem: Kmail2 syncs the folders several times a day. You move a few
mails to the Trash and voila, a folder sync. You keep running Kmail2 for a
few hours without interacting with it and then read your mails... again,
folder sync. It can even happen two times in a row, if you are particularly
unlucky.
This has gotten to the point, where it is really very annoying. I love
Kmail2 and I just cannot believe that this is how it is supposed to be.
Since I have a rather large inbox with many subfolders, a sync of the main
folder can take a bit of time.
Is there anything I am missing? Is this a known issue? Can I somehow fix or
workaround it?
Thanks so much in advance for any help or advice.
So long,
Matthias

[René J.V. Bertin]

Since I have a rather large inbox with many subfolders, a sync of the main
folder can take a bit of time.

kdepim 4.1x allowed one to set the retrieval properties per folder, including the use of the parent folder's properties.
Those properties also included the option to sync at an interval or when selecting the folder, whether or not to retrieve full messages at once as well as the duration for which messages bodies are to be cached locally.

Local caching of course doesn't make sense when your IMAP server is already local (it's the reason for which I stopped using Apple Mail on my Mac).

The 4.1x range only supported IMAP IDLE ("new email notification without explicit checking") on a single folder. If that's still accurate, you're likely to miss new messages in the subfolders under the inbox if you do the sorting outside (upstream) of kmail. In that case it'd be up to you to (re)define your priorities...

BTW, what's the point of sorting email in subfolders under the inbox? I've always seen that as a nuisance imposed by certain IMAP servers, and have set my own local imap server to use folders that are independent of my inbox (local inbox isn't used) and that are stored under my home directory (and thus backed up). Then again I'm using a simple UW imap server with just a home-made patch so it doesn't necessarily serve my whole $HOME as its root maildir; it's good enough for my purposes.

R.

[René J.V. Bertin]

several times a day, randomly, I open a new mail and Kmail2 only shows its
standard "Retrieving Folder Contents" screen while it is syncing the whole

You're lucky that it doesn't get stuck in that screen (anymore)!

This cannot be normal. Kmail2 is the only client connected to the folder(s),
so no changes really happen from the outside that could lead to it thinking
that the folder is somehow "dirty" and needs re-syncing. What does happen
though is that the sieve scripts I have in place move mails into their
appropriate folders -- and those naturally run in the context of Dovecot. But
that shouldn't trip up Kmail2.

KMail claims to be very standard-compliant, so there is a chance that Dovecot fails to do something it should do *according to "The Standard"* when a filter moves mail into a folder. Do you ever get this behaviour with the Inbox, for instance (presuming you access that one via IMAP)?

It also sounds like you could at least experiment with running the filters via KMail. You claim it's the only client connecting, so you apparently don't also have your phone, tablet, toilet-pc or whatever that's connecting to the IMAP server ;) . So it seems just as natural to run those Sieve filters from KMail: it knows how to do that.

KMail is a great client, but you have to accept its mindset (and that of its developers). This is why I ended up patching my IMAP server myself, because KMail refused to add support for a server root directory (= the one containing the imap"folders"). It was either that, or live with KMail dredging through my full $HOME or go back to Thunderbird ...

R.

[Sandro Knauß]

Hey,

I'm using a external dovecot on a regular basis and don't your explained
behaviour. But I'm using, because it is an external one DIMAP to have the
offline functionallity.

I have a standard debian stable version of dovecot running, are you using any
newer dovecot?

But the think the problem lives in akonadi <-> kmail. Normally the behaviour
is any job that never ends. So please open akonadiconsole and go to the "job"
tab and aktivate the job logging and than use kmail uuntil it gets stuck and
than look to the job tab if there is one job running, that never ends. If yes
than we know that the problem is related to akonadi job handling.

Regards,

sandro

[Sandro Knauß]

Hey Matthias,

Interesting. Do you happen to use any Sieve filters on your IMAP?

I'm using sieve a lot on my imap - yes.

Checked that and once the syncing started (for the xth time today), it
filled up the log quite a bit. But all ended in due time and there were no
"hanging" jobs whatsoever. Sorry. :(

okay so every job is ended state ? and no is waiting. Okay than here isn't the
problem - good.

Next "standard thing" - go to the "Browser tab" in akonadiconsole and search
for items in the collections without a Remote Id. These items doesn't managed
it to got uploaded to the IMAP server. But because they stay there akonadi
thinks, that the sync is broken, because there are more items in the cache
than on the IMAP server and triggers a complete resync.

Regards,

sandro

[René J.V. Bertin]

Hi,

Another thing you might check: I just discovered another instance where the agents browser in akonadiconsole showed that one IMAP account was at over 2000000% done synchronising a folder (a big one, "All Mail" of a gmail account). Evidently kmail's progress indicator remained stuck at 100% ...

This may be a trivial bug, but if you see it systematically when one of those unexpected full syncs occur (and not otherwise) there almost has to be a connection.

R.

[Martin Steigerwald]

Post by Sandro Knauß
Hey,

Hey Sandro,

First I hope you have a merry christmas and still enjoy it. Please feel free

Post by Sandro Knauß
Do you really complain about, that akonadi syncs multiple times with the
imap? I don't understand that, if akonnadi would not sync with imap server
you won't see new mails nor any changes from other imap clients.
The logic is, that if you change anything in kmail this is synced directly
to the server, but to not push outdated changes to the server, akonadi
updates the folder. That is independend for them the "check for new mail"
round. That's why it can happen that you sync two times in a row with the
server. But in good state Akonadi only sync the changes.

Since quite some time I do believe Akonadi does excessive syncing. Not only
with IMAP, but also with local maildir.

For IMAP an example of today: I delete some spam mails from inbox. After about
three of them Akonadi stats syncing the inbox. And then it takes about a
minute or so on a bit flaky network before I can do anything again. And then I
delete 3 more spam mails and it again syncs. And I then think: "WTF?"

Example with local maildir filled by POP3: I press Ctrl-J in inbox after
having deleted spam there to apply all filters. I do this manually cause right
now I still have not brought my CRM114 local spam filter setup within kmail to
work again. It synchronizes every folder the filters sorted mail to. Again I
think "WTF?". Cause: Akonadi is the *only* user of that maildir. I want to
tell it: "Stop syncing!". Cause: I know if Akonadi puts a file there, it stays
there. Even with BTRFS. It will be there tomorrow and it will be there in ten
days. Unless I delete it.

I do believe Akonadi is doing a ton of unnecessary work.

I could understand if it would to an initial sync on starting up. Or in case
of IMAP in some intervals. But not after deleting 3 mails from inbox.

I brought this issue up before in kdepim mailing list. And up to know I do not
understand when exactly Akonadi triggers a sync and how to influence it.

And really if I am usually the only user of an IMAP box, why not give me the
option to say: Only sync once a day or once every 4 hours or not at all unless
on starting of Akonadi and enjoy full performance? (Ideally IMAP would have
change tracking, but well.)

I do have a Dovecot IMAP server. From what I know its blazingly fast, yet
still in my KMail setup I have delays of half a minute at times cause Akonadi
is busy elsewhise. And that is the second thing that makes this so unbearable:

If Akonadi is busy syncing a folder it does not talk to KMail again. From what
I understood one of the main purposes of Akonadi was to give an always
responsive GUI to KMail. And yet Akonadi fails miserably with this goal in a
good number of setups. If I in KMail click on a mail I mean the following:
Akonadi interrupt syncing and other background stuff and give me that mail
*now*. Akonadi doesn´t and that is what creates immense frustrations for users
of KMail.

I did look in the code at some time, but I didn´t find where it actually
triggers that syncing. For maildir I would like to disable that syncing
completely just to see what performance boost it would give. And even for IMAP
I would like to be able to tell it to do it much more rarely. I never got any
good answer to the question: In which situation does it trigger syncing and
*why*?

I know I have a huge setup, but I sincerely believe that the delays in
response time in KMail are unnecessary and related to doing work that does not
really need to be done (or at least not that often).

Thanks,