Kmail not reflecting contents of ~/mail (maildir)

[Maurice]

The problem is although my (maildir) ~/mail database contains an
.inbox.directory of around 20 sub-folders, Kmail now fails to show ANY of
them, even shows an empty Sent folder, and fails to show any Inbox emails
older than the date after which I became aware of the problem, and now
fails to download any emails.

N.B. I have temporarily migrated to Thunderbird for email purposes.

Presumably there is an Akonadi/Baloo mixup somewhere, and I ask if anyone
can here vrecommend a way of achieving re-indexing of the files in ~/
mail,
please.

[I don't think it's a Kmail problem, as with another user on the system
it is behaving normally.)

[Ianseeks]

This post might be inappropriate. Click to display it.

[Maurice]

Post by Ianseeks
Does kmail point to your mail folder directory?

In Kmail/Configure/Accounts/Local Folders/ it shows:

"/home/<user>/mail" (correct)

and the note: "The selected path is a valid Maildir."

So Akonadi (/Baloo?) are not providing a correct index of the contents of
that mail file.

Kmail is downloading email but does not show them anywhere...

N.B. A copy of the same mail directory on my laptop IS correctly
reflected
in its Kmail.

[Ian Douglas]

Post by Maurice
Kmail is downloading email but does not show them anywhere...

I had similar issues since update in January.

They're likely in a different folder now.

eg
/home/[user]/.local/share/.local-mail.directory/
vs
/home/[user]/.local/share/local-mail/
vs
/home/[user]/.local/share/akonadi/file_db_data/[numbered directories]
(which is where my new mail seems to be)

and other mail from 2013 "upgrade" in
/home/[user]/.local/share/akonadi/file_lost+found/

Hope it helps you find the mail...

Cheers, Ian

[Maurice]

Post by Ian Douglas
Hope it helps you find the mail...

True, there are foloders there, but they all contain "0 files".

Still need to get the true ~/mail file re-indexed somehow...

Regards,

[Ianseeks]

Post by Maurice

Post by Ian Douglas
Hope it helps you find the mail...

True, there are foloders there, but they all contain "0 files".
Still need to get the true ~/mail file re-indexed somehow...
Regards,

I'm not sure what the problem might be but if it can't "see" the emails, i doubt it can index them. Perhaps have a look at this page to see if this might help. Be aware that it does mess up the current kmail config so you'll need to redo that if you follow these instructions.
https://docs.kde.org/trunk5/en/pim/kmail2/clean-start-after-a-failed-migration.html

[Maurice]

Post by Ianseeks
I'm not sure what the problem might be but if it can't "see" the emails,
i doubt it can index them.

Well, on the same Linux (Mageia-6) install, another user's Kmail picks up
and runs with its own ~/mail structure - first time, and shows the
files in each folder. So it's not Kmail itself that is the problem, but
something in my /home that us causing it, hence my suspicion of Akonadi.
But perhaps it's something else.

In my user session, it was doing until last week, when it suddenly
decided
not to behave. It shows an enpty Sent folder, none of the subfolders,
and in Inbox only a few recent emails. Unusable.

What could be causing the difference between the two users on the same
system? I've never before seen Kmail fail to reflect the folder structure
and maildir files in ~/mail, and I've been using Kmail on several
different machines for years.

Actually, I'm not stuck for email function, as some months ago I migrated
my maildir DB (contents of ~/mail) to Thunderbird, which - although it
has a terrible GUI - I've found much more reliable as well as simpler to
clone onto other machines, and has allowed me to have an email service
that does not get hung up on "Retrieving Folder Contents" until the time
when that Kmail problem has been solved.
But in the meantime I have kept running Kmail so as to monitor its
progress with the "Retrieving Folder Contents" problem.

If I can't get Kmail on my desktop user to honour the contents of ~/mail
I
shall have to rely on Kmail on my laptop (same Linux level using copy of
my ~/mail) to do that monitoring - or an extra user on this one.

I looked at the reference to 'failed migration' you pointed to, but this
is not a migration problem; it had been working well until recently.
But I will try its suggestions about deleting Akonadi folders -
something I may not have done thoroughly enough.

The $64 question is: How to get Kmail to properly reflect the folder
structure and maildir contents of ~/mail, which it appears to completely
ignore even though it shows that as a valid maildir directory...

Regards,

[Ianseeks]

Post by Maurice

Post by Ianseeks
I'm not sure what the problem might be but if it can't "see" the emails,
i doubt it can index them.

Well, on the same Linux (Mageia-6) install, another user's Kmail picks up
and runs with its own ~/mail structure - first time, and shows the
files in each folder. So it's not Kmail itself that is the problem, but
something in my /home that us causing it, hence my suspicion of Akonadi.
But perhaps it's something else.

I've always used opensuse. Is the akonadi/mysql install all upto date? e.g. i'm running akonadiserver 5.8.3. and postgresql 9.6

Post by Maurice
In my user session, it was doing until last week, when it suddenly
decided
not to behave. It shows an enpty Sent folder, none of the subfolders,
and in Inbox only a few recent emails. Unusable.
What could be causing the difference between the two users on the same
system? I've never before seen Kmail fail to reflect the folder structure
and maildir files in ~/mail, and I've been using Kmail on several
different machines for years.

Have you tried creating a new user, transferring the mail structure, setting correct permissions/ownership etc and seeing if it works with the new user?

Post by Maurice
Actually, I'm not stuck for email function, as some months ago I migrated
my maildir DB (contents of ~/mail) to Thunderbird, which - although it
has a terrible GUI - I've found much more reliable as well as simpler to
clone onto other machines, and has allowed me to have an email service
that does not get hung up on "Retrieving Folder Contents" until the time
when that Kmail problem has been solved.
But in the meantime I have kept running Kmail so as to monitor its
progress with the "Retrieving Folder Contents" problem.

I've had plenty of those but they went away when i started to use Postgresql.

Post by Maurice
If I can't get Kmail on my desktop user to honour the contents of ~/mail
I
shall have to rely on Kmail on my laptop (same Linux level using copy of
my ~/mail) to do that monitoring - or an extra user on this one.
I looked at the reference to 'failed migration' you pointed to, but this
is not a migration problem; it had been working well until recently.
But I will try its suggestions about deleting Akonadi folders -
something I may not have done thoroughly enough.
The $64 question is: How to get Kmail to properly reflect the folder
structure and maildir contents of ~/mail, which it appears to completely
ignore even though it shows that as a valid maildir directory...

Seems like it might need one of the PIM devs to offer an idea.

Post by Maurice
Regards,

[Maurice]

Post by Ianseeks
Have you tried creating a new user, transferring the mail structure,
setting correct permissions/ownership etc and seeing if it works with
the new user?

On the list for tomorrow

Post by Ianseeks

[quoted text muted]

I've had plenty of those but they went away when i started to use Postgresql.

I suspect it occurs only when using filtering on downloaded emails

Post by Ianseeks
Seems like it might need one of the PIM devs to offer an idea.

Is there q recommended way of finding wuch a dev?

Regards,

[Ianseeks]

Post by Maurice

Post by Ianseeks
Have you tried creating a new user, transferring the mail structure,
setting correct permissions/ownership etc and seeing if it works with
the new user?

On the list for tomorrow

Post by Ianseeks

[quoted text muted]

I've had plenty of those but they went away when i started to use Postgresql.

I suspect it occurs only when using filtering on downloaded emails

I stopped it filtering during the download. i now left the inbox fill up and then press the "Apply filters" and that stopped a lot of nonsense

Post by Maurice

Post by Ianseeks
Seems like it might need one of the PIM devs to offer an idea.

Is there q recommended way of finding wuch a dev?

they usually read this list but you could subscribe to the kde-pim.kde.org list as its a Dev list. Daniel Vatril, David Faure, Laurent Montel are the big names and there are others like Ingo Klocker. I don't approach them directly but they do sometimes answer a question but i'm not sure if they are okay with questions on the dev list. You might want to create bug and then put the number in the email question.

Post by Maurice
Regards,

[Maurice]

Post by Ianseeks
Have you tried creating a new user, transferring the mail structure,
setting correct permissions/ownership etc and seeing if it works with
the new user?

Now done that. Kmail did pick up the subfolder structure in the copy
of my ~/mail and gradually showed all the emails in there.
Just had to start kmail and it all started to appear automatically.

(Ditto with the Mageia-6 on my laptop.)

So - as I had suspected - something in /home/me is now preventing kmail
from properly reflecting the contents of ~/mail any more.

The $64 question is: How to find the culprit?!

Regards,

[Maurice]

something in /home/me is now preventing kmail from properly reflecting
the contents of ~/mail any more.
The $64 question is: How to find the culprit?!

Added a folder with odd name, then did search for that name in ~/.

Name appeared in ~/.mail.directory, which somehow Kmail was somehow
regarding as ~/mail!

Renamed ~/mail to ~/.mail.directory and Kmail soon picked that up and
eventually displayed all the missing subfolders and contents!

$64:
Why did Kmail suddenly decide to change to ~/.mail.directory instead
of ~./mail (despite it still declaring under /Configure/Local Folders
that
the maildir files were in ~/mail)?

[Jörg Schaible]

Post by Maurice

something in /home/me is now preventing kmail from properly reflecting
the contents of ~/mail any more.
The $64 question is: How to find the culprit?!

Added a folder with odd name, then did search for that name in ~/.
Name appeared in ~/.mail.directory, which somehow Kmail was somehow
regarding as ~/mail!
Renamed ~/mail to ~/.mail.directory and Kmail soon picked that up and
eventually displayed all the missing subfolders and contents!
Why did Kmail suddenly decide to change to ~/.mail.directory instead of
~./mail (despite it still declaring under /Configure/Local Folders that
the maildir files were in ~/mail)?

Because KMail has its own idea about a proper maildir(++) structure ... :-/

Cheers,
Jörg

[Ingo Klöcker]

Post by Jörg Schaible

Post by Maurice

something in /home/me is now preventing kmail from properly reflecting
the contents of ~/mail any more.
The $64 question is: How to find the culprit?!

Added a folder with odd name, then did search for that name in ~/.
Name appeared in ~/.mail.directory, which somehow Kmail was somehow
regarding as ~/mail!
Renamed ~/mail to ~/.mail.directory and Kmail soon picked that up and
eventually displayed all the missing subfolders and contents!
Why did Kmail suddenly decide to change to ~/.mail.directory instead of
~./mail (despite it still declaring under /Configure/Local Folders that
the maildir files were in ~/mail)?

This sounds like a bug to me. All folders in Local Folders should be inside
the configured root directory ~/mail. Top-level folders directly in ~/mail,
subfolders in .foo.directory subdirectories.

Post by Jörg Schaible
Because KMail has its own idea about a proper maildir(++) structure ... :-/

Yeah, well. Back when we added maildir support additionally to mbox support we
kept the tried and trusted way for handling subfolders. Also, in KMail mbox
and maildir folders could (and still can) be mixed in the same folder
hierarchy which, for obvious reason, isn't possible with "a proper maildir(++)
structure".

Regards,
Ingo

[Ianseeks]

Post by Maurice

something in /home/me is now preventing kmail from properly reflecting
the contents of ~/mail any more.
The $64 question is: How to find the culprit?!

Added a folder with odd name, then did search for that name in ~/.
Name appeared in ~/.mail.directory, which somehow Kmail was somehow
regarding as ~/mail!
Renamed ~/mail to ~/.mail.directory and Kmail soon picked that up and
eventually displayed all the missing subfolders and contents!
Why did Kmail suddenly decide to change to ~/.mail.directory instead
of ~./mail (despite it still declaring under /Configure/Local Folders
that
the maildir files were in ~/mail)?

Great that you've solved the issue. As a result of your discovery I thought i'd better check my folders too.

I've got a "/home/XX/.local/share/local-mail" (as per kmail config for Local Folders) which as a few old folders in it plus standard ones like inbox and all the folders are empty and "/home/XX/.local/share/.local-mail.directory" where all the mail folders and mails are actually resident.

Both ~/local-mail and ~/.local-mail.directory have Inbox folders that are completely empty but the view inside kmail shows me a lot of emails in the Inbox.

I'm not sure i understand this kmail directory structure anymore

[Maurice]

Post by Ianseeks
'm not sure i understand this kmail directory structure anymore

No problem with the structure, just the whereabouts of the maildir DB!

Also found it's Addressbook was empty, but was able to restore it from an
export copy I had stored for the purpose of cloning to the laptop.

Have now restored the Folder View settings and also re-selected the
various email filter folder destinations.

Now it's ust a question of waiting to see which new maildir
directory Kmail will decide to change to next... :-)

But I would like to find where in /home the use of '.mail.directory' as
the maildir directory is pointed to.

Any takers?!

[Ianseeks]

Post by Maurice

Post by Ianseeks
'm not sure i understand this kmail directory structure anymore

No problem with the structure, just the whereabouts of the maildir DB!

I have no idea - have you looked a folder called something like "akonadir_maildir_resource_0" , this has a duplication of the folder structure with email in it. i think the use of meaningful names to the user is no longer an option.

Post by Maurice
Also found it's Addressbook was empty, but was able to restore it from an
export copy I had stored for the purpose of cloning to the laptop.
Have now restored the Folder View settings and also re-selected the
various email filter folder destinations.
Now it's ust a question of waiting to see which new maildir
directory Kmail will decide to change to next... :-)

:)

Post by Maurice
But I would like to find where in /home the use of '.mail.directory' as
the maildir directory is pointed to.

i'm guessing its somewhere in the internal akonadi config as the setting in kmail seems redundant

Post by Maurice
Any takers?!

I seem to have multiple folders in ~/.local/share referencing mail in some way namely "local-mail", ".local.mail.directory", "kmail2", "akonadir_maildir_resource_0" - none of them seem to have the full compliment of mails as seen via kmail, the oldest seems to be 2017 and i have no idea how the "cur"/"new"/tmp" folders work in each mail folder - i would have thought new unread emails into the "new" and then transferred to "cur" when read - but no, i've got both new and cur containing old read emails and i know all emails are read.
Its seems a black art.. :)

[Peter Humphrey]

Post by Maurice
I would like to find where in /home the use of '.mail.directory' as
the maildir directory is pointed to.
Any takers?!

$ find . -xdev -type f -exec grep '-l' '.mail.directory' {} +
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1533251931166.R43.peak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1533248549494.R992.peak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1532890470768.R75.peak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1533286690845.R508.peak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1532886859937.R830.peak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1533303043111.R191.peak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1533295827003.R907.peak:2,S
./.local/share/akonadi/file_db_data/15/63315_r0
./.local/share/akonadi/file_db_data/21/63321_r0

HtH, but I doubt it does, much. ;-) I assume that the ../cur/.. files are
the ones containing this conversation, but why there are two file_db_data
directories is a mystery to me.

(Peak is the name of this box.)

[Peter Humphrey]

Post by Peter Humphrey

Post by Maurice
I would like to find where in /home the use of '.mail.directory' as
the maildir directory is pointed to.
Any takers?!

$ find . -xdev -type f -exec grep '-l' '.mail.directory' {} +
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1533251931166.R43.p
eak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1533248549494.R992
.peak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1532890470768.R75.
peak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1533286690845.R508
.peak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1532886859937.R830
.peak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1533303043111.R191
.peak:2,S
./.local/share/local-mail/.Linux.directory/KDE-PIM/cur/1533295827003.R907
.peak:2,S ./.local/share/akonadi/file_db_data/15/63315_r0
./.local/share/akonadi/file_db_data/21/63321_r0
HtH, but I doubt it does, much. ;-) I assume that the ../cur/.. files are
the ones containing this conversation, but why there are two file_db_data
directories is a mystery to me.

In fact there are 100 of them, all with two-digit numerical names and all
but two with the same timestamp; those two changed when I started this
reply.

[Ingo Klöcker]

Post by Ianseeks
I've got a "/home/XX/.local/share/local-mail" (as per kmail config for Local
Folders) which as a few old folders in it plus standard ones like inbox
and all the folders are empty and
"/home/XX/.local/share/.local-mail.directory" where all the mail folders
and mails are actually resident.
Both ~/local-mail and ~/.local-mail.directory have Inbox folders that are
completely empty but the view inside kmail shows me a lot of emails in the
Inbox.
I'm not sure i understand this kmail directory structure anymore

It's possible that KMail failed to write the emails to those (file system)
folders. In this case the emails are only stored in the database KMail uses
for faster access (as cache). You could try to copy those emails from your
Inbox to a different folder with KMail and then see if they appear somewhere
inside ~/local-mail or ~/.local-mail.directory.


Regards,
Ingo

[Ianseeks]

Post by Ingo Klöcker

Post by Ianseeks
I've got a "/home/XX/.local/share/local-mail" (as per kmail config for Local
Folders) which as a few old folders in it plus standard ones like inbox
and all the folders are empty and
"/home/XX/.local/share/.local-mail.directory" where all the mail folders
and mails are actually resident.
Both ~/local-mail and ~/.local-mail.directory have Inbox folders that are
completely empty but the view inside kmail shows me a lot of emails in the
Inbox.
I'm not sure i understand this kmail directory structure anymore

It's possible that KMail failed to write the emails to those (file system)
folders. In this case the emails are only stored in the database KMail uses
for faster access (as cache). You could try to copy those emails from your
Inbox to a different folder with KMail and then see if they appear somewhere
inside ~/local-mail or ~/.local-mail.directory.

Thanks, i'll give that a go

Post by Ingo Klöcker
Regards,
Ingo