FolderTagSync, a simplified mechanism maintaining 1-1 coorespondence between maildir folders and tags.

[naegling]

Intended an an RFC analogue for defining the behavior of a new afew feature, FolderTagSync (herein FTS). FTS is intended to exist side-by-side with MailMover. afew users can select whichever feature set most closely matches their use-case scenario.

The idea is to add a new section to afew config that define a set of mappings between a maildir folder and a set of search tags as per the following example:

[FolderTagSync]
personal/Sent Items = +personal +sent -inbox -spam -trash
personal/Inbox = +personal -sent +inbox -spam -trash
personal/Junk Email = +personal -sent -inbox +spam -trash
personal/Deleted Items = +personal -sent -inbox -spam +trash
personal/Archive = +personal -sent -inbox -spam -trash

FolderTagSync functions in 2 passes.

1. --move-mails. This pass must precede mail sync with imap server (isync, offlineimap, etc). email matching the tag settings that are not in the folder are relocated.
2. --tag. This pass must proceed mail sync with imap server. email in the folder that do not match the mapped tag settings are tag adjusted to comply.

Implementation Details:

• To prevent unexpected results, the tags should correspond precisely with folder location. For example, the sent tag can be set by an afew filter based upon criteria other than folder location. For those of us that keep record of prior sent email in the archive may not expect the resulting folder move. Similar issues exist with notmuch special flags.
• Currently, the tagging pass must consider all email, not just those tagged 'new'. Although that works, it is more inefficient than it need be. notmuch has already located the moved email, but just updates in location in database and without changing its tags. My intention is to modify notmuch to add a 'mov' tag to complement the 'new' tag. Then the FTS 2nd pass need only consider a comparatively small number of messages.

[naegling]

BTW, there is a simpler way to handle the folder tags, but less flexible.
define a set of folder tags: { sent inbox spam trash}
then for each folder select which tag should be set
Sent Items = sent
Inbox = inbox
Junk Email = spam
Deleted Items = trash

But this lacks flexibility,

[naegling]

No comments? The above FTS is implemented and currently under test (with my own mail), but if there is no interest then I will not pester the group with a PR.

[anarcat]

this approach is exactly what i am missing in afew right now. i am not using afew because it assumes it runs on a single workstation, while I have multiple. therefore i use state in the maildir as the reference to create tags in notmuch. for example, files moved to the "ham" folder get their "spam" tag removed and vice-versa. it's super tricky to implement, because you can get into a bouncing contest between the two workstations, for example.

i also use the F flag on messages to represent the inbox tag, which is also particularly tricky to implement.

this idea would possibly help me adopt afew.

[naegling]

@anarcat , my deployment config is multi-client, multi-account. I'm using a cloud-based server to pull imap and gmail accounts with mbsync and lieer respectively. 3 different workstations then sync to server with muchsync. My intent is to run afew only on server as well, then muchsync will replicate its tagging on the workstations.

In what I have seen so far, I have 2 cases in which the bouncing does not converge to a single state.

1. a message is stored in multiple maildirs. I resolve this by simply refusing to FolderTag such messages. Just find and remove such duplicates. IMHO, seems contrary to the entire intent of notmuch, but would welcome enlightenment on usecase.
2. a FolderTag tag is ambiguous due to overloaded use. For example, the default inbox filter removes new (as it must), but also applies the inbox tag. That could be a problem if the new msg does not happen to reside in a inbox folder. Similarly with the spam and sent mail filters (I renamed the filter's sent to from-me tag to reserve sent as a FolderTag.

My current implementation requires the FolderTags to be added and removed atomically. E.g. moving a message from o365's inbox to deleted items requires +trash and -inbox. I thought that would be easy to handle from alot. But turns out that currently alot handles adds and removes as separate commands and the selected message can change between commands. Still trying to decide if the right answer is to add atomic add/remove to alot, or add configurable radio Tag Groups (like UI radio buttons) to notmuch (adding a tag from a configured group automagically removes all other group tags).

[anarcat]

1. a message is stored in multiple maildirs. I resolve this by simply refusing to FolderTag such messages. Just find and remove such duplicates. IMHO, seems contrary to the entire intent of notmuch, but would welcome enlightenment on usecase.

how do you actually deduplicate those? checksum? message-id? typically this happens on mailing lists where you receive a copy of your message...

[naegling]

I just leveraged afew itself. Since notmuch tagging is applied at the message level, the de duplication needs to be to the notmuch message. I believe this is message-id and is definitely not a checksum. Fortunately, notmuch itself can find the duplicates for us.

I just temporarily used the message file iteration in my FolderTagSync implementation (similar to the one already in MailMover). When multiple files are found for a single message, the modified handler selected one to keep and moved the overs to a temp directory outside of the maildir. After de-duping the messages, I removed the temporary code.

Not really sure how I got my dupes. They were not from mailing lists (at least not the sample I looked at).

[agenbite]

Fwiw, I use a similar setup with folder/tag replication. In my case, I don't mind about duplicates (they are transparent with notmuch and disk space is not expensive) but I do mind about listing my >100 folders. I operate with a requirement to have identical names for folders and corresponding tags, i.e., messages in folder Folder/Subfolder will be tagged "+Folder/Subfolder -inbox". Then I use a pre-new script which moves files around before syncronizing. I'm still testing this setup: while it is true that it has worked very well for almost a year now, it is also true that I have some corner cases that need polishing.

I'll look into your solution (thanks for contributing!), but folder listing is a no-go.

[naegling]

I did not remove the duplicates to save some disk space. Its that messages are tagged, not files. So its easier to prevent message bouncing between folders if the tags uniquely identify a folder and vice-versa. i.e. a 1-1 map.

However, I am reconsidering the above scheme. Its implemented and seems to be working well, but adding a folder is too fault prone. (In other words, I f'ed up adding a new folder myself. that's generally a bad sign :))

The major requirements I have is to support multiple accounts, each with a set of primary folders (inbox, trash, spam, sent). within inbox, I have topical folders. When I remove the inbox tag (without adding trash, spam) I want the topical tag to remain. That way I can (in a limited way) tag archived email from my phone by moving the email to a topical folder and then archiving it.