Add recipient-aware Message-ID deduplication

[iafred-bot]

Summary

This PR adds an optional recipient-aware Message-ID deduplication mode.

By default, Piler keeps its current behavior: duplicate detection is based on Message-ID only. When enabled with:

deduplicate_messages_by_recipient=1

or, in the Docker entrypoint:

DEDUPLICATE_MESSAGES_BY_RECIPIENT=1

a message is considered a duplicate only if an archived message already has the same Message-ID, the same stored sender, and the exact same recipient set.

Use case

When Piler receives an email addressed to a functional/shared mailbox, the initial message is archived correctly and the indexed fields match that first delivery.

On the final mail server hosting that functional mailbox, mailbox forwarding rules may expand the message to the final recipients. When those expanded deliveries come back to Piler, they can currently be treated as duplicates, probably because the Message-ID is unchanged, even though the recipient is no longer the same as in the original delivery.

The impact is that Piler may not archive those expanded deliveries, creating a blind spot for emails received by some users through functional mailbox forwarding.

Behavior

With deduplicate_messages_by_recipient=1:

• same Message-ID, same sender, same exact recipient set: duplicate
• same Message-ID, same sender, different recipient set: archived separately
• same Message-ID, different sender: archived separately
• if sender or recipients cannot be parsed, Piler falls back to the existing Message-ID-only behavior for that message

The default remains 0, so existing installations are unchanged unless they explicitly enable the option.

Validation

Tested on Debian 13 runner:

./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make -j$(nproc)
./unit_tests/run.sh

The unit test suite passes, including the new recipient-aware duplicate detection cases.

Prepared by iafred-bot
GitLab: @fredbcode
GitHub: @fredbcode

[jsuto]

I think the problem can be solved with proper grouping. The UI supports creating groups, so the functional mailbox members can get an extra email address, ie. the functional mailbox's address so that they can see emails sent to "To: functional@example.com" address. Or another possible solution if you are using LDAP or similar authentication is that you add the members to an ldap group that has the address of the functional mailbox. I think the problem can be solved with existing functionality.

[jsuto]

Please see the comment above.

[iafred-bot]

I understand the idea behind using groups, but unfortunately that does not cover our use case.

The mailbox concerned here is not a functional/group mailbox with a list of members that can be managed or exposed through LDAP. It is a regular mailbox with forwarding rules configured at the mail system level.

For example, a message is sent to:

functional@example.com

and this mailbox has forwarding rules sending the message to:

user1@example.com
user2@example.com
user3@example.com

The forwarded messages keep the original Message-ID, which means that with the current deduplication mechanism they are considered duplicates of the original message and are therefore not archived separately.

The main issue is that these forwarding rules cannot be automatically discovered by the archive application. They are mail-routing rules, not group membership information, and in our environment there is no LDAP group containing these recipients that the application could use.

Manually creating groups in the archive application would therefore not be a viable solution: it would require maintaining a duplicate representation of all forwarding rules, and those rules can change independently of the archive system.

The Message-ID + To approach in this merge request addresses precisely this situation. It keeps the current deduplication behavior for the same message delivered to the same recipient set, while allowing the same original message to be archived when it is forwarded to a different To address.

This would also remain fully compatible with the existing grouping functionality, while covering forwarding scenarios that cannot be represented as groups.

For this reason, I believe the proposed change addresses a real use case that cannot be reliably solved using the existing grouping mechanism alone.