Skip to content

feat(media): add scoped attachment policies and excluded-media purge - #620

Open
salmonumbrella wants to merge 1 commit into
kenn-io:mainfrom
salmonumbrella:feat/attachment-policy
Open

feat(media): add scoped attachment policies and excluded-media purge#620
salmonumbrella wants to merge 1 commit into
kenn-io:mainfrom
salmonumbrella:feat/attachment-policy

Conversation

@salmonumbrella

Copy link
Copy Markdown
Contributor

What changed

  • Add a shared attachment policy for Beeper, Slack, Discord, and Teams: media, max_media_mb, media_scope (all, direct, none), and media_max_participants, with per-account overrides through accounts_config (Beeper account IDs, Slack team IDs, Teams emails) and the existing Discord guilds map.
  • Record a typed outcome on every attachment occurrence — pending, stored, skipped, or failed, plus a skip reason — so a deliberate policy skip is distinguishable from a failed download, and backfills retry only eligible items.
  • Add purge-excluded-media, which removes already-stored media that current policy excludes. --dry-run previews occurrence, blob, and logical byte counts; deletion requires --yes or interactive confirmation.
  • Keep SQLite and PostgreSQL behavior aligned, preserve the 100 MiB Beeper/Slack/Teams and 50 MiB Discord caps as defaults, and leave configurations with omitted policy keys allow-all.

Why

Attachment capture is all-or-nothing per provider, so archiving a busy chat network means storing every meme and forwarded video from large public rooms in order to keep the direct-message media that actually matters. --no-media on a manual run cannot express an ongoing policy because the daemon scheduler reads config, and it writes no marker, so a deliberate skip is indistinguishable from an attachment that was never seen.

Usage

[beeper]
media_scope = "all"
media_max_participants = 12
max_media_mb = 100

[beeper.accounts_config.accountid]
media = false
msgvault purge-excluded-media --dry-run   # preview what current policy excludes
msgvault purge-excluded-media --yes       # remove it

Closes #618

@roborev-ci

roborev-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown

roborev: Combined Review (a7f6add)

One-line verdict: Three medium-severity issues could bypass media-policy boundaries or lose durable attachment outcomes.

Medium

  • internal/teams/importer.go:425 — Teams channel imports set Type: "channel" but omit ParticipantCount, so media_max_participants evaluates the channel as having zero participants and never excludes large-channel media. Populate the context with an authoritative participant count and persist it for backfill/purge evaluation; if unknown while a limit is configured, skip the download.

  • internal/teams/importer.go:779 — Failed/skipped Teams inline markers use an HTTP URL as storage_path, but the same normal sync later deletes all HTTP-backed attachment rows via ReplaceMessageLinkAttachments. This erases durable policy/oversize outcomes and observed sizes, so --only-incomplete may re-fetch oversized media. Exclude source_attachment_id LIKE 'teams:inline:%' from link-row replacement and cover the normal import path.

  • cmd/msgvault/cmd/sync_slack.go:203[slack].media = false is represented as a disabled MediaPolicy instead of NoMedia. Because BackfillMedia only clears NoMedia, explicit backfills no longer download deferred files while the setting remains false, contrary to documented behavior. Preserve the explicit-backfill override for the provider toggle while still enforcing scope/account policies, or document the intentional breaking change.


Reviewers: 2 done | Synthesis: codex, 11s | Total: 18m5s

@salmonumbrella
salmonumbrella force-pushed the feat/attachment-policy branch from a7f6add to 49119c5 Compare August 15, 2026 01:10
@roborev-ci

roborev-ci Bot commented Aug 15, 2026

Copy link
Copy Markdown

roborev: Combined Review (49119c5)

Code requires changes: two medium-severity Teams media-policy issues can cause incorrect skipping and later unintended downloads.

Medium

  • internal/teams/client.go:281ListTeamMembers requires a delegated team-member read permission, but GraphScopes does not request one. Existing and newly authorized Teams tokens will reject the endpoint, causing media_max_participants to treat membership as unknown and skip channel media on every sync. Add the required scope to GraphScopes, stale-token validation, tests, and authorization documentation so users are prompted to reauthorize.

  • internal/teams/importer.go:454 — When team membership lookup fails, the fail-closed participant count is only stored in transient import options. The archived conversation retains a zero count, so a later media backfill may reinterpret the skipped marker as allowed and download it while membership remains unknown. Persist an explicit unknown-membership policy state, or refresh membership during backfill and exclude these markers until an authoritative count is available.


Reviewers: 2 done | Synthesis: codex, 10s | Total: 8m8s

@salmonumbrella
salmonumbrella force-pushed the feat/attachment-policy branch from 49119c5 to 3aa954e Compare August 15, 2026 01:32
@roborev-ci

roborev-ci Bot commented Aug 15, 2026

Copy link
Copy Markdown

roborev: Combined Review (3aa954e)

Verdict: Three policy-enforcement gaps can bypass participant caps or incorrectly exclude media, including two high-severity fail-open paths.

High

  • Participant cap fails open for ordinary Discord channelsinternal/discord/importer.go:210
    container.Channel.MemberCount is authoritative only for threads; ordinary guild channels receive zero. Falling back to participants observed in archived messages undercounts large channels, allowing attachments to be downloaded despite media_max_participants.
    Fix: Treat unavailable membership counts as unknown and fail closed, or obtain and persist a conservative authoritative count for consistent sync, backfill, and purge decisions.

  • Teams roster failures become policy bypasses during backfillinternal/teams/importer.go:454
    The temporary MaxParticipants + 1 fail-closed value is not persisted. After roster lookup fails, archived membership may contain only observed senders; a later media backfill then considers previously skipped content eligible and downloads it without retrying the roster lookup.
    Fix: Persist an explicit “membership unknown” state and keep media ineligible until an authoritative membership sync succeeds.

Medium

  • Teams sync lacks the required roster permissioninternal/teams/client.go:281
    Sync calls /teams/{team-id}/members, but GraphScopes() does not request TeamMember.Read.All. Production tokens can reject the call, causing channels to be treated as oversized and media to be skipped.
    Fix: Add the delegated scope to GraphScopes, stale-token detection, OAuth tests, and documentation so existing users are prompted to reauthorize.

  • Private and shared Teams channels use the wrong rosterinternal/teams/importer.go:269
    The team-wide roster is applied to every channel without considering Channel.MembershipType. Private and shared channels have channel-specific membership, producing incorrect participant counts and potentially skipping or purging valid media.
    Fix: Use the team roster only for standard channels; fetch channel-specific membership for private/shared channels and request the necessary OAuth permission.


Reviewers: 2 done | Synthesis: codex, 18s | Total: 14m28s

@salmonumbrella
salmonumbrella force-pushed the feat/attachment-policy branch from 3aa954e to e96ed81 Compare August 15, 2026 03:00
@roborev-ci

roborev-ci Bot commented Aug 15, 2026

Copy link
Copy Markdown

roborev: Combined Review (e96ed81)

Verdict: Four medium-severity issues could bypass or inconsistently enforce participant-based media limits.

Medium

Participant limits fail open when Slack or Teams membership lookup fails

Locations: internal/slack/importer.go:363, internal/slack/importer.go:464, internal/teams/importer.go:190, internal/teams/importer.go:194

Roster failures leave the participant count at zero while message and media import continues. This makes affected conversations appear eligible under any positive participant cap, and later backfills may continue trusting the archived zero.

Treat unavailable membership as ineligible for media. Persist an explicit unknown state or reuse a trustworthy archived count, and re-fetch membership before backfill reconsideration.

Discord guild participant floors are not persisted

Location: internal/discord/importer.go:205

The guild-wide participant floor exists only during sync. Purge and retry evaluation can therefore rely only on observed authors and incorrectly retain media from guilds exceeding media_max_participants.

Persist a policy-specific guild membership floor and update and consult it consistently during sync, purge, and backfill.

Teams channel roster counts become stale and monotonically increase

Location: internal/teams/importer.go:311

Current roster members are added to the historical participant set but departed members are never removed. Sync uses the current Graph roster, while purge and backfill use the growing archived set, so media may remain excluded after membership shrinks.

Store the exact current roster count separately from historical participants and use it consistently for sync, purge, and backfill.


Reviewers: 2 done | Synthesis: codex, 16s | Total: 13m49s

Add a provider-neutral attachment policy (media on/off, per-item size
cap, conversation scope, participant limit) with per-account overrides
for Beeper, Slack, Discord, and Teams. Importers persist typed download
outcomes so policy-excluded media is distinguishable from failed
downloads, backfills honor the active policy and retry only eligible
items, and the new purge-excluded-media command removes already-stored
media that the current policy excludes.

Fixes kenn-io#618

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@salmonumbrella
salmonumbrella force-pushed the feat/attachment-policy branch from e96ed81 to 83a9076 Compare August 15, 2026 04:45
@roborev-ci

roborev-ci Bot commented Aug 15, 2026

Copy link
Copy Markdown

roborev: Combined Review (83a9076)

High-severity purge handling can permanently delete eligible media when conversation membership is unknown; one medium-severity backfill path can also leave media permanently skipped.

High

  • internal/store/attachment_policy.go:81 — Purge candidates with member_count_unknown=true fall back to the accumulated participant count. If that stale count exceeds the threshold, purge-excluded-media permanently deletes media even though the current roster is unknown and may now be below the limit.
    • Fix: Preserve the unknown-membership state on candidates and suppress participant-threshold exclusions during purge, while still applying scope, account, and size exclusions.

Medium

  • internal/store/messages.go:2853 — Teams --only-incomplete evaluates skipped markers against the accumulated participant count before invoking the callback that refreshes an unknown roster. If the accumulated count exceeds the limit, the callback is never reached, leaving media permanently skipped even when the refreshed roster would fall below the threshold.
    • Fix: Use AttachmentConversationMembership and always yield unresolved-roster messages when a participant limit is configured, allowing the importer to refresh membership before final policy evaluation.

Reviewers: 2 done | Synthesis: codex, 12s | Total: 13m56s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Add per-account and conversation-scoped attachment policy, with a purge path for excluded media

1 participant