Add gated submissions API and agent intake skill

devops/deploy/docker-compose.yml

@@ -69,6 +69,8 @@ services:
     volumes:
       - "$WILDBOOK_BASE_DIR/wb-docker-deploy/wildbook_docker_webapps/:/usr/local/tomcat/webapps/"
       - "$WILDBOOK_BASE_DIR/wildbook_data_dir/:/usr/local/tomcat/webapps/wildbook_data_dir/"
+      # Private submissions staging; set submissions.stagingDirectory to this container path.
+      - "$WILDBOOK_BASE_DIR/wildbook_submissions_staging:/srv/wildbook-private/submissions"
       - "$WILDBOOK_BASE_DIR/wb-docker-deploy/logs/wildbook:/usr/local/tomcat/logs/"
       - /var/run/docker.sock:/var/run/docker.sock
       - .dockerfiles/tomcat/server.xml:/usr/local/tomcat/conf/server.xml

docs/design/2026-09-23-submissions-engineer-brief.md

@@ -0,0 +1,124 @@
+# Generic-client intake: accepted design direction
+
+Based on [your bulk-import proposal](https://qa.wildme.org/bulk-import-next.html)
+and implementation checkout `24cc99aede` (the PR is based on `main` at
+`dcf6f460de`; verification provenance is recorded in the workbench). The senior engineer accepted the three design
+decisions below on 2026-09-23, as relayed by the user. Detailed implementation
+mechanics are documented in the [implementation workbench](submissions/README.md).
+Runtime implementation is locally verified and disabled by default; nothing has
+been deployed. The workbench records test results and remaining QA gates.
+
+## Implementation for engineering review
+
+The sibling resource, durable drafts/uploads, strict validation, commit-once queue,
+worker, result mapping and resumable Python client are implemented locally. All
+six stages received Claude reviews; final rounds report no Critical or Major
+findings. Review transcripts and executed checks are in the workbench above.
+
+The main shared-code change is an opt-in importer mode: it persists through the
+caller's transaction and defers indexing/derivatives. Existing callers keep their
+defaults. This boundary matters because several existing Shepherd creation helpers
+commit internally. New PostgreSQL tests exercise rollback, concurrent acceptance,
+lost commit acknowledgments, recovery holds and actual two-image import mappings.
+
+The pilot intentionally accepts new encounters only, JPEG/PNG uploads and
+import-only processing. It requires configured locations and an explicit account
+allowlist. Uncertain imports require operator reconciliation; search-index
+dispatch is reported without claiming indexing completion. See the
+[pilot runbook](submissions/pilot-runbook.md) for configuration and the remaining
+QA/browser release gate. The Python client provides a human/integration entry
+point; no new browser form is included. The final addition is a public agent skill
+at `/api/v3/agent-skill/submit-sightings`, linked from the existing base toolbox,
+with complete field formats, validation examples and retry guidance.
+
+## Recommendation
+
+Agree with the central approach: reuse the existing bulk-import JSON rows,
+validators, media creation, and importer. Keep the working React workflow and its
+API contract stable. No new spreadsheet parser or encounter-creation pipeline.
+
+I recommend a small **submissions API alongside bulk import** to handle the
+additional lifecycle that agents and third-party integrations need:
+
+**Create draft → upload images → validate → commit once → poll results.**
+
+The new API owns authorization, staged files, validation revisions, and retries.
+The existing importer owns the conversion into Wildbook records. A submission
+can contain one encounter or a batch. Humans can use the same contract through
+a CLI now and a form later.
+
+## Suggested changes to the original proposal
+
+| Topic | Recommendation |
+| --- | --- |
+| Authentication | Reuse JWT infrastructure, but explicitly issue an import capability. A new filter name alone does not distinguish an import token from existing read-only tokens. Keep existing token behavior stable. |
+| API boundary | Add `/api/v3/submissions` for the new lifecycle. Preserve `/api/v3/bulk-import` and browser upload routing. Reuse Java components behind a small adapter. |
+| Retry safety | Persist an owned draft before upload; freeze it at commit; accept one import per submission. Require idempotency keys for creation/commit. A timed-out request must not lead to duplicate encounters. |
+| Upload | Start with streamed, one-file multipart uploads. Use owned staging and a size/digest manifest. Same filename/content can be retried; changed content is a conflict. Add resumable upload afterward using existing chunk mechanics. |
+| Validation | Reuse `BulkValidator` and `BulkImportUtil`. Apply stricter new-API policy at its boundary rather than changing shared defaults immediately. Require configured location, reject unknown fields, and validate media before commit. |
+| Limits | Separate per-file bytes, request bytes, draft storage and row limits. Apply `maximumMediaCountEncounter` to the resulting encounter after row grouping, not to the whole upload batch. |
+| Completion | Report data import separately from indexing, detection and identification. A downstream IA failure must not imply that submitted records disappeared. |
+
+The current checkout has evolved since parts of the proposal: upload requests
+already have configured byte bounds, chunk state keys include the destination
+path, and ImportTask authorization includes collaboration/admin rules. Token TTL
+is server-configured. Implementation should start from these current behaviors.
+
+## Smallest useful first release
+
+- Authenticated, explicitly enrolled integrations; existing browser bulk import
+  continues unchanged.
+- Discovery of supported fields, configured values, processing choices and limits.
+- JSON rows using existing `Class.fieldName` names, plus a client row ID for
+  diagnostics and mapping results back to source records.
+- Simple image uploads; strict validate-and-commit workflow; polling and links to
+  created records and the existing ImportTask.
+- Explicit processing choice: import only, detect, or detect and identify.
+  Recommend import-only as the new API default, preserving old API defaults.
+- Durable commit acceptance and conservative recovery. If a worker crashes and
+  commit outcome is uncertain, expose a reconciliation state instead of blindly
+  retrying record creation.
+
+Defer anonymous intake, general upserts, webhooks, CSV/XLSX parsing, new human UI,
+and broad third-party OAuth. Do not give an agent a user's password; use a trusted
+client/service to obtain its short-lived credential.
+
+## Engineering boundary and rollout
+
+The importer is reusable, but its servlet orchestration is not already a service
+interface. Extract only needed lifecycle helpers, preserving existing defaults
+and sequencing with characterization tests. `BulkImporter` also starts some
+indexing/media-child work before its caller commits; a new durable worker must
+account for that with a small, tested deferred-side-effects seam.
+
+Implement in three reviewable stages:
+
+1. Characterize the existing flow and agree on the new contract.
+2. Add gated authentication, owned drafts, simple uploads and validation.
+3. Add the execution adapter, commit deduplication and recovery tests; pilot one
+   installation/integration before broader enablement.
+
+This is more work than enabling Bearer authentication on the existing POST, but
+the additional work addresses unattended-client behavior without a broad importer
+rewrite. If delivery must be reduced, cut resumable upload and client conveniences
+first; retain ownership and safe commit retry semantics.
+
+## Accepted decisions — 2026-09-23
+
+1. **Use a sibling submissions resource.** Reuse the bulk-import pipeline behind
+   the new lifecycle API.
+2. **Require configured location and strict validation; default to import-only.**
+   Apply these defaults to the new API and preserve legacy behavior. Requiring
+   location universally may be a reasonable later correction, but is outside
+   this rollout to avoid disrupting working imports.
+3. **Pilot with a few approved partners.** Only accounts explicitly enabled for
+   the pilot can use the new API initially. This is what the proposed enrollment
+   allowlist means; the API is not opened to all users at launch.
+
+The [implementation plan](../plans/2026-09-23-submissions-api-implementation.md)
+breaks delivery into six reviewable changes, starting with the OpenAPI contract
+and compatibility tests, then gated drafts, uploads and validation. Pilot partners,
+installation, resource limits and retention need operational selection before rollout.
+
+The [supporting design](2026-09-23-submissions-api.md) includes proposed endpoints,
+payloads, state transitions, recovery behavior and regression acceptance gates.

docs/design/submissions/README.md

@@ -0,0 +1,165 @@
+> Processing-default update: new submissions now default to `detect-and-identify`.
+> Explicit `import-only` and all existing saved submissions remain unchanged.
+> See the pilot runbook for the additive schema rollout and AI handoff recovery.
+> Follow-up validation: 1,123 Java tests, zero failures/errors (seven skipped), eight client tests,
+> 21 contract examples; final 20-test resource/phase run and WAR packaging passed.
+> Claude approved the backend and final agent skill; see reviews/default-processing-disposition.md.
+> Earlier stage reports below describe the original import-only implementation.
+
+# Submissions contract workbench
+
+This folder contains the local implementation contract, review evidence and pilot
+handoff. The API has not been deployed. The accepted direction and
+implementation sequence are in [the implementation plan](../../plans/2026-09-23-submissions-api-implementation.md).
+
+- `openapi.yaml`: version-one contract; published OpenAPI reflects the implemented pilot subset.
+- `examples.json`: example envelopes and structured validation issues.
+- `scripts/submissions/check_contract.py` (repository root): checks local references,
+  example schema validity and key routing/precondition invariants. Requires PyYAML
+  and jsonschema; it is not a complete OpenAPI conformance validator.
+- `BulkSubmissionCompatibilityTest`: new characterization of legacy optional
+  location, year-only dates, equivalent row encodings and unknown-field policy.
+
+Run the local contract checks from the repository root:
+
+```bash
+python3 scripts/submissions/check_contract.py
+```
+
+Stage-one completion requires the targeted legacy baseline, new characterization
+tests and Claude review. Each later implementation stage also requires Claude
+review before proceeding. Review findings and dispositions are recorded under reviews/.
+
+## Review status
+
+The user explicitly approved sending relevant project files to Claude for
+read-only reviews at every stage, excluding credentials and unrelated material.
+Stages 1–6 have converged with no Critical or Major findings after corrections.
+Claude reviewed each stage read-only; execution evidence below comes from local checks.
+Full review transcripts and dispositions are under reviews/.
+
+## Local verification
+
+Baseline on implementation checkout `24cc99aede`, before runtime changes
+(PR base is `dcf6f460de`; see the provenance note below):
+
+```bash
+mvn -o test -Dtest=BulkApiPostTest,BulkApiOtherTest,BulkGeneralTest,BulkImagesTest,BulkImporterMissingAssetTest,AuthTokenTest,AuthTokenStepUpTest,WildbookTokenAuthenticationFilterTest,'UploadPaths*Test'
+```
+
+Result: **106 tests, 0 failures, 0 errors, 0 skipped; BUILD SUCCESS**.
+This is the selected unit-test baseline, not a full build or database recovery test.
+The run emitted background datastore diagnostics from existing mocked importer
+fixtures but completed successfully. Maven needed execution outside the sandbox
+because the canonical capitalized checkout path was treated as read-only.
+
+Initial stage-one contract check (historical): **10 operations/seven examples passed**.
+The current contract check covers 11 operations and 18 examples.
+
+New characterization suite:
+
+```bash
+mvn -o test -Dtest=BulkSubmissionCompatibilityTest
+```
+
+Result: **4 tests, 0 failures, 0 errors, 0 skipped; BUILD SUCCESS**.
+The new suite ran separately after the baseline, before runtime implementation.
+
+After Claude round-one fixes, the contract check passes 11 operations and 16
+positive/negative examples. The strengthened tests pass:
+
+```bash
+mvn -o test -Dtest=BulkSubmissionCompatibilityTest,BulkApiPostTest
+```
+
+**21 tests, 0 failures, 0 errors, 0 skipped; BUILD SUCCESS.**
+
+
+## Implementation verification
+
+Stage 2 authentication/draft/persistence tests: **38 passed**, including PostgreSQL
+restart, concurrent create/edit, quota race and rollback checks.
+
+Latest combined upload/validation/importer/queue run: **37 passed**, zero failures,
+errors or skips. Command:
+
+```bash
+mvn -o test -Dtest=SubmissionFilesTest,SubmissionValidatorTest,SubmissionStoreDbTest,BulkImporterSubmissionBoundaryTest,BulkSubmissionCompatibilityTest,BulkApiPostTest,BulkImporterMissingAssetTest
+```
+
+Client: `python3 -m unittest discover -s scripts/submissions -p test_client.py`:
+**7 passed**, including command-flow create/commit recovery, row correction,
+pagination and state-lock tests. Contract checker: **11 operations and 18 examples passed**.
+
+The full clean build ran the frontend: **21 bulk-import suites passed**; the entire
+frontend had **130 suites passed, 16 failed; 1,362 tests passed, 40 failed**.
+Failures are in unchanged frontend sources (including a missing Citation module
+and existing component test expectations); no base-commit frontend comparison was
+run, so these are not asserted to be proven pre-existing failures. Production
+frontend compilation completed. The clean build's first Java run had **1,108
+tests, one failure, one error, seven skipped**. Both failures were new test
+fixtures missing usernames. Those fixtures are corrected; the final full Java
+rerun passed as recorded below.
+
+Corrected PostgreSQL rerun: `mvn -o test -Dtest=SubmissionStoreDbTest`:
+**15 passed, zero failures/errors/skips; BUILD SUCCESS**. This includes the actual
+two-image adapter import, caller rollback, cleanup retention/locking, daily quota,
+invalid-validation rejection and replay pagination.
+
+`check_contract.py --runtime` passes both captured server responses (capabilities
+and commit acceptance) against both the draft and published OpenAPI schemas.
+These local tests do not replace the QA/browser release gate in
+[pilot-runbook.md](pilot-runbook.md).
+No installation has been deployed or enrolled.
+
+Final full Java regression: `mvn -o install`: **1,109 tests, zero failures, zero
+errors, seven skipped**. This includes all submissions/authentication tests and
+the existing bulk-import, upload, permissions and database suites. **BUILD SUCCESS**,
+including WAR packaging and local Maven installation. Frontend tests were run by the preceding clean invocation and
+retain the separate failure limitation above.
+
+Existing published OpenAPI paths and schemas were compared with the original
+checkout and remain unchanged; new route/auth documentation is additive.
+
+## Final addition: published agent skill
+
+At the user's request, added after the full implementation/build: the public
+`/api/v3/agent-skill/submit-sightings` resource, registered in AgentSkill and linked
+from the base toolbox and read-only API reference. It includes all supported
+fields, installation settings discovery, wire formats, field-specific validation
+failures, authorization boundaries, retry reconciliation and result mapping.
+`mvn -o test -Dtest=AgentSkillTest,AgentSkillContentTest`: **15 passed, zero
+failures/errors/skips; BUILD SUCCESS**. Existing skill routing/content checks and
+new runtime-parsed request examples passed. Two Claude review rounds converged
+with no Critical or Major issues; transcripts and disposition are under reviews/.
+This addition does not change submissions runtime behavior.
+
+Final artifact after the agent-skill addition: `mvn -o -DskipTests package`: **BUILD
+SUCCESS**. Tests were deliberately not rerun during packaging; the full API and
+subsequent skill test runs are recorded above. Verified the WAR contains byte-for-byte
+current submit-sightings, toolbox and API-reference resources, the new catalog
+registration, submissions servlet and JDO metadata. Artifact:
+`target/wildbook-10.14.war`. The deployment descriptor is handled separately as
+explained in the pilot runbook. No deployment or account enrollment was performed.
+
+## PR base and verification provenance
+
+The PR branch is based on `main` at `dcf6f460de`, excluding the separate mobile-layout
+commit `24cc99aede` present during implementation/testing. The difference between
+those bases contains only frontend files; Java sources and dependencies are identical.
+The frontend results and built WAR above therefore describe the earlier checkout,
+not a fresh frontend build of the PR base. No frontend changes are part of this PR.
+QA/browser verification on the final branch remains a release gate.
+
+Claude also approved the final PR handoff with no blockers; see
+[PR handoff review](reviews/pr-handoff-review.md). The suggested link and
+verification-provenance wording clarifications were incorporated.
+
+### Role enrollment and human token issuance
+
+The pilot's explicit enrollment is now managed through the `api-submission` role
+in the administrator user editor (context0). The previous UUID configuration is
+ignored; operators must grant the role to existing pilot users during rollout.
+Global admission/commit/worker switches remain unchanged. API Access offers an
+explicit Data import token purpose, while Read data remains the default. See the
+pilot runbook for migration, revocation, and token scope details.