feat(collectives-publish) inform publish service about archive ready

[code4candy]

📝 Summary

Goal

The tar.gz archive has already been created and stored.
The goal is now to notify the Publish Service that the archive is ready for download (API call).
And to provide an API to download the archive.

Description

• Two new configuration options need to be added to the Admin UI:
  • Publish Service URL – A field to configure the URL of the Publish Service.
  • Shared Secret – A password field to configure the shared secret used for authentication with the Publish Servic
• Send a POST request to the Publish Service (PS) to notify it that the tar.gz archive is ready for retrieval.
• Once the request has been successfully processed, close the modal and display a notification in the Collectives view.
• Provide public GET API to enable access for the external SSG Service to fetch the provided archive

API of the ext. SSG Service:
POST /build

{
  static_site_id: "a9608942-9484-4cd6-8672-017a55d1910b",
  content_download_url: "https://cloud.schnubbel.com/collective/publish/<static_site_id>/build_files"
  callback_status_url: "https://cloud.schnubbel.com/collectives/api/v1.0/p/static-sites/{staticSiteId}",
  slug: "<collective_name>",
	title: "my fancy site"
}

returns:

{
  "status": "enqueued"
  "status": "invalid", "error": "missing required fields" ||
  "status": "error", "error": "could not enqueue build"
}

Limitations:

• Currently no SSG-Stub in CI-Pipeline - so e2e & integration tests won't work for now (currently commented or skipped)
• The SSG expects the callback URL callback_status_url coming with the notification - this is part of another PR: feat(collectives-publish) static site update endpoint #2830 - Therefore the SSG does not accept the request and returns a 502 Bad Gateway that results in an error message shown in the UI.
• Missing SSG endpoint to validate URL & Shared-Secret (currently using /health to validate)

Acceptance Criteria

• A URL can be configured in the Admin UI.
• A shared secret can be configured in the Admin UI.
• 1. The PS is notified that a tar.gz archive is ready for retrieval.
• 2. The modal closes once the tar.gz archive has been successfully stored and the PS has been successfully notified. (Limitation: see above, so modal wont close by itself)
• 3. The status in the Static Site record is updated after informing the ssg service
• 4. The SSG service is provided with a download URL

🖼️ Screenshots

---

🚧 TODO

• ...

🏁 Checklist

• Code is properly formatted (npm run lint / npm run stylelint / composer run cs:check)
• Sign-off message is added to all commits
• Tests (unit, integration and/or end-to-end) passing and the changes are covered with tests

🤖 AI (if applicable)

• The content of this PR was partly or fully generated using AI tools
• The AI-generated content was reviewed, comprehended and tested by a human

Mermaid BPMN Diagramm for the Publish Process in Collectives

flowchart TD
    %% ---------- Lane: User / Frontend ----------
    subgraph FE["User / Frontend (CollectivePublishModal)"]
        start([Click 'Publish website'])
        feValid{Title and slug valid?}
        feDisabled([Button disabled])
        fePages{At least one page selected?}
        feNoPages([Error toast: select at least one page])
        post[POST /static-sites]
        feOk([Success toast 'Website publishing started', modal closes])
        feWarn([Warning toast with message, modal stays open, input kept])
        feErr([Error toast with message, modal stays open, input kept])
    end

    %% ---------- Lane: Nextcloud backend ----------
    subgraph NC["Nextcloud (StaticSiteController / StaticSiteService)"]
        perm{Collective found and user can edit?}
        conf{Publish service configured?}
        valid{Title, slug and pages valid?}
        insert[Insert static site, status pending]
        slugFree{Slug free?}
        collect[Collect files of the collective]
        size{Total size within static_site_max_size?}
        archive[Build tar.gz and store in appdata]
        archOk{Archive stored? paths valid, not empty}
        notify[POST /build to publish service]
        accepted{Response 2xx + enqueued?}
        temp{Not reachable, timeout or 5xx?}
        cleanup[Delete static site and archive]
        finish[Store title, page selection, status provided]
        stillThere{Static site still exists?}
        delArch[Delete archive]
    end

    %% ---------- Lane: Publish service ----------
    subgraph SSG["External publish service"]
        enqueue[Enqueue build]
        download[GET archive with bearer token]
    end

    %% ---------- Lane: Archive download (async) ----------
    subgraph DL["Nextcloud (StaticSiteArchiveController)"]
        blocked{More than 10 failed attempts in 30 min?}
        token{Bearer token valid?}
        exists{Static site and archive exist?}
        r429([429 Too Many Requests])
        r401([401, attempt counted])
        r404([404])
        r200([200, tar.gz streamed])
    end

    start --> feValid
    feValid -- no --> feDisabled
    feValid -- yes --> fePages
    fePages -- no --> feNoPages
    fePages -- yes --> post --> perm

    perm -- "no: 403 / 404" --> feWarn
    perm -- yes --> conf
    conf -- "no: 503 'not set up', logged" --> feErr
    conf -- yes --> valid
    valid -- "no: 400 / 404" --> feWarn
    valid -- yes --> insert --> slugFree
    slugFree -- "no: 400 'slug in use'" --> feWarn
    slugFree -- yes --> collect --> size
    size -- "no: 400, names 3 largest files" --> cleanup
    size -- yes --> archive --> archOk
    archOk -- "no: 400 path too long / no content, 500 storage error" --> cleanup
    archOk -- yes --> notify --> enqueue
    enqueue --> accepted
    accepted -- yes --> finish --> stillThere
    accepted -- no --> temp
    temp -- "yes: 503 'try again later'" --> cleanup
    temp -- "no: 502 'rejected', error logged" --> cleanup
    cleanup -- "4xx" --> feWarn
    cleanup -- "5xx" --> feErr
    stillThere -- yes --> feOk
    stillThere -- "no: 404" --> delArch --> feWarn

    enqueue -. "async, may start before status is provided" .-> download
    download --> blocked
    blocked -- yes --> r429
    blocked -- no --> token
    token -- no --> r401
    token -- yes --> exists
    exists -- no --> r404
    exists -- yes --> r200

Loading

[kjoscha]

I gonna review. only a meta thing in advance: Let us use our defined pattern for the PR naming:
So, this PR would change to: "feat(collectives-publish): inform publish service about archive ready"

[kjoscha]

I think this is called more often than necessary. It is called explicitly here and on update and then in StaticSiteGeneratorClient:108 again on each build

[code4candy]

The calls are all making sense:

1. StaticSiteService::create()
   --> assures the correct configuration before build()
2. StaticSiteService::update()
   --> same - also if not configured would set statsu to pending, and set status to failed on build() - we want an early fail
3. StaticSiteGeneratorClient::request()
   --> assures that every request asserts the config - double check ist only read process and sored after first call.

[kjoscha]

probably there are reasons for wirting every excpetion to a seperate file and not all in StaticSiteGeneratorException.php ?!

[mejo-]

From what I understand, they're used to distinguish different error states and show separate error messages. Makes sense to me.

[code4candy]

Exactly - the idea was that the user could be informed if this is a temporary error and to retry later or if it is neccessary to contact an admin.

[kjoscha]

sure. i just wodnered, wether they all can be in one file, because the yseem to be inhereted from one class, but never mind :)

[mejo-]

Nice that you introduce a config lexicon. I'm not sure though if we can have one that only contains parts of the app config settings. I guess it should be comprehensive if we add it. I can look into opening a PR that adds a config lexicon to Collectives main branch already and then you can add the new settings into it with your PRs, what do you think?

[mejo-]

#2856

[code4candy]

Super great - I wanted to ask you about that anyway.
It would have worked for now because of return Strictness::IGNORE;

And I introduced it because I needed it to automatically save the shared secret to the db as encrypted...
Perfect.
thx