diff --git a/README.md b/README.md index 0f14771..0cf9faf 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ The plugin has no HTTP routes of its own: core calls the internal-only `POST /_g ### Backend - `spi_*.go` implement the provider operations; the SDK conformance suite (`spi_conformance_test.go`) runs them against a fake GitLab (`spi_fakegitlab_test.go`), and `spi_test.go` / `spi_ops_test.go` cover the GitLab-specific behaviour. -- Owns the schema `plugin_data_com_paca_gitlab` (`gitlab_identities`, `gitlab_setup_sessions`). Access tokens and webhook secrets are stored encrypted with AES-256-GCM using the `ENCRYPTION_KEY` config value. +- Owns the schema `plugin_data_com_paca_gitlab` (`gitlab_identities`, `gitlab_setup_sessions`, and `gitlab_legacy_webhooks` for an upgrade from 0.5.x). Access tokens and webhook secrets are stored encrypted with AES-256-GCM using the `ENCRYPTION_KEY` config value. - Personal access tokens work, but the recommended identity is a **project or group access token**: GitLab creates a bot user for it, which can be mentioned and is limited to that project or group. ### Configuration @@ -111,11 +111,63 @@ Two limits come from core, not from this plugin: ## Upgrading from 0.5.x -Migration `0002_git_provider.sql` **drops** the legacy tables (they only exist if the plugin's schema held them: see the ID note below): `gitlab_integrations`, `gitlab_repositories`, `gitlab_merge_requests`, `gitlab_task_mr_links`, `gitlab_task_branches`. Nothing is migrated, because a project-bound token has no webhook secret and cannot back a provider identity on its own, and repositories, merge-request links and branch links now live in Paca core. Add the token again from a project's Git settings. +The community plugin (0.5.x) is migrated: migration `0004_migrate_community_plugin.sql` carries its data over to Paca core when the plugin is installed or upgraded. -The plugin ID changed from `com.gorlix.gitlab` to `com.paca.gitlab`, so this installs as a new plugin with its own schema `plugin_data_com_paca_gitlab`. The old plugin's data stays in `plugin_data_com_gorlix_gitlab` untouched (and is not read); export anything you still need from there, then remove the old plugin. Afterwards set up a GitLab identity (above) and relink the projects in Paca. +### What is carried over -The plugin's own screens, MCP tools, automation nodes (`gitlab.mr_linked`, `gitlab.merge_mr`, …), custom permission `gitlab.manage` and the `paca-gitlab-workflow` skill are gone: core's Git integration (neutral `change_request_*` / `review_*` tools, `git:*` permissions, Git triggers on agents) replaces them. Flows that used the old automation nodes need rebuilding on core's Git triggers. +| 0.5.x | After the upgrade | +|---|---| +| A project's access token | A connection of that project (*This project only*) in **Settings → Git → Connections**, with the same token. | +| Linked repositories | Linked repositories of the project, through that connection. A repository already linked by hand is kept as it is. | +| Merge requests and their links to tasks | Change requests in core, linked to the same tasks. | +| Branches linked to tasks | The same branch links. | + +What is not known at migration time is filled in the first time the connection is used: the token's user is asked of GitLab, a webhook secret is generated and the token's kind (personal, project, group) is read from GitLab. A token that GitLab no longer accepts (revoked, expired) reports *connect GitLab again*: add a new connection with a new token. + +Not carried over: the plugin's own screens, MCP tools, automation nodes (`gitlab.mr_linked`, `gitlab.merge_mr`, …), the `gitlab.manage` permission and the `paca-gitlab-workflow` skill. Core's Git integration replaces them (neutral `change_request_*` / `review_*` tools, `git:*` permissions, Git triggers on agents); flows that used the old automation nodes need rebuilding on core's Git triggers. + +The tokens are copied encrypted, as they are: they stay readable as long as the instance's `ENCRYPTION_KEY` is the one they were stored with. If the key changed, the connection says so; restore the key or add the token again. + +### Webhooks + +The webhooks 0.5.x registered on GitLab point at the old plugin, and they stop working with it. For each linked repository, open **Settings → Git → Repositories** and press *Try again*: Paca registers its new hook, and once that has worked it deletes the old plugin's hook from the GitLab project. Only a hook whose URL is still the old plugin's (`…/api/v1/plugins/com.gorlix.gitlab/projects/…/webhook`) is deleted; one that was pointed somewhere else is left alone. If the new hook cannot be registered, the old one stays. + +### Order of the steps + +0.5.0 and 0.5.1 already had the id `com.paca.gitlab`, 0.5.2 had `com.gorlix.gitlab`. Check which one you run in Paca's plugin list. + +1. **Back up the database** (below). The migration cannot be undone without it. +2. Install this plugin: + - from **0.5.0 or 0.5.1**: upgrade the plugin in place. The old tables in `plugin_data_com_paca_gitlab` are carried over and then dropped. + - from **0.5.2**: install `com.paca.gitlab` **before** removing `com.gorlix.gitlab`. The data is read from `plugin_data_com_gorlix_gitlab`, which is left untouched. +3. Check a project that used GitLab: its connection, its repositories, and the merge requests and branches on its tasks. +4. Press *Try again* on each repository (see Webhooks). +5. Only then remove `com.gorlix.gitlab` (0.5.2). Paca keeps its schema `plugin_data_com_gorlix_gitlab` after the plugin is removed; drop it by hand once you no longer need it. + +### Backup and rollback + +Take a backup right before the upgrade, and check that it can be read. With the bundled Postgres of `deploy/docker-compose.prod.yml`: + +```sh +docker compose -f deploy/docker-compose.prod.yml exec -T postgres \ + pg_dump -U paca -d paca -Fc > paca-before-gitlab-0.6.dump +docker run --rm -v "$PWD":/d:ro postgres:16 pg_restore -l /d/paca-before-gitlab-0.6.dump | head +``` + +With a managed database, run `pg_dump "$DATABASE_URL" -Fc` instead. The dump holds the encrypted tokens: keep it as private as the database. + +To roll back, stop the API, restore the dump and reinstall the plugin version you had: + +```sh +docker compose -f deploy/docker-compose.prod.yml stop api +docker compose -f deploy/docker-compose.prod.yml exec -T postgres dropdb -U paca --force paca +docker compose -f deploy/docker-compose.prod.yml exec -T postgres createdb -U paca paca +docker compose -f deploy/docker-compose.prod.yml exec -T postgres \ + pg_restore -U paca -d paca --no-owner < paca-before-gitlab-0.6.dump +docker compose -f deploy/docker-compose.prod.yml start api +``` + +Replace `paca` with your `POSTGRES_USER` and `POSTGRES_DB` if you changed them. A restore also brings back everything else as it was at the time of the dump, so roll back soon after the upgrade. Hooks that *Try again* deleted on GitLab are not restored with the database; register them again from the old plugin if you go back to it. --- @@ -133,6 +185,12 @@ go vet ./... && go test -race ./... tinygo build -target=wasip1 -buildmode=c-shared -o backend.wasm . ``` +The migration from 0.5.x is tested against a real Postgres 16 (in docker) with core's Git tables, taken from a Paca checkout: + +```bash +PACA_CORE=/path/to/paca scripts/test-legacy-migration/run.sh +``` + To install into a local development Paca instance, use Paca's development script: ```bash diff --git a/backend/migrations/0002_git_provider.sql b/backend/migrations/0002_git_provider.sql index 31dea88..dd7785b 100644 --- a/backend/migrations/0002_git_provider.sql +++ b/backend/migrations/0002_git_provider.sql @@ -1,8 +1,7 @@ -- 0002_git_provider.sql -- The GitLab provider's own state: the access-token identities Paca connects -- with (tokens and webhook secrets encrypted) and the setup sessions that tie a --- token-form round trip together. Drops the previous release's project-bound --- tables (see the end). +-- token-form round trip together. -- Run with search_path = plugin_data_com_paca_gitlab, public. CREATE TABLE IF NOT EXISTS gitlab_identities ( @@ -37,11 +36,6 @@ CREATE TABLE IF NOT EXISTS gitlab_setup_sessions ( -- The plugin is a Git provider: repositories, merge requests, task links and the -- per-project connection now live in Paca core (git_* tables). The previous --- release's project-bound tables are dropped; a project-scoped token cannot back --- a provider identity on its own (it has no webhook secret), so nothing is --- migrated. -DROP TABLE IF EXISTS gitlab_task_mr_links; -DROP TABLE IF EXISTS gitlab_task_branches; -DROP TABLE IF EXISTS gitlab_merge_requests; -DROP TABLE IF EXISTS gitlab_repositories; -DROP TABLE IF EXISTS gitlab_integrations; +-- release's project-bound tables (0001) are kept here until +-- 0004_migrate_community_plugin.sql has carried their rows over to core, and +-- dropped there. diff --git a/backend/migrations/0004_migrate_community_plugin.sql b/backend/migrations/0004_migrate_community_plugin.sql new file mode 100644 index 0000000..2799a2b --- /dev/null +++ b/backend/migrations/0004_migrate_community_plugin.sql @@ -0,0 +1,172 @@ +-- 0004_migrate_community_plugin.sql +-- Carries over the data of the community GitLab plugin (0.5.x), which this +-- provider replaces. Run with search_path = plugin_data_com_paca_gitlab, public. +-- +-- The community plugin kept one access token per project (gitlab_integrations) +-- and the repositories linked with it. Each becomes a project-scoped token +-- connection in core with its repositories, branch links and merge request +-- links. Its tables are in one of two places: +-- - 0.5.0 and 0.5.1 had this plugin's id (com.paca.gitlab), so their tables +-- are in this schema: 0001 created them and 0002 kept them. They are +-- dropped at the end, as the GitHub provider drops its previous release's. +-- - 0.5.2 had its own id (com.gorlix.gitlab), so its tables are in +-- plugin_data_com_gorlix_gitlab. That schema belongs to the old plugin and +-- is left as it is. +-- A project found in both is taken from 0.5.2, the newer. +-- +-- What cannot be known here is completed on the token's first use (see +-- completeMigratedIdentity in spi_provider.go): the token's user is asked of +-- GitLab and a webhook secret is generated. The webhooks the old plugin +-- registered on GitLab point at the old plugin and use per-repository secrets. +-- They are kept in gitlab_legacy_webhooks and each is deleted from GitLab once +-- the repository's new hook is registered ("Try again" on the repository), if +-- it still points at the old plugin (see removeLegacyHooks in +-- spi_legacy_hooks.go). +-- Nothing happens when the old plugin was never installed. Core's tables are +-- created before any plugin migration runs; if they are not there (a +-- plugin-only database) only the tokens are copied. + +-- The community plugin's hooks still to remove, per migrated identity. A row +-- goes once its hook is dealt with. +CREATE TABLE IF NOT EXISTS gitlab_legacy_webhooks ( + identity_id UUID NOT NULL REFERENCES gitlab_identities (id) ON DELETE CASCADE, + gitlab_project_id BIGINT NOT NULL DEFAULT 0, + full_path TEXT NOT NULL, + webhook_id BIGINT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + PRIMARY KEY (identity_id, webhook_id) +); + +DO $$ +DECLARE + sources TEXT[] := '{}'; + src TEXT; +BEGIN + IF to_regclass('plugin_data_com_gorlix_gitlab.gitlab_integrations') IS NOT NULL THEN + sources := sources || 'plugin_data_com_gorlix_gitlab'::text; + END IF; + IF to_regclass(format('%I.gitlab_integrations', current_schema())) IS NOT NULL THEN + sources := sources || current_schema()::text; + END IF; + IF cardinality(sources) = 0 THEN + RETURN; + END IF; + + -- Both releases' rows, gathered in one place; src says which schema each + -- came from, so a row only ever joins rows of its own release. + CREATE TEMP TABLE legacy_integrations (src TEXT, id UUID, project_id UUID, access_token_enc TEXT, instance_url TEXT) ON COMMIT DROP; + CREATE TEMP TABLE legacy_repositories (src TEXT, id UUID, project_id UUID, integration_id UUID, full_name TEXT, + gitlab_project_id BIGINT, webhook_id BIGINT, default_branch TEXT) ON COMMIT DROP; + CREATE TEMP TABLE legacy_merge_requests (src TEXT, id UUID, repo_id UUID, pr_number INT, title TEXT, state TEXT, html_url TEXT, + head_branch TEXT, base_branch TEXT, author TEXT, merged_at TIMESTAMPTZ) ON COMMIT DROP; + CREATE TEMP TABLE legacy_task_mr_links (task_id UUID, merge_request_id UUID) ON COMMIT DROP; + CREATE TEMP TABLE legacy_task_branches (task_id UUID, repo_id UUID, branch_name TEXT) ON COMMIT DROP; + + FOREACH src IN ARRAY sources LOOP + EXECUTE format($q$ + INSERT INTO pg_temp.legacy_integrations + SELECT %1$L, i.id, i.project_id, i.access_token_enc, i.instance_url FROM %1$I.gitlab_integrations i + WHERE NOT EXISTS (SELECT 1 FROM pg_temp.legacy_integrations l WHERE l.project_id = i.project_id OR l.id = i.id)$q$, src); + EXECUTE format($q$ + INSERT INTO pg_temp.legacy_repositories + SELECT %1$L, r.id, r.project_id, r.integration_id, r.full_name, r.gitlab_project_id, r.webhook_id, r.default_branch + FROM %1$I.gitlab_repositories r JOIN pg_temp.legacy_integrations l ON l.src = %1$L AND l.id = r.integration_id$q$, src); + EXECUTE format($q$ + INSERT INTO pg_temp.legacy_merge_requests + SELECT %1$L, m.id, m.repo_id, m.pr_number, m.title, m.state, m.html_url, m.head_branch, m.base_branch, m.author, m.merged_at + FROM %1$I.gitlab_merge_requests m JOIN pg_temp.legacy_repositories r ON r.src = %1$L AND r.id = m.repo_id$q$, src); + EXECUTE format($q$ + INSERT INTO pg_temp.legacy_task_mr_links + SELECT l.task_id, l.merge_request_id + FROM %1$I.gitlab_task_mr_links l JOIN pg_temp.legacy_merge_requests m ON m.src = %1$L AND m.id = l.merge_request_id$q$, src); + EXECUTE format($q$ + INSERT INTO pg_temp.legacy_task_branches + SELECT b.task_id, b.repo_id, b.branch_name + FROM %1$I.gitlab_task_branches b JOIN pg_temp.legacy_repositories r ON r.src = %1$L AND r.id = b.repo_id$q$, src); + END LOOP; + + -- The ciphertexts are copied as they are: the same ENCRYPTION_KEY (the + -- instance's, which core hands every plugin) protects both. + -- host is the host (and port) of the instance URL; the API is always https. + INSERT INTO gitlab_identities (id, host, username, token_enc, created_by) + SELECT i.id, + COALESCE(NULLIF(lower(substring(i.instance_url FROM '^(?:[A-Za-z][A-Za-z0-9+.-]*://)?([^/?#]+)')), ''), 'gitlab.com'), + '', i.access_token_enc, 'community-plugin' + FROM pg_temp.legacy_integrations i + ON CONFLICT (id) DO NOTHING; + + INSERT INTO gitlab_legacy_webhooks (identity_id, gitlab_project_id, full_path, webhook_id) + SELECT r.integration_id, r.gitlab_project_id, r.full_name, r.webhook_id + FROM pg_temp.legacy_repositories r + JOIN gitlab_identities g ON g.id = r.integration_id + WHERE r.webhook_id > 0 + ON CONFLICT DO NOTHING; + + IF to_regclass('public.git_identities') IS NULL THEN + RETURN; + END IF; + + INSERT INTO public.git_identities (provider_id, host, provider_identity_id, kind, scope, project_id, handle, display_name) + SELECT 'com.paca.gitlab', g.host, g.id::text, 'token', 'project', i.project_id, '', 'GitLab access token' + FROM pg_temp.legacy_integrations i + JOIN gitlab_identities g ON g.id = i.id + WHERE NOT EXISTS ( + SELECT 1 FROM public.git_identities c + WHERE c.provider_id = 'com.paca.gitlab' AND c.provider_identity_id = i.id::text + ); + + -- Webhook deliveries for a connection arrive at its own endpoint. + INSERT INTO public.git_webhook_endpoints (key, provider_id, identity_id) + SELECT replace(gen_random_uuid()::text || gen_random_uuid()::text, '-', ''), c.provider_id, c.id + FROM public.git_identities c + JOIN pg_temp.legacy_integrations i ON c.provider_identity_id = i.id::text + WHERE c.provider_id = 'com.paca.gitlab' AND c.kind = 'token' AND c.scope = 'project' + AND NOT EXISTS (SELECT 1 FROM public.git_webhook_endpoints e WHERE e.identity_id = c.id); + + -- external_id is GitLab's numeric project id, which survives renames. + INSERT INTO public.git_repositories (project_id, provider_id, host, external_id, full_path, default_branch, clone_url, identity_id) + SELECT r.project_id, 'com.paca.gitlab', c.host, + CASE WHEN r.gitlab_project_id > 0 THEN r.gitlab_project_id::text ELSE '' END, + r.full_name, r.default_branch, 'https://' || c.host || '/' || r.full_name || '.git', c.id + FROM pg_temp.legacy_repositories r + JOIN public.git_identities c ON c.provider_id = 'com.paca.gitlab' AND c.provider_identity_id = r.integration_id::text + ON CONFLICT DO NOTHING; + + INSERT INTO public.git_change_requests (repository_id, number, title, state, head_ref, base_ref, author, html_url, merged_at) + SELECT cr.id, m.pr_number::text, m.title, + CASE WHEN m.state IN ('open', 'closed', 'merged') THEN m.state ELSE 'open' END, + m.head_branch, m.base_branch, m.author, m.html_url, m.merged_at + FROM pg_temp.legacy_merge_requests m + JOIN pg_temp.legacy_repositories r ON r.id = m.repo_id + JOIN public.git_identities c ON c.provider_id = 'com.paca.gitlab' AND c.provider_identity_id = r.integration_id::text + JOIN public.git_repositories cr ON cr.project_id = r.project_id AND cr.provider_id = 'com.paca.gitlab' + AND cr.host = c.host AND lower(cr.full_path) = lower(r.full_name) + ON CONFLICT DO NOTHING; + + INSERT INTO public.git_task_links (task_id, change_request_id) + SELECT l.task_id, cr.id + FROM pg_temp.legacy_task_mr_links l + JOIN pg_temp.legacy_merge_requests m ON m.id = l.merge_request_id + JOIN pg_temp.legacy_repositories r ON r.id = m.repo_id + JOIN public.git_identities c ON c.provider_id = 'com.paca.gitlab' AND c.provider_identity_id = r.integration_id::text + JOIN public.git_repositories gr ON gr.project_id = r.project_id AND gr.provider_id = 'com.paca.gitlab' + AND gr.host = c.host AND lower(gr.full_path) = lower(r.full_name) + JOIN public.git_change_requests cr ON cr.repository_id = gr.id AND cr.number = m.pr_number::text + ON CONFLICT DO NOTHING; + + INSERT INTO public.git_task_branches (task_id, repository_id, branch_name) + SELECT b.task_id, gr.id, b.branch_name + FROM pg_temp.legacy_task_branches b + JOIN pg_temp.legacy_repositories r ON r.id = b.repo_id + JOIN public.git_identities c ON c.provider_id = 'com.paca.gitlab' AND c.provider_identity_id = r.integration_id::text + JOIN public.git_repositories gr ON gr.project_id = r.project_id AND gr.provider_id = 'com.paca.gitlab' + AND gr.host = c.host AND lower(gr.full_path) = lower(r.full_name) + ON CONFLICT DO NOTHING; +END $$; + +-- 0.5.0 and 0.5.1's tables in this schema, now carried over. +DROP TABLE IF EXISTS gitlab_task_mr_links; +DROP TABLE IF EXISTS gitlab_task_branches; +DROP TABLE IF EXISTS gitlab_merge_requests; +DROP TABLE IF EXISTS gitlab_repositories; +DROP TABLE IF EXISTS gitlab_integrations; diff --git a/backend/plugin.go b/backend/plugin.go index 0c2b1d4..e817535 100644 --- a/backend/plugin.go +++ b/backend/plugin.go @@ -22,7 +22,9 @@ func (p *gitlabPlugin) Init(ctx *plugin.Context) error { p.db = ctx.DB() p.log = ctx.Log() p.cfg = ctx.Config() - serve.Register(ctx, newGLProvider(newDBStore(p))) + prov := newGLProvider(newDBStore(p)) + prov.warn = p.log.Warn + serve.Register(ctx, prov) return nil } diff --git a/backend/spi_conformance_test.go b/backend/spi_conformance_test.go index 0a1f990..a4cda04 100644 --- a/backend/spi_conformance_test.go +++ b/backend/spi_conformance_test.go @@ -3,6 +3,7 @@ package main import ( "encoding/json" "strconv" + "strings" "sync" "testing" "time" @@ -17,10 +18,12 @@ type memStore struct { identities map[string]identityRecord setups map[string]setupSession seq int + // legacyHooks are each identity's community-plugin hooks still to remove. + legacyHooks map[string][]legacyWebhook } func newMemStore() *memStore { - return &memStore{identities: map[string]identityRecord{}, setups: map[string]setupSession{}} + return &memStore{identities: map[string]identityRecord{}, setups: map[string]setupSession{}, legacyHooks: map[string][]legacyWebhook{}} } func (m *memStore) loadIdentity(id string) (identityRecord, error) { @@ -61,6 +64,48 @@ func (m *memStore) saveReach(id string, r tokenReach) error { return nil } +func (m *memStore) completeIdentity(r identityRecord) error { + m.mu.Lock() + defer m.mu.Unlock() + rec, ok := m.identities[r.ID] + if !ok { + return errNoSuchIdentity + } + if rec.Username == "" { + rec.UserID, rec.Username, rec.Name, rec.Bot, rec.ExpiresAt = r.UserID, r.Username, r.Name, r.Bot, r.ExpiresAt + } + if rec.WebhookSecret == "" { + rec.WebhookSecret = r.WebhookSecret + } + m.identities[r.ID] = rec + return nil +} + +func (m *memStore) legacyWebhooks(identityID string, gitlabProjectID int64, fullPath string) ([]legacyWebhook, error) { + m.mu.Lock() + defer m.mu.Unlock() + var out []legacyWebhook + for _, h := range m.legacyHooks[identityID] { + if (gitlabProjectID > 0 && h.GitLabProjectID == gitlabProjectID) || strings.EqualFold(h.FullPath, fullPath) { + out = append(out, h) + } + } + return out, nil +} + +func (m *memStore) forgetLegacyWebhook(identityID string, webhookID int64) error { + m.mu.Lock() + defer m.mu.Unlock() + kept := m.legacyHooks[identityID][:0] + for _, h := range m.legacyHooks[identityID] { + if h.WebhookID != webhookID { + kept = append(kept, h) + } + } + m.legacyHooks[identityID] = kept + return nil +} + func (m *memStore) saveSetup(s setupSession) error { m.mu.Lock() defer m.mu.Unlock() diff --git a/backend/spi_dbstore.go b/backend/spi_dbstore.go index d9d34a0..200b4f4 100644 --- a/backend/spi_dbstore.go +++ b/backend/spi_dbstore.go @@ -1,6 +1,7 @@ package main import ( + "fmt" "time" plugin "github.com/Paca-AI/plugin-sdk-go" @@ -35,7 +36,11 @@ func (s dbStore) loadIdentity(id string) (identityRecord, error) { if enc == "" { return "", nil } - return s.p.decrypt(enc) + v, err := s.p.decrypt(enc) + if err != nil { + return "", fmt.Errorf("%w: %s: %v", errUndecryptable, col, err) + } + return v, nil } token, err := open("token_enc") if err != nil { @@ -110,6 +115,55 @@ func (s dbStore) saveReach(id string, r tokenReach) error { return err } +func (s dbStore) completeIdentity(x identityRecord) error { + secret := "" + if x.WebhookSecret != "" { + var err error + if secret, err = s.p.encrypt(x.WebhookSecret); err != nil { + return err + } + } + var expires any + if !x.ExpiresAt.IsZero() { + expires = x.ExpiresAt.UTC().Format(time.RFC3339Nano) + } + // Every CASE reads the row as it was before this statement, so the user + // fields are written together, and only while the username is still blank. + _, err := s.db.Exec(` + UPDATE gitlab_identities SET + user_id = CASE WHEN username = '' THEN $2 ELSE user_id END, + username = CASE WHEN username = '' THEN $3 ELSE username END, + display_name = CASE WHEN username = '' THEN $4 ELSE display_name END, + bot = CASE WHEN username = '' THEN $5 ELSE bot END, + expires_at = CASE WHEN username = '' THEN $6::timestamptz ELSE expires_at END, + webhook_secret_enc = CASE WHEN webhook_secret_enc = '' THEN $7 ELSE webhook_secret_enc END, + updated_at = $8 + WHERE id = $1`, + x.ID, x.UserID, x.Username, x.Name, x.Bot, expires, secret, nowStr()) + return err +} + +func (s dbStore) legacyWebhooks(identityID string, gitlabProjectID int64, fullPath string) ([]legacyWebhook, error) { + r, err := s.db.Query(` + SELECT gitlab_project_id, full_path, webhook_id FROM gitlab_legacy_webhooks + WHERE identity_id = $1 AND (($2 > 0 AND gitlab_project_id = $2) OR lower(full_path) = lower($3))`, + identityID, gitlabProjectID, fullPath) + if err != nil { + return nil, err + } + var out []legacyWebhook + for _, row := range r.Rows { + sc := newRowScanner(r.Columns, row) + out = append(out, legacyWebhook{GitLabProjectID: sc.int64Val("gitlab_project_id"), FullPath: sc.str("full_path"), WebhookID: sc.int64Val("webhook_id")}) + } + return out, nil +} + +func (s dbStore) forgetLegacyWebhook(identityID string, webhookID int64) error { + _, err := s.db.Exec(`DELETE FROM gitlab_legacy_webhooks WHERE identity_id = $1 AND webhook_id = $2`, identityID, webhookID) + return err +} + func (s dbStore) saveSetup(x setupSession) error { _, err := s.db.Exec(` INSERT INTO gitlab_setup_sessions (state, scope, agent_id, host, expires_at, created_at) diff --git a/backend/spi_fakegitlab_test.go b/backend/spi_fakegitlab_test.go index 8a4149e..0612460 100644 --- a/backend/spi_fakegitlab_test.go +++ b/backend/spi_fakegitlab_test.go @@ -646,7 +646,10 @@ func (g *fakeGitLab) hookAPI(method string, rest []string, payload map[string]an if _, ok := g.hooks[id]; !ok { return notFound() } - if method == "DELETE" { + switch method { + case "GET": + return resp(200, map[string]any{"id": id, "url": g.hooks[id]}) + case "DELETE": delete(g.hooks, id) return text(204, "") } diff --git a/backend/spi_hooks.go b/backend/spi_hooks.go index 1892bde..c146e66 100644 --- a/backend/spi_hooks.go +++ b/backend/spi_hooks.go @@ -44,15 +44,25 @@ func (p *glProvider) EnsureWebhook(_ context.Context, req gp.EnsureWebhookReques if rec.WebhookSecret == "" { return gp.WebhookRef{}, gp.Errorf(gp.CodeConflict, "the identity has no webhook secret") } - api := rec.api() - body := hookBody(u.String(), rec.WebhookSecret) + ref, err := ensureHook(rec.api(), proj, u.String(), hookBody(u.String(), rec.WebhookSecret)) + if err != nil { + return gp.WebhookRef{}, err + } + // Only now that the new hook is in place: a repository keeps its old hook + // until it has a working one. + p.removeLegacyHooks(rec, req.Target.Repo, proj) + return ref, nil +} + +// ensureHook updates the project's hook that points at hookURL, or creates one. +func ensureHook(api *glAPI, proj, hookURL string, body map[string]any) (gp.WebhookRef, error) { for pg := 1; pg <= maxHookPages; pg++ { var hooks []glHook if err := api.get(paged(proj+"/hooks", pg), &hooks); err != nil { return gp.WebhookRef{}, err } for _, h := range hooks { - if h.URL == u.String() { + if h.URL == hookURL { if err := api.send("PUT", proj+"/hooks/"+strconv.FormatInt(h.ID, 10), body, nil); err != nil { return gp.WebhookRef{}, err } diff --git a/backend/spi_legacy_hooks.go b/backend/spi_legacy_hooks.go new file mode 100644 index 0000000..aecf01c --- /dev/null +++ b/backend/spi_legacy_hooks.go @@ -0,0 +1,70 @@ +package main + +import ( + "net/url" + "strconv" + "strings" + + gp "github.com/Paca-AI/plugin-sdk-go/gitprovider" +) + +// legacyHookPath is in the URL of every hook the community plugin (0.5.x) +// registered: /api/v1/plugins/com.gorlix.gitlab/projects//webhook. +const legacyHookPath = "/api/v1/plugins/com.gorlix.gitlab/projects/" + +// legacyWebhook is a hook the community plugin registered on a repository with +// the token an identity was migrated from (see 0004_migrate_community_plugin.sql). +type legacyWebhook struct { + GitLabProjectID int64 + FullPath string + WebhookID int64 +} + +func isLegacyHookURL(raw string) bool { + u, err := url.Parse(raw) + return err == nil && strings.Contains(u.Path, legacyHookPath) && strings.HasSuffix(u.Path, "/webhook") +} + +// removeLegacyHooks deletes the hooks the community plugin registered on the +// repository, once its new hook is in place: they point at the old plugin, +// whose route is gone once it is removed, so GitLab would keep failing them +// until it disables them. Only a hook whose URL is still the old plugin's is +// deleted; one that was pointed elsewhere is left alone. +// +// It is best effort and never fails EnsureWebhook. Every outcome but a +// retryable failure is recorded, so a hook is tried once, not on every call. +func (p *glProvider) removeLegacyHooks(rec identityRecord, repo gp.RepoRef, proj string) { + var projectID int64 + if allDigits(repo.ExternalID) { + projectID, _ = strconv.ParseInt(repo.ExternalID, 10, 64) + } + hooks, err := p.store.legacyWebhooks(rec.ID, projectID, repo.FullPath) + if err != nil || len(hooks) == 0 { + return + } + api := rec.api() + for _, h := range hooks { + path := proj + "/hooks/" + strconv.FormatInt(h.WebhookID, 10) + var cur glHook + err := api.get(path, &cur) + switch { + case err == nil && isLegacyHookURL(cur.URL): + err = api.send("DELETE", path, nil, nil) + case err == nil: + p.warn("GitLab hook " + strconv.FormatInt(h.WebhookID, 10) + " on " + h.FullPath + + " no longer points at the previous GitLab plugin; it was left in place") + } + if err != nil { + e := gp.AsError(err) + if e.Retryable() { + continue + } + // Already gone, or the token may not manage hooks: nothing to retry. + if e.Code != gp.CodeNotFound && e.Code != gp.CodeForbidden { + p.warn("could not remove the previous GitLab plugin's hook " + strconv.FormatInt(h.WebhookID, 10) + + " on " + h.FullPath + ": " + e.Message) + } + } + _ = p.store.forgetLegacyWebhook(rec.ID, h.WebhookID) + } +} diff --git a/backend/spi_migrated_test.go b/backend/spi_migrated_test.go new file mode 100644 index 0000000..a51cd50 --- /dev/null +++ b/backend/spi_migrated_test.go @@ -0,0 +1,288 @@ +package main + +import ( + "errors" + "strings" + "testing" + + gp "github.com/Paca-AI/plugin-sdk-go/gitprovider" +) + +// migratedIdentity seeds what 0004_migrate_community_plugin.sql leaves behind: +// the host and the token, nothing else. +func migratedIdentity(st *memStore, id, token string) gp.IdentityRef { + st.identities[id] = identityRecord{ID: id, Host: fakeHost, Token: token} + return gp.IdentityRef{ID: id} +} + +func (g *fakeGitLab) count(call string) int { + g.mu.Lock() + defer g.mu.Unlock() + n := 0 + for _, c := range g.calls { + if c == call { + n++ + } + } + return n +} + +// A migrated token is completed on first use: its user is asked of GitLab, a +// webhook secret is generated and its reach is classified, and all of it is +// stored so the next use asks GitLab nothing. +func TestMigratedIdentity_CompletedOnFirstUse(t *testing.T) { + g := newFakeGitLab(t) + st := newMemStore() + p := newGLProvider(st) + ref := migratedIdentity(st, "migrated-1", fakeProjectToken) + + got, err := p.ResolveRepository(bg, gp.ResolveRepoRequest{Candidates: []gp.IdentityRef{ref}, PathOrURL: "acme/ui"}) + if err != nil { + t.Fatal(err) + } + if got.Repository.Ref.FullPath != "acme/ui" { + t.Errorf("resolved %+v", got.Repository) + } + rec := st.identities["migrated-1"] + if rec.Username != "project_4_bot_ab12" || rec.UserID != 77 || !rec.Bot { + t.Errorf("user not stored: %+v", rec) + } + if len(rec.WebhookSecret) != 64 { + t.Errorf("webhook secret %q, want 32 random bytes in hex", rec.WebhookSecret) + } + if rec.ExpiresAt.IsZero() { + t.Error("the token's expiry was not stored") + } + if rec.Reach != projectReach { + t.Errorf("reach %+v, want %+v", rec.Reach, projectReach) + } + + secret := rec.WebhookSecret + if _, err := p.ResolveRepository(bg, gp.ResolveRepoRequest{Candidates: []gp.IdentityRef{ref}, PathOrURL: "acme/ui"}); err != nil { + t.Fatal(err) + } + if n := g.count("GET /user"); n != 1 { + t.Errorf("GET /user called %d times, want once", n) + } + if st.identities["migrated-1"].WebhookSecret != secret { + t.Error("the webhook secret changed on the second use") + } +} + +// The completed identity registers hooks with its new secret and verifies the +// deliveries signed with it. +func TestMigratedIdentity_WebhookUsesTheGeneratedSecret(t *testing.T) { + newFakeGitLab(t) + st := newMemStore() + p := newGLProvider(st) + ref := migratedIdentity(st, "migrated-1", fakeToken) + + body := []byte(`{"object_kind":"push","project":{"id":1,"path_with_namespace":"acme/web"},"ref":"refs/heads/main"}`) + _, err := p.VerifyAndParseWebhook(bg, gp.WebhookRequest{Identity: ref, Headers: map[string]string{"x-gitlab-token": "guess"}, Body: body}) + code(t, err, gp.CodeInvalidSignature) + + secret := st.identities["migrated-1"].WebhookSecret + if secret == "" { + t.Fatal("no webhook secret was generated") + } + if _, err := p.VerifyAndParseWebhook(bg, gp.WebhookRequest{Identity: ref, Headers: map[string]string{"x-gitlab-token": secret}, Body: body}); err != nil { + t.Fatalf("delivery signed with the generated secret: %v", err) + } +} + +// A token GitLab no longer accepts cannot be completed: the error says to +// connect again, and nothing is stored. +func TestMigratedIdentity_RevokedToken(t *testing.T) { + newFakeGitLab(t) + st := newMemStore() + p := newGLProvider(st) + ref := migratedIdentity(st, "migrated-1", "glpat-revoked") + + _, err := p.ListRepositories(bg, gp.ListReposRequest{Identity: ref}) + code(t, err, gp.CodeForbidden) + if !strings.Contains(err.Error(), "connect GitLab again") { + t.Errorf("error does not say what to do: %v", err) + } + if rec := st.identities["migrated-1"]; rec.Username != "" || rec.WebhookSecret != "" { + t.Errorf("a failed completion stored %+v", rec) + } +} + +// racingStore stores another secret just before the provider's completion, as a +// concurrent first use on another replica would. +type racingStore struct { + *memStore + other string +} + +func (s racingStore) completeIdentity(r identityRecord) error { + first := r + first.WebhookSecret = s.other + if err := s.memStore.completeIdentity(first); err != nil { + return err + } + return s.memStore.completeIdentity(r) +} + +// Two first uses at once must agree on the secret, or the hook one registers +// rejects the deliveries the other verifies: the stored one wins. +func TestMigratedIdentity_ConcurrentCompletionKeepsTheStoredSecret(t *testing.T) { + newFakeGitLab(t) + st := newMemStore() + p := newGLProvider(racingStore{memStore: st, other: "secret-stored-by-the-other-replica"}) + ref := migratedIdentity(st, "migrated-1", fakeToken) + + rec, err := p.loadIdentity(ref.ID) + if err != nil { + t.Fatal(err) + } + if rec.WebhookSecret != "secret-stored-by-the-other-replica" { + t.Errorf("provider uses %q, not the stored secret", rec.WebhookSecret) + } +} + +// undecryptableStore is a store whose instance key does not open the token. +type undecryptableStore struct{ *memStore } + +func (undecryptableStore) loadIdentity(string) (identityRecord, error) { + return identityRecord{}, errors.Join(errUndecryptable, errors.New("cipher: message authentication failed")) +} + +// A token stored under another ENCRYPTION_KEY gets an error naming the key, +// not a generic failure. +func TestMigratedIdentity_WrongEncryptionKey(t *testing.T) { + p := newGLProvider(undecryptableStore{newMemStore()}) + _, err := p.loadIdentity("migrated-1") + code(t, err, gp.CodeInternal) + if !strings.Contains(err.Error(), "ENCRYPTION_KEY") { + t.Errorf("error does not name the key: %v", err) + } +} + +const ( + newHookURL = "https://paca.test/api/v1/git/webhooks/abc" + legacyHookURL = "https://paca.test/api/v1/plugins/com.gorlix.gitlab/projects/0b6f1c1e-3c1a-4e0e-9a51-7f1f0b3c2d4e/webhook" +) + +// legacyHooksProvider is the fixture identity with community-plugin hooks on +// acme/web: hook 10 (registered by project id) and hook 11 (by path only). +func legacyHooksProvider(t *testing.T, urls map[int64]string) (*glProvider, *fakeGitLab, *memStore, *[]string) { + t.Helper() + p, g, st := seededProvider(t) + var warned []string + p.warn = func(m string) { warned = append(warned, m) } + for id, u := range urls { + g.hooks[id] = u + h := legacyWebhook{GitLabProjectID: 1, FullPath: "acme/web", WebhookID: id} + if id == 11 { + h = legacyWebhook{FullPath: "Acme/Web", WebhookID: id} + } + st.legacyHooks[fakeIdentityID] = append(st.legacyHooks[fakeIdentityID], h) + } + return p, g, st, &warned +} + +func ensureFixtureHook(p *glProvider) error { + _, err := p.EnsureWebhook(bg, gp.EnsureWebhookRequest{Target: fixtureTarget, URL: newHookURL}) + return err +} + +// Once the new hook is registered, the old plugin's hooks on the repository go, +// whether they were recorded by project id or by path. +func TestLegacyHooks_RemovedAfterTheNewHook(t *testing.T) { + p, g, st, warned := legacyHooksProvider(t, map[int64]string{10: legacyHookURL, 11: legacyHookURL}) + if err := ensureFixtureHook(p); err != nil { + t.Fatal(err) + } + if _, ok := g.hooks[10]; ok { + t.Error("legacy hook 10 is still on GitLab") + } + if _, ok := g.hooks[11]; ok { + t.Error("legacy hook 11 (recorded by path) is still on GitLab") + } + if len(g.hooks) != 1 { + t.Errorf("hooks left: %v, want only the new one", g.hooks) + } + if n := len(st.legacyHooks[fakeIdentityID]); n != 0 { + t.Errorf("%d legacy hooks still recorded", n) + } + if len(*warned) != 0 { + t.Errorf("unexpected warnings: %v", *warned) + } +} + +// A hook someone has since pointed elsewhere is not the old plugin's any more: +// it stays, with a warning, and is not looked at again. +func TestLegacyHooks_RepointedHookIsLeft(t *testing.T) { + p, g, st, warned := legacyHooksProvider(t, map[int64]string{10: "https://ci.example.com/hook"}) + if err := ensureFixtureHook(p); err != nil { + t.Fatal(err) + } + if g.hooks[10] != "https://ci.example.com/hook" { + t.Error("a hook that does not point at the old plugin was deleted") + } + if len(*warned) != 1 || !strings.Contains((*warned)[0], "left in place") { + t.Errorf("warnings: %v", *warned) + } + if n := len(st.legacyHooks[fakeIdentityID]); n != 0 { + t.Errorf("%d legacy hooks still recorded", n) + } +} + +// Without a working new hook the repository keeps its old one. +func TestLegacyHooks_KeptWhenRegistrationFails(t *testing.T) { + p, g, st, _ := legacyHooksProvider(t, map[int64]string{10: legacyHookURL}) + g.forceError["POST /projects/1/hooks"] = fakeResponse{status: 403, body: `{"message":"403 Forbidden"}`} + if err := ensureFixtureHook(p); err == nil { + t.Fatal("registration was expected to fail") + } + if _, ok := g.hooks[10]; !ok { + t.Error("legacy hook deleted although the new one was not registered") + } + if n := len(st.legacyHooks[fakeIdentityID]); n != 1 { + t.Errorf("%d legacy hooks recorded, want 1 still to do", n) + } +} + +// A hook already deleted on GitLab is done with; a transient failure is retried +// on the next call. Neither fails the registration. +func TestLegacyHooks_GoneOrTransient(t *testing.T) { + p, g, st, warned := legacyHooksProvider(t, map[int64]string{10: legacyHookURL, 11: legacyHookURL}) + delete(g.hooks, 10) + g.forceError["GET /projects/1/hooks/11"] = fakeResponse{status: 502, body: `{"message":"502 Bad Gateway"}`} + if err := ensureFixtureHook(p); err != nil { + t.Fatal(err) + } + left := st.legacyHooks[fakeIdentityID] + if len(left) != 1 || left[0].WebhookID != 11 { + t.Fatalf("recorded after the first call: %+v, want only hook 11", left) + } + delete(g.forceError, "GET /projects/1/hooks/11") + if err := ensureFixtureHook(p); err != nil { + t.Fatal(err) + } + if _, ok := g.hooks[11]; ok { + t.Error("legacy hook 11 was not removed on the retry") + } + if n := len(st.legacyHooks[fakeIdentityID]); n != 0 { + t.Errorf("%d legacy hooks still recorded", n) + } + if len(*warned) != 0 { + t.Errorf("unexpected warnings: %v", *warned) + } +} + +func TestIsLegacyHookURL(t *testing.T) { + for u, want := range map[string]bool{ + legacyHookURL: true, + "https://paca.example.com/sub/api/v1/plugins/com.gorlix.gitlab/projects/x/webhook": true, + newHookURL: false, + "https://paca.test/api/v1/plugins/com.paca.gitlab/projects/x/webhook": false, + "https://evil.test/?u=/api/v1/plugins/com.gorlix.gitlab/projects/x/webhook": false, + "::": false, + } { + if got := isLegacyHookURL(u); got != want { + t.Errorf("isLegacyHookURL(%q) = %v, want %v", u, got, want) + } + } +} diff --git a/backend/spi_ops_test.go b/backend/spi_ops_test.go index 0a77c29..fe49ef1 100644 --- a/backend/spi_ops_test.go +++ b/backend/spi_ops_test.go @@ -353,7 +353,7 @@ func TestOptionalOps_ArgumentChecks(t *testing.T) { } func TestWebhookManagement_UpdatesInsteadOfDuplicating(t *testing.T) { - p, g, st := seededProvider(t) + p, g, _ := seededProvider(t) req := gp.EnsureWebhookRequest{Target: fixtureTarget, URL: "https://paca.test/api/v1/git/webhooks/abc"} a, err := p.EnsureWebhook(bg, req) if err != nil { @@ -378,12 +378,6 @@ func TestWebhookManagement_UpdatesInsteadOfDuplicating(t *testing.T) { g.forceError["POST /projects/1/hooks"] = fakeResponse{status: 403, body: `{"message":"403 Forbidden"}`} _, err = p.EnsureWebhook(bg, gp.EnsureWebhookRequest{Target: fixtureTarget, URL: "https://paca.test/api/v1/git/webhooks/ghi"}) code(t, err, gp.CodeForbidden) - - rec := st.identities[fakeIdentityID] - rec.WebhookSecret = "" - st.identities[fakeIdentityID] = rec - _, err = p.EnsureWebhook(bg, req) - code(t, err, gp.CodeConflict) } func TestDeleteIdentity_ForgetsTheTokens(t *testing.T) { diff --git a/backend/spi_provider.go b/backend/spi_provider.go index ada8901..dc3ffc1 100644 --- a/backend/spi_provider.go +++ b/backend/spi_provider.go @@ -3,6 +3,7 @@ package main import ( "context" "encoding/json" + "errors" "net/url" "strconv" "strings" @@ -20,10 +21,12 @@ import ( type glProvider struct { store spiStore now func() time.Time + // warn reports what an operator may want to know but no caller can act on. + warn func(msg string) } func newGLProvider(store spiStore) *glProvider { - return &glProvider{store: store, now: time.Now} + return &glProvider{store: store, now: time.Now, warn: func(string) {}} } // Describe implements gp.Provider. Capabilities are declared honestly: @@ -90,11 +93,21 @@ func (p *glProvider) loadIdentity(identityID string) (identityRecord, error) { } rec, err := p.store.loadIdentity(identityID) if err != nil { - if err == errNoSuchIdentity { + switch { + case errors.Is(err, errNoSuchIdentity): return identityRecord{}, notFoundIdentity() + case errors.Is(err, errUndecryptable): + return identityRecord{}, gp.Errorf(gp.CodeInternal, + "the stored GitLab token cannot be decrypted: the instance's ENCRYPTION_KEY is not the one it was stored with; "+ + "restore that key, or connect GitLab again") } return identityRecord{}, gp.Errorf(gp.CodeInternal, "could not load the GitLab identity") } + if rec.Username == "" || rec.WebhookSecret == "" { + if err := p.completeMigratedIdentity(&rec); err != nil { + return identityRecord{}, err + } + } if rec.Reach.Type == "" { // Stored before token types were recorded: classify from the username // once and save it. A failed save only means doing this again next time. @@ -106,6 +119,47 @@ func (p *glProvider) loadIdentity(identityID string) (identityRecord, error) { return rec, nil } +// completeMigratedIdentity fills in what an identity carried over from the +// community plugin (0.5.x) lacks: the token's user (asked of GitLab) and a +// webhook secret (generated; the old per-repository secrets belonged to hooks +// that pointed at the old plugin). It runs on first use and stores the result; +// the token's reach is classified afterwards like any unclassified identity. +func (p *glProvider) completeMigratedIdentity(rec *identityRecord) error { + if rec.Username == "" { + // The token was checked when it was saved, possibly long ago: a token + // GitLab rejects now has to be replaced, not retried. + user, info, err := validateToken(rec.api(), false) + if err != nil { + if e := gp.AsError(err); e.Code == gp.CodeValidation { + return gp.Errorf(gp.CodeForbidden, "the access token carried over from the previous GitLab plugin no longer works (%s): connect GitLab again with a new token", e.Message) + } + return err + } + rec.UserID, rec.Username, rec.Name = user.ID, user.Username, user.Name + rec.Bot = user.Bot || accessTokenBotRe.MatchString(user.Username) + rec.ExpiresAt = info.expiry() + } + if rec.WebhookSecret == "" { + secret, err := randomHex(32) + if err != nil { + return gp.Errorf(gp.CodeInternal, "could not generate a webhook secret") + } + rec.WebhookSecret = secret + } + if err := p.store.completeIdentity(*rec); err != nil { + return gp.Errorf(gp.CodeInternal, "could not store the GitLab identity") + } + // completeIdentity only fills blanks, so a concurrent first use may have + // stored a different secret first: read back what won, or hooks signed with + // ours fail. + won, err := p.store.loadIdentity(rec.ID) + if err != nil { + return gp.Errorf(gp.CodeInternal, "could not load the GitLab identity") + } + *rec = won + return nil +} + func (r identityRecord) api() *glAPI { return &glAPI{base: baseURL(r.Host), token: r.Token} } // sameHost reports whether a repository's host is the identity's. An empty host diff --git a/backend/spi_store.go b/backend/spi_store.go index 0d23518..30bb198 100644 --- a/backend/spi_store.go +++ b/backend/spi_store.go @@ -14,6 +14,15 @@ type spiStore interface { deleteIdentity(id string) error // saveReach records what an identity's token covers (see identityRecord). saveReach(id string, r tokenReach) error + // completeIdentity stores the user and webhook secret of an identity carried + // over from the community plugin (0.5.x). It only fills fields that are still + // blank, so a concurrent completion keeps whichever was stored first. + completeIdentity(r identityRecord) error + // legacyWebhooks lists the community plugin's hooks on a repository, by + // GitLab project id or path, still to be removed for the identity. + legacyWebhooks(identityID string, gitlabProjectID int64, fullPath string) ([]legacyWebhook, error) + // forgetLegacyWebhook records that a legacy hook was dealt with. + forgetLegacyWebhook(identityID string, webhookID int64) error saveSetup(s setupSession) error loadSetup(state string) (setupSession, error) @@ -26,6 +35,10 @@ type spiStore interface { var errNoSuchIdentity = errors.New("gitlab identity not found") var errNoSuchSetup = errors.New("setup session not found") +// errUndecryptable is a stored secret the instance key does not open: the key +// changed since the row was stored. +var errUndecryptable = errors.New("gitlab identity secret cannot be decrypted") + // identityRecord is one GitLab access-token identity. type identityRecord struct { ID string // row id; the SPI IdentityRef.ID diff --git a/scripts/test-legacy-migration/run.sh b/scripts/test-legacy-migration/run.sh new file mode 100755 index 0000000..446f2ea --- /dev/null +++ b/scripts/test-legacy-migration/run.sh @@ -0,0 +1,178 @@ +#!/usr/bin/env bash +# Tests backend/migrations/0004_migrate_community_plugin.sql against Postgres 16 +# with core's real git tables (000073 + 000076) and the community plugin's +# schema (0.5.x's 0001 is this plugin's 0001). Each migration runs the way core +# runs it: one transaction with SET LOCAL search_path TO ,public. +# +# Usage: PACA_CORE=/path/to/paca [PACA_CORE_REF=origin/master] scripts/test-legacy-migration/run.sh +# Needs docker. Starts and removes a throwaway container, paca-migtest-pg. +set -euo pipefail + +HERE=$(cd "$(dirname "$0")" && pwd) +PLUGIN=$(cd "$HERE/../.." && pwd) +CORE=${PACA_CORE:?set PACA_CORE to a checkout of Paca-AI/paca} +CORE_REF=${PACA_CORE_REF:-HEAD} +PASS=0; FAIL=0 +NAME=paca-migtest-pg +PASS=0; FAIL=0 + +docker rm -f "$NAME" >/dev/null 2>&1 || true +docker run -d --name "$NAME" -e POSTGRES_PASSWORD=t -e POSTGRES_DB=paca postgres:16 >/dev/null +trap 'docker rm -f "$NAME" >/dev/null 2>&1 || true' EXIT +until docker exec "$NAME" pg_isready -U postgres -d paca >/dev/null 2>&1; do sleep 0.5; done +sleep 1 + +psql() { docker exec -i "$NAME" psql -U postgres -v ON_ERROR_STOP=1 -q -X "$@"; } +val() { psql -d "$1" -At -c "$2"; } +check() { # db description sql expected + local got; got=$(val "$1" "$3") + if [[ "$got" == "$4" ]]; then PASS=$((PASS+1)); echo " ok $2" + else FAIL=$((FAIL+1)); echo " FAIL $2: got [$got] want [$4]"; fi +} + +# migrate : core's runner, one transaction per file. +migrate() { + local db=$1 schema=$2; shift 2 + psql -d "$db" -c "CREATE SCHEMA IF NOT EXISTS $schema" + for f in "$@"; do + { echo "BEGIN; SET LOCAL search_path TO $schema,public;"; cat "$f"; echo "; COMMIT;"; } | psql -d "$db" + done +} + +core_schema() { # the parts of core the git tables reference, then the git tables + psql -d "$1" <<'SQL' +CREATE TABLE users (id UUID PRIMARY KEY DEFAULT gen_random_uuid()); +CREATE TABLE projects (id UUID PRIMARY KEY DEFAULT gen_random_uuid(), name TEXT NOT NULL); +CREATE TABLE agents (id UUID PRIMARY KEY DEFAULT gen_random_uuid()); +CREATE TABLE tasks (id UUID PRIMARY KEY DEFAULT gen_random_uuid(), project_id UUID NOT NULL REFERENCES projects(id) ON DELETE CASCADE, deleted_at TIMESTAMPTZ); +SQL + git -C "$CORE" show "$CORE_REF":services/api/migrations/000073_add_git_integration.sql | psql -d "$1" + git -C "$CORE" show "$CORE_REF":services/api/migrations/000076_add_git_repository_description.sql | psql -d "$1" +} + +M=$PLUGIN/backend/migrations +LEGACY=$M/0001_create_gitlab_tables.sql # the tables 0.5.x created +NEW=("$M/0001_create_gitlab_tables.sql" "$M/0002_git_provider.sql" "$M/0003_identity_token_reach.sql" "$M/0004_migrate_community_plugin.sql") + +S=plugin_data_com_paca_gitlab +full_checks() { # db + local D=$1 +check $D "one plugin identity per old integration" "SELECT count(*) FROM $S.gitlab_identities" 3 +check $D "ids kept, ciphertext copied unchanged" "SELECT string_agg(id || '=' || token_enc, ',' ORDER BY id) FROM $S.gitlab_identities" "30000000-0000-0000-0000-000000000001=aa01,30000000-0000-0000-0000-000000000002=aa02,30000000-0000-0000-0000-000000000003=aa03" +check $D "username, secret and reach left for first use" "SELECT count(*) FROM $S.gitlab_identities WHERE username = '' AND webhook_secret_enc = '' AND token_type = '' AND read_token_enc = ''" 3 +check $D "hosts from instance_url (case, port, path, empty)" "SELECT string_agg(host, ',' ORDER BY host) FROM $S.gitlab_identities" "gitlab.com,gitlab.com,gitlab.example.com:8443" +check $D "core identities: project-scoped tokens" "SELECT count(*) FROM git_identities WHERE provider_id = 'com.paca.gitlab' AND kind = 'token' AND scope = 'project'" 3 +check $D "core identity per old project" "SELECT string_agg(provider_identity_id || '@' || project_id, ',' ORDER BY provider_identity_id) FROM git_identities WHERE provider_id = 'com.paca.gitlab'" "30000000-0000-0000-0000-000000000001@10000000-0000-0000-0000-000000000001,30000000-0000-0000-0000-000000000002@10000000-0000-0000-0000-000000000002,30000000-0000-0000-0000-000000000003@10000000-0000-0000-0000-000000000003" +check $D "core identity host = plugin identity host" "SELECT count(*) FROM git_identities c JOIN $S.gitlab_identities g ON g.id::text = c.provider_identity_id AND g.host = c.host" 3 +check $D "GitHub identity untouched" "SELECT count(*) FROM git_identities WHERE provider_id = 'com.paca.github'" 1 +check $D "one webhook endpoint per migrated identity" "SELECT count(*) FROM git_webhook_endpoints e JOIN git_identities c ON c.id = e.identity_id WHERE c.provider_id = 'com.paca.gitlab'" 3 +check $D "endpoint keys are 64 hex chars" "SELECT count(*) FROM git_webhook_endpoints WHERE provider_id = 'com.paca.gitlab' AND key ~ '^[0-9a-f]{64}$'" 3 +check $D "repositories (one already linked by hand)" "SELECT count(*) FROM git_repositories WHERE provider_id = 'com.paca.gitlab'" 4 +check $D "the hand-linked one is not duplicated" "SELECT count(*) FROM git_repositories WHERE lower(full_path) = 'acme/web'" 1 +check $D "the hand-linked one keeps its identity" "SELECT identity_id IS NULL FROM git_repositories WHERE lower(full_path) = 'acme/web'" t +check $D "external_id = GitLab project id, '' when unknown" "SELECT string_agg(full_path || '=' || external_id, ',' ORDER BY full_path COLLATE ucs_basic) FROM git_repositories WHERE provider_id = 'com.paca.gitlab'" "Acme/Web=,acme/empty=12,group/sub/infra=,ops/tools=33" +check $D "clone URLs on the identity's host" "SELECT string_agg(clone_url, ',' ORDER BY full_path COLLATE ucs_basic) FROM git_repositories WHERE provider_id = 'com.paca.gitlab' AND identity_id IS NOT NULL" "https://gitlab.com/acme/empty.git,https://gitlab.example.com:8443/group/sub/infra.git,https://gitlab.com/ops/tools.git" +check $D "default branch kept" "SELECT default_branch FROM git_repositories WHERE full_path = 'ops/tools'" develop +check $D "merge requests" "SELECT count(*) FROM git_change_requests" 5 +check $D "states (unknown → open)" "SELECT string_agg(number || ':' || state, ',' ORDER BY number::int) FROM git_change_requests" "1:open,2:merged,3:closed,4:open,7:merged" +check $D "merged_at kept" "SELECT count(*) FROM git_change_requests WHERE merged_at IS NOT NULL" 2 +check $D "MRs of the hand-linked repo attach to it" "SELECT count(*) FROM git_change_requests cr JOIN git_repositories r ON r.id = cr.repository_id WHERE r.full_path = 'Acme/Web'" 2 +check $D "MR fields kept" "SELECT title || '|' || head_ref || '|' || base_ref || '|' || author || '|' || html_url FROM git_change_requests WHERE number = '2'" "Add login|feat/login|main|alice|https://gitlab.com/acme/web/-/merge_requests/2" +check $D "task ↔ MR links (soft-deleted task included)" "SELECT count(*) FROM git_task_links" 4 +check $D "task branches" "SELECT count(*) FROM git_task_branches" 3 +check $D "legacy hooks recorded per identity" "SELECT string_agg(webhook_id || '@' || full_path || '#' || gitlab_project_id, ',' ORDER BY webhook_id) FROM $S.gitlab_legacy_webhooks" "101@acme/web#11,103@group/sub/infra#0,104@ops/tools#33" +check $D "this schema's old tables dropped" "SELECT count(*) FROM information_schema.tables WHERE table_schema = '$S' AND table_name IN ('gitlab_integrations','gitlab_repositories')" 0 + +} + +echo "== upgrade from 0.5.2 (com.gorlix.gitlab, its own schema)" +psql -d paca -c "CREATE DATABASE from_052" +core_schema from_052 +migrate from_052 plugin_data_com_gorlix_gitlab "$LEGACY" +psql -d from_052 -v legacy=plugin_data_com_gorlix_gitlab -f - < "$HERE/seed.sql" +migrate from_052 $S "${NEW[@]}" +full_checks from_052 +check from_052 "0.5.2's schema left as it was" "SELECT (SELECT count(*) FROM plugin_data_com_gorlix_gitlab.gitlab_integrations) || '/' || (SELECT count(*) FROM plugin_data_com_gorlix_gitlab.gitlab_merge_requests)" "3/5" + +echo "== upgrade in place from 0.5.0/0.5.1 (com.paca.gitlab, this schema)" +psql -d paca -c "CREATE DATABASE from_050" +core_schema from_050 +migrate from_050 $S "${NEW[0]}" +psql -d from_050 -v legacy=$S -f - < "$HERE/seed.sql" +migrate from_050 $S "${NEW[@]:1}" +full_checks from_050 + +D=from_052 +echo "== running 0004 again creates nothing new" +before=$(val $D "SELECT (SELECT count(*) FROM $S.gitlab_identities),(SELECT count(*) FROM git_identities),(SELECT count(*) FROM git_webhook_endpoints),(SELECT count(*) FROM git_repositories),(SELECT count(*) FROM git_change_requests),(SELECT count(*) FROM git_task_links),(SELECT count(*) FROM git_task_branches)") +migrate $D $S "$M/0004_migrate_community_plugin.sql" +check $D "same row counts" "SELECT (SELECT count(*) FROM $S.gitlab_identities),(SELECT count(*) FROM git_identities),(SELECT count(*) FROM git_webhook_endpoints),(SELECT count(*) FROM git_repositories),(SELECT count(*) FROM git_change_requests),(SELECT count(*) FROM git_task_links),(SELECT count(*) FROM git_task_branches)" "$before" + +echo "== completeIdentity's statement (spi_dbstore.go) against the real table" +ID=$(val $D "SELECT id FROM $S.gitlab_identities ORDER BY id LIMIT 1") +UPD=$(sed -n '/^\s*UPDATE gitlab_identities SET$/,/WHERE id = \$1/p' "$PLUGIN/backend/spi_dbstore.go" | sed 's/`,$//') +complete() { # secret expires + psql -d $D <