diff --git a/.github/workflows/track-releases.yml b/.github/workflows/track-releases.yml index b2cac6e..25eb3d3 100644 --- a/.github/workflows/track-releases.yml +++ b/.github/workflows/track-releases.yml @@ -1,15 +1,17 @@ name: Track plugin releases -# Scans the GitHub Releases of every listed plugin and opens one pull request -# that adds versions not yet in the index. Plugin authors only need to publish -# a release that follows docs/publishing.md. +# Scans the GitHub Releases of every listed plugin, and of every unlisted +# public `lithe-plugin-*` repository in the Lithe-IDEA organization except +# `lithe-plugin-docs`, then opens one pull request that adds the versions and +# plugins not yet in the index. Plugin authors only need to publish a release +# that follows docs/publishing.md. on: schedule: - cron: "41 */6 * * *" workflow_dispatch: inputs: plugin: - description: "Only track this plugin ID (empty = all)" + description: "Only track this plugin ID, listed or a new organization plugin (empty = all)" required: false default: "" @@ -58,6 +60,7 @@ jobs: if: steps.track.outputs.added != '0' env: SUMMARY: ${{ steps.track.outputs.summary }} + NEW_PLUGINS: ${{ steps.track.outputs.new_plugins }} run: | git config user.name "lithe-store-bot" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" @@ -69,6 +72,10 @@ jobs: git commit -m "chore(index): track new plugin releases" -m "$SUMMARY" git push --force origin "$BRANCH" body=$(printf '自动追踪到新的插件版本：\n\n%s\n\n已校验 SHA-256 与包内 plugin.json（id、version、pluginAPIVersion、signatureRequirement），README 总览已一并重新生成。维护者请按 docs/maintainers.md 的清单审查后合并。' "$(echo "$SUMMARY" | tr ',' '\n' | sed 's/^ */- /')") + if [ -n "$NEW_PLUGINS" ]; then + # The next scan rebuilds this branch from main, so metadata fixes belong in a follow-up PR. + body="$body$(printf '\n\n首次收录：%s。条目由插件仓库的简介、许可证、topics 和包内 plugin.json 自动生成，没有图标和中文简介，description 沿用仓库简介的语言。请按 docs/maintainers.md 的首次收录清单审查；合并后另提 PR 补充 icon、localizedDescriptions 和 tags，不要直接改这个分支，下一轮扫描会覆盖它。' "$NEW_PLUGINS")" + fi if gh pr view "$BRANCH" --json state -q .state 2>/dev/null | grep -q OPEN; then gh pr edit "$BRANCH" --body "$body" else diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 10f432b..a1ac50d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,7 +2,7 @@ | 我想… | 怎么做 | | --- | --- | -| 收录自己的插件 | 读 [docs/publishing.md](docs/publishing.md)，提 PR 或开“插件收录申请” issue | +| 收录自己的插件 | 读 [docs/publishing.md](docs/publishing.md)，提 PR 或开“插件收录申请” issue；Lithe-IDEA 组织的 `lithe-plugin-*` 仓库由 bot 自动收录（见 [首次收录](docs/publishing.md#3-首次收录)） | | 发布插件新版本 | 在自己仓库发布 Release 即可，bot 会自动开 PR（见 [发布新版本](docs/publishing.md#4-发布新版本)） | | 修改插件信息 | 提 PR 修改 `lithe-plugin-index.json` 中自己的条目（含 `icon`） | | 报告有问题的插件 | 开“报告插件问题” issue；安全问题请勾选“安全问题”，维护者会优先处理 | @@ -57,7 +57,7 @@ Lithe 会把仓库地址 `https://github.com/Lithe-IDEA/Lithe-store` 解析为 ```text lithe-plugin-index.json # 唯一的数据文件，Lithe 客户端读取它 scripts/validate-index.mjs # 与 Lithe Rust Core 同规则的校验 + 下载包校验 -scripts/track-releases.mjs # 追踪已收录插件的新 Release 并生成版本条目 +scripts/track-releases.mjs # 追踪已收录插件和 Lithe-IDEA 组织 lithe-plugin-* 仓库的新 Release，生成索引条目 scripts/render-readme.mjs # 根据索引生成 README 的插件总览 scripts/store.test.mjs # 校验脚本的测试（fixtures 复制自插件契约） assets/ # README 用到的图片（无图标插件的默认图标） diff --git a/docs/maintainers.md b/docs/maintainers.md index 9e852a6..7fc59a9 100644 --- a/docs/maintainers.md +++ b/docs/maintainers.md @@ -15,11 +15,18 @@ CI（`Validate index`）已经自动保证： - [ ] 首次收录：`author` 与仓库所有者对应，名称不冒充其他项目或 Lithe 官方； - [ ] 插件声明要运行的程序与描述相符（看包内 `plugin.json` 的 `languageServers` 与 `executionProviders`）； - [ ] 描述、标签、图标没有误导内容； -- [ ] 新版本 PR（bot 或作者提交）：版本号递增，没有删除旧版本。 +- [ ] 新版本 PR（bot 或作者提交）：版本号递增，没有删除旧版本； +- [ ] bot 首次收录的插件（PR 描述里的“首次收录”）：仓库简介适合作为 `description`，仓库 topics 适合作为 `tags`。 + 合并后另提 PR 补充 `icon`、`localizedDescriptions`，必要时修改 `description`、`tags`； + 不要直接改 `bot/track-releases`，下一轮扫描会从 main 重建这个分支。 ## 自动追踪新版本 `Track plugin releases` 每 6 小时运行一次，也可在 Actions 中手动运行（可只指定一个插件 ID）。 +它扫描已收录插件的 Release，以及 Lithe-IDEA 组织下尚未收录的公开 `lithe-plugin-*` 仓库 +（`lithe-plugin-docs` 不发布插件，固定排除），只收录正式 Release；新插件的条目生成规则见 +[发布社区插件的首次收录](publishing.md#3-首次收录)。手动运行时指定的插件 ID 可以是已收录插件， +也可以是组织中尚未收录的插件。 它把结果统一提交到 `bot/track-releases` 分支并开一个 PR，PR 里同时包含索引和重新生成的 README 总览（合并后不需要再等 `Sync README`）；新的扫描会覆盖这个分支并更新 PR 描述。 需要的仓库设置： diff --git a/docs/publishing.md b/docs/publishing.md index 1ca8d44..4b51b2c 100644 --- a/docs/publishing.md +++ b/docs/publishing.md @@ -77,6 +77,25 @@ shasum -a 256 zig-support-2.0.0-darwin-arm64.zip ## 3. 首次收录 +Lithe-IDEA 组织下名为 `lithe-plugin-*` 的公开仓库（`lithe-plugin-docs` 除外，它不发布插件） +不需要手工提交首次收录 PR。`Track plugin releases` 发现这类仓库还没收录、并且已经有带插件包的 +正式 Release 时，会把它连同全部正式版本一起放进 bot PR，条目按以下来源生成： + +| 字段 | 来源 | +| --- | --- | +| `id`、`name`、`author` | 最新正式 Release 的包内 `plugin.json`：`id`、`displayName`、`vendor.displayName` | +| `repository`、`homepage` | 仓库地址，`homepage` 为 `<仓库地址>#readme` | +| `description` | GitHub 仓库简介，原样使用 | +| `license` | GitHub 识别出的许可证（SPDX ID） | +| `tags` | 仓库 topics；没有 topics 时省略 | + +仓库是私有、已归档或 fork，仓库没有简介，GitHub 识别不出许可证，包内 `plugin.json` 缺字段或 ID 不符合索引规则， +或者 ID 已经被别的条目占用时，自动收录会跳过该仓库，并在工作流日志里写明原因。自动生成的条目 +没有 `icon` 和 `localizedDescriptions`，bot PR 合并后另提 PR 补充。和版本追踪一样，只发布过 +pre-release 的仓库不会被收录。 + +其他仓库按下面的步骤收录： + 1. Fork 本仓库，在 `lithe-plugin-index.json` 的 `plugins` 中**按 `id` 字母序**插入你的条目： ```json @@ -145,7 +164,8 @@ shasum -a 256 zig-support-2.0.0-darwin-arm64.zip 收录后，新版本**不需要你再提 PR**： 1. 在你的仓库递增 `plugin.json` 的 `version`，打包，发布新的正式 Release（资产命名同上）。 -2. `Track plugin releases` 工作流每 6 小时扫描一次已收录插件最近 20 个 Release， +2. `Track plugin releases` 工作流每 6 小时扫描一次已收录插件最近 20 个 Release + （Lithe-IDEA 组织尚未收录的 `lithe-plugin-*` 仓库也一起扫描，见 [首次收录](#3-首次收录)）， 把新版本写进索引（版本号、`pluginAPIVersion`、`requires` 取自包内 `plugin.json`， 能力表按键排序写入；发布日期取 Release 发布时间，SHA-256 由 CI 计算）， 然后开一个 `chore(index): 追踪插件新版本` PR。 @@ -162,7 +182,8 @@ shasum -a 256 zig-support-2.0.0-darwin-arm64.zip 稳定版用户安装；稳定版是否支持，以 Lithe 的发行说明为准。支持之前，把 GitHub Release 标为 预发布，并手工提交索引条目（LLDB Debugger `0.1.0` 就是这样收录的）。稳定版 Lithe 遇到自己 不支持的 API 等级只会显示“需要更新 Lithe”，不会安装，所以不会装出坏插件。稳定版支持之后， -把 Release 转为正式发布（或发布新版本），自动追踪就会接手后续版本。调试适配器插件的写法见 +把 Release 转为正式发布（或发布新版本），自动追踪就会接手后续版本；还没收录的 Lithe-IDEA +组织插件会在这时被自动首次收录。调试适配器插件的写法见 [编写调试适配器插件](https://github.com/Lithe-IDEA/lithe-plugin-docs/blob/main/guides/debug-adapter-plugin.md)。 自动追踪会在日志中说明归档无法收录的原因：没有符合命名的 zip、包内 `plugin.json` diff --git a/scripts/store.test.mjs b/scripts/store.test.mjs index 063c3f8..b8f8de3 100644 --- a/scripts/store.test.mjs +++ b/scripts/store.test.mjs @@ -6,8 +6,12 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { test } from "node:test"; import { END, START, categoryOf, renderOverview, renderReadme, systemsOf, truncate } from "./render-readme.mjs"; -import { addVersion, platformsForAsset, versionFromRelease } from "./track-releases.mjs"; -import { checkManifest, formatIndex, inspectArchive, validateEntry, validateIndex, verifyArchives } from "./validate-index.mjs"; +import { + addVersion, discoverPlugins, platformsForAsset, unlistedPluginRepositories, versionFromRelease, +} from "./track-releases.mjs"; +import { + checkManifest, formatIndex, inspectArchive, readPackageManifests, validateEntry, validateIndex, verifyArchives, +} from "./validate-index.mjs"; // Fixtures are copied from the plugin contract (lithe-plugin-docs contract/v1/fixtures/marketplace) // so this validator reports the same code as Lithe's Rust Core for each case. @@ -207,6 +211,179 @@ test("release tracking preserves API 1/2 releases without adding requires", { ti } }); +// Organization discovery fixtures: GitHub responses are plain objects, and each +// archive's bytes are its plugin.json text, so no test touches the network or unzip. +const REPOSITORIES_PATH = "orgs/Lithe-IDEA/repos?type=public&per_page=100&page=1"; +const ZIG_ID = "io.github.lithe-idea.zig-support"; + +function organizationRepository(name, overrides = {}) { + return { + name, + full_name: `Lithe-IDEA/${name}`, + html_url: `https://github.com/Lithe-IDEA/${name}`, + private: false, + archived: false, + fork: false, + description: "Zig language support for Lithe", + license: { spdx_id: "Apache-2.0" }, + topics: ["language", "zig"], + ...overrides, + }; +} + +function organizationRelease(repository, version, overrides = {}) { + const name = `zig-support-${version}-darwin.zip`; + return { + tag_name: `v${version}`, + draft: false, + prerelease: false, + published_at: "2026-11-01T00:00:00Z", + assets: [{ name, browser_download_url: `${repository.html_url}/releases/download/v${version}/${name}` }], + ...overrides, + }; +} + +function zigManifest(version) { + return { + schemaVersion: 2, + id: ZIG_ID, + displayName: "Zig Support", + version, + pluginAPIVersion: 2, + vendor: { displayName: "Lithe Community", signatureRequirement: "none" }, + }; +} + +function organizationFixture(responses, manifestsByURL) { + const downloads = []; + return { + downloads, + request: async (path) => { + assert.ok(Object.hasOwn(responses, path), `unexpected GitHub request: ${path}`); + return structuredClone(responses[path]); + }, + downloadArchive: async (url) => { + assert.ok(Object.hasOwn(manifestsByURL, url), `unexpected download: ${url}`); + downloads.push(url); + return Buffer.from(JSON.stringify(manifestsByURL[url])); + }, + readManifests: (bytes, pluginID) => { + const manifest = JSON.parse(bytes.toString()); + return { manifests: pluginID === undefined || manifest.id === pluginID ? [manifest] : [], unsafe: [] }; + }, + }; +} + +function zigOrganization() { + const repository = organizationRepository("lithe-plugin-zig"); + const releases = [ + organizationRelease(repository, "1.2.0", { prerelease: true }), + organizationRelease(repository, "1.1.0"), + organizationRelease(repository, "1.0.0"), + ]; + const manifestsByURL = Object.fromEntries(releases.map((release) => + [release.assets[0].browser_download_url, zigManifest(release.tag_name.slice(1))])); + const fixture = organizationFixture({ + [REPOSITORIES_PATH]: [repository, organizationRepository("lithe-plugin-docs")], + "repos/Lithe-IDEA/lithe-plugin-zig/releases?per_page=20": releases, + }, manifestsByURL); + return { releases, fixture }; +} + +test("organization discovery considers only active, unlisted lithe-plugin- repositories", () => { + const index = { schemaVersion: 1, plugins: [{ ...validEntry(), repository: "https://github.com/Lithe-IDEA/Lithe-Plugin-Listed" }] }; + const names = unlistedPluginRepositories([ + organizationRepository("lithe-plugin-zig"), + organizationRepository("lithe-plugin-docs"), + organizationRepository("lithe-plugin-listed"), + organizationRepository("lithe-plugin-old", { archived: true }), + organizationRepository("lithe-plugin-copy", { fork: true }), + organizationRepository("lithe-plugin-secret", { private: true }), + organizationRepository("Lithe-store"), + ], index).map((repository) => repository.name); + assert.deepEqual(names, ["lithe-plugin-zig"]); +}); + +test("organization discovery lists a new plugin with its stable releases", { timeout: 5_000 }, async () => { + const { releases, fixture } = zigOrganization(); + const index = { schemaVersion: 1, plugins: [validEntry(), { ...validEntry(), id: "org.example.zig-tools" }] }; + const result = await discoverPlugins(index, fixture); + + assert.deepEqual(result.skipped, []); + assert.deepEqual(result.newPlugins, [ZIG_ID]); + assert.deepEqual(result.added, [`${ZIG_ID}@1.1.0 (v1.1.0) [new plugin]`, `${ZIG_ID}@1.0.0 (v1.0.0) [new plugin]`]); + assert.deepEqual(index.plugins.map((entry) => entry.id), ["io.example.zig-support", ZIG_ID, "org.example.zig-tools"]); + const { versions, ...metadata } = index.plugins[1]; + assert.deepEqual(metadata, { + id: ZIG_ID, + name: "Zig Support", + author: "Lithe Community", + repository: "https://github.com/Lithe-IDEA/lithe-plugin-zig", + homepage: "https://github.com/Lithe-IDEA/lithe-plugin-zig#readme", + description: "Zig language support for Lithe", + tags: ["language", "zig"], + license: "Apache-2.0", + }); + assert.deepEqual(versions.map((version) => version.version), ["1.1.0", "1.0.0"]); + assert.deepEqual(validateIndex(index), []); + // The draft and its first version share one download; the pre-release is never fetched. + assert.deepEqual(fixture.downloads, [releases[1], releases[2]].map((release) => release.assets[0].browser_download_url)); +}); + +test("organization discovery keeps only the plugin named by --plugin", { timeout: 5_000 }, async () => { + const other = zigOrganization(); + const otherIndex = { schemaVersion: 1, plugins: [validEntry()] }; + const skippedRun = await discoverPlugins(otherIndex, { ...other.fixture, onlyPlugin: "io.example.other" }); + assert.deepEqual(skippedRun.added, []); + assert.equal(otherIndex.plugins.length, 1); + + const named = zigOrganization(); + const namedIndex = { schemaVersion: 1, plugins: [validEntry()] }; + const namedRun = await discoverPlugins(namedIndex, { ...named.fixture, onlyPlugin: ZIG_ID }); + assert.deepEqual(namedRun.newPlugins, [ZIG_ID]); +}); + +test("organization discovery skips repositories that cannot be listed yet", { timeout: 5_000 }, async () => { + const preview = organizationRepository("lithe-plugin-preview"); + const unlicensed = organizationRepository("lithe-plugin-unlicensed", { license: { spdx_id: "NOASSERTION" } }); + const undescribed = organizationRepository("lithe-plugin-undescribed", { description: null }); + const copy = organizationRepository("lithe-plugin-copy"); + const copyRelease = organizationRelease(copy, "1.0.0"); + const anonymous = organizationRepository("lithe-plugin-anonymous"); + const anonymousRelease = organizationRelease(anonymous, "1.0.0"); + const official = organizationRepository("lithe-plugin-official"); + const officialRelease = organizationRelease(official, "1.0.0"); + const fixture = organizationFixture({ + [REPOSITORIES_PATH]: [preview, unlicensed, undescribed, copy, anonymous, official], + "repos/Lithe-IDEA/lithe-plugin-preview/releases?per_page=20": [organizationRelease(preview, "0.1.0", { prerelease: true })], + "repos/Lithe-IDEA/lithe-plugin-unlicensed/releases?per_page=20": [organizationRelease(unlicensed, "1.0.0")], + "repos/Lithe-IDEA/lithe-plugin-undescribed/releases?per_page=20": [organizationRelease(undescribed, "1.0.0")], + "repos/Lithe-IDEA/lithe-plugin-copy/releases?per_page=20": [copyRelease], + "repos/Lithe-IDEA/lithe-plugin-anonymous/releases?per_page=20": [anonymousRelease], + "repos/Lithe-IDEA/lithe-plugin-official/releases?per_page=20": [officialRelease], + }, { + [copyRelease.assets[0].browser_download_url]: { ...zigManifest("1.0.0"), id: "io.example.zig-support" }, + [anonymousRelease.assets[0].browser_download_url]: { ...zigManifest("1.0.0"), id: "io.github.lithe-idea.anonymous", vendor: { signatureRequirement: "none" } }, + [officialRelease.assets[0].browser_download_url]: { ...zigManifest("1.0.0"), id: "dev.lithe.official" }, + }); + const index = { schemaVersion: 1, plugins: [validEntry()] }; + const result = await discoverPlugins(index, fixture); + + assert.deepEqual(result.added, []); + assert.deepEqual(result.newPlugins, []); + assert.equal(index.plugins.length, 1); + const log = result.skipped.join("\n"); + assert.match(log, /lithe-plugin-preview: no stable release/); + assert.match(log, /lithe-plugin-unlicensed: GitHub detects no license/); + assert.match(log, /lithe-plugin-undescribed: the repository has no description/); + assert.match(log, /lithe-plugin-copy: io\.example\.zig-support is already listed for https:\/\/github\.com\/example\/lithe-zig/); + // A drafted entry that the index would reject never gains a version, so it is not listed. + assert.match(log, /io\.github\.lithe-idea\.anonymous.*author/); + assert.match(log, /dev\.lithe\.official.*officialNamespace/); + // Metadata problems are found before any archive is downloaded. + assert.deepEqual(fixture.downloads, [copyRelease, anonymousRelease, officialRelease].map((release) => release.assets[0].browser_download_url)); +}); + test("archive verification rechecks changed manifest expectations for the same URL and hash", { timeout: 5_000 }, async () => { const entry = capabilityEntry(); const manifest = manifestFor(entry); @@ -322,6 +499,9 @@ test("archives must hold one matching community plugin.json", () => { /signatureRequirement/, ); assert.match(inspectArchive(build({ ...manifest, id: "io.other" }), entry, version).join(), /exactly one plugin\.json/); + // Without an ID (an unlisted plugin) every manifest that declares one is returned. + assert.deepEqual(readPackageManifests(build({ ...manifest, id: "io.other" })).manifests.map((found) => found.id), ["io.other"]); + assert.deepEqual(readPackageManifests(build({ version: "1.0.0" })).manifests, []); } finally { rmSync(workDir, { recursive: true, force: true }); } diff --git a/scripts/track-releases.mjs b/scripts/track-releases.mjs index 25e49a9..40af35d 100644 --- a/scripts/track-releases.mjs +++ b/scripts/track-releases.mjs @@ -3,6 +3,12 @@ // adds them as new versions. The scheduled `track-releases` workflow runs this // and opens a pull request; maintainers review it like any other index change. // +// Public repositories of the Lithe-IDEA organization whose names start with +// `lithe-plugin-` are tracked before they are listed: once such a repository +// publishes a stable release, the tracker adds a new entry for it, drafted from +// the repository's metadata and the package's plugin.json. `lithe-plugin-docs` +// carries the prefix but is not a plugin, so it is excluded. +// // Release convention (see docs/publishing.md): // - a published, non-prerelease GitHub Release in the plugin's `repository`; // - one `.zip` asset per platform named `--.zip` @@ -11,6 +17,7 @@ // - every archive agrees on plugin.json `version`, `pluginAPIVersion`, and `requires`. // // Usage: node scripts/track-releases.mjs [index.json] [--plugin ] [--dry-run] +// `--plugin` accepts the ID of a listed plugin or of an unlisted organization plugin. // GITHUB_TOKEN is used for API requests when set. import { createHash } from "node:crypto"; @@ -18,6 +25,11 @@ import { readFileSync, writeFileSync } from "node:fs"; import { fileURLToPath } from "node:url"; import { checkManifest, formatIndex, readArchiveBytes, requirementsKey, validateIndex } from "./validate-index.mjs"; +export const DISCOVERY_OWNER = "Lithe-IDEA"; +export const DISCOVERY_PREFIX = "lithe-plugin-"; +// Organization repositories that carry the prefix but do not publish a plugin. +export const DISCOVERY_EXCLUDED = new Set(["lithe-plugin-docs"]); + export const KNOWN_ARCHITECTURES = { darwin: ["arm64", "x64"], windows: ["x64", "arm64"], linux: ["x64", "arm64"] }; const ASSET_PLATFORM = /-(darwin|windows|linux)(?:-(arm64|x64))?\.zip$/; @@ -120,6 +132,160 @@ export async function versionFromRelease(entry, release, { return versionProblems.length > 0 ? { problems: versionProblems } : { version }; } +/** + * Adds every stable release that the entry does not list yet, newest first. + * Returns the added versions and, with their reasons, the releases it refused. + */ +export async function trackEntry(entry, releases, archiveIO = {}) { + const added = []; + const skipped = []; + const listed = new Set(entry.versions.map((version) => version.version)); + const listedURLs = new Set(entry.versions.flatMap((version) => version.downloads.map((d) => d.downloadURL))); + for (const release of releases) { + if (release.draft || release.prerelease) continue; + if ((release.assets ?? []).some((asset) => listedURLs.has(asset.browser_download_url))) continue; + const result = await versionFromRelease(entry, release, archiveIO); + if (result.problems) { + skipped.push(`${entry.id} ${release.tag_name}: ${result.problems.join("; ")}`); + continue; + } + if (listed.has(result.version.version)) { + skipped.push(`${entry.id} ${release.tag_name}: version ${result.version.version} is already listed with other archives`); + continue; + } + addVersion(entry, result.version); + listed.add(result.version.version); + added.push(`${entry.id}@${result.version.version} (${release.tag_name})`); + } + return { added, skipped }; +} + +/** Selects the organization repositories that should be listed but are not yet. */ +export function unlistedPluginRepositories(repositories, index) { + const listed = new Set(index.plugins.map((entry) => repositoryPath(entry.repository).toLowerCase())); + return repositories.filter((repository) => + repository.name.startsWith(DISCOVERY_PREFIX) + && !DISCOVERY_EXCLUDED.has(repository.name) + && !repository.private && !repository.archived && !repository.fork + && !listed.has(repository.full_name.toLowerCase())); +} + +/** + * Drafts the index entry of an unlisted repository from its newest stable + * release. plugin.json supplies the ID, name and author; the repository + * supplies the description, license and topics as tags. The draft has no + * versions: trackEntry adds them, so an entry is only listed with a valid release. + */ +export async function entryFromRepository(repository, releases, { + downloadArchive = download, readManifests = readArchiveBytes, +} = {}) { + const release = releases.find((candidate) => !candidate.draft && !candidate.prerelease + && (candidate.assets ?? []).some((asset) => platformsForAsset(asset.name))); + if (!release) return { problems: ["no stable release with a plugin archive yet"] }; + + const license = repository.license?.spdx_id; + const problems = []; + if (!repository.description?.trim()) problems.push("the repository has no description"); + if (!license || license === "NOASSERTION") problems.push("GitHub detects no license in the repository"); + if (problems.length > 0) return { problems }; + + const asset = release.assets.find((candidate) => platformsForAsset(candidate.name)); + let manifests; + try { + ({ manifests } = readManifests(await downloadArchive(asset.browser_download_url))); + } catch (error) { + return { problems: [`${release.tag_name} ${asset.name}: ${error.message}`] }; + } + if (manifests.length !== 1) { + return { problems: [`${release.tag_name} ${asset.name}: archive must contain exactly one plugin.json (found ${manifests.length})`] }; + } + const [manifest] = manifests; + // Field order follows the index schema, so the formatted diff reads like a hand-written entry. + const entry = { + id: manifest.id, + name: manifest.displayName, + author: manifest.vendor?.displayName, + repository: repository.html_url, + homepage: `${repository.html_url}#readme`, + description: repository.description.trim(), + ...(repository.topics?.length > 0 ? { tags: [...repository.topics] } : {}), + license, + versions: [], + }; + return { entry }; +} + +/** Inserts an entry and keeps `plugins` sorted by id, as the store policy requires. */ +export function insertEntry(index, entry) { + index.plugins = [...index.plugins, entry].sort((left, right) => (left.id < right.id ? -1 : left.id > right.id ? 1 : 0)); +} + +/** Reads each URL once: drafting an entry and tracking its release use the same archive. */ +function downloadOnce(downloadArchive) { + const archives = new Map(); + return (url) => { + if (!archives.has(url)) archives.set(url, downloadArchive(url)); + return archives.get(url); + }; +} + +async function organizationRepositories(owner, request) { + const repositories = []; + for (let page = 1; ; page += 1) { + const batch = await request(`orgs/${owner}/repos?type=public&per_page=100&page=${page}`); + repositories.push(...batch); + if (batch.length < 100) return repositories; + } +} + +/** + * Lists the stable releases of organization plugin repositories that the index + * does not hold yet. `onlyPlugin` keeps just the plugin with that ID. GitHub + * and archive I/O are injectable so tests run without the network. + */ +export async function discoverPlugins(index, { + onlyPlugin, request = github, downloadArchive = download, readManifests = readArchiveBytes, +} = {}) { + const added = []; + const skipped = []; + const newPlugins = []; + let repositories; + try { + repositories = await organizationRepositories(DISCOVERY_OWNER, request); + } catch (error) { + return { added, skipped: [`${DISCOVERY_OWNER} repositories: ${error.message}`], newPlugins }; + } + for (const repository of unlistedPluginRepositories(repositories, index)) { + let releases; + try { + releases = await request(`repos/${repository.full_name}/releases?per_page=20`); + } catch (error) { + skipped.push(`${repository.full_name}: ${error.message}`); + continue; + } + const archiveIO = { downloadArchive: downloadOnce(downloadArchive), readManifests }; + const draft = await entryFromRepository(repository, releases, archiveIO); + if (draft.problems) { + skipped.push(`${repository.full_name}: ${draft.problems.join("; ")}`); + continue; + } + const { entry } = draft; + if (onlyPlugin && entry.id !== onlyPlugin) continue; + const owner = index.plugins.find((listed) => listed.id === entry.id); + if (owner) { + skipped.push(`${repository.full_name}: ${entry.id} is already listed for ${owner.repository}`); + continue; + } + const result = await trackEntry(entry, releases, archiveIO); + skipped.push(...result.skipped); + if (entry.versions.length === 0) continue; + insertEntry(index, entry); + added.push(...result.added.map((line) => `${line} [new plugin]`)); + newPlugins.push(entry.id); + } + return { added, skipped, newPlugins }; +} + async function main(argv) { const pluginFlag = argv.indexOf("--plugin"); const onlyPlugin = pluginFlag >= 0 ? argv[pluginFlag + 1] : undefined; @@ -132,8 +298,6 @@ async function main(argv) { const skipped = []; for (const entry of index.plugins) { if (onlyPlugin && entry.id !== onlyPlugin) continue; - const listed = new Set(entry.versions.map((version) => version.version)); - const listedURLs = new Set(entry.versions.flatMap((version) => version.downloads.map((d) => d.downloadURL))); let releases; try { releases = await github(`repos/${repositoryPath(entry.repository)}/releases?per_page=20`); @@ -141,23 +305,13 @@ async function main(argv) { skipped.push(`${entry.id}: ${error.message}`); continue; } - for (const release of releases) { - if (release.draft || release.prerelease) continue; - if ((release.assets ?? []).some((asset) => listedURLs.has(asset.browser_download_url))) continue; - const result = await versionFromRelease(entry, release); - if (result.problems) { - skipped.push(`${entry.id} ${release.tag_name}: ${result.problems.join("; ")}`); - continue; - } - if (listed.has(result.version.version)) { - skipped.push(`${entry.id} ${release.tag_name}: version ${result.version.version} is already listed with other archives`); - continue; - } - addVersion(entry, result.version); - listed.add(result.version.version); - added.push(`${entry.id}@${result.version.version} (${release.tag_name})`); - } + const result = await trackEntry(entry, releases); + added.push(...result.added); + skipped.push(...result.skipped); } + const discovered = await discoverPlugins(index, { onlyPlugin }); + added.push(...discovered.added); + skipped.push(...discovered.skipped); const problems = validateIndex(index); for (const line of added) console.log(`+ ${line}`); @@ -169,7 +323,8 @@ async function main(argv) { if (!dryRun && added.length > 0) writeFileSync(indexPath, formatIndex(index)); if (process.env.GITHUB_OUTPUT) { const summary = added.join(", "); - writeFileSync(process.env.GITHUB_OUTPUT, `added=${added.length}\nsummary=${summary}\n`, { flag: "a" }); + const output = `added=${added.length}\nsummary=${summary}\nnew_plugins=${discovered.newPlugins.join(", ")}\n`; + writeFileSync(process.env.GITHUB_OUTPUT, output, { flag: "a" }); } return 0; } diff --git a/scripts/validate-index.mjs b/scripts/validate-index.mjs index 2b55c1e..3512d53 100644 --- a/scripts/validate-index.mjs +++ b/scripts/validate-index.mjs @@ -258,7 +258,11 @@ export function listZipEntries(archive) { .filter(Boolean); } -/** Returns every plugin.json in the archive whose id equals `pluginID`. */ +/** + * Returns every plugin.json in the archive whose id equals `pluginID`. Without + * `pluginID` (a plugin that is not listed yet) it returns every plugin.json that + * declares a string id. + */ export function readPackageManifests(archive, pluginID) { const names = listZipEntries(archive); const unsafe = names.filter((name) => name.startsWith("/") || name.includes("\\") || name.split("/").includes("..")); @@ -266,7 +270,8 @@ export function readPackageManifests(archive, pluginID) { for (const name of names.filter((name) => name === "plugin.json" || name.endsWith("/plugin.json"))) { try { const manifest = JSON.parse(execFileSync("unzip", ["-p", archive, name], { encoding: "utf8" })); - if (manifest?.id === pluginID) manifests.push(manifest); + const matches = pluginID === undefined ? typeof manifest?.id === "string" : manifest?.id === pluginID; + if (matches) manifests.push(manifest); } catch { // Not a JSON manifest; Lithe ignores it the same way. }