Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/opensak-ref
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The OpenSAK commit whose src/opensak/macro/metadata.py the checks use to
# parse macro headers. Change it deliberately when the header rules change
# The OpenSAK commit whose src/opensak/macro/metadata.py (and the
# deprecation.py it imports) the checks use to parse macro headers. Change it deliberately when the header rules change
# in OpenSAK (see MAINTAINERS.md). The first line without "#" is used.
# OpenSAK v1.22.0-beta.9
7f2bf6daafb91aee9357c9b5363655af8424a4fc
# OpenSAK v1.22.0
2f05c474b1dd7e688919c8f3070b8e2dec2b8be3
13 changes: 9 additions & 4 deletions .github/workflows/auto-merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,16 +70,21 @@ jobs:
run: |
pip install "lupa>=2.0,<3"
ref=$(grep -v '^#' .github/opensak-ref | head -n 1 | tr -d '[:space:]')
mkdir -p "$RUNNER_TEMP/tools"
curl -fsSL -o "$RUNNER_TEMP/tools/macro_metadata.py" \
"https://raw.githubusercontent.com/OpenSAK-Org/OpenSAK/${ref}/src/opensak/macro/metadata.py"
dir="$RUNNER_TEMP/opensak-src/opensak/macro"
url="https://raw.githubusercontent.com/OpenSAK-Org/OpenSAK/${ref}/src/opensak/macro"
mkdir -p "$dir"
curl -fsSL -o "$dir/metadata.py" "$url/metadata.py"
# metadata.py's imports, as PARSER_DEPS in check.yml
for f in deprecation.py; do
curl -fsSL -o "$dir/$f" "$url/$f" || echo "::notice::No $f at $ref (not needed there)"
done

- name: Check the pull request
if: steps.fetch.outputs.current == 'true'
run: |
python -I scripts/check_macro_pr.py \
--repo . --base "$(git rev-parse HEAD)" --head "$HEAD_SHA" \
--metadata "$RUNNER_TEMP/tools/macro_metadata.py" \
--metadata "$RUNNER_TEMP/opensak-src/opensak/macro/metadata.py" \
--json "$RUNNER_TEMP/check.json" || true

- name: Decide
Expand Down
30 changes: 23 additions & 7 deletions .github/workflows/check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,9 @@
# "Test checker" runs the tests with the pull request's version of the
# scripts, for pull requests that change them.
#
# The header parser is OpenSAK's src/opensak/macro/metadata.py, downloaded
# from the commit in .github/opensak-ref.
# The header parser is OpenSAK's src/opensak/macro/metadata.py (with the
# deprecation.py it imports), downloaded from the commit in
# .github/opensak-ref into an opensak/macro/ folder.

name: Check

Expand All @@ -28,7 +29,11 @@ concurrency:

env:
LUPA: "lupa>=2.0,<3"
PARSER_URL: https://raw.githubusercontent.com/OpenSAK-Org/OpenSAK/%s/src/opensak/macro/metadata.py
PARSER_URL: https://raw.githubusercontent.com/OpenSAK-Org/OpenSAK/%s/src/opensak/macro
# The modules metadata.py imports (deprecation.py from v1.22.0). An older
# ref doesn't have them and doesn't need them; if one is missing that is
# needed, loading the parser fails with ModuleNotFoundError.
PARSER_DEPS: deprecation.py

jobs:
check-macro:
Expand All @@ -55,7 +60,13 @@ jobs:
from_main() { git show "$BASE:$1" 2>/dev/null || cat "$1"; }
from_main scripts/check_macro_pr.py > "$RUNNER_TEMP/tools/check_macro_pr.py"
ref=$(from_main .github/opensak-ref | grep -v '^#' | head -n 1 | tr -d '[:space:]')
curl -fsSL -o "$RUNNER_TEMP/tools/macro_metadata.py" "$(printf "$PARSER_URL" "$ref")"
dir="$RUNNER_TEMP/opensak-src/opensak/macro"
url="$(printf "$PARSER_URL" "$ref")"
mkdir -p "$dir"
curl -fsSL -o "$dir/metadata.py" "$url/metadata.py"
for f in $PARSER_DEPS; do
curl -fsSL -o "$dir/$f" "$url/$f" || echo "::notice::No $f at $ref (not needed there)"
done

- name: Check macro
env:
Expand All @@ -64,7 +75,7 @@ jobs:
run: |
python -I "$RUNNER_TEMP/tools/check_macro_pr.py" \
--repo . --base "$BASE" --head "$HEAD" \
--metadata "$RUNNER_TEMP/tools/macro_metadata.py"
--metadata "$RUNNER_TEMP/opensak-src/opensak/macro/metadata.py"

test-checker:
name: Test checker
Expand All @@ -83,9 +94,14 @@ jobs:

- name: Get OpenSAK header parser
run: |
mkdir -p .tools
ref=$(grep -v '^#' .github/opensak-ref | head -n 1 | tr -d '[:space:]')
curl -fsSL -o .tools/macro_metadata.py "$(printf "$PARSER_URL" "$ref")"
dir=".tools/opensak/macro"
url="$(printf "$PARSER_URL" "$ref")"
mkdir -p "$dir"
curl -fsSL -o "$dir/metadata.py" "$url/metadata.py"
for f in $PARSER_DEPS; do
curl -fsSL -o "$dir/$f" "$url/$f" || echo "::notice::No $f at $ref (not needed there)"
done

- name: Run tests
run: pytest -v tests/
106 changes: 106 additions & 0 deletions .github/workflows/index.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Builds index.json — the list of macros with versions, file sizes and
# SHA-256 hashes that OpenSAK's Macro manager reads — after every push to
# main, and publishes it on the "index" branch:
#
# https://raw.githubusercontent.com/OpenSAK-Org/OpenSAK-Macros/index/index.json
#
# The index branch has no history of main: it holds only index.json (and a
# README saying where it comes from), one commit per change. Nobody edits it
# by hand; a run that gives the same file pushes nothing.
#
# Runs scripts/build_index.py from main (only pushes to main trigger it), so
# what lands in the index went through the pull request checks first. The
# script reads git objects only; no macro code is run.

name: Index

on:
push:
branches: [main]
workflow_dispatch:

permissions: {}

concurrency:
group: index
cancel-in-progress: false

env:
INDEX_BRANCH: index
PARSER_URL: https://raw.githubusercontent.com/OpenSAK-Org/OpenSAK/%s/src/opensak/macro
# The modules metadata.py imports (deprecation.py from v1.22.0). An older
# ref doesn't have them and doesn't need them; if one is missing that is
# needed, loading the parser fails with ModuleNotFoundError.
PARSER_DEPS: deprecation.py

jobs:
index:
name: Build and publish index.json
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v7
with:
ref: ${{ github.event.repository.default_branch }}
fetch-depth: 0 # the versions come from main's history
persist-credentials: false

- uses: actions/setup-python@v7
with:
python-version: "3.12"

- name: Get OpenSAK header parser
run: |
ref=$(grep -v '^#' .github/opensak-ref | head -n 1 | tr -d '[:space:]')
dir="$RUNNER_TEMP/opensak-src/opensak/macro"
url="$(printf "$PARSER_URL" "$ref")"
mkdir -p "$dir"
curl -fsSL -o "$dir/metadata.py" "$url/metadata.py"
for f in $PARSER_DEPS; do
curl -fsSL -o "$dir/$f" "$url/$f" || echo "::notice::No $f at $ref (not needed there)"
done

- name: Build index.json
run: |
mkdir -p "$RUNNER_TEMP/out"
python -I scripts/build_index.py --repo . --ref HEAD \
--metadata "$RUNNER_TEMP/opensak-src/opensak/macro/metadata.py" \
--repository "$GITHUB_REPOSITORY" \
--output "$RUNNER_TEMP/out/index.json"

- name: Publish on the index branch
env:
GH_TOKEN: ${{ github.token }}
run: |
out="$RUNNER_TEMP/out"
cat > "$out/README.md" <<'EOF'
# Macro index

`index.json` lists the macros on `main` with their versions, files,
sizes and SHA-256 hashes, for OpenSAK's Macro manager. It is built by
`.github/workflows/index.yml` after every push to `main`; don't edit
this branch by hand. The format is described in MAINTAINERS.md on
`main`.
EOF

remote="https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
work="$RUNNER_TEMP/index-branch"
if git ls-remote --exit-code --heads "$remote" "$INDEX_BRANCH" >/dev/null; then
git clone -q --depth 1 --branch "$INDEX_BRANCH" "$remote" "$work"
else
git init -q -b "$INDEX_BRANCH" "$work"
git -C "$work" remote add origin "$remote"
fi
cp "$out/index.json" "$out/README.md" "$work/"
cd "$work"
git add index.json README.md
if git diff --cached --quiet; then
echo "index.json is unchanged."
exit 0
fi
git -c user.name="github-actions[bot]" \
-c user.email="41898282+github-actions[bot]@users.noreply.github.com" \
commit -q -m "Index for ${GITHUB_SHA::7}"
git push -q origin "HEAD:$INDEX_BRANCH"
echo "Published index.json for ${GITHUB_SHA}." >> "$GITHUB_STEP_SUMMARY"
19 changes: 17 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,10 @@ Declare everything the macro can do, not just what it usually does.

List plain host names (or IP addresses): `api.example.com`, not
`https://api.example.com/path`, no ports and no wildcards.
`geocaching.com`, `coord.info` and their subdomains are not allowed — the
Geocaching.com terms of use don't permit automated access outside their API,
Geocaching HQ's sites — `geocaching.com`, `coord.info`, `groundspeak.com`,
`waymarking.com`, `wherigo.com` and related domains — and their
subdomains are not allowed: the Geocaching.com terms of use don't permit
automated access outside their API,
so OpenSAK will never let a macro contact them.

## Automatic checks
Expand Down Expand Up @@ -138,6 +140,19 @@ maintainer too.
Every change to the `.lua` file needs a higher `Version`. A change to only the
README or example files doesn't — but if in doubt, bump it.

Add a line for the new version to the `## Changes` section of the README,
starting with the version:

```markdown
## Changes

- 1.1 — lists the last 10 runs instead of 5
- 1.0 — first version
```

OpenSAK's Macro manager shows these lines when it offers the update (see
[The macro index](MAINTAINERS.md#the-macro-index)).

## License

By opening a pull request you agree that your contribution is published under
Expand Down
52 changes: 50 additions & 2 deletions MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,9 @@ the README explains why.
## Changing the header rules

The checks parse headers with OpenSAK's `src/opensak/macro/metadata.py` from
the commit in `.github/opensak-ref`. When OpenSAK changes the header rules
the commit in `.github/opensak-ref`, together with the `deprecation.py` it
imports (from v1.22.0; OpenSAK keeps that module free of other OpenSAK
imports for this repository's sake). When OpenSAK changes the header rules
(new permission words, new keys, new blocked hosts):

1. Put the new OpenSAK commit (ideally a tagged release) in
Expand All @@ -69,7 +71,51 @@ the commit in `.github/opensak-ref`. When OpenSAK changes the header rules

The checker also uses two internal helpers of the parser (`_header_lines`,
`_KEY_LINE`) to see which keys are actually written. If "Test checker"
fails after a bump, look there first.
fails after a bump, look there first. If it fails with `ModuleNotFoundError`,
`metadata.py` has gained another top-level import: add that file to
`PARSER_DEPS` in `check.yml` and `index.yml`, to the list in
`auto-merge.yml` and to the download in `tests/test_check_macro_pr.py`.

## The macro index

After every push to `main`, **Index** (`.github/workflows/index.yml`) runs
`scripts/build_index.py` and publishes `index.json` on the `index` branch:

https://raw.githubusercontent.com/OpenSAK-Org/OpenSAK-Macros/index/index.json

OpenSAK's Macro manager reads it to list, install and update macros, so it
doesn't have to walk the repository through the GitHub API. Everything in it
comes from `main`'s git history, so the same commit always gives the same
file; a push that changes nothing in it (e.g. to MAINTAINERS.md) leaves the
`index` branch alone. Run it by hand from the Actions tab ("Run workflow")
after changing the script or `.github/opensak-ref`.

Locally (with the header parser in `.tools/`, see `tests/`):

python scripts/build_index.py --metadata .tools/opensak/macro/metadata.py --ref main

Format (`schema` 1; fields are only ever added, a breaking change raises
`schema`):

| Field | |
|---|---|
| `schema`, `commit`, `generated` | format version; the `main` commit indexed and its date (UTC) |
| `macros[]` | one entry per folder under `macros/` that has `<folder>/<folder>.lua` |
| `.id`, `.folder`, `.file`, `.readme`, `.page` | folder name, paths in the repository, link to the folder on GitHub |
| `.name`, `.author`, `.version`, `.description`, `.requires_api`, `.permissions`, `.network`, `.headless` | the header, read with OpenSAK's parser (`headless` stays null until OpenSAK's parser has a Headless key; v1.22.0 doesn't) |
| `.files[]` | every file in the folder: `path`, `size`, `sha256`, and a download `url` pinned to `commit` |
| `.versions[]` | every version that was on `main`, newest first: `version`, `published` (date it first reached `main`), `commit`, `requires_api`, `permissions`, `network`, and `size`, `sha256`, `url` of the macro file |
| `.published`, `.updated` | when the first and the current version reached `main` |
| `.changelog[]` | `version` and `text` from the `## Changes` list in the README, newest first |
| `.error` | only when the header couldn't be read |
| `withdrawn[]` | macros that were on `main` and were removed: `id`, `folder`, `name`, `last_version`, `withdrawn` (date), and `versions[]` with the `sha256` of each, so OpenSAK can recognise an installed copy |

Hashes are SHA-256 of the bytes in git, which are the bytes the `url`s
serve. A macro is identified by its folder name, so renaming a folder lists
the old name as withdrawn and the new one as a new macro.

The `index` branch must stay writable by GitHub Actions: don't put it under
the "Protect main" ruleset or any other ruleset that blocks pushes.

## Repository settings this relies on

Expand All @@ -80,3 +126,5 @@ fails after a bump, look there first.
- Settings → General → Pull Requests → **Allow auto-merge** on.
- Labels `macro problem`, `auto-merge` and `needs review`.
- Discussions on.
- The `index` branch is created by the first **Index** run and left
unprotected (see [The macro index](#the-macro-index)).
5 changes: 3 additions & 2 deletions docs/github-website-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,9 @@ the GitHub website — no git, no command line. The rules themselves are in
`read-files`, `write-files`, `open`, `run-macros`). If it only reads
data, write `none`.
- **Network:** add this line only if the macro contacts web sites. Give
plain host names such as `api.example.com`. `geocaching.com` and
`coord.info` are not allowed.
plain host names such as `api.example.com`. Geocaching HQ's sites
(`geocaching.com`, `coord.info`, `waymarking.com`, `wherigo.com` and
related domains) are not allowed.

4. Write the `README.md`: what the macro does, how to use it, and what it
needs.
Expand Down
Loading
Loading