Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .github/workflows/process.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ jobs:
process:
runs-on: ubuntu-latest
timeout-minutes: 60
outputs:
review: ${{ steps.process.outputs.review }}
infected: ${{ steps.process.outputs.infected }}
defaults:
run:
working-directory: api
Expand Down Expand Up @@ -54,7 +57,51 @@ jobs:
done

- name: Process pending versions
id: process
env:
API_URL: ${{ vars.API_URL || 'https://lilka-catalog-api.black-ghost-ukraine.workers.dev' }}
SCAN_SECRET: ${{ secrets.SCAN_SECRET }}
run: python3 scripts/process.py

# Tells the moderators that something waits for them: one open issue
# assigned to them, with a comment for every later batch, so GitHub sends an
# email and a mobile notification. Close the issue once the queue is empty;
# the next upload opens a new one.
#
# A job of its own: only it may write issues, and it never touches the
# uploaded files. The text holds counts only, nothing an uploader wrote,
# because the issue is public.
notify:
needs: process
if: ${{ !cancelled() && (needs.process.outputs.review > 0 || needs.process.outputs.infected > 0) }}
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
issues: write
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
# GitHub logins, comma-separated
NOTIFY: ${{ vars.MODERATION_NOTIFY || 'black-ghost-off' }}
REVIEW: ${{ needs.process.outputs.review }}
INFECTED: ${{ needs.process.outputs.infected }}
SITE: https://catalog.lilka.dev/?type=mod
steps:
- name: Open or update the moderation issue
run: |
set -euo pipefail
[[ "$REVIEW" =~ ^[0-9]*$ && "$INFECTED" =~ ^[0-9]*$ ]] || exit 1
[[ "$NOTIFY" =~ ^[A-Za-z0-9,-]+$ ]] || { echo "MODERATION_NOTIFY must be logins"; exit 1; }
mentions=$(echo "$NOTIFY" | tr ',' '\n' | sed '/^$/d; s/^/@/' | paste -sd' ' -)
body="${REVIEW:-0} new item(s) waiting for review"
[ "${INFECTED:-0}" -gt 0 ] && body="$body, ${INFECTED} blocked by the antivirus"
body="$body: $SITE"

gh label create moderation --color FBCA04 --description "Uploads waiting for a moderator" 2>/dev/null || true
issue=$(gh issue list --label moderation --state open --limit 1 --json number --jq '.[0].number // empty')
if [ -n "$issue" ]; then
gh issue comment "$issue" --body "$mentions $body"
else
gh issue create --title "Uploads waiting for review" --label moderation \
--assignee "$NOTIFY" --body "$mentions $body"$'\n\n'"Close this issue once the queue is empty; the next upload opens a new one."
fi
10 changes: 7 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,15 @@ The copied `site/apps/`, `site/wallpapers/`, `site/mods/` and
Moderators are GitHub numeric user ids in `MODERATORS` of `api/wrangler.toml`.
They see a **Moderation** tab: versions waiting for review with code and image
preview, checksums and the antivirus result; approve, reject with a reason, ban
an author; open reports.
an author; open reports. **All items** lists every app, wallpaper and mod with
an owner selector, **Users** everyone who has signed in, with ban and unban.

New uploads are announced in a GitHub issue labelled `moderation` (see
[api/README.md](api/README.md#lifecycle-of-an-upload)).

Apps imported from the old repository belong to the `lilka-catalog` account
and keep their original author line. A moderator can hand one over to its
author once they have signed in: `PATCH /apps/<id>` with `{"owner": "<login>"}`.
and keep their original author line. A moderator hands one over to its author
in **All items** once the author has signed in at least once.

## License

Expand Down
4 changes: 3 additions & 1 deletion TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@ has to be ready first.
Actions secrets; the repository only holds a pointer README now.
- Set the `DISPATCH_TOKEN` secret (fine-grained PAT for `lilka-dev/catalog`,
Contents: Read and write) so uploads are processed and approvals reach
devices within minutes instead of the 6-hourly and hourly schedules.
devices within minutes instead of the 6-hourly and hourly schedules. The
moderation issue comes from the processing run, so without it the
notification about a new upload can also be up to 6 hours late.
- Branch protection on `main`: require a PR, the **API / test** check and
code-owner review; optionally a reviewer for the `production` environment.
- "Publish" from the web IDE (lilka-web-ide), and running Lua apps from the
Expand Down
13 changes: 11 additions & 2 deletions api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,13 @@ mirror is a plain copy of that tree.

Imported apps (below) skip step 4: they were reviewed as repo pull requests.

When a run of the processing workflow leaves items waiting for review, its
`notify` job opens an issue labelled `moderation` assigned to the moderators
(or comments on the open one), so GitHub sends them an email and a push
notification. The issue only holds counts, because the repository is public.
Who gets it: the `MODERATION_NOTIFY` repository variable, comma-separated
logins, `black-ghost-off` by default. Close the issue once the queue is empty.

```mermaid
sequenceDiagram
autonumber
Expand Down Expand Up @@ -179,6 +186,8 @@ All bodies are JSON unless noted. Authenticated calls send
| GET | `/users/:login/apps[?versions]` | user | Apps of a user (all of them, with versions, for the user themself or a moderator) |
| GET | `/files/:slug/:version/:name` | owner, mod | File by version id, for "My apps" and moderation previews |
| GET | `/mod/queue?status=` | mod | Versions by status, default `pending_review` |
| GET | `/mod/apps` | mod | Every item with its owner, drafts and hidden ones included |
| GET | `/mod/users` | mod | Everyone who has signed in, with role and item count |
| POST | `/mod/versions/:id/approve` | mod | `{ "note"?: "..." }` |
| POST | `/mod/versions/:id/reject` | mod | `{ "note": "reason" }` |
| GET | `/mod/reports` | mod | Open reports |
Expand Down Expand Up @@ -278,6 +287,8 @@ the code controls is what a request costs:
few rows per item instead of a scan per item;
- rate limits per IP (300/min), per user for writes (60/min) and uploads
(10/min) run before any database work;
- per account: 3 new items a day, 20 in all, 5 versions waiting for review
(moderators are exempt);
- on the free plan the limits are hard caps: a flood can stop the API until
00:00 UTC, but never costs money. Do not move to a paid plan without a
custom domain in front of the Worker.
Expand All @@ -295,6 +306,4 @@ The old repository's apps were imported with `scripts/import_catalog.py`
## Not done yet

- "Publish" from the web IDE.
- A moderator UI for handing imported apps to their authors (the API has it:
`PATCH /apps/:slug` with `owner`).
- A custom domain with WAF rules in front of the Worker (needs DNS access).
20 changes: 16 additions & 4 deletions api/scripts/process.py
Original file line number Diff line number Diff line change
Expand Up @@ -202,8 +202,7 @@ def process(session, api, version_id, prefix=""):
if infected:
detail = "\n".join(f"{name}: {v['detail']}" for name, v in infected.items())
log(" INFECTED")
report(session, api, version_id, {"result": "infected", "detail": detail}, [], info)
return
return report(session, api, version_id, {"result": "infected", "detail": detail}, [], info)
if errors:
# Leave it pending: the next run retries instead of passing unscanned files
log(f" scan failed for {len(errors)} file(s), will retry")
Expand Down Expand Up @@ -258,7 +257,7 @@ def process(session, api, version_id, prefix=""):
"av_scan": verdicts[name],
})

report(session, api, version_id, {
return report(session, api, version_id, {
"result": "clean",
"detail": "ClamAV: " + (subprocess.run([CLAMSCAN or CLAMDSCAN, "--version"], capture_output=True, text=True).stdout.strip()),
"security": json.dumps(security),
Expand All @@ -283,6 +282,17 @@ def report(session, api, version_id, fields, files, info):
log(f" -> approved: {info['app']['dir']}/{info['app']['slug']} {info['version']['version']}")
else:
log(f" -> {status}")
return status


def write_outputs(statuses):
"""Counts for the notify job; numbers only, nothing an uploader wrote."""
path = os.environ.get("GITHUB_OUTPUT")
if not path:
return
with open(path, "a") as out:
out.write(f"review={statuses.count('pending_review')}\n")
out.write(f"infected={statuses.count('infected')}\n")


def main():
Expand All @@ -305,16 +315,18 @@ def main():
log(f"{len(ids)} item(s) to process")

failed = 0
statuses = []
for number, version_id in enumerate(ids, 1):
try:
process(session, api, version_id, f"[{number}/{len(ids)}]")
statuses.append(process(session, api, version_id, f"[{number}/{len(ids)}]"))
except Exception as e:
failed += 1
# Exception texts can hold URLs with file names of a private draft;
# the status code says enough to debug and names nothing
status = getattr(getattr(e, "response", None), "status_code", None)
log(f"[{number}/{len(ids)}] version id {version_id}: failed: {type(e).__name__}"
+ (f" (HTTP {status})" if status else ""))
write_outputs(statuses)
if failed:
sys.exit(f"{failed} item(s) failed")

Expand Down
52 changes: 45 additions & 7 deletions api/scripts/smoke.sh
Original file line number Diff line number Diff line change
Expand Up @@ -158,12 +158,44 @@ call 404 "import is not public before processing" "${mirror[@]}" "$API/catalog/a
# --- security regressions -----------------------------------------------
call 200 "huge page number is clamped" "${fan[@]}" "$API/apps?page=99999999999999999999"
call 409 "second report on the same app" "${fan[@]}" "${json[@]}" -X POST "$API/apps/smoke-hello/reports" -d '{"reason":"again"}'
made=0; for i in $(seq 1 25); do
s=$(curl -s -o /dev/null -w '%{http_code}' "${fan[@]}" "${json[@]}" -X POST "$API/apps" \
-d "{\"slug\":\"fan-spam-$i\",\"kind\":\"app\",\"name\":\"s\",\"short_description\":\"s\",\"sources_url\":\"https://github.com/a/b\"}")
[ "$s" = 201 ] && made=$((made+1))
done
check "apps per account capped at 20" 20 "$made"
spam() { # count of 201s out of $1 new apps by the fan, slugs from $2
local made=0 i s
for i in $(seq 1 "$1"); do
s=$(curl -s -o "$work/body" -w '%{http_code}' "${fan[@]}" "${json[@]}" -X POST "$API/apps" \
-d "{\"slug\":\"$2-$i\",\"kind\":\"app\",\"name\":\"s\",\"short_description\":\"s\",\"sources_url\":\"https://github.com/a/b\"}")
[ "$s" = 201 ] && made=$((made+1))
done
echo "$made"
}
check "new items per day capped at 3" 3 "$(spam 6 fan-spam)"
grep -q "per day" "$work/body" && msg=ok || msg=other
check "daily cap says so" ok "$msg"
# Yesterday's items no longer count, but 20 per account still holds
npx wrangler d1 execute lilka-catalog --local --command "
UPDATE apps SET created_at = datetime('now', '-2 days') WHERE owner_id = 3;
INSERT INTO apps (slug, owner_id, kind, sources_url, created_at)
SELECT 'fan-old-' || value, 3, 'app', 'https://github.com/a/b', datetime('now', '-2 days')
FROM json_each('[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]');" >/dev/null
check "a day later the fan can add again" 1 "$(spam 1 fan-next)"
check "apps per account capped at 20" 0 "$(spam 1 fan-over)"
grep -q "20 apps" "$work/body" && msg=ok || msg=other
check "account cap says so" ok "$msg"

# --- moderator lists and hand-over ---------------------------------------
call 403 "users list is for moderators" "${user[@]}" "$API/mod/users"
call 200 "moderator lists users" "${mod[@]}" "$API/mod/users"
jq -e '(.items[0].system == true) and ([.items[] | select(.login == "smoke-fan")][0].items == 20)' "$work/body" >/dev/null && ul=ok || ul=bad
check "users list: catalog account first, item counts" ok "$ul"
call 403 "apps list is for moderators" "${fan[@]}" "$API/mod/apps"
call 200 "moderator lists every item" "${mod[@]}" "$API/mod/apps"
jq -e '[.items[] | select(.slug == "fan-old-1")][0] | .owner == "smoke-fan" and .status == "draft"' "$work/body" >/dev/null && al=ok || al=bad
check "apps list has drafts with their owner" ok "$al"
call 403 "owner cannot hand an app over" "${fan[@]}" "${json[@]}" -X PATCH "$API/apps/fan-old-1" -d '{"owner":"smoke-dev"}'
call 200 "moderator hands an app to another user" "${mod[@]}" "${json[@]}" -X PATCH "$API/apps/fan-old-1" -d '{"owner":"smoke-dev"}'
check "the new owner has it" smoke-dev "$(jq -r .owner.login "$work/body")"
call 200 "moderator hands it back to the catalog account" "${mod[@]}" "${json[@]}" -X PATCH "$API/apps/fan-old-1" -d '{"owner":"lilka-catalog"}'
npx wrangler d1 execute lilka-catalog --local --command "INSERT INTO users (id, login, role) VALUES (4, 'smoke-banned', 'banned');" >/dev/null
call 409 "no hand-over to a banned user" "${mod[@]}" "${json[@]}" -X PATCH "$API/apps/fan-old-1" -d '{"owner":"smoke-banned"}'
call 200 "download name is encoded" -I "${user[@]}" "$API/files/smoke-hello/$version_id/hello.lua?download"
grep -qi "filename\*=UTF-8''hello.lua" "$work/body" && dl=ok || dl=raw
check "Content-Disposition uses filename*" ok "$dl"
Expand Down Expand Up @@ -199,11 +231,17 @@ check "latest_version_id is the approved version, not the infected one" "$versio
location=$(curl -sS -o /dev/null -w '%{redirect_url}' "$API/auth/login?return=http://localhost:8000/")
state=$(printf '%s' "$location" | sed -n 's/.*[?&]state=\([^&]*\).*/\1/p')
check "OAuth state is a signed token, not a stored row" ok "$(printf '%s' "$state" | grep -qE '^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$' && echo ok || echo "$state")"
call 400 "tampered OAuth state refused" "$API/auth/callback?code=x&state=${state%?}A"
# Flip the signature's first character: the last one also carries padding
# bits the decoder ignores, so changing it can leave the same bytes
sig=${state#*.}; first=${sig:0:1}; [ "$first" = A ] && swap=B || swap=A
call 400 "tampered OAuth state refused" "$API/auth/callback?code=x&state=${state%%.*}.$swap${sig:1}"
call 400 "made-up OAuth state refused" "$API/auth/callback?code=x&state=eyJyIjoiaHR0cHM6Ly9ldmlsLmV4YW1wbGUvIn0.AAAA"

# Last, because it uses up this IP's budget for a minute
limited=$(seq 1 330 | xargs -P 30 -I{} curl -s -o /dev/null -w '%{http_code}\n' "$API/" | grep -c '^429$' || true)
check "per-IP rate limit kicks in" yes "$([ "$limited" -gt 0 ] && echo yes || echo "no 429 in 330 requests")"
cors=$(curl -sS -o /dev/null -D - -H 'Origin: http://localhost:8000' "${user[@]}" "$API/auth/me" | tr -d '\r')
check "a 429 still has CORS headers, so the site can read it" ok \
"$(grep -q '^HTTP/[0-9.]* 429' <<<"$cors" && grep -qi '^access-control-allow-origin: http://localhost:8000' <<<"$cors" && echo ok || echo "$cors")"

echo "all $pass checks passed"
17 changes: 15 additions & 2 deletions api/src/apps.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ const PAGE_SIZE = 24;
const MAX_PENDING_PER_USER = 5;
/** Apps per account, drafts included: stops slug squatting and spam. */
const MAX_APPS_PER_USER = 20;
/** New apps, wallpapers and mods per account in any 24 hours. */
export const MAX_NEW_ITEMS_PER_DAY = 3;
/** Open reports per account; one per app is enough to reach a moderator. */
const MAX_OPEN_REPORTS_PER_USER = 10;
const MAX_PAGE = 1000;
Expand Down Expand Up @@ -322,6 +324,16 @@ apps.post('/', async (c) => {
.bind(user.id)
.first<{ n: number }>();
if ((owned?.n ?? 0) >= MAX_APPS_PER_USER) fail(429, `at most ${MAX_APPS_PER_USER} apps per account`);
// Counted from the table: it holds across Worker instances and over a
// whole day, which the per-minute rate limiters cannot do
const today = await c.env.DB.prepare(
`SELECT COUNT(*) AS n FROM apps WHERE owner_id = ? AND created_at > datetime('now', '-1 day')`,
)
.bind(user.id)
.first<{ n: number }>();
if ((today?.n ?? 0) >= MAX_NEW_ITEMS_PER_DAY) {
fail(429, `at most ${MAX_NEW_ITEMS_PER_DAY} new items per day, try again tomorrow`);
}
}

await c.env.DB.prepare(
Expand Down Expand Up @@ -410,10 +422,11 @@ apps.patch('/:slug', async (c) => {
}
if (body.owner !== undefined) {
if (typeof body.owner !== 'string') fail(400, 'owner must be a GitHub login');
const owner = await c.env.DB.prepare(`SELECT id FROM users WHERE login = ? COLLATE NOCASE`)
const owner = await c.env.DB.prepare(`SELECT id, role FROM users WHERE login = ? COLLATE NOCASE`)
.bind(body.owner)
.first<{ id: number }>();
.first<{ id: number; role: string }>();
if (!owner) fail(404, 'that user has never signed in');
if (owner.role === 'banned') fail(409, 'that user is banned');
set('owner_id', owner.id);
}
}
Expand Down
10 changes: 6 additions & 4 deletions api/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,8 @@ const writeLimit: MiddlewareHandler<AppEnv> = async (c, next) => {
app.use('/scan/*', machineSecret('SCAN_SECRET'));
app.use('/catalog/*', machineSecret('MIRROR_SECRET'));

app.use('*', async (c, next) => {
if (c.req.path.startsWith('/scan/') || c.req.path.startsWith('/catalog/')) return next();
return ipLimit(c, next);
});
// CORS comes before the rate limit, so the site can read a 429 and say what
// happened instead of seeing a network error
app.use('*', async (c, next) => {
// Only the catalog site calls the API from a browser
if (c.req.path.startsWith('/scan/') || c.req.path.startsWith('/catalog/')) return next();
Expand All @@ -64,6 +62,10 @@ app.use('*', async (c, next) => {
maxAge: 86400,
})(c, next);
});
app.use('*', async (c, next) => {
if (c.req.path.startsWith('/scan/') || c.req.path.startsWith('/catalog/')) return next();
return ipLimit(c, next);
});
app.use('*', sessionMiddleware);
app.use('*', writeLimit);

Expand Down
59 changes: 59 additions & 0 deletions api/src/moderation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,65 @@ moderation.post('/reports/:id/resolve', async (c) => {
return c.json({ ok: true });
});

/**
* Everyone who has signed in, for the Users tab and the owner selector.
* One query with the item counts; the catalog has a few hundred users at most.
*/
moderation.get('/users', async (c) => {
const { results } = await c.env.DB.prepare(
`SELECT u.id, u.login, u.name, u.avatar_url, u.role, u.created_at,
(SELECT COUNT(*) FROM apps a WHERE a.owner_id = u.id) AS items
FROM users u ORDER BY u.id = 0 DESC, u.login COLLATE NOCASE ASC LIMIT 1000`,
).all<{
id: number;
login: string;
name: string | null;
avatar_url: string | null;
role: string;
created_at: string;
items: number;
}>();
return c.json({
items: results.map((u) => ({
...u,
// The catalog account is not a person: it holds imports until handed over
system: u.id === 0,
})),
});
});

/** Every item with its owner, drafts and hidden ones included, for the Apps tab. */
moderation.get('/apps', async (c) => {
const { results } = await c.env.DB.prepare(
`SELECT a.slug, a.kind, a.status, a.author, a.name_uk, a.name_en, a.owner_id, a.created_at,
u.login AS owner_login
FROM apps a JOIN users u ON u.id = a.owner_id
ORDER BY a.kind, a.slug LIMIT 2000`,
).all<{
slug: string;
kind: string;
status: string;
author: string | null;
name_uk: string | null;
name_en: string | null;
owner_id: number;
created_at: string;
owner_login: string;
}>();
return c.json({
items: results.map((a) => ({
slug: a.slug,
kind: a.kind,
status: a.status,
name: { uk: a.name_uk, en: a.name_en },
author: displayAuthor(a),
owner: a.owner_login,
unclaimed: a.owner_id === 0,
created_at: a.created_at,
})),
});
});

/** A ban also hides every app of the user and ends their sessions. */
moderation.post('/users/:login/ban', async (c) => {
const user = await c.env.DB.prepare(`SELECT id, role FROM users WHERE login = ? COLLATE NOCASE`)
Expand Down
Loading
Loading