Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions TODO.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# TODO

## Move the API to catalog-api.lilka.org (paused)

Why: on a domain in our own Cloudflare zone a WAF rate-limiting rule can stop
floods before they reach the Worker. Blocked requests do not count against the
Worker's free 100k/day limit; on `*.workers.dev` nothing can be filtered.
`lilka.org` is already a zone in the Cloudflare account (registered through
Cloudflare Registrar).

The code is ready on branch `yboichuk/feat/api_domain` (commit `37c6772`,
tested locally: `wrangler deploy --dry-run`, 86 smoke checks). Rebase it on
`main` before merging.

### Step 1: switch to the new address (workers.dev stays on)

Do these in this order: the site switches as soon as it deploys, so the API
has to be ready first.

1. Cloudflare → My Profile → API Tokens → edit the token stored as the
`CLOUDFLARE_API_TOKEN` secret: add **Zone → Workers Routes: Edit** and
**Zone → DNS: Edit**, limited to `lilka.org`. The value stays the same.
2. GitHub OAuth App (Client ID `Ov23li85…`): add the redirect URI
`https://catalog-api.lilka.org/auth/callback`. Keep the workers.dev one.
3. Merge `yboichuk/feat/api_domain`. Check `https://catalog-api.lilka.org/`
answers, sign-in works, the mirror deploy passes.
4. Cloudflare → lilka.org → Security → Security rules → Create rule →
**Rate limiting rule**:
- If incoming requests match: Hostname equals `catalog-api.lilka.org`
- With the same characteristics: IP
- When rate exceeds: 100 requests per 10 seconds
- Then: Block, for 10 seconds

Leave Bot Fight Mode off: it would challenge the GitHub Actions jobs. The
mirror (5 req/s) and the processor (one per 0.2 s) stay under the rule.

### Step 2: close the old address

1. `api/wrangler.toml`: `workers_dev = false`, so the WAF cannot be bypassed
through `lilka-catalog-api.black-ghost-ukraine.workers.dev`.
2. `site/index.html`: remove the workers.dev host from the CSP `connect-src`.
3. GitHub OAuth App: remove the workers.dev redirect URI.
4. Update the API URL in `api/README.md` (Deploying, diagrams) if anything
still names workers.dev.

## Other open items

- Archive `lilka-dev/catalog-api` (Settings → Danger Zone) and delete its
Actions secrets; the repository only holds a pointer README now.
- Set the `DISPATCH_TOKEN` secret (fine-grained PAT for `lilka-dev/catalog`,
Contents: Read and write) so uploads are processed and approvals reach
devices within minutes instead of the 6-hourly and hourly schedules.
- Branch protection on `main`: require a PR, the **API / test** check and
code-owner review; optionally a reviewer for the `production` environment.
- "Publish" from the web IDE (lilka-web-ide), and running Lua apps from the
catalog in the browser.
139 changes: 139 additions & 0 deletions api/migrations/0004_dates.sql

Large diffs are not rendered by default.

7 changes: 5 additions & 2 deletions api/scripts/process.py
Original file line number Diff line number Diff line change
Expand Up @@ -310,8 +310,11 @@ def main():
process(session, api, version_id, f"[{number}/{len(ids)}]")
except Exception as e:
failed += 1
# Exception texts can hold URLs with file names of a private draft
log(f"[{number}/{len(ids)}] version id {version_id}: failed: {type(e).__name__}")
# Exception texts can hold URLs with file names of a private draft;
# the status code says enough to debug and names nothing
status = getattr(getattr(e, "response", None), "status_code", None)
log(f"[{number}/{len(ids)}] version id {version_id}: failed: {type(e).__name__}"
+ (f" (HTTP {status})" if status else ""))
if failed:
sys.exit(f"{failed} item(s) failed")

Expand Down
7 changes: 7 additions & 0 deletions api/scripts/smoke.sh
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,13 @@ call 200 "catalog package download" "${mirror[@]}" "$API/catalog/apps/smoke-hell
call 200 "catalog authors.json" "${mirror[@]}" "$API/catalog/authors.json"
jq -e '."@smoke-dev"[0].path == "smoke-hello"' "$work/body" >/dev/null || { echo "FAIL authors"; exit 1; }
call 404 "catalog unknown kind" "${mirror[@]}" "$API/catalog/games/index_0.json"
call 401 "listing.json needs the mirror secret" "$API/catalog/apps/listing.json"
call 200 "listing.json for the site" "${mirror[@]}" "$API/catalog/apps/listing.json"
jq -e '.items[] | select(.path == "smoke-hello") | (.added != null) and (.updated != null) and .rating.average == 4 and .rating.count == 1 and .author == "@smoke-dev"' \
"$work/body" >/dev/null && li=ok || li="$(jq -c '.items[] | select(.path == "smoke-hello")' "$work/body")"
check "listing has dates, rating and author" ok "$li"
pub=$(npx wrangler d1 execute lilka-catalog --local --json --command "SELECT published_at FROM apps WHERE slug = 'smoke-hello'" 2>/dev/null | jq -r '.[0].results[0].published_at')
check "published_at set on first approval" ok "$([ -n "$pub" ] && [ "$pub" != null ] && echo ok || echo missing)"

# wrangler dev's local explorer lists the bucket; R2 keys are <app id>-<slug>/v/<version>/<file>
r2_keys=$(curl -sS "$API/cdn-cgi/local/explorer/api/r2/buckets/lilka-catalog-files/objects?limit=1000" | jq -r '.result[].key')
Expand Down
44 changes: 43 additions & 1 deletion api/src/catalog.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ const DIRS: Record<string, AppRow['kind']> = { apps: 'app', wallpapers: 'wallpap

interface CatalogRow extends AppRow {
v_id: number;
published_at: string | null;
approved_at: string | null;
rating_sum: number | null;
keira_version: string | null;
entry_type: string | null;
entry_file: string | null;
Expand Down Expand Up @@ -57,10 +60,12 @@ async function published(env: Env, kind?: AppRow['kind'], slug?: string): Promis
const { results } = await env.DB.prepare(
`SELECT a.*, u.login AS owner_login, u.avatar_url AS owner_avatar,
v.id AS v_id, v.keira_version, v.entry_type, v.entry_file, v.files, v.files_meta, v.icon, v.icon_min,
v.screenshots, v.changelog, v.changelog_en, v.package, v.security, v.storage_prefix
v.screenshots, v.changelog, v.changelog_en, v.package, v.security, v.storage_prefix,
v.approved_at, COALESCE(r.n, 0) AS rating_count, r.total AS rating_sum
FROM apps a
JOIN users u ON u.id = a.owner_id
JOIN versions v ON v.id = a.latest_version_id
LEFT JOIN (SELECT app_id, COUNT(*) AS n, SUM(stars) AS total FROM ratings GROUP BY app_id) r ON r.app_id = a.id
WHERE ${where.join(' AND ')}
ORDER BY a.slug COLLATE BINARY`,
)
Expand Down Expand Up @@ -222,6 +227,40 @@ function authorsDoc(rows: CatalogRow[]): Record<string, unknown[]> {
return sorted;
}

/**
* listing.json: everything the website's list needs for one kind, in one file.
*
* Not part of the old build.py layout and not read by Keira, so it can carry
* what the catalog list sorts by (dates, rating) without touching the
* manifests devices parse. It also turns a list page from 1 index plus 12
* manifests into one request.
*/
function listingDoc(rows: CatalogRow[]) {
return {
items: rows.map((row) => {
const loc = localization(row);
const entry: Record<string, unknown> = {
path: row.slug,
name: pick(loc, 'name'),
short_description: pick(loc, 'short_description'),
author: displayAuthor(row),
icon: row.icon ?? '',
added: row.published_at ?? row.approved_at,
updated: row.approved_at,
rating: {
average: row.rating_count ? Math.round((row.rating_sum! / row.rating_count) * 10) / 10 : null,
count: row.rating_count,
},
};
if (Object.keys(loc).length) {
entry.languages = Object.keys(loc);
entry.localization = onlyFields(loc, ['name', 'short_description']);
}
return entry;
}),
};
}

/** index_<n>.json documents of one kind; build.py wrote none for an empty kind, this keeps an empty page 0. */
function indexDocs(rows: CatalogRow[]) {
const ids = rows.map((row) => row.slug);
Expand All @@ -235,6 +274,8 @@ function indexDocs(rows: CatalogRow[]) {

catalog.get('/authors.json', async (c) => json(c, authorsDoc(await published(c.env))));

catalog.get('/:dir/listing.json', async (c) => json(c, listingDoc(await published(c.env, dirKind(c.req.param('dir'))))));

catalog.get('/:dir/:page{index_[0-9]+\\.json}', async (c) => {
const kind = dirKind(c.req.param('dir'));
const page = Number(/^index_(\d+)\.json$/.exec(c.req.param('page'))![1]);
Expand All @@ -259,6 +300,7 @@ catalog.get('/export.json', async (c) => {
for (const dir of Object.keys(DIRS)) {
const ofKind = rows.filter((row) => kindDir(row.kind) === dir);
indexDocs(ofKind).forEach((doc, page) => (documents[`${dir}/index_${page}.json`] = text(doc)));
documents[`${dir}/listing.json`] = text(listingDoc(ofKind));
for (const row of ofKind) {
const base = `${dir}/${row.slug}`;
documents[`${base}/index.json`] = text(fullDoc(row));
Expand Down
6 changes: 4 additions & 2 deletions api/src/moderation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,8 @@ moderation.post('/versions/:id/approve', async (c) => {
// A hidden app stays hidden: approving a version is not a decision to republish it
c.env.DB.prepare(
`UPDATE apps SET status = CASE WHEN status = 'hidden' THEN 'hidden' ELSE 'published' END,
latest_version_id = ?, updated_at = datetime('now')
latest_version_id = ?, published_at = COALESCE(published_at, datetime('now')),
updated_at = datetime('now')
WHERE id = ?`,
).bind(version.id, version.app_id),
]);
Expand Down Expand Up @@ -403,7 +404,8 @@ scan.post('/versions/:id', async (c) => {
statements.push(
c.env.DB.prepare(
`UPDATE apps SET status = CASE WHEN status = 'hidden' THEN 'hidden' ELSE 'published' END,
latest_version_id = ?, updated_at = datetime('now')
latest_version_id = ?, published_at = COALESCE(published_at, datetime('now')),
updated_at = datetime('now')
WHERE id = ?`,
).bind(version.id, version.app_id),
);
Expand Down
79 changes: 79 additions & 0 deletions site/community.js
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,18 @@ const COMMUNITY_STRINGS = {
'status-rejected': 'відхилено',
'status-infected': 'знайдено загрозу',
'review-note': 'Коментар модератора',
'rating-title': 'Оцінка',
'rating-none': 'Ще ніхто не оцінив.',
// Plural forms by Intl.PluralRules: one / few / many (and other)
'rating-summary-one': '★ {average} · {count} оцінка',
'rating-summary-few': '★ {average} · {count} оцінки',
'rating-summary-many': '★ {average} · {count} оцінок',
'rating-summary-other': '★ {average} · {count} оцінки',
'rating-yours': 'Ваша оцінка:',
'rating-star': '{n} з 5',
'rating-sign-in': 'Увійдіть, щоб оцінити',
'rating-thanks': 'Дякуємо! Ваша оцінка ({stars}★) з’явиться в списку протягом години.',
'rating-own': 'Свій застосунок оцінити не можна.',
'files-removed': 'Файли цієї версії видалено після відхилення.',
'file-gone': 'Цього файлу вже немає.',
'scan-detail': 'Результат антивірусу',
Expand Down Expand Up @@ -194,6 +206,15 @@ const COMMUNITY_STRINGS = {
'status-rejected': 'rejected',
'status-infected': 'threat found',
'review-note': 'Moderator note',
'rating-title': 'Rating',
'rating-none': 'No ratings yet.',
'rating-summary-one': '★ {average} · {count} rating',
'rating-summary-other': '★ {average} · {count} ratings',
'rating-yours': 'Your rating:',
'rating-star': '{n} of 5',
'rating-sign-in': 'Sign in to rate',
'rating-thanks': 'Thanks! Your rating ({stars}★) shows in the list within an hour.',
'rating-own': 'You cannot rate your own app.',
'files-removed': 'The files of this version were deleted when it was rejected.',
'file-gone': 'This file is no longer available.',
'scan-detail': 'Antivirus result',
Expand Down Expand Up @@ -470,6 +491,64 @@ class CommunityUI {
}
}

// --- Rating in the details view ------------------------------------------

// The average comes from listing.json on Pages (no Worker request); only
// sending a rating calls the API. The list picks it up on the next mirror run.
async mountRating(modalBody, type, slug, entry) {
if (!this.enabled || !['apps', 'wallpapers', 'mods'].includes(type)) return;
const section = document.createElement('div');
section.className = 'modal-section rating-section';
const header = modalBody.querySelector('.modal-header');
if (header) header.after(section); else modalBody.prepend(section);

const rating = entry && entry.rating;
const plural = rating && rating.count ? new Intl.PluralRules(this.app.currentLang).select(rating.count) : '';
const summary = rating && rating.count ?
this.t(`rating-summary-${plural}`, {average: rating.average, count: rating.count}) :
this.t('rating-none');
section.innerHTML = `
<h3>${this.esc(this.t('rating-title'))}</h3>
<p class="rating-summary">${this.esc(summary)}</p>
<div class="rating-action"></div>
<p class="community-hint rating-message" hidden></p>`;

await this.ready;
const action = section.querySelector('.rating-action');
if (!this.user) {
action.innerHTML = `<button class="btn-sm rating-sign-in">${this.esc(this.t('rating-sign-in'))}</button>`;
action.querySelector('button').addEventListener('click', () => this.signIn());
return;
}
action.innerHTML = `<span class="community-hint">${this.esc(this.t('rating-yours'))}</span>
<span class="rating-stars" role="group">${[1, 2, 3, 4, 5].map(n =>
`<button class="rating-star" data-stars="${n}" title="${this.esc(this.t('rating-star', {n}))}"
aria-label="${this.esc(this.t('rating-star', {n}))}">★</button>`).join('')}</span>`;
const message = section.querySelector('.rating-message');
const stars = [...action.querySelectorAll('.rating-star')];
const show = (n) => stars.forEach(b => b.classList.toggle('on', Number(b.dataset.stars) <= n));
action.querySelector('.rating-stars').addEventListener('mouseleave', () => show(Number(action.dataset.chosen || 0)));
stars.forEach(button => {
button.addEventListener('mouseenter', () => show(Number(button.dataset.stars)));
button.addEventListener('click', async () => {
const n = Number(button.dataset.stars);
stars.forEach(b => b.disabled = true);
try {
await this.request('PUT', `/apps/${encodeURIComponent(slug)}/rating`, {stars: n});
action.dataset.chosen = String(n);
show(n);
message.textContent = this.t('rating-thanks', {stars: n});
if (window.umami) window.umami.track('rate', {item: slug, stars: n});
} catch (error) {
message.textContent = error.status === 403 ? this.t('rating-own') : this.t('error', {message: error.message});
} finally {
message.hidden = false;
stars.forEach(b => b.disabled = false);
}
});
});
}

// --- API -------------------------------------------------------------------

async request(method, path, body) {
Expand Down
11 changes: 11 additions & 0 deletions site/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,17 @@
<div id="error" class="error" style="display: none;"></div>

<div id="content">
<div class="list-toolbar">
<label for="sortSelect" data-i18n="sort-label">Sort by</label>
<select id="sortSelect">
<option value="newest" data-i18n="sort-newest">Newest</option>
<option value="updated" data-i18n="sort-updated">Recently updated</option>
<option value="rating" data-i18n="sort-rating">Top rated</option>
<option value="name" data-i18n="sort-name">Name</option>
<option value="author" data-i18n="sort-author">Author</option>
</select>
</div>

<div class="pagination">
<button id="prevPage" class="btn" disabled>←</button>
<div id="pageNumbers" class="page-numbers"></div>
Expand Down
Loading
Loading