feat(providers): add DMOJ as a third problem source - #140
Merged
Merged
Conversation
DMOJ problem pages sit behind bot protection, so ingestion reads the public API v2 single-problem endpoint through an allowlisted proxy that reduces caller input to a validated problem code before building the upstream URL. Unlike Kattis, bare problem codes are rejected: DMOJ codes are opaque short strings, so accepting them unqualified would let any stray word classify as a DMOJ problem. DMOJ scores problems in points, which are not comparable to the stored Codeforces-style 800-3500 rating range, so difficulty is left unset rather than mapped through an invented curve.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds DMOJ as a third problem source, so
https://dmoj.ca/problem/ciw26p2andits siblings can be listed alongside Codeforces and Kattis.
Why the API and not the page
DMOJ's public problem pages sit behind bot protection — a plain non-browser
request gets
403, so the Kattis-style HTML scrape cannot work:Ingestion therefore reads the API v2 single-problem endpoint through
/api/dmoj, an allowlisted proxy that mirrors/api/kattis: caller input isreduced to a validated problem code before the upstream URL is built, so no
arbitrary-URL proxy is ever exposed. Timeout, abort, and upstream-status
handling are identical to the Kattis route.
Two decisions worth reviewing
Bare codes are rejected. Kattis accepts a bare
hellobecause its slugs arereadable and unambiguous. DMOJ codes are opaque short strings, so accepting
them unqualified would let any stray pasted word classify as a DMOJ problem —
and
getProblemSourceruns against every stored URL. DMOJ requires a fullyqualified
dmoj.ca/problem/<code>URL, which is also what makes the three-waysource classification safe.
Difficulty is left unset.
problems.difficultyis an integer on theCodeforces 800–3500 scale, and Kattis maps its 1–10 range onto it. DMOJ scores
problems in points (typically 3–50), which are not comparable across problem
groups the way a rating is. Rather than invent a curve that would sort DMOJ
problems wrongly against the existing catalog, difficulty stays undefined —
the same state the Kattis fetch-failure path already produces. DMOJ rows render
as unrated, with a tooltip explaining why. Happy to add a mapping if you'd
prefer one; it is a one-line change in the ingestion module.
DMOJ's
typesfield maps directly onto the existingtagsarray, so thatmetadata is preserved.
Changes
New:
src/lib/providers/dmoj/ingestion.ts— URL validation, extraction, resolvesrc/routes/api/dmoj/+server.ts— code-validated upstream proxysrc/lib/submit/providers/dmoj.ts— submit adapterstatic/images/dmoj.svg— an original mark, not DMOJ's logoWidened from two sources to three:
ProviderId,providerOrder,providerFromUrl,Problem['source'],getProblemSource,SOURCE_FILTER_STATES(the table filter is now a 4-state cycle), andProblemTable's logo lookup and difficulty tooltip. No schema change —problemsis source-agnostic and derives source from the URL.E2E redirects the DMOJ upstream to the mock server via
PUBLIC_DMOJ_API_BASE, so no live provider is contacted.Validation
lint,lint:es,check,build, andtest:e2e(211 passed, 7 live-layerskipped) all pass. Unit coverage stays at 100% lines/branches/functions with
test:coverage; 472 unit tests pass.Note on deployment
The API answered from a residential IP. Vercel egresses from datacenter IPs,
which Cloudflare treats more aggressively — worth confirming the first DMOJ
submit works from the deployed runtime. If it is blocked, the proxy is the only
place that would need to change.