diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 1c0b5d1..1244d52 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -50,7 +50,8 @@ Job pool fields.py per-field starting preferences (the wizard's "What do you study?"). intern.py Summer internships (Simplify's and vanshb03's lists, one job on both kept once): the second pool, built like jobs.py (refresh, triage, fetch, score, category, build_pool); approve() -> manage.add("other"); - update(), review_queue(), stats(). Data in data/intern/. + update(), review_queue(), stats(). Short links -> the posting's own (real_links, real_url; used by + assist.other_target). Data in data/intern/. postings.py reads one posting from a company's own job site, no login; public addresses only (allowed_host). daily.py what the homelab timers run. main() -> scan() (via step()), reminders(), record_hand_applications(), applications(), internships(); run_claude(), notify() / discord(), weekly(). @@ -209,7 +210,7 @@ use headless Firefox on local pages or demo mode. Run it before pushing; CI runs test_deploy deploy.py on a throwaway git repo test_doctor_remote doctor.check_remote_assist test_health health.py's checks test_insights Insights numbers test_selftest `nuauto selftest` test_setup the wizard (onboard.py) - test_intern internship pool, postings.py (links, parsing, the network guard, fetch tries), Review queue + test_intern internship pool, postings.py (links, parsing, the network guard, fetch tries), short links, Review queue test_assist the agent's guard, answer bank, slots, passwords, uploads, by-hand rows test_accounts accounts.py: sites, Bitwarden export / import of your own logins test_mail mail.py: Gmail's format, which mail counts, the `email` command diff --git a/docs/ASSIST.md b/docs/ASSIST.md index c724794..236b6bf 100644 --- a/docs/ASSIST.md +++ b/docs/ASSIST.md @@ -123,7 +123,9 @@ ACCOUNTS (added 2026-10-10, my call; accounts.py) - Never on Google / Microsoft / Apple / LinkedIn / GitHub / Northeastern sign-ins, nor the other providers in accounts.NEVER (accounts.never). "Sign in with Google" buttons are mine. - A site is its host; on a host companies share (Workday's myworkdaysite.com, Greenhouse / Lever / Ashby / Workable) - host/ (accounts.site; assist.site_key). + host/ (accounts.site; assist.site_key). An Other jobs row with an internship list's short link + (go.vansh.nyc/...) is its posting's own link once the internship update knows it (assist.other_target -> + intern.real_url), so its site is the job site, not the shortener (2026-10-26). - `nuauto accounts export` (or Settings) writes Bitwarden's import file (one login per site, matched on its host; the jobs in its notes); `--all` every login again. `nuauto accounts export --bitwarden` (2026-10-11) puts the logins NUauto made and Bitwarden doesn't have yet straight into my vault with `bw` (I unlock it: BW_SESSION; NUauto never diff --git a/docs/PIPELINE.md b/docs/PIPELINE.md index 1737091..142eb61 100644 --- a/docs/PIPELINE.md +++ b/docs/PIPELINE.md @@ -118,6 +118,14 @@ year's lists are new repos: changing the term moves the addresses with it); off - Lists: a listing counts when it is for the term (Simplify's "terms"; vanshb03's has only a season, "Summer", on a list whose address says the term's year). One its list closes (or drops) stays, marked inactive; an approved one that closes sends a Discord message (intern.closed_notice). Ids: "s..." Simplify's, "sv..." vanshb03's. +- Short links (2026-10-26, my request): vanshb03's list gives go.vansh.nyc/ links, not the posting's. Each update + asks where each new one leads (intern.real_links: its redirect only, the posting is not opened; 0.2 s apart; public + addresses only; 5 failures in a row and the rest wait for the next run) and the listing keeps the posting's own link + (list_url: the short one). The answers are in data/intern/links.json, so each is asked once; a later download of the + list keeps the real link while the short one is the same. So the job site's own data, the same job on two lists and + new Other jobs rows all use the real link. A sheet row added earlier with a short link is matched through links.json + (intern.real_url: Review leaves it out, closed_notice finds it), and the assistant opens and holds the real job site + (assist.other_target), so several such rows run side by side. The sheet's cells are never rewritten. - Dedupe (intern.mark_dupes): the same job on both lists is kept once: the listing seen first (the same day: Simplify's) stays, the other gets dup_of and is left out of everything. Once a duplicate, always. The same job = the same posting link (no www / case / tracking / trailing /apply / locale), or the same company and job number in @@ -147,7 +155,7 @@ year's lists are new repos: changing the term moves the addresses with it); off trained). The model learns from co-op and internship decisions together, plus Other jobs rows (Approved, Applied...) as yes; one data/ratings.json. Approve = an Approved row in the Other jobs tab (Notes " internship (); match ..."), so no weekly or total cap; Undo = that row back to Proposed. Apply: `nuauto assist other `. -- Data: data/intern/ (listings, source, triage, details/, failed, scores, categories, category_sets, pool, unread, +- Data: data/intern/ (listings, source, links, triage, details/, failed, scores, categories, category_sets, pool, unread, scans). Discord: new pool internships after a run; the morning message adds the last day's internship counts. - `nuauto intern fetch [--browser]` reads one posting now (nothing saved); `intern stats` shows each step. diff --git a/docs/REMOTE.md b/docs/REMOTE.md index 93fa884..69f5360 100644 --- a/docs/REMOTE.md +++ b/docs/REMOTE.md @@ -112,6 +112,8 @@ THE QUEUE (remote.py; nuauto-assist.timer every 10 minutes runs `nuauto assist a - One session per job site (assist.site_key): the host, but each company on a board companies share by path is its own site (job-boards.greenhouse.io/, jobs.lever.co/, jobs.ashbyhq.com/, apply.workable.com/, wd1.myworkdaysite.com/), so two companies' Greenhouse jobs run side by side. + A row with a short link (vanshb03's go.vansh.nyc/...) counts as the site it leads to (intern.real_url), not the + shortener. - Some sites refuse an automated browser (they show it a sign-in page with no way to register). End such a session with "End: do it on the laptop" (or "No: do it on the laptop" at "Did it go through?"): no question after, and the row's Notes get "DO ON THE LAPTOP..." (assist.LAPTOP_MARK, at the end): the homelab never starts it again, and the diff --git a/src/nuauto/assist.py b/src/nuauto/assist.py index ea40406..10f44e6 100644 --- a/src/nuauto/assist.py +++ b/src/nuauto/assist.py @@ -211,10 +211,13 @@ def check_link(url): def other_target(row, url_override=None): """(url, None) if the agent may work on this Other jobs tab row (Approved, an https link that is not NUworks), - else (None, reason).""" + else (None, reason). A short link (an internship list's go.vansh.nyc/...) is the posting's own link once the + internship update knows it (intern.real_url), so the session holds that job site, not the shortener.""" + from nuauto import intern if row.status != "Approved": return None, f"it is {row.status or 'not marked'}, not Approved" - return check_link(url_override or row.url) + url = url_override or row.url + return check_link(intern.real_url(url) if intern.short(url) else url) def parse_refs(text): diff --git a/src/nuauto/intern.py b/src/nuauto/intern.py index 7ee60fc..9d8c905 100644 --- a/src/nuauto/intern.py +++ b/src/nuauto/intern.py @@ -29,7 +29,7 @@ import urllib.error import urllib.request from datetime import date, datetime, timedelta -from urllib.parse import parse_qsl, urlencode, urlparse, urlunparse +from urllib.parse import parse_qsl, urlencode, urljoin, urlparse, urlunparse from nuauto import config from nuauto import jobs @@ -63,6 +63,8 @@ TRIES = 3 # runs that read a posting every way (browser too) before it shows as unreadable STALE_DAYS = 45 # older postings are flagged (internships fill on a rolling basis) TRIAGE_BATCH = 250 +SHORT_HOSTS = ("go.vansh.nyc",) # link shorteners a list gives instead of the posting's link (vanshb03's) +SHORT_SPACING = 0.2 # seconds between two short links asked # ---------------------------------------------------------------- storage @@ -168,6 +170,8 @@ def merge(listings, data, today=None, key="simplify", year=None): x = slim(sim, key) old = out.get(x["id"]) seen.add(x["id"]) + if old and old.get("list_url") == x["url"]: # the same short link: where it leads is known (real_links) + x.update(url=old["url"], list_url=old["list_url"]) if old is None: if x["active"]: # (the first download also has every listing Simplify already closed) new.append(x["id"]) @@ -183,6 +187,105 @@ def merge(listings, data, today=None, key="simplify", year=None): return out, new, closed +# ---------------------------------------------------------------- short links (vanshb03's go.vansh.nyc/...) + +def short(url): + return (urlparse(str(url or "")).hostname or "").lower() in SHORT_HOSTS + + +class _Stay(urllib.request.HTTPRedirectHandler): + def redirect_request(self, *a, **k): + return None # the redirect is the answer: the posting itself is not opened + + +def unshorten(url, hops=3): + """Where a short link leads (its redirect; one to another short link is followed), without opening the posting. + ValueError (or postings.Unreadable: not a public address) if it doesn't lead to a web link.""" + from nuauto import postings + opener = urllib.request.build_opener(_Stay) + for _ in range(hops): + postings.check_url(url) + try: + with opener.open(urllib.request.Request(url, headers={"User-Agent": "NUauto (internship list check)"}), + timeout=30) as r: + raise ValueError(f"no redirect (HTTP {r.status})") + except urllib.error.HTTPError as e: + e.close() + if e.code not in (301, 302, 303, 307, 308) or not e.headers.get("Location"): + raise ValueError(f"HTTP {e.code}") + url = urljoin(url, e.headers["Location"]) + if not url.startswith(("https://", "http://")): + raise ValueError(f"not a web link: {url[:80]!r}") + if not short(url): + return clean_url(url) + raise ValueError("too many redirects") + + +_LINKS = {} # links.json as last read: {"mtime", "links"} + + +def links(): + """{short link: the posting's own link} (links.json; real_links fills it).""" + try: + mtime = os.path.getmtime(path("links.json")) + except OSError: + return {} + if _LINKS.get("mtime") != mtime: + _LINKS.update(mtime=mtime, links=load("links.json", {})) + return _LINKS["links"] + + +def real_url(url): + """A link as the pool has it: a short link -> the posting's own link once known (links.json); any other link + without its tracking parameters. For sheet rows added with a short link, and the assistant's start link + (assist.other_target: one session per job site).""" + url = clean_url(url) + return links().get(url, url) if short(url) else url + + +def real_links(listings, log=print): + """Each listing's short link -> the posting's own link (in place; list_url keeps the short one), so the job site's + own data (postings.py), the same job on two lists (mark_dupes) and one assistant session per job site all see the + real one. Each short link is asked once (links.json keeps the answers); one that fails is asked again next run. + A listing whose link changes gets it in its read posting too, and is read again if no way could. Returns how many + listings changed.""" + known = dict(links()) + todo = [] if config.DEMO else sorted({x["url"] for x in listings.values() if short(x["url"]) and x["url"] not in known}) + if todo: + log(f"short links: asking where {len(todo)} lead") + bad, in_row = 0, 0 + try: + for n, u in enumerate(todo, 1): + try: + known[u] = unshorten(u) + in_row = 0 + except Exception as e: + bad, in_row = bad + 1, in_row + 1 + if in_row == 5: # the shortener is down: the rest wait for the next run + log(f"short links: 5 in a row failed ({type(e).__name__}: {str(e)[:80]}); the rest wait for the next run") + break + if n % 100 == 0: + save("links.json", known) + time.sleep(SHORT_SPACING) + finally: + if todo: + save("links.json", known) + if todo: + log(f"short links: {len(todo) - bad} found" + (f", {bad} not yet" if bad else "")) + changed, failed = 0, load("failed.json", {}) + for i, x in listings.items(): + if short(x["url"]) and x["url"] in known: + listings[i] = {**x, "url": known[x["url"]], "list_url": x["url"]} + changed += 1 + d = load(f"details/{i}.json", None) + if d: + save(f"details/{i}.json", {**d, "url": known[x["url"]]}) + failed.pop(i, None) + if changed: + save("failed.json", failed) + return changed + + # ---------------------------------------------------------------- the same job on two lists JOB_PARAMS = {"gh_jid", "token", "for", "jr_id", "job", "jobid", "job_id", "jid", "id", "pid", "req", "reqid", @@ -390,7 +493,7 @@ def refresh(log=print): year = re.search(r"20\d\d", url) listings, new[key], closed[key] = merge(listings, data, key=key, year=year and year.group()) got = True - if got: + if real_links(listings, log) or got: listings, _ = mark_dupes(listings) save("listings.json", listings) save("source.json", src) @@ -700,12 +803,12 @@ def save_pool(today=None): def sheet_urls(rows): """The Other jobs tab's links, except Proposed rows (Review shows those first, like your Proposed NUworks rows).""" - return {clean_url(r.url) for r in rows or [] if r.url and r.status != "Proposed"} + return {real_url(r.url) for r in rows or [] if r.url and r.status != "Proposed"} def proposed_rows(rows): """{link: row number} of the Other jobs tab's Proposed rows (e.g. an approval you undid).""" - return {clean_url(r.url): r.number for r in rows or [] if r.url and r.status == "Proposed"} + return {real_url(r.url): r.number for r in rows or [] if r.url and r.status == "Proposed"} def ranked_pool(ratings, other_rows, nuworks_rows=()): @@ -717,7 +820,7 @@ def ranked_pool(ratings, other_rows, nuworks_rows=()): labels = jobs.my_labels(every, ratings, nuworks_rows) by_url = {d.get("url"): i for i, d in details.items()} for r in other_rows or []: - i = by_url.get(clean_url(r.url)) + i = by_url.get(real_url(r.url)) if i and r.status in ("Approved", "Applied", "Accepted", "Rejected", "Needs Human", "Failed") and i not in ratings: labels[i] = 1 taste = jobs.taste_for(every, labels, [r["id"] for r in pool]) @@ -877,13 +980,13 @@ def closed_notice(closed, daily): listings = load("listings.json", {}) urls = {listings[i]["url"]: listings[i] for i in closed if i in listings} try: - rows = [r for r in other_rows(interactive=False) if r.status == "Approved" and clean_url(r.url) in urls] + rows = [r for r in other_rows(interactive=False) if r.status == "Approved" and real_url(r.url) in urls] except BaseException as e: # the sheet can't be read now (e.g. the Google login expired): no message daily.log(f"closed internships: could not read the sheet ({type(e).__name__})") return if rows: daily.notify(f"{len(rows)} approved internship{'s' if len(rows) > 1 else ''} closed on " - + (list_name(urls[clean_url(rows[0].url)]["id"]) if len(rows) == 1 else "their lists"), + + (list_name(urls[real_url(rows[0].url)]["id"]) if len(rows) == 1 else "their lists"), "\n".join(f"Other jobs row {r.number}: {r.title[:40]} | {r.company[:22]}" for r in rows) + "\nCheck the posting before starting the assistant (or remove the row on the Sheet screen).") diff --git a/tests/test_intern.py b/tests/test_intern.py index f1a9ed5..c50c6e4 100644 --- a/tests/test_intern.py +++ b/tests/test_intern.py @@ -307,12 +307,17 @@ def van(n, title="Firmware Intern", season="Summer", active=True, url=None, plac assert later[ID(21)]["active"] and later[ID(23)]["active"] is False # refresh: both lists from a local server (ETag), duplicates left out of "new", a list that fails doesn't stop the other -FILES, ASKED = {}, [] +FILES, ASKED, REDIRECTS = {}, [], {} class Lists(http.server.BaseHTTPRequestHandler): def do_GET(self): ASKED.append((self.path, self.headers.get("If-None-Match"))) + if self.path in REDIRECTS: # a short link: where it leads + self.send_response(302) + self.send_header("Location", REDIRECTS[self.path]) + self.end_headers() + return if self.path not in FILES: self.send_response(404) self.end_headers() @@ -357,6 +362,46 @@ def log_message(self, *a): FILES["/Summer2026/vansh.json"] = [van(40)] intern.refresh(logged.append) assert VID(40) not in intern.load("listings.json", {}) +# short links (vanshb03's go.vansh.nyc/...; here the list server): a listing saved with one (before NUauto read them) +# gets where it leads on the next update even when no list changed; its read posting too, and it is read again +intern.LIST_URLS["vansh"] = lbase + "/Summer2027/vansh2.json" +FILES["/Summer2027/vansh2.json"] = [van(41, title="Sensors Intern", url=lbase + "/go/a"), van(42, url=lbase + "/go/dead")] +intern.refresh(logged.append) +assert intern.load("listings.json", {})[VID(41)]["url"] == lbase + "/go/a" # not a short host yet: kept as it is +intern.save(f"details/{VID(41)}.json", {"id": VID(41), "url": lbase + "/go/a", "description": "x"}) +intern.save("failed.json", {VID(41): {"tries": 3, "last": "2026-10-01", "why": "page: no posting text"}}) +real_short, intern.SHORT_HOSTS, intern.SHORT_SPACING = intern.SHORT_HOSTS, ("127.0.0.1",), 0 +REDIRECTS.update({"/go/a": "/go/b", "/go/b": "https://jobs.example.org/sensors?utm_source=vansh"}) # one hop to another short link +ASKED.clear() +assert intern.refresh(logged.append) is None # no list changed +got = intern.load("listings.json", {}) +assert got[VID(41)]["url"] == "https://jobs.example.org/sensors" and got[VID(41)]["list_url"] == lbase + "/go/a", got[VID(41)] +assert got[VID(42)]["url"] == lbase + "/go/dead" and "list_url" not in got[VID(42)] # 404: asked again next run +assert intern.load(f"details/{VID(41)}.json", {})["url"] == "https://jobs.example.org/sensors" +assert VID(41) not in intern.load("failed.json", {}) # read again, now from the job site's own link +assert intern.load("links.json", {}) == {lbase + "/go/a": "https://jobs.example.org/sensors"} +assert [p for p, _ in ASKED if p.startswith("/go/")] == ["/go/a", "/go/b", "/go/dead"], ASKED +assert "short links: 1 found, 1 not yet" in logged, logged +ASKED.clear() +FILES["/Summer2027/vansh2.json"].append(van(43)) # the list changes: the known link stays, the failed one is asked again +intern.refresh(logged.append) +assert intern.load("listings.json", {})[VID(41)]["url"] == "https://jobs.example.org/sensors" +assert [p for p, _ in ASKED if p.startswith("/go/")] == ["/go/dead"], ASKED +# a row added with the short link is the same job (Review leaves it out); the assistant opens and holds the real site +assert intern.real_url(lbase + "/go/a?utm_source=x") == "https://jobs.example.org/sensors" +assert intern.real_url(lbase + "/go/new") == lbase + "/go/new" and intern.real_url("https://x.com/a?utm_source=v") == "https://x.com/a" +assert intern.sheet_urls([sheet.Row(3, lbase + "/go/a", "Co", "T", "Approved", "", "")]) == {"https://jobs.example.org/sensors"} +from nuauto import assist # noqa: E402 +assert assist.other_target(sheet.Row(3, lbase + "/go/a", "Co", "T", "Approved", "", "")) == ("https://jobs.example.org/sensors", None) +assert assist.site_key(assist.other_target(sheet.Row(3, lbase + "/go/a", "Co", "T", "Approved", "", ""))[0]) == "jobs.example.org" +assert "https" in assist.other_target(sheet.Row(3, lbase + "/go/new", "Co", "T", "Approved", "", ""))[1] # not known: as it is +# the shortener down: 5 failures in a row and the rest wait for the next run +ASKED.clear() +dead = {f"d{k}": {"id": f"d{k}", "url": f"{lbase}/go/dead{k}"} for k in range(8)} +assert intern.real_links(dead, logged.append) == 0 and len(ASKED) == 5 and "5 in a row failed" in logged[-2], (ASKED, logged[-2:]) +intern.SHORT_HOSTS = real_short +for name in ("links.json", "failed.json", f"details/{VID(41)}.json"): + os.remove(intern.path(name)) lsrv.shutdown() intern.LIST_URLS = real_urls for name in ("listings.json", "source.json"):