Internships: vanshb03's short links become the posting's own link

docs/ARCHITECTURE.md

@@ -50,7 +50,8 @@ Job pool
   fields.py       per-field starting preferences (the wizard's "What do you study?").
   intern.py       Summer internships (Simplify's and vanshb03's lists, one job on both kept once): the second pool, built
                   like jobs.py (refresh, triage, fetch, score, category, build_pool); approve() -> manage.add("other");
-                  update(), review_queue(), stats(). Data in data/intern/.
+                  update(), review_queue(), stats(). Short links -> the posting's own (real_links, real_url; used by
+                  assist.other_target). Data in data/intern/.
   postings.py     reads one posting from a company's own job site, no login; public addresses only (allowed_host).
   daily.py        what the homelab timers run. main() -> scan() (via step()), reminders(), record_hand_applications(),
                   applications(), internships(); run_claude(), notify() / discord(), weekly().
@@ -209,7 +210,7 @@ use headless Firefox on local pages or demo mode. Run it before pushing; CI runs
   test_deploy    deploy.py on a throwaway git repo
   test_doctor_remote   doctor.check_remote_assist          test_health   health.py's checks
   test_insights  Insights numbers          test_selftest  `nuauto selftest`          test_setup  the wizard (onboard.py)
-  test_intern    internship pool, postings.py (links, parsing, the network guard, fetch tries), Review queue
+  test_intern    internship pool, postings.py (links, parsing, the network guard, fetch tries), short links, Review queue
   test_assist    the agent's guard, answer bank, slots, passwords, uploads, by-hand rows
   test_accounts  accounts.py: sites, Bitwarden export / import of your own logins
   test_mail      mail.py: Gmail's format, which mail counts, the `email` command

docs/ASSIST.md

@@ -123,7 +123,9 @@ ACCOUNTS (added 2026-10-10, my call; accounts.py)
 - Never on Google / Microsoft / Apple / LinkedIn / GitHub / Northeastern sign-ins, nor the other providers in
   accounts.NEVER (accounts.never). "Sign in with Google" buttons are mine.
 - A site is its host; on a host companies share (Workday's myworkdaysite.com, Greenhouse / Lever / Ashby / Workable)
-  host/<company> (accounts.site; assist.site_key).
+  host/<company> (accounts.site; assist.site_key). An Other jobs row with an internship list's short link
+  (go.vansh.nyc/...) is its posting's own link once the internship update knows it (assist.other_target ->
+  intern.real_url), so its site is the job site, not the shortener (2026-10-26).
 - `nuauto accounts export` (or Settings) writes Bitwarden's import file (one login per site, matched on its host; the
   jobs in its notes); `--all` every login again. `nuauto accounts export --bitwarden` (2026-10-11) puts the logins
   NUauto made and Bitwarden doesn't have yet straight into my vault with `bw` (I unlock it: BW_SESSION; NUauto never

docs/PIPELINE.md

@@ -118,6 +118,14 @@ year's lists are new repos: changing the term moves the addresses with it); off
 - Lists: a listing counts when it is for the term (Simplify's "terms"; vanshb03's has only a season, "Summer", on a
   list whose address says the term's year). One its list closes (or drops) stays, marked inactive; an approved one
   that closes sends a Discord message (intern.closed_notice). Ids: "s..." Simplify's, "sv..." vanshb03's.
+- Short links (2026-10-26, my request): vanshb03's list gives go.vansh.nyc/<code> links, not the posting's. Each update
+  asks where each new one leads (intern.real_links: its redirect only, the posting is not opened; 0.2 s apart; public
+  addresses only; 5 failures in a row and the rest wait for the next run) and the listing keeps the posting's own link
+  (list_url: the short one). The answers are in data/intern/links.json, so each is asked once; a later download of the
+  list keeps the real link while the short one is the same. So the job site's own data, the same job on two lists and
+  new Other jobs rows all use the real link. A sheet row added earlier with a short link is matched through links.json
+  (intern.real_url: Review leaves it out, closed_notice finds it), and the assistant opens and holds the real job site
+  (assist.other_target), so several such rows run side by side. The sheet's cells are never rewritten.
 - Dedupe (intern.mark_dupes): the same job on both lists is kept once: the listing seen first (the same day:
   Simplify's) stays, the other gets dup_of and is left out of everything. Once a duplicate, always. The same job =
   the same posting link (no www / case / tracking / trailing /apply / locale), or the same company and job number in
@@ -147,7 +155,7 @@ year's lists are new repos: changing the term moves the addresses with it); off
   trained). The model learns from co-op and internship decisions together, plus Other jobs rows (Approved, Applied...)
   as yes; one data/ratings.json. Approve = an Approved row in the Other jobs tab (Notes "<term> internship (<list>);
   match ..."), so no weekly or total cap; Undo = that row back to Proposed. Apply: `nuauto assist other <row>`.
-- Data: data/intern/ (listings, source, triage, details/, failed, scores, categories, category_sets, pool, unread,
+- Data: data/intern/ (listings, source, links, triage, details/, failed, scores, categories, category_sets, pool, unread,
   scans). Discord: new pool internships after a run; the morning message adds the last day's internship counts.
 - `nuauto intern fetch <url> [--browser]` reads one posting now (nothing saved); `intern stats` shows each step.
 

docs/REMOTE.md

@@ -112,6 +112,8 @@ THE QUEUE (remote.py; nuauto-assist.timer every 10 minutes runs `nuauto assist a
 - One session per job site (assist.site_key): the host, but each company on a board companies share by path is its own
   site (job-boards.greenhouse.io/<company>, jobs.lever.co/<company>, jobs.ashbyhq.com/<company>,
   apply.workable.com/<company>, wd1.myworkdaysite.com/<company>), so two companies' Greenhouse jobs run side by side.
+  A row with a short link (vanshb03's go.vansh.nyc/...) counts as the site it leads to (intern.real_url), not the
+  shortener.
 - Some sites refuse an automated browser (they show it a sign-in page with no way to register). End such a session
   with "End: do it on the laptop" (or "No: do it on the laptop" at "Did it go through?"): no question after, and the
   row's Notes get "DO ON THE LAPTOP..." (assist.LAPTOP_MARK, at the end): the homelab never starts it again, and the

src/nuauto/assist.py

@@ -211,10 +211,13 @@ def check_link(url):
 
 def other_target(row, url_override=None):
     """(url, None) if the agent may work on this Other jobs tab row (Approved, an https link that is not NUworks),
-    else (None, reason)."""
+    else (None, reason). A short link (an internship list's go.vansh.nyc/...) is the posting's own link once the
+    internship update knows it (intern.real_url), so the session holds that job site, not the shortener."""
+    from nuauto import intern
     if row.status != "Approved":
         return None, f"it is {row.status or 'not marked'}, not Approved"
-    return check_link(url_override or row.url)
+    url = url_override or row.url
+    return check_link(intern.real_url(url) if intern.short(url) else url)
 
 
 def parse_refs(text):

src/nuauto/intern.py

@@ -29,7 +29,7 @@
 import urllib.error
 import urllib.request
 from datetime import date, datetime, timedelta
-from urllib.parse import parse_qsl, urlencode, urlparse, urlunparse
+from urllib.parse import parse_qsl, urlencode, urljoin, urlparse, urlunparse
 
 from nuauto import config
 from nuauto import jobs
@@ -63,6 +63,8 @@
 TRIES = 3            # runs that read a posting every way (browser too) before it shows as unreadable
 STALE_DAYS = 45      # older postings are flagged (internships fill on a rolling basis)
 TRIAGE_BATCH = 250
+SHORT_HOSTS = ("go.vansh.nyc",)  # link shorteners a list gives instead of the posting's link (vanshb03's)
+SHORT_SPACING = 0.2  # seconds between two short links asked
 
 
 # ---------------------------------------------------------------- storage
@@ -168,6 +170,8 @@ def merge(listings, data, today=None, key="simplify", year=None):
         x = slim(sim, key)
         old = out.get(x["id"])
         seen.add(x["id"])
+        if old and old.get("list_url") == x["url"]:  # the same short link: where it leads is known (real_links)
+            x.update(url=old["url"], list_url=old["list_url"])
         if old is None:
             if x["active"]:  # (the first download also has every listing Simplify already closed)
                 new.append(x["id"])
@@ -183,6 +187,105 @@ def merge(listings, data, today=None, key="simplify", year=None):
     return out, new, closed
 
 
+# ---------------------------------------------------------------- short links (vanshb03's go.vansh.nyc/...)
+
+def short(url):
+    return (urlparse(str(url or "")).hostname or "").lower() in SHORT_HOSTS
+
+
+class _Stay(urllib.request.HTTPRedirectHandler):
+    def redirect_request(self, *a, **k):
+        return None  # the redirect is the answer: the posting itself is not opened
+
+
+def unshorten(url, hops=3):
+    """Where a short link leads (its redirect; one to another short link is followed), without opening the posting.
+    ValueError (or postings.Unreadable: not a public address) if it doesn't lead to a web link."""
+    from nuauto import postings
+    opener = urllib.request.build_opener(_Stay)
+    for _ in range(hops):
+        postings.check_url(url)
+        try:
+            with opener.open(urllib.request.Request(url, headers={"User-Agent": "NUauto (internship list check)"}),
+                             timeout=30) as r:
+                raise ValueError(f"no redirect (HTTP {r.status})")
+        except urllib.error.HTTPError as e:
+            e.close()
+            if e.code not in (301, 302, 303, 307, 308) or not e.headers.get("Location"):
+                raise ValueError(f"HTTP {e.code}")
+            url = urljoin(url, e.headers["Location"])
+        if not url.startswith(("https://", "http://")):
+            raise ValueError(f"not a web link: {url[:80]!r}")
+        if not short(url):
+            return clean_url(url)
+    raise ValueError("too many redirects")
+
+
+_LINKS = {}  # links.json as last read: {"mtime", "links"}
+
+
+def links():
+    """{short link: the posting's own link} (links.json; real_links fills it)."""
+    try:
+        mtime = os.path.getmtime(path("links.json"))
+    except OSError:
+        return {}
+    if _LINKS.get("mtime") != mtime:
+        _LINKS.update(mtime=mtime, links=load("links.json", {}))
+    return _LINKS["links"]
+
+
+def real_url(url):
+    """A link as the pool has it: a short link -> the posting's own link once known (links.json); any other link
+    without its tracking parameters. For sheet rows added with a short link, and the assistant's start link
+    (assist.other_target: one session per job site)."""
+    url = clean_url(url)
+    return links().get(url, url) if short(url) else url
+
+
+def real_links(listings, log=print):
+    """Each listing's short link -> the posting's own link (in place; list_url keeps the short one), so the job site's
+    own data (postings.py), the same job on two lists (mark_dupes) and one assistant session per job site all see the
+    real one. Each short link is asked once (links.json keeps the answers); one that fails is asked again next run.
+    A listing whose link changes gets it in its read posting too, and is read again if no way could. Returns how many
+    listings changed."""
+    known = dict(links())
+    todo = [] if config.DEMO else sorted({x["url"] for x in listings.values() if short(x["url"]) and x["url"] not in known})
+    if todo:
+        log(f"short links: asking where {len(todo)} lead")
+    bad, in_row = 0, 0
+    try:
+        for n, u in enumerate(todo, 1):
+            try:
+                known[u] = unshorten(u)
+                in_row = 0
+            except Exception as e:
+                bad, in_row = bad + 1, in_row + 1
+                if in_row == 5:  # the shortener is down: the rest wait for the next run
+                    log(f"short links: 5 in a row failed ({type(e).__name__}: {str(e)[:80]}); the rest wait for the next run")
+                    break
+            if n % 100 == 0:
+                save("links.json", known)
+            time.sleep(SHORT_SPACING)
+    finally:
+        if todo:
+            save("links.json", known)
+    if todo:
+        log(f"short links: {len(todo) - bad} found" + (f", {bad} not yet" if bad else ""))
+    changed, failed = 0, load("failed.json", {})
+    for i, x in listings.items():
+        if short(x["url"]) and x["url"] in known:
+            listings[i] = {**x, "url": known[x["url"]], "list_url": x["url"]}
+            changed += 1
+            d = load(f"details/{i}.json", None)
+            if d:
+                save(f"details/{i}.json", {**d, "url": known[x["url"]]})
+            failed.pop(i, None)
+    if changed:
+        save("failed.json", failed)
+    return changed
+
+
 # ---------------------------------------------------------------- the same job on two lists
 
 JOB_PARAMS = {"gh_jid", "token", "for", "jr_id", "job", "jobid", "job_id", "jid", "id", "pid", "req", "reqid",
@@ -390,7 +493,7 @@ def refresh(log=print):
         year = re.search(r"20\d\d", url)
         listings, new[key], closed[key] = merge(listings, data, key=key, year=year and year.group())
         got = True
-    if got:
+    if real_links(listings, log) or got:
         listings, _ = mark_dupes(listings)
         save("listings.json", listings)
     save("source.json", src)
@@ -700,12 +803,12 @@ def save_pool(today=None):
 
 def sheet_urls(rows):
     """The Other jobs tab's links, except Proposed rows (Review shows those first, like your Proposed NUworks rows)."""
-    return {clean_url(r.url) for r in rows or [] if r.url and r.status != "Proposed"}
+    return {real_url(r.url) for r in rows or [] if r.url and r.status != "Proposed"}
 
 
 def proposed_rows(rows):
     """{link: row number} of the Other jobs tab's Proposed rows (e.g. an approval you undid)."""
-    return {clean_url(r.url): r.number for r in rows or [] if r.url and r.status == "Proposed"}
+    return {real_url(r.url): r.number for r in rows or [] if r.url and r.status == "Proposed"}
 
 
 def ranked_pool(ratings, other_rows, nuworks_rows=()):
@@ -717,7 +820,7 @@ def ranked_pool(ratings, other_rows, nuworks_rows=()):
     labels = jobs.my_labels(every, ratings, nuworks_rows)
     by_url = {d.get("url"): i for i, d in details.items()}
     for r in other_rows or []:
-        i = by_url.get(clean_url(r.url))
+        i = by_url.get(real_url(r.url))
         if i and r.status in ("Approved", "Applied", "Accepted", "Rejected", "Needs Human", "Failed") and i not in ratings:
             labels[i] = 1
     taste = jobs.taste_for(every, labels, [r["id"] for r in pool])
@@ -877,13 +980,13 @@ def closed_notice(closed, daily):
     listings = load("listings.json", {})
     urls = {listings[i]["url"]: listings[i] for i in closed if i in listings}
     try:
-        rows = [r for r in other_rows(interactive=False) if r.status == "Approved" and clean_url(r.url) in urls]
+        rows = [r for r in other_rows(interactive=False) if r.status == "Approved" and real_url(r.url) in urls]
     except BaseException as e:  # the sheet can't be read now (e.g. the Google login expired): no message
         daily.log(f"closed internships: could not read the sheet ({type(e).__name__})")
         return
     if rows:
         daily.notify(f"{len(rows)} approved internship{'s' if len(rows) > 1 else ''} closed on "
-                     + (list_name(urls[clean_url(rows[0].url)]["id"]) if len(rows) == 1 else "their lists"),
+                     + (list_name(urls[real_url(rows[0].url)]["id"]) if len(rows) == 1 else "their lists"),
                      "\n".join(f"Other jobs row {r.number}: {r.title[:40]} | {r.company[:22]}" for r in rows)
                      + "\nCheck the posting before starting the assistant (or remove the row on the Sheet screen).")