Repository navigation
Migrate the crawler modules to TypeScript - #51
Merged
Merged
Conversation
… entry points Browser session, form scanner, login handling, crawler and explorer move to core/crawler/typed with named timeouts and limits and unknown-safe error handling. Output in a real browser matches the recorded results.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
There was a problem hiding this comment.
🟡 Changes recommended
The new test lacks Chromium installation in two CI workflows, blocking clean-runner builds, and registers cleanup too late to prevent server leaks on startup failure.
2 open issues
What changed in this PR
Migrates the crawler subsystem to strict TypeScript while preserving existing CommonJS entry points and adding browser-based characterization tests.
Changes:
- Moves browser sessions, form scanning, login, crawling, and exploration into typed modules.
- Extracts shared URL and error helpers and includes crawler modules in the core TypeScript build.
- Adds a local test site and golden files covering forms, navigation, login, logs, and screenshot filenames.
| File | Description |
|---|---|
tsconfig.core.json |
Includes typed crawler modules in compilation. |
test/golden/crawler/scan-standalone.json |
Records standalone input scanning. |
test/golden/crawler/scan-forms.json |
Records form fields and constraints. |
test/golden/crawler/navigation.json |
Records home-page navigation candidates. |
test/golden/crawler/navigation-about.json |
Records secondary-page navigation candidates. |
test/golden/crawler/login-log.json |
Records successful login logs. |
test/golden/crawler/gate.json |
Records agreement-gate traversal. |
test/golden/crawler/explore.json |
Records exploration results and skipped controls. |
test/golden/crawler/explore-shots.json |
Records exploration screenshot filenames. |
test/golden/crawler/explore-log.json |
Records exploration logs. |
test/golden/crawler/crawl.json |
Records discovered forms and page snapshots. |
test/golden/crawler/crawl-shots.json |
Records crawler screenshot filenames. |
test/golden/crawler/crawl-log.json |
Records crawl and click-probe logs. |
test/crawlerCharacterization.test.js |
Adds real-browser characterization tests. |
core/crawler/typed/urls.ts |
Extracts shared URL helpers. |
core/crawler/typed/login.ts |
Migrates login handling to TypeScript. |
core/crawler/typed/formScanner.ts |
Types form and navigation extraction. |
core/crawler/typed/explorer.ts |
Migrates exploration and state capture. |
core/crawler/typed/errors.ts |
Adds a shared error-log formatter. |
core/crawler/typed/crawler.ts |
Migrates crawling, probing, and snapshots. |
core/crawler/typed/browser.ts |
Migrates browser lifecycle and options. |
core/crawler/login.js |
Becomes a login compatibility entry point. |
core/crawler/formScanner.js |
Becomes a scanner compatibility entry point. |
core/crawler/explorer.js |
Becomes an explorer compatibility entry point. |
core/crawler/crawler.js |
Becomes a crawler compatibility entry point. |
core/crawler/browser.js |
Becomes a browser compatibility entry point. |
Review effort: Balanced
| const site = await startSite(); | ||
| const origin = `http://127.0.0.1:${site.address().port}`; | ||
| const session = new BrowserSession({ headful: false }); | ||
| const page = await session.start(); |
Comment on lines
+77
to
+81
| const session = new BrowserSession({ headful: false }); | ||
| const page = await session.start(); | ||
| const lines = []; | ||
| const log = line => lines.push(line.split(origin).join('http://SITE')); | ||
| t.after(async () => { await session.stop(); site.close(); }); |
…ven if the browser cannot start
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the browser session, form scanner, login handling, crawler and explorer to strict TypeScript. Existing entry points stay as small compatibility files, and output in a real browser matches the recorded results.