Repository navigation
Feature/core crawler typescript - #50
Closed
masahiroid wants to merge 3 commits into
Closed
masahiroid wants to merge 3 commits into
masahiroid wants to merge 3 commits into
Conversation
… entry points Browser session, form scanner, login handling, crawler and explorer move to core/crawler/typed with named timeouts and limits and unknown-safe error handling. Output in a real browser matches the recorded results.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
There was a problem hiding this comment.
🟡 Changes recommended
The test can leave its fixture server open if browser startup fails or browser shutdown throws, potentially preventing the test worker from exiting.
1 open issue
What changed in this PR
Migrates the core crawler to TypeScript while retaining CommonJS entry points for existing callers. Adds browser-based characterization tests to capture existing behavior.
Changes:
- Moves browser lifecycle, crawling, form scanning, login, and exploration into typed modules.
- Extracts shared URL and error helpers and includes crawler sources in the core TypeScript build.
- Adds golden fixtures for scan results, navigation, login, logs, and screenshot filenames.
| File | Description |
|---|---|
tsconfig.core.json |
Includes typed crawler sources. |
test/golden/crawler/scan-standalone.json |
Records standalone field scanning. |
test/golden/crawler/scan-forms.json |
Records form fields and constraints. |
test/golden/crawler/navigation.json |
Records home navigation candidates. |
test/golden/crawler/navigation-about.json |
Records secondary-page navigation. |
test/golden/crawler/login-log.json |
Records successful login logs. |
test/golden/crawler/gate.json |
Records agreement-gate crawling. |
test/golden/crawler/explore.json |
Records exploration results. |
test/golden/crawler/explore-shots.json |
Records exploration screenshot filenames. |
test/golden/crawler/explore-log.json |
Records exploration logs. |
test/golden/crawler/crawl.json |
Records discovered forms and page snapshots. |
test/golden/crawler/crawl-shots.json |
Records crawl screenshot filenames. |
test/golden/crawler/crawl-log.json |
Records crawl and click-probe logs. |
test/crawlerCharacterization.test.js |
Adds local-site browser characterization tests. |
core/crawler/typed/urls.ts |
Provides shared URL helpers. |
core/crawler/typed/login.ts |
Migrates login handling to TypeScript. |
core/crawler/typed/formScanner.ts |
Types form and navigation extraction. |
core/crawler/typed/explorer.ts |
Migrates site exploration to TypeScript. |
core/crawler/typed/errors.ts |
Provides compact error formatting. |
core/crawler/typed/crawler.ts |
Types crawl orchestration and shared retry logic. |
core/crawler/typed/browser.ts |
Types browser lifecycle and options. |
core/crawler/login.js |
Forwards to compiled login implementation. |
core/crawler/formScanner.js |
Forwards to compiled scanner implementation. |
core/crawler/explorer.js |
Forwards to compiled explorer implementation. |
core/crawler/crawler.js |
Forwards to compiled crawler implementation. |
core/crawler/browser.js |
Forwards to compiled browser implementation. |
Review effort: Balanced
Comment on lines
+78
to
+81
| const page = await session.start(); | ||
| const lines = []; | ||
| const log = line => lines.push(line.split(origin).join('http://SITE')); | ||
| t.after(async () => { await session.stop(); site.close(); }); |
…ven if the browser cannot start
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.