A composable Playwright automation framework built on a handler-registry pattern — retry with backoff, structured JSON logging, failure-artifact capture, proxy/stealth support, async multi-target sweeps, and OSINT-style structured data extraction, all pluggable without touching the core runner.
- Overview
- How It Works
- Sample Output
- Features
- Project Structure
- Installation
- Usage
- Configuration
- Limitations
- Roadmap
- Links
- License
Point it at a target, register the handlers you need, get back structured results — actions taken, data extracted, and (on failure) a screenshot and HTML dump for debugging.
Built as a modular framework, not a one-off script: new behavior is a new handler class, not a change to the runner. Scaling from one target to a batch sweep across hundreds is a config/CLI flag, not a rewrite.
Each handler declares two things: whether it applies to the current page (can_handle), and what to do if so (execute). The registry owns everything else — retry, pacing, logging, and failure capture — so handlers stay small and independently testable.
| Step | What happens |
|---|---|
| 1. Configure | AutomationConfig resolved from defaults → YAML → env vars → CLI flags |
| 2. Launch | Playwright browser/context launched per target, proxy + stealth patches applied if configured |
| 3. Match | Each registered handler's can_handle(page) checked against current DOM state |
| 4. Execute | Matching handlers run with retry/backoff; extraction handlers return structured fields, action handlers return a status |
| 5. Capture | On failure: screenshot + HTML dump saved to artifacts/failures/, attached to the result |
| 6. Export | All results aggregated and written to JSON and/or CSV |
| Feature | Detail |
|---|---|
| Handler registry | Conditional (can_handle-gated) or sequential execution, with per-handler retry |
| Retry with backoff | Exponential backoff + jitter; PLAYWRIGHT_TRANSIENT_EXCEPTIONS covers common transient failures out of the box |
| Structured JSON logging | One JSON object per log line — ready for ELK / Datadog / CloudWatch |
| Config-driven | AutomationConfig from YAML, BAF_* env vars, or constructed directly — no code changes to retarget a run |
| CLI | Single --url or batch --url-file; export via --json-out / --csv-out |
| Failure artifacts | Screenshot + HTML dump auto-captured on handler failure |
| Rate limiting | Configurable delay + jitter between handler executions |
| Proxy + stealth | Proxy server/list config, plus JS-level anti-detection patches |
| Async multi-target sweeps | playwright.async_api registry with bounded concurrency via semaphore |
| Result export | ResultCollector → clean JSON (full detail) or CSV (flattened, one row per handler) |
| Extraction handlers | ExtractionHandler base class for structured data pulls (OSINT-style), not just pass/fail actions |
| Real test suite | 32 unit tests against fake Page objects — no browser install needed to run pytest |
cli.py CLI entrypoint — single or batch targets
config.example.yaml example AutomationConfig
example_usage.py minimal scripted example
docs/assets/ README diagrams (SVG)
core/
config.py AutomationConfig — YAML / env / dict
retry.py retry decorator + PLAYWRIGHT_TRANSIENT_EXCEPTIONS
results.py RunResult / ResultCollector — JSON/CSV export
stealth.py anti-detection JS patches
handlers/
base.py BaseAutomationHandler (sync, action-oriented)
registry.py HandlerRegistry — retry, artifacts, pacing
async_base.py AsyncAutomationHandler
async_registry.py AsyncHandlerRegistry + run_many_targets()
extraction.py ExtractionHandler — structured data scraping
logging_utils/
json_logger.py structured JSON logging setup
tests/ 32 tests, no browser required
git clone https://github.com/Badarulnisa/browser-automation-framework.git
cd browser-automation-framework
pip install -r requirements.txt
playwright install chromiumSingle target, sync:
from handlers.base import BaseAutomationHandler
from handlers.registry import HandlerRegistry
from logging_utils.json_logger import configure_json_logging
from playwright.sync_api import sync_playwright
class CookieBannerHandler(BaseAutomationHandler):
def can_handle(self, page):
return page.locator("#cookie-consent-accept").count() > 0
def execute(self, page):
page.click("#cookie-consent-accept", timeout=5000)
return {"status": "success", "action": "dismissed_cookie_banner"}
logger = configure_json_logging()
registry = HandlerRegistry(screenshot_on_failure=True, artifact_dir="artifacts")
registry.register(CookieBannerHandler())
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://example.com")
results = registry.run_conditional(page, max_attempts_per_handler=2)
browser.close()Structured extraction (OSINT-style):
from handlers.extraction import ExtractionHandler, ExtractionRunSummary
class ProfileHandler(ExtractionHandler):
def can_handle(self, page):
return page.locator(".profile-name").count() > 0
def extract_fields(self, page):
return {
"name": self.text_or_none(page, ".profile-name"),
"bio": self.text_or_none(page, ".profile-bio"),
"socials": self.find_social_links(page),
}
registry.register(ProfileHandler())
results = registry.run_conditional(page)
records = ExtractionRunSummary.records_from_results(results)CLI, batch run with export:
python cli.py --url-file targets.txt --config config.example.yaml \
--csv-out out/results.csv --delay 1.0 --jitter 0.5Async, concurrent multi-target sweep:
import asyncio
from core.config import AutomationConfig
from handlers.async_registry import AsyncHandlerRegistry, run_many_targets
def make_registry():
registry = AsyncHandlerRegistry()
# registry.register(YourAsyncHandler())
return registry
cfg = AutomationConfig(headless=True, request_delay_seconds=0.5, request_jitter_seconds=0.5)
targets = ["https://a.example", "https://b.example", "https://c.example"]
collector = asyncio.run(run_many_targets(targets, cfg, make_registry, concurrency=5))
collector.to_json("out/sweep_results.json")Logs: console + structured JSON lines. Failure artifacts: artifacts/failures/.
| Setting | Controls |
|---|---|
browser_type |
chromium | firefox | webkit |
headless |
Headless vs headed launch |
timeout_ms |
Default navigation/action timeout |
proxy_server / proxy_list |
Proxy routing for the browser context |
stealth |
Applies anti-detection JS patches (navigator.webdriver, plugins/languages, WebGL spoof) |
max_attempts_per_handler |
Retry attempts per handler before marking failed |
request_delay_seconds / request_jitter_seconds |
Pacing between handler executions |
screenshot_on_failure / artifact_dir |
Failure-artifact capture toggle and output location |
user_agent / locale / timezone_id / headers |
Context-level fingerprint overrides |
All of the above are set via config.example.yaml, BAF_* environment variables, or CLI flags — see the README section in the repo's CLI help (python cli.py --help) for the full flag list.
| Limitation | Detail |
|---|---|
| No built-in CAPTCHA handling | Stealth patches reduce basic bot-detection signals but won't solve CAPTCHAs |
| Stealth is best-effort | JS-level patches only — not a substitute for residential proxies or full TLS fingerprint spoofing |
| Sync registry runs one target at a time | Use the async registry (handlers/async_registry.py) for concurrent multi-target sweeps |
| Extraction handlers assume Playwright locators | Non-Playwright page objects (e.g. raw HTTP scraping) aren't supported by the extraction helpers |
- Distributed run coordination (Celery/RQ) for very large target lists
- Built-in HAR capture per run for network-level forensic review
- Pluggable notification hooks (Slack/webhook) on run completion
- Handler result schema validation (pydantic) for stricter pipelines
MIT — see LICENSE.