Skip to content

resolveworks/mine

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

18 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

mine

A pi extension that adds a web_fetch tool for fetching web pages as clean markdown. It launches Chrome via Playwright on a virtual X display (Xvfb).

How it works

  1. Opens the target URL in a headed Chrome instance
  2. Waits for the page's load event (JS-heavy SPAs work fine)
  3. Cookie consent banners are dismissed by I Still Don't Care About Cookies, force-installed via Chrome enterprise policy (see below)
  4. Extracts the page content and converts it to clean markdown using Defuddle

Requirements

mine renders Chrome to a virtual X display so its window never appears on screen. Linux only. You need on your system:

  • Google Chrome — Playwright launches the system Chrome (channel: "chrome"). Install from google.com/chrome or via your package manager.
  • Xvfb — virtual framebuffer X server.
Distro Xvfb install
Arch sudo pacman -S xorg-server-xvfb
Debian/Ubuntu sudo apt install xvfb
Fedora/RHEL sudo dnf install xorg-x11-server-Xvfb

The extension manages the Xvfb process itself — start, display assignment, and shutdown are all automatic.

One-time: force-install the cookie-banner extension

Drop this JSON into /etc/opt/chrome/policies/managed/idcac.json (requires sudo):

{
  "ExtensionInstallForcelist": [
    "edibdbjcniadpccecjdfdjjppcpchdlm"
  ]
}

Chrome reads the policy on launch and pulls I Still Don't Care About Cookies from the Chrome Web Store. The policy is system-wide, so every Chrome instance on the machine will have the extension active.

Install

pi install git:github.com/resolveworks/mine

For project-local install:

pi install git:github.com/resolveworks/mine -l

Usage

Once installed, the web_fetch tool is available to pi:

> Fetch the content of https://example.com

License

MIT

About

O(n) archive lookup

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors