Lightpanda Proxy

Transparent HTTP proxy that renders JavaScript using Lightpanda before returning HTML to text browsers like w3m.

Why This Exists

Modern websites rely heavily on JavaScript to render content. Text browsers like w3m can't execute JavaScript, making many sites unusable. This proxy bridges that gap:

1. AI Agents - Enable LLMs and coding assistants to browse JS-heavy sites
2. Text Browsers - Make w3m, lynx, and links useful for modern web
3. Automation - Scrape sites that require JavaScript without heavy browsers
4. Low Bandwidth - Get rendered content without loading a full browser

Lightpanda is a headless browser designed for AI and automation. It renders JavaScript efficiently without the overhead of Chrome or Firefox.

Installation

# Clone the repository
git clone https://github.com/KakkoiDev/lightpanda-proxy.git
cd lightpanda-proxy

# Install Lightpanda binary
./setup.sh

# Add to PATH
export PATH="$HOME/.local/bin:$PATH"

Quick Start

# Install wjs to your PATH
ln -s "$(pwd)/wrappers/wjs" ~/.local/bin/wjs

# Browse with JS rendering
wjs https://github.com

# Or just pass a domain - https:// is added automatically
wjs github.com

Usage

Option 1: wjs wrapper (easiest)

# Auto-starts proxy and opens URL
wjs github.com/anthropics/claude

# Manage proxy manually
wjs -start-proxy
wjs -stop-proxy
wjs -status

# Direct render (HTML to stdout, no w3m)
wjs -direct https://example.com

Option 2: Manual proxy

# Terminal 1: Start proxy
python3 proxy.py --port 8080 -v

# Terminal 2: Browse
w3m http://localhost:8080/https://github.com

Option 3: HTTP_PROXY environment

# Start proxy
python3 proxy.py &

# Set proxy for all HTTP requests
export http_proxy=http://127.0.0.1:8080
w3m http://example.com

Option 4: Programmatic usage (AI Agents)

import subprocess
import urllib.request

# Start proxy
proxy = subprocess.Popen(['python3', 'proxy.py', '--port', '8080'])

# Fetch rendered page
url = 'http://127.0.0.1:8080/https://github.com'
html = urllib.request.urlopen(url).read().decode()

# Process HTML...
proxy.terminate()

See examples/agent-usage.py for a complete example.

Proxy Options

--host HOST       Bind address (default: 127.0.0.1)
--port PORT       Listen port (default: 8080)
--lightpanda PATH Path to lightpanda binary
--no-cache        Disable response caching
--cache-ttl SECS  Cache TTL (default: 300)
--timeout SECS    Render timeout (default: 30)
--verbose, -v     Verbose logging
--clear-cache     Clear cache and exit

How It Works

w3m request → Proxy → Lightpanda renders JS → HTML returned → w3m displays

1. w3m sends request to proxy
2. Proxy forwards URL to Lightpanda
3. Lightpanda executes JavaScript, waits for page to render
4. Returns fully-rendered HTML
5. Proxy caches and serves to w3m

Caching

Rendered pages are cached in ~/.cache/js-proxy/ for 5 minutes by default. Clear with:

python3 proxy.py --clear-cache

Requirements

• Python 3.7+
• Lightpanda (install via ./setup.sh)
• w3m (or any text browser) - for the wrapper
• curl (fallback for simple pages)

Limitations

• HTTPS CONNECT tunneling not supported (use direct URL mode)
• Some sites may block headless browsers
• Complex SPAs may not render correctly (Lightpanda is still in beta)
• First load of uncached pages takes a few seconds

Contributing

Contributions are welcome! Ideas for improvement:

• Additional browser wrappers (lynx, links)
• Docker support
• Response filtering/transformation
• Better error handling for blocked sites

Credits

• Lightpanda - The headless browser that powers this proxy
• Inspired by the need for AI agents to browse modern websites

License

MIT License - see LICENSE for details.