🕷️ FieldScrape

Multi-threaded web scraper with intelligent contact extraction and a real-time dashboard

---

🎯 Overview

FieldScrape is a multi-threaded web scraping engine with a Flask-powered real-time dashboard that crawls websites to extract contact information — emails and phone numbers — using intelligent link prioritization, concurrent workers, and a custom SQLite storage layer with built-in deduplication.

Built for lead generation, market research, and OSINT data collection — with responsible crawling practices baked in.

Scrape smarter, not harder — FieldScrape prioritizes contact pages, respects robots.txt, filters false positives, and deduplicates results automatically.

---

✨ Key Features

Feature	Description
🧵 Multi-Threaded Crawling	3 concurrent daemon workers processing jobs from a shared queue
🧠 Smart Link Prioritization	Contact, about, support, and team pages are automatically crawled first
📧 Dual Contact Extraction	Emails via mailto: links + regex; phones via tel: links + regex
🚫 False Positive Filtering	Blocklist filters junk emails (e.g., @example.com, @sentry.io, image extensions)
🤖 Polite Crawling	robots.txt compliance, 0.5s request delays, custom User-Agent
📡 Real-Time Dashboard	Frontend polls every 2.5s for live job progress with animated status badges
📥 CSV Export	Download extracted contacts per job (type, value, source URL)
💾 Persistent Storage	SQLite database with contact deduplication per job
🔄 Job Management	Create, monitor, expand, and delete scraping jobs from the UI
🌐 RESTful API	Clean JSON API for programmatic access

---

🛠️ Tech Stack

Technology	Purpose
	Core scraping engine & application logic
	Web framework & REST API
	Persistent data storage with deduplication
BeautifulSoup 4	HTML parsing & contact extraction
Requests	HTTP client with configurable delays
Threading + Queue	Concurrent crawling engine (3 daemon workers)
	Frontend interactivity, polling & dynamic UI
urllib.robotparser	Ethical crawling compliance

---

📁 Project Structure

FieldScrape/
├── app.py              # Flask application & REST API routes
├── scraper.py          # Multi-threaded crawling engine with BFS
├── db.py               # SQLite database layer (jobs + contacts)
├── templates/
│   └── index.html      # Single-page dashboard interface
├── static/
│   ├── app.js          # Polling, job management & UI interactions
│   └── style.css       # Custom CSS with earthy color palette
├── scraper.sqlite3     # SQLite database (auto-generated, gitignored)
├── requirements.txt    # Python dependencies
├── .gitignore
└── README.md

---

🚀 Getting Started

Prerequisites

• Python 3.10+
• pip (Python package manager)

Installation

1. Clone the repository

   git clone https://github.com/musamamaher-afk/FieldScrape.git
   cd FieldScrape

2. Create a virtual environment

   python -m venv venv
   source venv/bin/activate        # Linux/Mac
   venv\Scripts\activate           # Windows

3. Install dependencies

   pip install -r requirements.txt

4. Run the application

   python app.py

5. Open in browser

   http://localhost:5050
   

---

📖 Usage

1. Enter Target URL — Provide the website URL you want to scrape
2. Configure Max Pages — Set the crawl limit (1–200 pages, default 25)
3. Start Scraping — Launch the job and watch progress update in real-time
4. Expand Job Details — Click a job to see the crawl log and extracted contacts table
5. Export — Download results as CSV for further use
6. Clean Up — Delete completed or failed jobs

---

🔌 API Endpoints

Method	Endpoint	Description
GET	/	Dashboard interface
POST	/api/jobs	Start a new scraping job
GET	/api/jobs	List all jobs with email/phone counts
GET	/api/jobs/<id>	Get job details with contacts & crawl log
DELETE	/api/jobs/<id>	Delete a completed/failed job
GET	/api/jobs/<id>/csv	Download extracted contacts as CSV

---

🏗️ Architecture

┌─────────────────┐     ┌──────────────────┐     ┌────────────────────┐
│   Flask Web UI   │────▶│  Job Queue        │────▶│  Thread Pool        │
│   (Dashboard)    │     │  (queue.Queue)    │     │  (3 Daemon Workers) │
└─────────────────┘     └──────────────────┘     └────────┬───────────┘
        ▲                                                  │
        │  Polling       ┌──────────────────┐              │  BFS Crawl
        │  (2.5s)        │     SQLite        │              │  + Extraction
        └────────────────│   (db.py)         │◀─────────────┘
                         │  jobs + contacts  │
                         └──────────────────┘

Data Flow:

1. User submits a URL → JS sends POST /api/jobs
2. Flask normalizes the URL, creates a job record, enqueues it
3. A daemon worker picks up the job and crawls using BFS with priority sorting
4. Contacts are deduplicated and stored; crawl logs are appended in real-time
5. Frontend polls every 2.5s to update status badges and job details

---

🧠 How the Scraper Thinks

• Priority Queue — Links containing keywords like contact, about, support, team, connect, reach, get-in-touch are crawled first
• Dual Extraction — First checks mailto: and tel: link hrefs (most reliable), then runs regex on full page text (broader coverage)
• Smart Filtering — Rejects emails from domains like example.com, sentry.io, wixpress.com, and those matching image file extensions
• Phone Validation — Only accepts numbers with 7–15 digits to filter noise

---

⚠️ Disclaimer

This tool is intended for ethical and legal use only. Always:

• Respect website Terms of Service
• Comply with robots.txt directives
• Use reasonable crawl delays
• Only scrape publicly available data
• Adhere to applicable data protection laws (GDPR, CCPA, etc.)

The author is not responsible for misuse of this tool.

---

👤 Author

Usama Younas

• GitHub: @musamamaher-afk
• Aspiring AI Engineer | Python Developer | Automation Enthusiast

---

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.

---

_{⭐ Star this repo if you found it useful!}