Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,6 @@ BUGCROWD_OTP=your_totp_secret

# API Authentication
API_KEY=your_api_key_here

# Webhook Notifications (optional, referenced from config/settings.yml as ${WEBHOOK_TOKEN})
WEBHOOK_TOKEN=your_webhook_token
2 changes: 1 addition & 1 deletion .rubocop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Metrics/ClassLength:
Max: 200

Metrics/ModuleLength:
Max: 120
Max: 140

Metrics/MethodLength:
Max: 30
Expand Down
38 changes: 38 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,43 @@
# Changelog

## Version 2.1.0 - Generic HTTP Webhook Notifications

### ✨ Features

#### Generic HTTP Webhook Notifications

Notifications are no longer Discord-only: a generic HTTP webhook notifier can be enabled alongside (or instead of) Discord.

**Changes:**
- **New `Notifiers::Webhook`**: POSTs every event as JSON to a configurable endpoint
- **New `Notifiers::Multi`**: fans notifications out to all configured notifiers, a failing one never blocks the others
- **Custom headers**: optional headers (typically for authentication) with `${VAR}` environment variable interpolation, so secrets stay in `.env`
- **Same filtering as Discord**: `events` and `new_scope_types` filters, plus a dedicated `error` event

**Configuration** (`config/settings.yml`):

```yaml
webhook:
enabled: true
url: "https://example.com/hooks/scopes"
headers:
Authorization: "Bearer ${WEBHOOK_TOKEN}"
events: ["new_program", "removed_program", "new_scope", "removed_scope", "ignored_asset", "error"]
new_scope_types: ["web"]
```

**Payload:**

```json
{
"event": "new_scope",
"timestamp": "2025-07-30T10:17:00Z",
"data": { "platform": "yeswehack", "program": "Example Program", "value": "*.example.com", "type": "web" }
}
```

---

## Version 2.0.1 - Bugcrowd Auth Fix

### 🐛 Bug Fixes
Expand Down
93 changes: 86 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
![Image](https://github.com/user-attachments/assets/8fa9dd2a-04c8-48d4-a0d7-6057c102436c)

A tool to automatically synchronize and track bug bounty program scopes from multiple platforms. Monitor new programs, scope changes, and receive Discord notifications for updates.
A tool to automatically synchronize and track bug bounty program scopes from multiple platforms. Monitor new programs, scope changes, and receive Discord or HTTP webhook notifications for updates.

[![Ruby](https://img.shields.io/badge/Ruby-3.4.7-red.svg)](https://www.ruby-lang.org/en/)
[![Docker](https://img.shields.io/badge/Docker-Supported-blue.svg)](https://www.docker.com/)
Expand All @@ -17,7 +17,7 @@ A tool to automatically synchronize and track bug bounty program scopes from mul

- 🔄 **Multi-Platform Support**: YesWeHack, HackerOne, Intigriti, Bugcrowd
- 📊 **Automatic Synchronization**: Continuously monitor programs and detect scope changes
- 🔔 **Discord Notifications**: Get notified about new programs, scope changes, and removals
- 🔔 **Notifications**: Get notified about new programs, scope changes, and removals via Discord or a generic HTTP webhook (JSON)
- 🗄️ **SQLite Database**: Persistent storage with historical change tracking
- 🌐 **REST API**: Query scopes and changes programmatically
- 🎯 **Smart Scope Processing**: Automatic validation and normalization with platform-specific rules
Expand All @@ -39,7 +39,7 @@ A tool to automatically synchronize and track bug bounty program scopes from mul

2. **Configure platform credentials** in `.env` (email, password, API tokens, TOTP secrets)

3. **Configure application settings** in `config/settings.yml` (enable/disable platforms, Discord webhooks, etc.)
3. **Configure application settings** in `config/settings.yml` (enable/disable platforms, notification webhooks, etc.)

### Using Docker (Recommended)

Expand Down Expand Up @@ -383,9 +383,10 @@ curl -H "X-API-KEY: your_api_key" "http://localhost:4567/exclusions"

</details>

## Discord Notifications
## Notifications

Configure Discord webhooks to receive real-time notifications:
Two notifiers are available and can be enabled independently (both at once if needed):
Discord webhooks and a generic HTTP webhook receiving JSON payloads.

### Notification Types

Expand All @@ -394,10 +395,11 @@ Configure Discord webhooks to receive real-time notifications:
- **new_scope**: New scope added to a program
- **removed_scope**: Scope removed from a program
- **ignored_asset**: Asset failed validation and was ignored
- **error**: Synchronization error (HTTP webhook only, Discord uses a dedicated `errors` webhook)

### Scope Type Filtering

Use `new_scope_types` to filter which scope types trigger notifications:
Use `new_scope_types` to filter which scope types trigger `new_scope` notifications:

```yaml
discord:
Expand All @@ -407,6 +409,83 @@ discord:
# Or leave empty/null to notify for all types
```

### Discord Notifications

```yaml
discord:
enabled: true
webhooks:
main:
url: "https://discord.com/api/webhooks/.../xxx"
events: ["new_program", "removed_program", "new_scope", "removed_scope", "ignored_asset"]
new_scope_types: ["web"]
errors:
url: "https://discord.com/api/webhooks/.../yyy"
```

### HTTP Webhook Notifications

Send every event as a JSON `POST` to your own endpoint:

```yaml
webhook:
enabled: true
url: "https://example.com/hooks/scopes"
headers:
Authorization: "Bearer ${WEBHOOK_TOKEN}" # ${VAR} is replaced by the environment variable
events: ["new_program", "removed_program", "new_scope", "removed_scope", "ignored_asset", "error"]
new_scope_types: ["web"]
```

Custom headers are optional and mainly meant for authentication. Any `${VAR}` inside a header
value (or inside the URL) is replaced by the matching environment variable, so secrets stay in
`.env` instead of `config/settings.yml`.

#### Payload Format

Every request shares the same envelope, `data` depends on the event:

```json
{
"event": "new_scope",
"timestamp": "2025-07-30T10:17:00Z",
"data": {
"platform": "yeswehack",
"program": "Example Program",
"value": "*.example.com",
"type": "web"
}
}
```

<details>
<summary><strong>Payload per event</strong></summary>

| Event | `data` fields |
|-------|---------------|
| `new_program` | `platform`, `program`, `slug`, `scopes_count`, `scopes` (count per type) |
| `removed_program` | `platform`, `program`, `slug` |
| `new_scope` | `platform`, `program`, `value`, `type` |
| `removed_scope` | `platform`, `program`, `value` |
| `ignored_asset` | `platform`, `program`, `value`, `reason` |
| `error` | `title`, `message` |

```json
{
"event": "new_program",
"timestamp": "2025-07-30T10:17:00Z",
"data": {
"platform": "yeswehack",
"program": "Example Program",
"slug": "example-program",
"scopes_count": 3,
"scopes": { "web": 2, "mobile": 1 }
}
}
```

</details>

## Scope Processing

ScopesExtractor includes intelligent scope processing with automatic normalization and validation.
Expand Down Expand Up @@ -531,7 +610,7 @@ scopes_refactor/
│ ├── sync_manager.rb # Platform synchronization orchestration
│ ├── validator.rb # Scope validation logic
│ ├── models/ # Dry-Struct models
│ ├── notifiers/ # Discord notifications
│ ├── notifiers/ # Discord & HTTP webhook notifications
│ └── platforms/ # Platform-specific implementations
│ ├── base_platform.rb
│ ├── yeswehack/
Expand Down
10 changes: 10 additions & 0 deletions config/settings.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,16 @@ discord:
errors:
url: "" # https://discord.com/api/webhooks/.../yyy

# Generic HTTP webhook notifications (JSON payloads)
webhook:
enabled: false
url: "" # https://example.com/hooks/scopes
headers: {} # Optional custom headers, "${VAR}" is replaced by the environment variable
# Example: { "Authorization": "Bearer ${WEBHOOK_TOKEN}" }
events: ["new_program", "removed_program", "new_scope", "removed_scope", "ignored_asset", "error"]
new_scope_types: ["web"] # Filter new_scope notifications by type (web, mobile, android, ios, etc.)
# Empty or null = notify all types

platform_exclusions:
yeswehack: []
hackerone: []
Expand Down
2 changes: 2 additions & 0 deletions lib/scopes_extractor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,8 @@ def colorize_severity(severity)

# Require notifiers
require_relative 'scopes_extractor/notifiers/discord'
require_relative 'scopes_extractor/notifiers/webhook'
require_relative 'scopes_extractor/notifiers/multi'

# Require models
require_relative 'scopes_extractor/models/scope'
Expand Down
30 changes: 30 additions & 0 deletions lib/scopes_extractor/config.rb
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,32 @@ def discord_new_scope_types
discord_main_webhook[:new_scope_types] || []
end

def webhook
load[:webhook] || {}
end

def webhook_enabled?
webhook[:enabled] == true
end

def webhook_url
resolve_env(webhook[:url])
end

# Custom headers sent with every webhook call (e.g. authentication).
# Values may reference environment variables using the "${VAR}" syntax.
def webhook_headers
(webhook[:headers] || {}).transform_keys(&:to_s).transform_values { |value| resolve_env(value) }
end

def webhook_events
webhook[:events] || []
end

def webhook_new_scope_types
webhook[:new_scope_types] || []
end

def platform_exclusions
load[:platform_exclusions] || {}
end
Expand All @@ -143,6 +169,10 @@ def excluded?(platform_name, program_slug)

private

def resolve_env(value)
value.to_s.gsub(/\$\{(\w+)\}/) { ENV.fetch(Regexp.last_match(1), '') }
end

def config_path
File.join(ScopesExtractor.root, 'config', 'settings.yml')
end
Expand Down
4 changes: 2 additions & 2 deletions lib/scopes_extractor/diff_engine.rb
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ module ScopesExtractor
class DiffEngine
def initialize(notifier: nil)
@db = ScopesExtractor.db
@notifier = notifier || Notifiers::Discord.new
@notifier = notifier || Notifiers::Multi.default
end

def process_program(platform_name, fetched_program, skip_notifications: false)
Expand Down Expand Up @@ -45,7 +45,7 @@ def process_program(platform_name, fetched_program, skip_notifications: false)
details: 'Brand new program discovered'
)

# Only send Discord notification if not skipping notifications
# Only send notifications if not skipping them
return if skip_notifications

@notifier.notify_new_program(
Expand Down
33 changes: 33 additions & 0 deletions lib/scopes_extractor/notifiers/multi.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# frozen_string_literal: true

module ScopesExtractor
module Notifiers
# Fans out every notification to all configured notifiers.
# A failing notifier never prevents the others from being notified.
class Multi
NOTIFY_METHODS = %i[
notify_new_program notify_removed_program notify_new_scope
notify_removed_scope notify_ignored_asset notify_error
].freeze

# Default set of notifiers, each one self-disables when not configured.
def self.default
new([Discord.new, Webhook.new])
end

def initialize(notifiers)
@notifiers = notifiers
end

NOTIFY_METHODS.each do |method_name|
define_method(method_name) do |*args, **kwargs|
@notifiers.each do |notifier|
notifier.public_send(method_name, *args, **kwargs)
rescue StandardError => e
ScopesExtractor.logger.error "#{notifier.class} ##{method_name} failed: #{e.message}"
end
end
end
end
end
end
Loading
Loading