Skip to content

Commit 145298e

Browse files
heiskrheiskr
authored
Expand README for src/secret-scanning (#58890)
1 parent 7c1ed5d commit 145298e

1 file changed

Lines changed: 140 additions & 5 deletions

File tree

‎src/secret-scanning/README.md‎

Lines changed: 140 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,153 @@
11
# Secret scanning
22

3-
This secret scanning pipeline automates a table displayed on the [Supported secret scanning patterns](https://docs.github.com/code-security/secret-scanning/introduction/supported-secret-scanning-patterns#supported-secrets) page.
3+
The secret-scanning subject automates the generation and maintenance of the "Supported secret scanning patterns" table on docs.github.com. It fetches data from upstream sources, transforms it, and renders it as a Liquid-powered Markdown table.
44

5-
Each day a workflow checks if the [data](src/secret-scanning/data/public-docs.yml) is up-to-date. When there are changes, the workflow automatically creates a pull request to update the `src/secret-scanning/data/public-docs.yml` file. The workflow runs `npm run sync-secret-scanning` to check for updates.
5+
## Purpose & Scope
66

7-
This pipeline uses middleware to check if the path of the URL matches the page that contains the table. The middleware decorates the context with the data, which is displayed on the page using a Markdown table and Liquid. For example:
7+
This subject is responsible for:
8+
- Syncing secret scanning pattern data from upstream sources
9+
- Storing pattern data in YAML files by version
10+
- Middleware that injects data into the supported patterns page
11+
- Rendering the patterns table with Liquid in Markdown
12+
- Automated daily workflow to check for and update pattern changes
13+
14+
The table appears on: [Supported secret scanning patterns](https://docs.github.com/code-security/secret-scanning/introduction/supported-secret-scanning-patterns#supported-secrets)
15+
16+
## Architecture & Key Assets
17+
18+
### Key capabilities and their locations
19+
20+
- `middleware/secret-scanning.ts` - Middleware that loads YAML data and adds to `req.context.secretScanningData`
21+
- `scripts/sync.ts` - Script that syncs pattern data from upstream sources and updates YAML files
22+
- `lib/config.json` - Configuration specifying which page gets the data (`targetFilename`)
23+
- `data/pattern-docs/*.yml` - YAML files containing pattern data per version
24+
25+
## Setup & Usage
26+
27+
### Daily automated sync
28+
29+
A GitHub Actions workflow runs daily to check for pattern updates:
30+
1. Runs `npm run sync-secret-scanning`
31+
2. If changes detected, creates a PR to update YAML files
32+
3. Team reviews and merges PR
33+
34+
### Manual sync
35+
36+
To manually sync pattern data:
37+
38+
```bash
39+
npm run sync-secret-scanning
40+
```
41+
42+
This fetches latest pattern data and updates YAML files in `data/pattern-docs/`.
43+
44+
### How the table is rendered
45+
46+
1. Middleware checks if current page matches `targetFilename` from config
47+
2. Loads appropriate YAML file based on version (FPT/GHEC/GHES)
48+
3. Adds data to `req.context.secretScanningData`
49+
4. Markdown uses Liquid to render table rows
50+
51+
Example Markdown with Liquid:
852

953
```markdown
10-
<!-- FPT version of table -->
1154
{% ifversion fpt %}
1255

1356
| Provider | Token | Partner | User | Push protection | Base64 |
1457
|----|:----|:----:|:----:|:----:|
1558
{%- for entry in secretScanningData %}
16-
| {{ entry.provider }} | {{ entry.secretType }} | {% if entry.isPublic %}{% octicon "check" aria-label="Supported" %}{% else %}{% octicon "x" aria-label="Unsupported" %}{% endif %} | {% if entry.isPrivateWithGhas %}{% octicon "check" aria-label="Supported" %}{% else %}{% octicon "x" aria-label="Unsupported" %}{% endif %} | {% if entry.hasPushProtection %}{% octicon "check" aria-label="Supported" %}{% else %}{% octicon "x" aria-label="Unsupported" %}{% endif %} | {% if entry.base64Supported %}{% octicon "check" aria-label="Supported" %}{% else %}{% octicon "x" aria-label="Unsupported" %}{% endif %} |
59+
| {{ entry.provider }} | {{ entry.secretType }} | {% if entry.isPublic %}{% octicon "check" %}{% else %}{% octicon "x" %}{% endif %} | ...
1760
{%- endfor %}
1861
```
62+
63+
### Data structure
64+
65+
Each pattern entry includes:
66+
- `provider` - Service provider name
67+
- `secretType` - Type of secret/token
68+
- `isPublic` - Available on public repos
69+
- `isPrivateWithGhas` - Available on private repos with GHAS
70+
- `hasPushProtection` - Has push protection enabled
71+
- `hasValidityCheck` - Has validity checking
72+
- `base64Supported` - Supports base64-encoded secrets
73+
74+
## Data & External Dependencies
75+
76+
### Data inputs
77+
- Upstream secret scanning pattern sources (internal APIs)
78+
- Existing YAML files in `data/pattern-docs/`
79+
- Version information from `@/versions/lib/all-versions`
80+
81+
### Dependencies
82+
- `js-yaml` - YAML parsing and generation
83+
- `@/content-render` - Liquid rendering for table
84+
- `@/versions` - Version detection and mapping
85+
- GitHub Actions workflow for automated sync
86+
87+
### Data outputs
88+
- Updated YAML files in `data/pattern-docs/`
89+
- `req.context.secretScanningData` - Array of pattern objects
90+
- Rendered Markdown table on docs page
91+
92+
## Cross-links & Ownership
93+
94+
### Related subjects
95+
- [`src/content-render`](../content-render/README.md) - Liquid rendering for table
96+
- [`src/versions`](../versions/README.md) - Version detection for loading correct data file
97+
- Content page: `content/code-security/secret-scanning/introduction/supported-secret-scanning-patterns.md`
98+
99+
### Internal documentation
100+
For upstream data source details and API access, see internal Docs Engineering documentation.
101+
102+
### Ownership
103+
- Team: Docs Engineering
104+
- Content: Code Security team
105+
106+
## Current State & Next Steps
107+
108+
### Automated workflow
109+
110+
GitHub Actions workflow (`.github/workflows/sync-secret-scanning.yml`) runs daily:
111+
- Checks for pattern updates
112+
- Creates PR if changes found
113+
- Runs `npm run sync-secret-scanning`
114+
115+
### Version-specific data
116+
117+
Different data files per version:
118+
- `dotcom.yml` - Free, Pro, Team (FPT)
119+
- `ghec.yml` - GitHub Enterprise Cloud
120+
- `ghes-{version}.yml` - GitHub Enterprise Server versions
121+
122+
Middleware automatically selects correct file based on `req.context.currentVersion`.
123+
124+
### Known limitations
125+
- Manual review required for auto-generated PRs
126+
- Pattern data schema must match between upstream and our YAML
127+
- Changes to upstream API may break sync script
128+
- Table only appears on one specific page (configured in `config.json`)
129+
130+
### Expanding to more pages
131+
132+
To display secret scanning data on additional pages:
133+
1. Update `config.json` with new target filenames (as array)
134+
2. Update middleware to handle multiple pages
135+
3. Add Liquid table rendering to those pages
136+
137+
### Troubleshooting sync issues
138+
139+
**Sync fails:**
140+
- Check upstream API access and credentials
141+
- Verify YAML file permissions
142+
- Check for schema changes in upstream data
143+
144+
**Table not rendering:**
145+
- Verify page path matches `targetFilename` in `config.json`
146+
- Check that `secretScanningData` is in context
147+
- Verify Liquid syntax in Markdown
148+
149+
**Wrong data version:**
150+
- Check version detection logic in middleware
151+
- Verify correct YAML file exists for version
152+
- Check version mapping in middleware
153+

0 commit comments

Comments
 (0)