Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
3af3a8d
feat(admin): put the Replicator page in the CMS menu and draw it with…
xiaodoudou Oct 11, 2026
3f9c042
feat(replication)!: replicate over a websocket on the HTTP port, not …
xiaodoudou Oct 11, 2026
139afa3
docs(replication): describe the types and directions as the code appl…
xiaodoudou Oct 11, 2026
13864e0
fix(replication): keep the resources of the CMS out of the global pee…
xiaodoudou Oct 11, 2026
7dcebd6
Merge branch 'feat/replicator-page' into feat/live-replication
xiaodoudou Oct 11, 2026
5cdd023
feat(replication)!: replicate live, each node pulling from the peers …
xiaodoudou Oct 11, 2026
3c0cfcc
feat(admin): make the Replicator page easier to read
xiaodoudou Oct 11, 2026
611327a
feat(admin): show the direction of each resource at a glance on the R…
xiaodoudou Oct 11, 2026
9ccb4e3
style(admin): give the table of the Replicator page a darker head
xiaodoudou Oct 11, 2026
1c37fb2
style(admin): one box on the Replicator page, with a light head
xiaodoudou Oct 11, 2026
97fcfd8
style(admin): drop the count of the resources that do not replicate f…
xiaodoudou Oct 11, 2026
755229c
fix(replicator): say why a link to a name with several addresses fail…
xiaodoudou Oct 11, 2026
744a7cf
docs(replication): show the Replicator page, and describe live replic…
xiaodoudou Oct 11, 2026
bd322a1
docs(replication): say how the admin shows a record made on another node
xiaodoudou Oct 11, 2026
4289f69
feat(admin): say in the editor that a record of another node is read-…
xiaodoudou Oct 11, 2026
f00758a
docs(replication): mention the read-only pill of the editor
xiaodoudou Oct 11, 2026
2a18da7
docs(replication): explain the record id and what a foreign record is
xiaodoudou Oct 11, 2026
beead2c
docs(changelog): note the read-only pill of a record made on another …
xiaodoudou Oct 11, 2026
263d9b4
fix(replication): send a change written as a live link connects, not …
xiaodoudou Oct 11, 2026
244f048
fix(replication): stop a link in its handshake at once, and reconnect…
xiaodoudou Oct 11, 2026
ed78e51
test(replication): wait for the connection of each test of the fake p…
xiaodoudou Oct 11, 2026
1427603
Merge remote-tracking branch 'origin/dev' into feat/live-replication
xiaodoudou Oct 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,13 @@ What changed in each version of Embed CMS, from 3.0.0 on, newest first. The vers
- **A token field.** `input: 'token'` holds a key or a secret that two systems share. Type it, paste it, or make a random one with the **Generate** button, and copy it with **Copy**. `options.length` (random bytes), `options.prefix` and `options.format` (`base64url` or `hex`) shape what Generate makes, and `options.autoGenerate` gives a new record a token by itself, in the admin and over the API. See [token](docs/reference/fields/token.md).
- **The Token under This CMS in the Sync settings is a token field.** It has a Generate and a Copy button, so the secret no longer has to be invented by hand.

### Changed

- **Replication is live, and needs no port of its own.** A node keeps a link open to each peer it receives a resource from: the peer sends what the node lacks, then each change as it is written, and the link connects again by itself after an outage. Records go over a websocket on the HTTP port of the CMS, at `/_replicator` next to the REST API, so they travel over `wss://` when the CMS is served over HTTPS; files still go over the REST API. A node a peer connects to only sends, so a connecting peer can no longer write anything. With MongoDB or PostgreSQL, a node pulls again every `replication.pollInterval` (30 seconds). **To upgrade:** remove `netPort`, give each peer the `url` of its REST API, set `replication.secret` on every node, and list on each node the peers it receives from. `settleDelay` and `POST /replicator/sync/:resource/:id` are gone. See [REPLICATION.md](docs/operations/REPLICATION.md#moving-from-3x).
- **The resources of the CMS stay out of the global peers list.** `_users`, `_groups`, `_settings`, `_sync` and `_xlsx` now replicate only when `replication.peersByResource` names them. Every node makes its own `admins` group and `localAdmin` on its first start, and a node that fetched another node's `admins` group could no longer update it, so its admin login stopped working. See [REPLICATION.md](docs/operations/REPLICATION.md#users-groups-and-settings).
- **The Replicator page shows the replication, live, and starts nothing.** For each resource that replicates, it shows the peers this node pulls it from, the state of each link (live, reconnecting with the error and the next try) and its last change, and the peers that pull it from here. The Sync all and Sync a record buttons are gone. The page moved from the System menu to the CMS menu, and is drawn with the plugin kit.
- **A record made on another node says so in the editor.** Its form showed "All changes saved" above fields that can't be edited. It now reads "Read-only, made on" and the `mid` of the node to edit it on. See [REPLICATION.md](docs/operations/REPLICATION.md#records-of-another-node).

## [3.1.1] - 2026-10-11

### Changed
Expand Down
18 changes: 9 additions & 9 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Each protection is one setting in the `security` block of `cms.json`. All of the
| `limits.json` | `"100kb"` | Size of a JSON request body, on every route (REST, sync, import, the admin): it is the only JSON limit. A larger body answers `413` with a JSON message naming this setting. 100kb is what every route accepted before the option existed |
| `limits.upload` | `{ fileSize: "256mb", files: 20, fields: 200, fieldSize: "1mb", parts: 260 }` | Limits of multipart uploads; a value given here replaces the matching default |
| `restrictRemoteUrls` | `true` | `importFromRemote` only fetches attachments from the remote it imports from (and `allowedHosts` of its `remote` block) |
| `strictReplication` | `true` | The replication port needs `replication.secret`, peers may only ask for resources this node has, changes from a peer are validated |
| `strictReplication` | `true` | Peers may only ask for resources this node has, and each record a node pulls from a peer is validated before it is stored. The replication websocket always needs `replication.secret` |
| `wsAuth` | `true` | The record update websocket needs a login and a same-origin (or listed) page |
| `wsMaxPayload` | `16384` | Largest message a websocket client may send |
| `authCacheTtl` | `60000` | Milliseconds a verified password stays verified in memory (Basic authentication runs the hash once per minute, not per request); `0` turns it off |
Expand All @@ -75,13 +75,13 @@ Each protection is one setting in the `security` block of `cms.json`. All of the
| `trustProxy` | not set | Passed to Express `trust proxy`: `req.ip`, `req.secure` and the login lockout follow the proxy chain. Set it to the number of proxies in front of the CMS; without it the address of the TCP peer is used |
| `imageConcurrency` | number of cores | Image operations (resize, crop, smart crop) that run at once; the others wait. `0` means no limit. At most 100 may wait, the next request gets 503 |
| `attachmentCleanupGrace` | `300000` | Milliseconds a file must have been idle before `cleanAttachment` may remove it as an orphan |
| `replication.secret` | not set | Shared secret of the replication port. Both sides prove they know it with an HMAC-SHA256 challenge before any resource is named |
| `replication.strictTypes` | `false` | Read the type (`normal`, `upstream`, `downstream`) of a resource from its options, so the direction rules of `replication.peers` apply. Off, every resource counts as `normal`, as in earlier releases |
| `replication.settleDelay` | `2000` | Milliseconds `replicate()` waits for a peer to flush before it syncs attachments |
| `replication.secret` | not set | Shared secret of the replication websocket (`/_replicator` on the HTTP port). Both sides prove they know it with an HMAC-SHA256 challenge before any resource is named; without it the node doesn't listen for peers |
| `replication.strictTypes` | `false` | Read the type (`normal`, `upstream`, `downstream`) of a resource from its options, so the `direction` of each peer is checked against it. Off, every peer is used for every resource |
| `replication.pollInterval` | `30000` | Milliseconds between two pulls from a peer whose store can't keep a link open (MongoDB, PostgreSQL) |
| `replication.maxRecordBytes` | `4194304` | Largest record accepted from a peer with `strictReplication` |
| `importFromRemote.remote.restrictUrls` / `.allowedHosts` | `false` / `[]` | Per remote, in addition to `restrictRemoteUrls` |

`replication.strictTypes` is worth a word. The replication manager reads the type of a resource from its options. Off, every resource counts as `normal` and the `direction` of a peer never matters. With the option on, a `downstream` resource (`_users`, `_groups`, `_settings` and the resources you declared so) is only synced with peers whose `direction` is `upstream` or `normal`, and an `upstream` resource with peers of direction `downstream` or `normal`. A peer without `direction` then stops syncing typed resources. Give every peer a direction before turning it on.
`replication.strictTypes` is worth a word. A node pulls each resource from the peers it lists, except a resource of type `upstream`, which it only sends. Off, a node opens a link to every peer it lists. With the option on, it opens links only to peers whose `direction` is `upstream` or `normal`, the peers it takes from; a peer without `direction` then gets no link. Give every peer a direction before turning it on. A peer that connects to a node can pull from it but never writes to it, with or without the option. [REPLICATION.md](docs/operations/REPLICATION.md#directions) explains the types and the directions.

### Always on

Expand Down Expand Up @@ -157,7 +157,7 @@ The values that depend on your setup:
- `trustProxy` is the number of reverse proxies in front of the CMS; change `1` to the real number.
- `allowedOrigins` takes the origin of the admin app when it is served from another host than the API.
- `limits.json` is `"100kb"`, the size of the largest JSON body (a record, an import); raise it when a record is bigger.
- `replication.strictTypes` is on, so every peer needs a `direction`, or it stops syncing `_users`, `_groups` and `_settings`.
- `replication.strictTypes` is on, so every peer needs a `direction`, or it syncs `normal` resources only.
- `localAdmin` is off. A `localAdmin` record that already exists stays, and every boot logs an error while it has its default password: change that password or delete the account.
- `safeAttachments` and `strictUploads` are on. Turning `safeAttachments` off displays HTML and SVG attachments inline, with the privileges of the site for whoever opens them. Turning `strictUploads` off removes the file name sanitising, the content-based type and the upload limits.

Expand All @@ -176,7 +176,7 @@ Run with `NODE_ENV=production`, then check:
3. Password hashes. Leave `passwordHash` on `scrypt`. If nodes replicate `_users`, upgrade every node first: a node that runs an earlier release cannot verify an scrypt hash.
4. Proxy. Behind a reverse proxy, set `trustProxy` to the number of proxies, forward `X-Forwarded-Proto`, and serve the admin app over https (cookies get `Secure`, HSTS is sent).
5. CSRF and origins. If the admin app is served from another origin than the API, list it in `allowedOrigins`.
6. Replication. Set the same `replication.secret` on every node, firewall the replication port to the peers, and set a `direction` on every peer before turning on `replication.strictTypes`.
6. Replication. Set the same `replication.secret` on every node, serve the CMS over HTTPS so peers replicate over `wss://`, and set a `direction` on every peer before turning on `replication.strictTypes`.
7. Uploads. Tune `limits.upload` to what your editors need. Do not add `text/html` or `image/svg+xml` to `inlineTypes` unless you trust everyone who can upload.
8. Configuration. `cms.json` is not editable from the admin: nothing on the web can read or rewrite the secrets.
9. Logs. `/api/_syslog` shows the log to every user who can log in: keep secrets out of log lines (the backend does not write passwords or tokens) and give the `plugins` right of the group only to administrators. The log page and the log file are bounded by `syslog.maxLineLength`, `syslog.maxFileSize` and `syslog.maxClientBuffer`.
Expand All @@ -185,7 +185,7 @@ Run with `NODE_ENV=production`, then check:
## Not covered

- The frontend (`src/`) was not changed by this work. Its own protections (DOMPurify) are separate.
- Transport encryption of the replication port: the shared secret authenticates the peers, it does not encrypt what they exchange. Run the replication port on a private network or through a tunnel.
- Transport encryption of the replication: the shared secret authenticates the peers, it does not encrypt what they exchange. Serve the CMS over HTTPS (peers then use `wss://`), or keep the nodes on a private network or a tunnel.
- MongoDB and PostgreSQL servers are configured by their operators; the CMS only guarantees the queries it sends.
- Regular expressions are checked, not sandboxed: a pattern that passes the check can still be slow on a very long text field. If that matters to you, keep `limits.json` small and consider a regular expression engine with a linear time guarantee in front of the query filter.

Expand All @@ -206,7 +206,7 @@ Options added, with defaults (see the tables above):

- `strongSecrets`, `generateSecrets`, `localAdmin`, `passwordHash`, `hideCredentials`, `genericLockout`, `csrf`, `allowedOrigins`, `cookies`, `strictSessions`, `redactConfig`, `strictAdmin`, `headers`, `contentSecurityPolicy`, `sseCors`, `safeRegex`, `uniformErrors`, `safeAttachments`, `inlineTypes`, `strictUploads`, `restrictRemoteUrls`, `strictReplication`, `wsAuth`, `wsMaxPayload`, `authCacheTtl`, `limits.json`, `limits.upload`
- top level: `trustProxy`, `imageConcurrency`, `attachmentCleanupGrace`; `blockRetry: false` turns the lockout off
- `replication.secret`, `replication.strictTypes`, `replication.settleDelay`, `replication.maxRecordBytes`
- `replication.secret`, `replication.strictTypes`, `replication.pollInterval`, `replication.maxRecordBytes`
- `importFromRemote.remote.restrictUrls`, `importFromRemote.remote.allowedHosts`

Files written by the backend: `<data>/.secrets.json` (only with `generateSecrets`), `<data>/.revoked-tokens.json`, `<data>/<resource>/json/db.json.corrupt-<time>` (only when a store file could not be read).
Expand Down
2 changes: 1 addition & 1 deletion docs/contributing/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ The record goes back up the same way, as JSON. The REST plugin folds the attachm

Meanwhile the broadcast of step 4 reaches `UpdatesManager` (`lib/UpdatesManager.js`), which sends `{ action, data: { resource, _id, _updatedBy } }` over a websocket to every open admin. Another editor looking at the same list sees it refresh. Resources whose name starts with `_` are not broadcast.

If the resource has peers and a replication type, the replicator (`lib/plugins/replicator`) now syncs the change to the other nodes: records over its TCP protocol, files over HTTP. See [REPLICATION.md](../operations/REPLICATION.md).
The write also reaches the peers that pull the resource from this node: each keeps a live link open (`lib/plugins/replicator`), and the change goes out on it as soon as it is stored. Records travel over a websocket on the HTTP port, files over HTTP. See [REPLICATION.md](../operations/REPLICATION.md).

## Where things live

Expand Down
4 changes: 2 additions & 2 deletions docs/contributing/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,14 @@ Setting `LOG_LEVEL` does the same (`LOG_LEVEL=debug` shows even the refused requ
| `test/helpers/` | What the backend tests share: `app.js` boots a CMS, `cmsInstance.js` holds the shared options, `engines.js` the storage engines of the contract suite, `quiet.js` the log level |
| `test/fixtures/` | Data the tests read: the `resources/` of the shared CMS, `contractResources/` for the contract suite, `ossResources/` for the OSS tests, `man.jpg` and the `smartCrop/` reference pictures |
| `test/bench/` | The benchmarks (`npm run bench:stores`, `bench.js`, `compare.js`) and their own `resources/` |
| `test/security/` | One regression test per finding of the backend security audit: `auth` (passwords, sessions, login limits, revocation), `http` (headers, CSRF, CORS, errors), `uploads`, `query`, `rest`, `importFromRemote` (SSRF), `replication` (handshake, peer validation), `websocket` |
| `test/security/` | One regression test per finding of the backend security audit: `auth` (passwords, sessions, login limits, revocation), `http` (headers, CSRF, CORS, errors), `uploads`, `query`, `rest`, `importFromRemote` (SSRF), `replication` (handshake, a node that only sends, the records a node pulls checked one by one), `websocket` |
| `test/unit/authentication*` | Passwords, login, retry blocking, group permissions, JWT handling |
| `test/unit/driver*`, `resource*`, `locking*` | CRUD, hooks, unique keys, concurrency, attachments, import maps |
| `test/unit/localEngines*` | The three local engines behind one interface (ranges, paging, scan, count, hooks, reopen), the choice in `cms.json`, and the migration between them |
| `test/unit/stores*`, `jsondown*`, `queryNeedles*`, `paging*`, `importMap*` | `JsonStore`, `FileStore` (including path traversal), the JSON file engine (ordering, ranges, atomic and sliced flush, corrupt files), the text prefilter of queries, paging, the indexed import map |
| `test/unit/drivers.contract.test.js` | The [driver contract suite](#driver-contract-suite) |
| `test/unit/authentication.cache*`, `hardening*`, `security.utils*` | The verified-password cache, ids, and the small security utilities (passwords, redaction, safe regular expressions, file types) |
| `test/unit/replicator.attachments*`, `sync*` | Attachment download between peers, the low-level store sync |
| `test/unit/replication.live*`, `replicator.attachments*`, `sync*` | Live links between two nodes (a change crosses as it is written, a link that drops connects again, files follow the records), attachment download between peers, the low-level store sync |
| `test/unit/rest.attachments*`, `ossHelper*`, `admin*`, `updates*`, `syslog.system*` | REST attachment routes, attachments kept in Alibaba Cloud OSS (the client stood in for), admin plugin, websocket updates, log and system streams |
| `test/unit/xlsx*`, `sync*`, `import*`, `importFromRemote*`, `replicator*` | The plugins, each checked through its real routes |
| `test/unit/pageHelper*` | The [PageHelper](../reference/PAGE_HELPER.md): the templates read and checked at the start, each way a kept page can be old (a template, a record, the age, a damaged file), the memory and the folder, errors and 404 pages, many requests at once. `pageHelper.edge.test.js` stands a small CMS in for the real one, so that each failure is made on purpose |
Expand Down
2 changes: 1 addition & 1 deletion docs/extending/PLUGINS.md
Original file line number Diff line number Diff line change
Expand Up @@ -357,6 +357,6 @@ A page can also have an `allowed` list: `allowed: ['admins']` limits it to those
| `anonymousRead` | `lib/plugins/anonymousRead/index.js` | The smallest: one bootstrap function. |
| `xlsx`, `import`, `importFromRemote` | `lib/plugins/xlsx`, `import`, `importFromRemote` | Routes with uploads, a resource for settings. |
| `sync` | `lib/plugins/sync/index.js` | Routes, a settings resource, hooks, and an admin page (`src/components/pages/SyncResource.vue`). |
| `replicator` | `lib/plugins/replicator/` | A TCP protocol next to the HTTP routes, and the **Replicator** page. |
| `replicator` | `lib/plugins/replicator/` | A websocket at `/_replicator` on the HTTP port, the HTTP routes, and the **Replicator** page. |

The built-in server plugins are switched on by the options in [CONFIG.md](../reference/CONFIG.md#features-you-can-switch). Yours are installed in code with `cms.use`: they don't need an option, and the CMS doesn't look for plugin files by itself.
Binary file added docs/img/replicator.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading