Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ jobs:
- name: Run unit tests with coverage
run: >
nix develop --command just test-coverage
- name: Validate independent fctl plugin
run: >
nix develop --command just test-fctl-plugin build-fctl-plugin
- name: Run internal tests with optional features and coverage
run: >
nix develop --command just test-internal-coverage
Expand Down
9 changes: 9 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ permissions:

jobs:
GoReleaser:
env:
PLUGIN_REVISION: "1"
runs-on: namespace-profile-linux-amd64-4vcpu
steps:
- uses: namespacelabs/nscloud-checkout-action@445c25d7009680597d73eb03c4e1cd5be522ed73 # v9
Expand All @@ -18,6 +20,9 @@ jobs:
dissociate: true
- name: Setup Env
uses: ./.github/actions/default
- name: Validate fctl plugin before co-release
run: >
nix develop --command just test-fctl-plugin
- name: Set up Docker Buildx
uses: namespacelabs/nscloud-setup-buildx-action@d059ed7184f0bc7c8b27e8810cea153d02bcc6dd # v0.0.23
- name: Login to GitHub Container Registry
Expand All @@ -31,6 +36,10 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GORELEASER_KEY: ${{ secrets.GORELEASER_KEY }}
- name: Publish Ledger fctl plugin and release catalogue
run: nix develop --command just publish-fctl-plugin-release
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push operator image
run: >
nix develop --command just operator-docker-build ${{ github.ref_name }}
Expand Down
34 changes: 34 additions & 0 deletions .goreleaser.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ project_name: ledger
# `formancehq/ledger` published image.
env:
- IMAGE_REPOSITORY={{ envOrDefault "IMAGE_REPOSITORY" "ghcr.io/formancehq/ledger" }}
- PLUGIN_REVISION={{ envOrDefault "PLUGIN_REVISION" "1" }}

builds:
- binary: ledger-server
Expand Down Expand Up @@ -66,6 +67,26 @@ builds:
- amd64
- arm64

# The product owns its HTTP fctl commands and co-releases their executable.
- binary: fctl-plugin-ledger
id: fctl-plugin-ledger
dir: misc/fctl-plugin
main: ./cmd/fctl-plugin-ledger
# Keep -ldflags in Go build metadata for publication identity checks.
ldflags:
- -s -w
- -X main.serviceVersion={{ .Version }}
- -X main.revision={{ .Env.PLUGIN_REVISION }}
env:
- CGO_ENABLED=0
goos:
- linux
- darwin
- windows
goarch:
- amd64
- arm64

nightly:
tag_name: nightly
publish_release: true
Expand Down Expand Up @@ -98,6 +119,19 @@ archives:
- zip
name_template: "{{ .ProjectName }}_{{ if not .IsNightly }}v{{ end }}{{ .Version }}_{{ .Os }}-{{ .Arch }}"
wrap_in_directory: true
- id: fctl-plugin-ledger
ids:
- fctl-plugin-ledger
formats:
- tar.gz
format_overrides:
- goos: windows
formats:
- zip
name_template: "fctl-plugin-ledger_v{{ .Version }}_r{{ .Env.PLUGIN_REVISION }}_{{ .Os }}-{{ .Arch }}"
files:
- src: misc/fctl-plugin/README.md
dst: README.md

checksum:
name_template: "checksums.txt"
Expand Down
8 changes: 8 additions & 0 deletions docs/ops/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

`ledgerctl` is the command-line client for interacting with Ledger v3 servers via gRPC.

Ledger also owns an independent HTTP command plugin for fctl v4. The
`fctl-plugin-ledger` executable is built from the separate `misc/fctl-plugin/` Go
module and co-released with the Ledger service for Linux, macOS, and Windows
on amd64 and arm64. fctl supplies profiles, authentication, interactive forms,
HTTP diagnostics, and output styling. The plugin supplies the Ledger command
manifest and HTTP operations. See the [fctl plugin guide](../../misc/fctl-plugin/README.md)
for local builds, service versions, plugin revisions, and distribution inputs.

## Getting Started

![Getting Started](../../misc/demo/demo_getting_started.gif)
Expand Down
1 change: 1 addition & 0 deletions docs/technical/architecture/subsystems/api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Client-facing transport layers (`internal/adapter/grpc`, `internal/adapter/http`
| [http-api.md](http-api.md) | HTTP endpoint lifecycle and failure shutdown, REST API endpoints, single-decoding metadata key paths, response formats, opt-in string monetary amounts, exact metadata number decoding, ledger-log JSON output and discriminators, error handling, and the `apierr`/`grpcerr` error boundary crossed by a forwarded write. |
| [auth.md](auth.md) | Client JWT authentication (OIDC + Ed25519), scope-based authorization, and the Raft inter-node cluster-secret auth layer. |
| [sensitive-projection.md](sensitive-projection.md) | Detached, descriptor-driven secret projection and its ownership boundary. |
| [fctl-plugin.md](fctl-plugin.md) | Product-owned HTTP command module, external SDK broker, co-release identity, and immutable OCI publication. |

Decoded leader errors retain their exact gRPC status through routing wrappers
and streaming cursors. See [forwarding and cancellation](http-api.md#forwarded-writes-and-the-transport-seam)
Expand Down
60 changes: 60 additions & 0 deletions docs/technical/architecture/subsystems/api/fctl-plugin.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Product-owned fctl plugin

With Ledger removed from the fctl binary, users need an installable exact-version
command provider. Ledger co-releases that provider so its API owner can keep
commands aligned with the service without pulling server or host CLI code into
the module. Auth PR #162 establishes the same release/distribution contract.

Ledger owns its HTTP command package in the separate `misc/fctl-plugin/` Go module.
Its sole direct dependency is the public fctl plugin SDK. It imports neither
Ledger service internals nor the fctl core, UI, profiles, or authentication.

The package implements the SDK's `GetManifest` and `Execute` contract. The
manifest declares command arguments, flags, confirmations, and forms. The same
package can run embedded or through the standalone `fctl-plugin-ledger`
executable. The external entry point calls `transport.Serve` with the factory
that accepts the host-provided HTTP client.

In external mode, the SDK broker returns each HTTP request to the fctl host.
The host applies its authentication, endpoint restrictions, transport, and
diagnostics. Exact JSON bytes and partial bulk responses stay on the public
SDK boundary. The executable has no Ledger service gRPC contract dependency;
the plugin SDK protocol version is distinct from `pkg/grpcprotocol.Version`.

The existing command contract includes shape validation before dispatch, raw JSON
numbers, page cursors and partial bulk errors. Validation rejects malformed input
without executing a mutation; state-dependent business rules remain server-owned.

Each plugin release names an exact Ledger service version and a positive
plugin revision. A CLI-only fix increments the revision for that service
version. It does not change the service version in the SDK manifest. The
host's catalogue maps this identity and platform to a digest-addressed native
executable. The host validates the running manifest against the locked one.

GoReleaser builds the six supported native platforms in the Ledger release
pipeline. The publisher reads the resulting GoReleaser artifact list and the
SDK manifest exported by the native executable. It validates each binary's Go
entry point, target platform, and injected service version and plugin revision,
computes its SHA-256 checksum, and uploads
one raw executable layer per OCI artifact through the Nix-pinned ORAS tool.
Paths from the artifact list are opened through `os.Root`; traversal and
symlink escapes cannot select a file outside the build root.

Publication builds retain Go linker metadata. They do not use `-trimpath`,
which removes the recorded `-ldflags` needed to verify foreign-platform
version and revision without executing the binary. The publisher checks every
platform before calling ORAS. A stale manifest or a mixed set of product
versions fails preflight and produces neither an upload nor a catalogue.

The publisher creates private staging files before uploading. It emits a
catalogue only after all six uploads succeed. If an upload fails, earlier
artifacts can remain in the registry, but no complete catalogue is emitted.
The canonical Just recipe replaces the prepared catalogue only on success.
The local layout recipe exercises the same ORAS serialization without any
registry upload. See the [build and publication guide](../../../../../misc/fctl-plugin/README.md).

Tagged release CI invokes the publisher after GoReleaser, verifies the exact
service tag as well as every platform identity, uses the job token for GHCR, and
attaches `fctl-plugin-catalogue.json` after all uploads. The first publication
requires Public GHCR visibility and anonymous checksum verification. Promotion
into `formancehq/fctl-plugin-registry` is a separate reviewed operation.
1 change: 1 addition & 0 deletions docs/technical/contributing/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ ledger/
├── pkg/ # Public packages (actions/, scenario/, testserver/)
├── tests/ # Test suites (e2e/, scenarios/, antithesis/, perf/, schemathesis/)
├── misc/
│ ├── fctl-plugin/ # Independent HTTP command plugin for fctl v4 (own Go module)
│ ├── proto/ # Protocol Buffer definitions
│ ├── demo/ # VHS tape files for CLI demos
│ ├── numscript/examples/ # Numscript examples
Expand Down
3 changes: 3 additions & 0 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@
acli
go_1_27
ffmpeg
gh
coreutils
ginkgo
gomarkdoc
go-jsonnet
Expand All @@ -92,6 +94,7 @@
k6
kubernetes-helm
nodejs_22
oras
python314
trufflehog
uv
Expand Down
94 changes: 94 additions & 0 deletions justfile
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ lint:
golangci-lint run --fix --build-tags it,local,enable_antithesis_sdk,{{all_tags}} --timeout 5m
echo "==> golangci-lint (operator)"
(cd misc/operator && golangci-lint run --fix --timeout 5m)
echo "==> golangci-lint (fctl plugin)"
(cd misc/fctl-plugin && golangci-lint run --fix --build-tags it --timeout 5m)
echo "==> golangci-lint (model workload)"
(cd tests/antithesis/workload && golangci-lint run --fix --build-tags enable_antithesis_sdk --timeout 5m)

Expand All @@ -40,6 +42,8 @@ tidy:
go mod tidy
echo "==> go mod tidy (operator)"
(cd misc/operator && go mod tidy)
echo "==> go mod tidy (fctl plugin)"
(cd misc/fctl-plugin && go mod tidy)
echo "==> go mod tidy (model workload)"
(cd tests/antithesis/workload && go mod tidy)

Expand Down Expand Up @@ -68,6 +72,45 @@ build-full:
build-client:
go build -o ./build/ledgerctl ./cmd/ledgerctl

# Build the HTTP fctl plugin, independently of the Ledger service module.
build-fctl-plugin version="dev" revision="1":
cd misc/fctl-plugin && CGO_ENABLED=0 go build -ldflags '-X main.serviceVersion={{version}} -X main.revision={{revision}}' -o ../../build/fctl-plugin-ledger ./cmd/fctl-plugin-ledger

test-fctl-plugin:
#!/usr/bin/env bash
set -euo pipefail
# The SDK uses Unix sockets; the canonical validator's nested TMPDIR can
# exceed macOS sockaddr_un.sun_path. Keep real IPC tests in a short root.
fixture_tmp="$(mktemp -d /tmp/fctl-plugin.XXXXXX)"
trap 'rm -rf "$fixture_tmp"' EXIT
export TMPDIR="$fixture_tmp"
cd misc/fctl-plugin
go test -race -tags it ./... -timeout 3m

# Prepare six native platform archives and checksums without publishing.
package-fctl-plugin version="dev" revision="1":
cd misc/fctl-plugin && LEDGER_SERVICE_VERSION={{version}} PLUGIN_REVISION={{revision}} goreleaser release --snapshot --clean --skip=publish

# Export the SDK manifest from the previously built native executable.
fctl-plugin-manifest:
./build/fctl-plugin-ledger --manifest > ./build/fctl-plugin-manifest.json

# Publish previously packaged raw binaries; advertise the catalogue only on success.
publish-fctl-plugin registry="https://ghcr.io" repository="formancehq/fctl-plugin-ledger" revision="1":
#!/usr/bin/env bash
set -euo pipefail
cd misc/fctl-plugin
go run ./cmd/fctl-plugin-publish --service-version "$(jq -er .version ../../build/fctl-plugin-manifest.json)" --manifest ../../build/fctl-plugin-manifest.json --artifacts ../../build/fctl-plugin/artifacts.json --source-root ../../build/fctl-plugin --registry '{{registry}}' --repository '{{repository}}' --revision '{{revision}}' > ../../build/fctl-plugin-catalogue.json.tmp
mv ../../build/fctl-plugin-catalogue.json.tmp ../../build/fctl-plugin-catalogue.json

# Exercise real ORAS artifact creation locally without accessing any registry.
fctl-plugin-oci-layout revision="1":
#!/usr/bin/env bash
set -euo pipefail
cd misc/fctl-plugin
go run ./cmd/fctl-plugin-publish --service-version "$(jq -er .version ../../build/fctl-plugin-manifest.json)" --manifest ../../build/fctl-plugin-manifest.json --artifacts ../../build/fctl-plugin/artifacts.json --source-root ../../build/fctl-plugin --layout ../../build/fctl-plugin-oci --revision '{{revision}}' > ../../build/fctl-plugin-catalogue.json.tmp
mv ../../build/fctl-plugin-catalogue.json.tmp ../../build/fctl-plugin-catalogue.json

# Run the application locally (single node)
run:
go run . run --node-id 1 --cluster-id local-dev --bootstrap --bind-addr 127.0.0.1:7777 --grpc-port 8888 --wal-dir ./wal/node-1 --data-dir ./data/node-1
Expand Down Expand Up @@ -548,3 +591,54 @@ test-sdk-bigint:
npm --prefix "$sdk_dir" ci --ignore-scripts --no-audit --no-fund
(cd "$sdk_dir" && node node_modules/typescript/bin/tsc)
LEDGER_SDK_TEST="$sdk_dir/esm/ledger-bigint-test.js" go test -tags sdk ./internal/adapter/http -run ^TestBigintOperations$ -count=1 -timeout=2m

# Release publication uses the native executable from this exact GoReleaser run.
publish-fctl-plugin-release:
#!/usr/bin/env bash
set -euo pipefail
: "${GITHUB_REF_NAME:?release tag is required}"
: "${GITHUB_TOKEN:?GitHub Actions token is required}"
: "${GITHUB_ACTOR:?publishing actor is required}"
[[ "$GITHUB_REF_NAME" == v* ]] || { echo 'Expected a Ledger v-prefixed release tag' >&2; exit 1; }
version="${GITHUB_REF_NAME#v}"
revision="${PLUGIN_REVISION:-1}"
native="$(jq -er '[.[] | select(.type == "Binary" and .extra.ID == "fctl-plugin-ledger" and .goos == "linux" and .goarch == "amd64")] | if length == 1 then .[0].path else error("expected one native Ledger plugin") end' dist/artifacts.json)"
: "${GITHUB_REPOSITORY:?release repository is required}"
release_root="$PWD"
auth_dir="$(mktemp -d)"
trap 'rm -rf "$auth_dir"; rm -f "$release_root/dist/fctl-plugin-catalogue.json.tmp" "$release_root/dist/fctl-plugin-manifest.json.tmp"' EXIT
export DOCKER_CONFIG="$auth_dir"
"$native" --manifest > dist/fctl-plugin-manifest.json.tmp
printf '%s' "$GITHUB_TOKEN" | oras login ghcr.io --username "$GITHUB_ACTOR" --password-stdin
cd misc/fctl-plugin
go run ./cmd/fctl-plugin-publish --manifest ../../dist/fctl-plugin-manifest.json.tmp --artifacts ../../dist/artifacts.json --source-root ../.. --service-version "$version" --revision "$revision" > ../../dist/fctl-plugin-catalogue.json.tmp
cd ../..
mv dist/fctl-plugin-catalogue.json.tmp dist/fctl-plugin-catalogue.json
GH_TOKEN="$GITHUB_TOKEN" gh release upload "$GITHUB_REF_NAME" dist/fctl-plugin-catalogue.json --clobber --repo "$GITHUB_REPOSITORY"


# Read-only check after an administrator enables public GHCR downloads.
verify-fctl-plugin-anonymous catalogue="dist/fctl-plugin-catalogue.json":
#!/usr/bin/env bash
set -euo pipefail
catalogue_path="$(realpath '{{catalogue}}')"
fixture="$(mktemp -d)"
trap 'rm -rf "$fixture"' EXIT
printf '{"auths":{}}' > "$fixture/config.json"
jq -e '.schemaVersion == 1 and (.releases | length == 6)' "$catalogue_path" > /dev/null
while IFS=$'\t' read -r registry repository digest checksum; do
reference="${registry#https://}/$repository@$digest"
oras pull --registry-config "$fixture/config.json" --output "$fixture/artifact" "$reference"
printf '%s %s\n' "$checksum" "$fixture/artifact/fctl-plugin-ledger" | sha256sum --check
rm -rf "$fixture/artifact"
done < <(jq -r '.releases[] | [.artifact.registry, .artifact.repository, .artifact.digest, .sha256] | @tsv' "$catalogue_path")


# Exercise the real external host without importing its core.
test-fctl-plugin-host binary:
#!/usr/bin/env bash
set -euo pipefail
fixture_tmp="$(mktemp -d /tmp/fctl-plugin.XXXXXX)"
trap 'rm -rf "$fixture_tmp"' EXIT
export TMPDIR="$fixture_tmp"
FCTL_BINARY="{{binary}}" go -C misc/fctl-plugin test -race -tags host ./cmd/fctl-plugin-ledger -run TestFctlLifecycle -count=1 -timeout 3m
51 changes: 51 additions & 0 deletions misc/fctl-plugin/.goreleaser.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# yaml-language-server: $schema=https://goreleaser.com/static/schema.json
version: 2
project_name: fctl-plugin-ledger
dist: ../../build/fctl-plugin

env:
- PLUGIN_REVISION={{ envOrDefault "PLUGIN_REVISION" "1" }}
- LEDGER_SERVICE_VERSION={{ envOrDefault "LEDGER_SERVICE_VERSION" .Version }}

builds:
- id: fctl-plugin-ledger
binary: fctl-plugin-ledger
main: ./cmd/fctl-plugin-ledger
env:
- CGO_ENABLED=0
# Keep -ldflags build metadata: the publisher checks version/revision on
# every foreign-platform executable before uploading any of them.
ldflags:
- -s -w
- -X main.serviceVersion={{ .Env.LEDGER_SERVICE_VERSION }}
- -X main.revision={{ .Env.PLUGIN_REVISION }}
goos:
- linux
- darwin
- windows
goarch:
- amd64
- arm64

archives:
- ids:
- fctl-plugin-ledger
name_template: "{{ .ProjectName }}_v{{ .Env.LEDGER_SERVICE_VERSION }}_r{{ .Env.PLUGIN_REVISION }}_{{ .Os }}-{{ .Arch }}"
formats:
- tar.gz
format_overrides:
- goos: windows
formats:
- zip
files:
- README.md

checksum:
name_template: checksums.txt

# The service release configuration attaches the same builds to Ledger releases.
# This configuration prepares independent plugin snapshots without publishing.
release:
disable: true
changelog:
disable: true
Loading
Loading