Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 0 additions & 27 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,32 +48,6 @@ jobs:
exit 1
fi

host:
name: Renderer host and live socket (macOS)
runs-on: macos-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: dtolnay/rust-toolchain@4360b52568e2003a75bf9bc1d59f33a8e3fc893c # stable
with:
components: clippy
- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
- name: Clippy host
run: cargo clippy -p qa-inspect-host --all-targets -- -D warnings
- name: Test real host transport
run: cargo test -p qa-inspect-host --release
- name: Test documented renderer-backed QA fixture
shell: zsh {0}
run: |
set -euo pipefail
cargo build -p qa-inspect-host -p ps-qa --release
target/release/ps-qa \
--app crates/qa-inspect-host/tests/fixture/ps-qa.ron \
qa-hosted fixture-text-entry \
--host target/release/qa-inspect-host \
--page crates/qa-inspect-host/tests/fixture/page.html \
--checks crates/qa-inspect-host/tests/fixture/checks

package:
name: Package boundaries
runs-on: macos-latest
Expand All @@ -96,6 +70,5 @@ jobs:
# protocol source without installing a persistent workspace patch;
# publish.yml repeats ordinary registry-backed `cargo package` after
# publishing the protocol and before uploading the driver.
cargo package -p qa-inspect-host
cargo --config 'patch.crates-io.blitz-control-protocol.path="crates/blitz-control-protocol"' \
package -p ps-qa
1 change: 0 additions & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,4 +79,3 @@ jobs:
# The host embeds tauri-runtime-blitz, which in turn consumes the
# protocol above. Keep it last so a new protocol and driver can ship
# before the corresponding runtime release reaches crates.io.
publish_if_new qa-inspect-host
1 change: 0 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ members = [
"crates/blitz-control-protocol",
"crates/ps-blitz-debug-control",
"crates/ps-qa",
"crates/qa-inspect-host",
]

[workspace.package]
Expand Down
65 changes: 43 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,32 @@
# ps-observability

The observability and QA stack for native Blitz applications. This workspace
keeps the protocol, transports, renderer host, driver, fixtures, and release
documentation together so the system has one ownership boundary.
keeps the protocol, transports, driver, and release documentation together so
the system has one ownership boundary.

```text
application ── tauri-runtime-blitz ── blitz-control-protocol ── ps-qa
qa-inspect-host ────────────────┘
chuzz-headless ────────┘

renderer embedder ── ps-blitz-debug-control ── WebDriver-style HTTP client
```

The headless host is not here. It used to be, as `qa-inspect-host`, and that
made two headless browsers: one a person browses with and one QA drives, with
the web platform in only the first of them. A host without `URLSearchParams`,
`matchMedia`, storage, the observers and `performance.getEntriesByType` blanks
every routed page before its first render, so every gap closed for the browser
had to be closed a second time here, by hand, or the harness measured a browser
nobody ships.

So the host is a mode of the browser now: `chuzz-headless`, in
[pathscale/chuzz](https://github.com/pathscale/chuzz), which loads through the
same loader and the same engine a tab uses and serves this protocol over the
same socket. `ps-qa` still links no renderer, because that constraint was
always about the socket rather than about which repository the host lives in.

These are two deliberate alternatives, not two stacked transports.
`blitz-control-protocol` is the typed MCP/JSON-RPC inspection plane used by
`tauri-runtime-blitz`, the headless host, and `ps-qa`.
Expand All @@ -30,34 +44,42 @@ instrumentation hooks but do not own a control server.
- `blitz-control-protocol`: transport-neutral observability domain types and
their MCP wire encoding. It deliberately has no renderer dependency.
- `ps-blitz-debug-control`: loopback WebDriver-style transport adapter.
- `qa-inspect-host`: a real renderer host for headless fixtures and CI.
- `ps-qa`: the lightweight driver, audit runner, and report generator.

## Quick start

From this workspace, install the driver and build the real headless renderer
host:
Install the driver, and build the host from the chuzz checkout beside this one:

```zsh
cargo install ps-qa
cargo build -p qa-inspect-host
cargo build --manifest-path ../chuzz/Cargo.toml --bin chuzz-headless --release
```

Serve one page in one terminal. A directory is served as a site, on a loopback
origin, so a built application's absolute asset paths and its client routing
both work; a single file or an `http(s)` URL is taken as given:

```zsh
../chuzz/target/release/chuzz-headless ../support.cafe/dist
```

Start the supplied renderer fixture in one terminal:
The host prints its descriptor path when ready. In a second terminal, drive it;
`ps-qa` discovers the live descriptor automatically:

```zsh
QA_INSPECT_PAGE="$PWD/crates/qa-inspect-host/tests/fixture/page.html" \
target/debug/qa-inspect-host
ps-qa find --role button
ps-qa audit
```

The host prints its descriptor path when ready. In a second terminal, run the
fixture's outcome check; `ps-qa` discovers the live descriptor automatically:
`qa-hosted` does both halves at once, launching the host, running a group of
checks and stopping it again:

```zsh
ps-qa \
--app crates/qa-inspect-host/tests/fixture/ps-qa.ron \
qa fixture-text-entry \
--checks crates/qa-inspect-host/tests/fixture/checks
ps-qa --app tests/ps-qa/ps-qa.ron \
qa-hosted \
--host ../chuzz/target/release/chuzz-headless \
--page dist \
--checks tests/ps-qa
```

This is a renderer-backed check: it enters text through the control protocol
Expand All @@ -82,10 +104,9 @@ browser remote-debugging port and must remain disabled in production builds.

The protocol, HTTP transport, and `ps-qa` driver are continuously checked on
Linux; `ps-qa` connects through a Unix-domain socket and currently supports
macOS and Linux, not Windows. The renderer-backed `qa-inspect-host` artifact is
currently validated on macOS. Linux renderer-host packaging remains explicit
follow-up work, so “headless” here means no window or display interaction—not a
claim that the current host package has completed Linux portability.
macOS and Linux, not Windows. The host's own platform status belongs to chuzz
now, and "headless" there means no window or display interaction rather than a
claim about which platforms the host has been packaged for.

## Releases

Expand Down
2 changes: 1 addition & 1 deletion crates/ps-qa/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[package]
name = "ps-qa"
description = "Drive a running Blitz app through its MCP control socket and assert what the renderer did"
version = "0.6.2"
version = "0.6.3"
edition = "2024"
rust-version = "1.88"
license = "MIT OR Apache-2.0"
Expand Down
13 changes: 9 additions & 4 deletions crates/ps-qa/src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,14 @@ pub struct Cli {
/// Multiply interaction and rendered-outcome deadlines on an overloaded
/// runner. The default remains the strict local latency contract; CI must
/// opt in explicitly rather than silently weakening every check.
///
/// `QA_TIMEOUT_SCALE` sets the same thing. A workflow sets it once for the
/// job rather than repeating a flag on every invocation, and the fleet's
/// workflows were already written that way while nothing read it.
#[arg(
long,
global = true,
env = "QA_TIMEOUT_SCALE",
default_value_t = 1.0,
value_parser = parse_timeout_scale
)]
Expand All @@ -112,8 +117,8 @@ pub struct Cli {
/// by node id.
///
/// For a host with no font catalogue, which is what a Linux CI runner is
/// and what `qa-inspect-host` is on any platform now that nothing enables
/// `system-fonts`. Text there shapes to no glyphs, so a control whose whole
/// and what any host built without `system-fonts` is on every platform.
/// Text there shapes to no glyphs, so a control whose whole
/// size comes from its label lays out at its line width and zero height --
/// `button:Open dialog` is in the tree, enabled, with a box, and is
/// rejected by the geometry gate that every coordinate-driven step needs.
Expand Down Expand Up @@ -764,7 +769,7 @@ mod tests {
"qa-hosted",
"fixture-text-entry",
"--host",
"qa-inspect-host",
"chuzz-headless",
"--page",
"page.html",
"--checks",
Expand All @@ -783,7 +788,7 @@ mod tests {
panic!("qa-hosted did not parse as the hosted QA command");
};
assert_eq!(selector.as_deref(), Some("fixture-text-entry"));
assert_eq!(host, PathBuf::from("qa-inspect-host"));
assert_eq!(host, PathBuf::from("chuzz-headless"));
assert_eq!(page, PathBuf::from("page.html"));
assert_eq!(checks, Some(PathBuf::from("checks")));
assert_eq!(startup_timeout, 30);
Expand Down
16 changes: 16 additions & 0 deletions crates/ps-qa/src/diagnostics.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,22 @@ pub(crate) async fn metrics(client: &mut Client) -> Result<RendererMetrics> {
}
}

/// Frame metrics when the host has them, and `None` when it says it does not.
///
/// A headless host is a document with no compositor: there are no frames, so
/// there is nothing to count. Commands that only print the numbers as context
/// around an action use this and carry on without them. Commands that exist to
/// judge frame timing -- `idle`, `frames`, `drift`, `blink` -- keep using
/// [`metrics`] and keep failing, because a zeroed reading from a host that
/// never painted would pass every one of them while proving nothing.
pub(crate) async fn metrics_if_supported(client: &mut Client) -> Result<Option<RendererMetrics>> {
match metrics(client).await {
Ok(metrics) => Ok(Some(metrics)),
Err(error) if crate::inspector::is_unsupported(&error) => Ok(None),
Err(error) => Err(error),
}
}

pub(crate) async fn transcript(client: &mut Client) -> Result<()> {
let answer = client
.diagnostics(&DiagnosticsRequest::Snapshot(SnapshotRequest {
Expand Down
14 changes: 14 additions & 0 deletions crates/ps-qa/src/inspector.rs
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,20 @@ impl std::fmt::Display for InspectorResponseError {

impl std::error::Error for InspectorResponseError {}

/// Whether a host refused a request because it does not have the thing asked
/// for, rather than because something went wrong.
///
/// The distinction matters for a headless host. It owns a document and no
/// compositor, so there are no frame metrics to report and saying "unsupported"
/// is the true answer. A caller that only wanted the numbers as context can
/// carry on without them; one that exists to judge frame timing cannot, and
/// should still fail.
pub fn is_unsupported(error: &eyre::Report) -> bool {
error
.downcast_ref::<InspectorResponseError>()
.is_some_and(|refusal| refusal.code == "unsupported")
}

impl Client {
fn queue_event(&mut self, event: DebugEvent) {
if self.events.len() == MAX_QUEUED_EVENTS {
Expand Down
38 changes: 27 additions & 11 deletions crates/ps-qa/src/interaction.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ use blitz_control_protocol::{
};
use eyre::{Result, bail, eyre};

use crate::diagnostics::metrics;
use crate::diagnostics::metrics_if_supported;
use crate::inspector::{Client, inspect};
use crate::target::{locate_control, selector_matches_node};
use crate::timing::{pace, sleep_pace};
Expand Down Expand Up @@ -315,7 +315,7 @@ pub(crate) async fn type_keys(client: &mut Client, count: usize, want: &str) ->
}))
.await?;

let before = metrics(client).await?;
let before = metrics_if_supported(client).await?;
let mut latencies = Vec::with_capacity(count);
for index in 0..count {
let letter = (b'a' + (index % 26) as u8) as char;
Expand All @@ -335,12 +335,10 @@ pub(crate) async fn type_keys(client: &mut Client, count: usize, want: &str) ->
latencies.push(started.elapsed().as_secs_f64() * 1000.0);
sleep_pace().await;
}
let after = metrics(client).await?;
let after = metrics_if_supported(client).await?;

report::show_latencies("keystrokes", count, &mut latencies);
report::show("before", &before);
report::show("after", &after);
report::show_delta(&before, &after, count);
report_frames(before.as_ref(), after.as_ref(), count);
Ok(())
}

Expand Down Expand Up @@ -385,19 +383,37 @@ pub(crate) async fn click_named(client: &mut Client, want: &str) -> Result<()> {
// of the viewport gets a `pointerdown` at a point nothing is at and no
// click at all. "Show 12 earlier messages" sat at y=-2246 and every attempt
// to press it read as the button doing nothing.
let before = metrics(client).await?;
let before = metrics_if_supported(client).await?;
let started = Instant::now();
client
.agent(&AgentControlRequest::Act(AgentAction::Click {
node_id: target_id,
}))
.await?;
let ack = started.elapsed().as_secs_f64() * 1000.0;
let after = metrics(client).await?;
let after = metrics_if_supported(client).await?;

println!("click acked in {ack:.1}ms");
report::show("before", &before);
report::show("after", &after);
report::show_delta(&before, &after, 1);
report_frames(before.as_ref(), after.as_ref(), 1);
Ok(())
}

/// The frame numbers either side of an action, when there are any.
///
/// A headless host has no compositor and says so. The click still happened and
/// its acknowledgement is still timed; what is missing is the frame context,
/// and saying that out loud is better than printing zeroes that read as a
/// renderer doing nothing.
fn report_frames(
before: Option<&blitz_control_protocol::RendererMetrics>,
after: Option<&blitz_control_protocol::RendererMetrics>,
actions: usize,
) {
let (Some(before), Some(after)) = (before, after) else {
println!("frames: the host reports no renderer metrics; it has no compositor");
return;
};
report::show("before", before);
report::show("after", after);
report::show_delta(before, after, actions);
}
12 changes: 10 additions & 2 deletions crates/ps-qa/src/qa.rs
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,8 @@ pub enum Expect {
///
/// That is not a hypothetical. All six of ps-blitz's activation fixtures
/// asserted their outcomes with [`PaintsNamed`](Expect::PaintsNamed) over
/// heading text, and all six failed the moment `qa-inspect-host` stopped
/// enabling `system-fonts`. They had only ever passed because the host
/// heading text, and all six failed the moment the host of the day stopped
/// enabling `system-fonts`. They had only ever passed because that host
/// carried a font catalogue, which is the thing a headless check must not
/// need.
///
Expand Down Expand Up @@ -575,6 +575,14 @@ pub fn checks(dir: Option<&std::path::Path>) -> Result<Vec<Check>, String> {
.filter_map(Result::ok)
.map(|entry| entry.path())
.filter(|path| path.extension().is_some_and(|ext| ext == "ron"))
// The application profile is not a check group.
//
// The documented layout puts `ps-qa.ron` beside the checks, and this
// glob then read it as one and failed the whole run with
// "Expected opening `[`" at line 5 -- pointing at the profile's
// syntax, which is correct, rather than at the file being included by
// mistake. The documented layout could not be used.
.filter(|path| path.file_name().is_some_and(|name| name != "ps-qa.ron"))
.collect();
// Name order, so a run is reproducible rather than dependent on whatever
// order the filesystem happens to hand back.
Expand Down
Loading
Loading