Repository navigation
main full-tier suite red: all 12 parity shards, 8/8 cargo-test-perry shards, doc-tests — full-suite-gate blocks releases #11560
Description
The full-tier suite on main is red across the board. The last two full runs:
- 2026-09-27 09:41, e379a7a (run 36310190277): 30 failed jobs, including all 12 parity shards and 8 of 8 cargo-test-perry shards.
- 2026-09-26 08:57, ebbc11f: 11 of 12 parity shards red.
The other failed jobs in the 09-27 run: doc-tests on macos and windows, binary-size (#11541), native-abi-evidence-packet, compiler-output-regression, lint (the grandfathered public-baseline step), then parity-aggregate and full-suite-gate. Releases wait on full-suite-gate, so nothing can be released. Every full-tier PR run inherits these failures too, as on #11413 and #11551.
The failures fall into buckets that look systemic rather than ~170 independent regressions:
- About 45 "compile TIMEOUT after 300s", e.g. test_gap_cron_cronjob, test_globalthis_builtins, test_gap_temporal_now, test_parity_vm and test_fastify_in_process. This suggests compile time (auto-optimize runtime rebuilds?) regressed, or a runner/cache problem.
- About 30 perry/ui "compile error", e.g. test_ui_comprehensive, test_ui_controls, test_issue_610_foreach and test_take_screenshot. Probably a single UI crate or link break.
- http2 wire tests, all "output mismatch", about 15 (test_gap_http2_wire_*).
- perry/tui, "output mismatch", about 12.
- The rest: individual output mismatches, e.g. test_inherited_store_chain_traps ("Cannot assign to read only property 'k'"), test_issue_449_new_target, test_parity_lodash / test_lodash_, test_jose_, test_jwt_*.
cargo-test-perry failures (09-27):
- issue_10104_runtime_data_imports (2)
- debug_symbols_* (3, file:line attribution)
- cjs_wrap object_create (cjs_wrap_builtin_require: object_create_on_builtin_namespace_get_prototype_of_not_sentinel fails on main (TypeError: Cannot convert undefined or null to object) #11542, fix in fix(runtime): a module namespace's own miss continues at its [[Prototype]] (#11542) #11557)
- issue_5989 strict global builtin assignment
- retained_chain_ring_keeps_every_next_edge_scannable
- async_hooks_lifecycle_regressions
- request_converts_dynamic_headers_init
- fetch lifecycle (fixed by fix(stdlib): scrub dead stack before the fetch lifecycle reader cases #11538, since merged)
doc-tests: getting-started/npm_packages.ts and stdlib/http/fastify_json.ts (both ~400–500 s, likely timeouts), and stdlib/utilities/snippets.ts.
The next step is triage by bucket: find the first bad full-tier run or bisect each bucket, starting with the compile timeouts and the UI compile errors.
Full failure list (09-27 run)
cargo-test-perry (1/8): test runtime_code_imports_keep_the_deferred_error ... FAILED
cargo-test-perry (1/8): test runtime_data_loaders_and_legacy_migration_match_bun ... FAILED
cargo-test-perry (2/8): test debug_symbols_attaches_file_line_to_property_read_on_undefined ... FAILED
cargo-test-perry (2/8): test debug_symbols_localizes_chained_read_at_the_throwing_access ... FAILED
cargo-test-perry (2/8): test default_build_keeps_anonymous_frame_for_property_read ... FAILED
cargo-test-perry (3/8): test cjs_wrap_object_create_on_builtin_namespace_get_prototype_of_not_sentinel ... FAILED
cargo-test-perry (4/8): test strict_assignment_to_existing_global_builtin_writes_through ... FAILED
cargo-test-perry (5/8): test retained_chain_ring_keeps_every_next_edge_scannable ... FAILED
cargo-test-perry (7/8): test async_hooks_lifecycle_regressions ... FAILED
cargo-test-perry (8/8): test request_converts_dynamic_headers_init ... FAILED
cargo-test: test fetch::lifecycle::lifecycle_tests::readers_survive_a_full_trace_that_releases_ids_of_their_own_kind ... FAILED
doc-tests (macos-14, per: FAIL getting-started/npm_packages.ts (411999 ms)
doc-tests (macos-14, per: FAIL stdlib/http/fastify_json.ts (333532 ms)
doc-tests (macos-14, per: FAIL stdlib/utilities/snippets.ts (514 ms)
doc-tests (windows-2022,: FAIL getting-started/npm_packages.ts (496535 ms)
doc-tests (windows-2022,: FAIL stdlib/http/fastify_json.ts (492399 ms)
doc-tests (windows-2022,: FAIL stdlib/utilities/snippets.ts (897 ms)
parity (1): FAIL test_gap_2159_defineproperty_class_prototype (output mismatch)
parity (1): FAIL test_gap_cron_cronjob (compile TIMEOUT after 300s — killed, not rejected)
parity (1): FAIL test_gap_http2_alpn_secure (output mismatch)
parity (1): FAIL test_gap_http2_wire_server_ping (output mismatch)
parity (1): FAIL test_gap_perfhooks_3088_3008_3010_3011 (output mismatch)
parity (1): FAIL test_inherited_store_chain_traps (output mismatch)
parity (1): FAIL test_issue_4913_atomics_cross_thread (output mismatch)
parity (1): FAIL test_issue_556_table_array (compile error)
parity (1): FAIL test_issue_645_chained_method_on_null_obj (output mismatch)
parity (1): FAIL test_issue_7302_thread_throws (output mismatch)
parity (1): FAIL test_issue_915_native_module_after_async_resume (output mismatch)
parity (1): FAIL test_lodash_max_min (output mismatch)
parity (1): FAIL test_reflect_metadata (output mismatch)
parity (1): FAIL test_v8_self_contained (output mismatch)
parity (10): FAIL test_crypto_hmac_dynamic_alg (compile TIMEOUT after 300s — killed, not rejected)
parity (10): FAIL test_effect_pipe_map (output mismatch)
parity (10): FAIL test_gap_10392_fetch_json_key_order (compile TIMEOUT after 300s — killed, not rejected)
parity (10): FAIL test_gap_array_dyndispatch_2803_2808_2791 (compile TIMEOUT after 300s — killed, not rejected)
parity (10): FAIL test_gap_fetch_handle_in_operator (compile TIMEOUT after 300s — killed, not rejected)
parity (10): FAIL test_gap_http2_wire_goaway_send (output mismatch)
parity (10): FAIL test_gap_readable_stream_tee_pull (compile TIMEOUT after 300s — killed, not rejected)
parity (10): FAIL test_gap_temporal_now (compile TIMEOUT after 300s — killed, not rejected)
parity (10): FAIL test_globalthis_builtins (compile TIMEOUT after 300s — killed, not rejected)
parity (10): FAIL test_issue_1120_fastify_buffer (compile TIMEOUT after 300s — killed, not rejected)
parity (10): FAIL test_issue_1723_require_stdlib_subnamespace (output mismatch)
parity (10): FAIL test_issue_1867_audio_playback (compile error)
parity (10): FAIL test_issue_237_streams_user_source (compile TIMEOUT after 300s — killed, not rejected)
parity (10): FAIL test_issue_403_404_animated_input (output mismatch)
parity (10): FAIL test_issue_553_mobile_widgets (compile error)
parity (10): FAIL test_issue_927_jwt_verify_returns_object (output mismatch)
parity (10): FAIL test_issue_pino_sorting_order_undefined (output mismatch)
parity (10): FAIL test_jwt_sign_dynamic_alg (output mismatch)
parity (10): FAIL test_parity_vm (compile TIMEOUT after 300s — killed, not rejected)
parity (11): FAIL test_gap_http2_wire_goaway_server (output mismatch)
parity (11): FAIL test_issue_405_perry_tui_phase3_5 (output mismatch)
parity (11): FAIL test_issue_859_native_promise_pin (output mismatch)
parity (11): FAIL test_issue_915_jwt_sign (output mismatch)
parity (11): FAIL test_parity_lodash (output mismatch)
parity (11): FAIL test_ramda_user_import (compile error)
parity (12): FAIL test_gap_http2_wire_refused_stream (output mismatch)
parity (12): FAIL test_issue_1777_prototype_borrow (output mismatch)
parity (12): FAIL test_issue_1934_spawn_reactor (output mismatch)
parity (12): FAIL test_issue_2153_http_typed_feedback_dispatch (output mismatch)
parity (12): FAIL test_issue_3580_arguments_object_semantics (output mismatch)
parity (12): FAIL test_issue_640_navstack_textfield (compile error)
parity (12): FAIL test_parity_lru_cache (output mismatch)
parity (2): FAIL test_ads_compile_smoke (output mismatch)
parity (2): FAIL test_bun_plugin (compile TIMEOUT after 300s — killed, not rejected)
parity (2): FAIL test_gap_http2_e2e_streams (output mismatch)
parity (2): FAIL test_gap_http2_wire_server_settings (output mismatch)
parity (2): FAIL test_gap_node_crypto_buffer (compile TIMEOUT after 300s — killed, not rejected)
parity (2): FAIL test_gap_typed_array_dynamic_index_access (compile TIMEOUT after 300s — killed, not rejected)
parity (2): FAIL test_issue_10155_textfield_singleline (compile error)
parity (2): FAIL test_issue_1495_image_systemname (compile error)
parity (2): FAIL test_issue_2022_canvas_draw_image (compile error)
parity (2): FAIL test_issue_320_readable_stream (compile TIMEOUT after 300s — killed, not rejected)
parity (2): FAIL test_issue_4971_tls_connect_options (output mismatch)
parity (2): FAIL test_issue_556_table_concat (compile error)
parity (2): FAIL test_lodash_sum (output mismatch)
parity (2): FAIL test_parity_cluster (output mismatch)
parity (2): FAIL test_parity_stream_web (output mismatch)
parity (3): FAIL test_fastify_in_process (compile TIMEOUT after 300s — killed, not rejected)
parity (3): FAIL test_gap_9616_readable_to_web_fs (compile TIMEOUT after 300s — killed, not rejected)
parity (3): FAIL test_gap_http2_wire_session_lifecycle (output mismatch)
parity (3): FAIL test_gap_intl_component_locale (compile TIMEOUT after 300s — killed, not rejected)
parity (3): FAIL test_gap_prop_plan_cache_invalidation (output mismatch)
parity (3): FAIL test_gap_webdata_2612_2606_2959 (compile TIMEOUT after 300s — killed, not rejected)
parity (3): FAIL test_hono_bundle (output mismatch)
parity (3): FAIL test_issue_610_foreach (compile error)
parity (3): FAIL test_issue_679_box_runtime_children (output mismatch)
parity (3): FAIL test_issue_9842_form_data_blob_upload (compile TIMEOUT after 300s — killed, not rejected)
parity (3): FAIL test_parity_commander (output mismatch)
parity (3): FAIL test_parity_streams_byob (compile TIMEOUT after 300s — killed, not rejected)
parity (3): FAIL test_ui_adbanner_smoke (compile error)
parity (3): FAIL test_ws_static_constants_6117 (output mismatch)
parity (4): FAIL test_dynamic_import_data_10104 (output mismatch)
parity (4): FAIL test_gap_10421_function_ctor_as_value (compile TIMEOUT after 300s — killed, not rejected)
parity (4): FAIL test_gap_10476_builtin_named_guarded_receivers (compile TIMEOUT after 300s — killed, not rejected)
parity (4): FAIL test_gap_10589_new_cast_builtin_shadow (compile TIMEOUT after 300s — killed, not rejected)
parity (4): FAIL test_gap_11447_crypto_sign_followups (compile TIMEOUT after 300s — killed, not rejected)
parity (4): FAIL test_gap_http2_session_isolation (output mismatch)
parity (4): FAIL test_gap_http2_wire_settings_ack (output mismatch)
parity (4): FAIL test_gap_intl_ctor_mechanics_5835 (compile TIMEOUT after 300s — killed, not rejected)
parity (4): FAIL test_gap_response_null_body_status_10360 (compile TIMEOUT after 300s — killed, not rejected)
parity (4): FAIL test_issue_351_media_playback (compile error)
parity (4): FAIL test_issue_358_perry_tui_phase1 (output mismatch)
parity (4): FAIL test_issue_442_inline_button_bg (compile error)
parity (4): FAIL test_issue_538_background_tasks (compile error)
parity (4): FAIL test_issue_610_smoke (compile error)
parity (4): FAIL test_issue_679_perry_tui_hooks (output mismatch)
parity (4): FAIL test_issue_763_reactive_textfield (compile error)
parity (4): FAIL test_jose_sign (output mismatch)
parity (4): FAIL test_perry_tui_inkcompat_usestdout (output mismatch)
parity (4): FAIL test_ui_comprehensive (compile error)
parity (5): FAIL test_gap_node_module_3118plus (output mismatch)
parity (5): FAIL test_issue_1021_rxjs_reexport_methods (output mismatch)
parity (5): FAIL test_issue_10890_parent_class_object_reads (output mismatch)
parity (5): FAIL test_issue_336_class_keys_collision (expected-output mismatch)
parity (5): FAIL test_issue_4449_thread_promise_void (output mismatch)
parity (5): FAIL test_issue_5912_url_new_local_shadow (output mismatch)
parity (5): FAIL test_issue_679_perry_tui_text_styling (output mismatch)
parity (5): FAIL test_issue_764_state_at_module_init (compile error)
parity (5): FAIL test_jose_signverify_roundtrip (compile error)
parity (5): FAIL test_parcel_watcher_facade (output mismatch)
parity (5): FAIL test_parity_crypto (output mismatch)
parity (5): FAIL test_phase2v3_3_show_toast_set_text (compile error)
parity (5): FAIL test_ui_controls (compile error)
parity (5): FAIL test_xmod_no_ctor_arg_forward (output mismatch)
parity (6): FAIL test_better_sqlite3_raw (compile TIMEOUT after 300s — killed, not rejected)
parity (6): FAIL test_date_fns_format (output mismatch)
parity (6): FAIL test_gap_10424_function_ctor_to_string_args (compile TIMEOUT after 300s — killed, not rejected)
parity (6): FAIL test_gap_10523_random_uuid_entropy_cache (compile TIMEOUT after 300s — killed, not rejected)
parity (6): FAIL test_gap_gc_next_request_import (compile TIMEOUT after 300s — killed, not rejected)
parity (6): FAIL test_gap_http2_wire_client_ping (output mismatch)
parity (6): FAIL test_gap_v8_2_3680plus (output mismatch)
parity (6): FAIL test_http_server_unref_5011 (output mismatch)
parity (6): FAIL test_issue_1140_buffer_index_runtime (expected-output mismatch)
parity (6): FAIL test_issue_2656_weakref_finalization_gc (output mismatch)
parity (6): FAIL test_issue_358_perry_tui_phase3_flexbox (output mismatch)
parity (6): FAIL test_issue_402_403_404_405_406_holistic (output mismatch)
parity (6): FAIL test_issue_4831_stripe_proto_methods (output mismatch)
parity (6): FAIL test_issue_617_inline_await_fetch_with_auth (output mismatch)
parity (6): FAIL test_issue_679_useState_gc_root (output mismatch)
parity (6): FAIL test_issue_7769_thread_class_dispatch (output mismatch)
parity (6): FAIL test_parity_dotenv (compile error)
parity (6): FAIL test_parity_util (output mismatch)
parity (6): FAIL test_ui_drag_drop (compile error)
parity (7): FAIL test_better_sqlite3_v8_bridge (compile TIMEOUT after 300s — killed, not rejected)
parity (7): FAIL test_decorators_builtin_metadata_9501 (compile TIMEOUT after 300s — killed, not rejected)
parity (7): FAIL test_gap_http2_wire_flow_control (output mismatch)
parity (7): FAIL test_gap_json_lazy_defineproperty_index (output mismatch)
parity (7): FAIL test_issue_2671_2672_2709_3108_3045_2985_3050_3052_2960_process_cohort (output mismatch)
parity (7): FAIL test_issue_358_perry_tui_phase4_5_widgets (output mismatch)
parity (7): FAIL test_issue_402_perry_tui_table_tabs (output mismatch)
parity (7): FAIL test_parity_uuid (compile error)
parity (7): FAIL test_pdf_create_smoke (output mismatch)
parity (7): FAIL test_ui_on_keydown_smoke (compile error)
parity (8): FAIL test_gap_10427_webcrypto_method_identity (compile TIMEOUT after 300s — killed, not rejected)
parity (8): FAIL test_gap_10479_instanceof_value_kinds (compile TIMEOUT after 300s — killed, not rejected)
parity (8): FAIL test_gap_2514_settracesigint (output mismatch)
parity (8): FAIL test_gap_gc_index_set_bounded_globalthis_ta_rooting (compile TIMEOUT after 300s — killed, not rejected)
parity (8): FAIL test_gap_http2_wire_frame_errors (output mismatch)
parity (8): FAIL test_gap_intl_host_timezone (compile TIMEOUT after 300s — killed, not rejected)
parity (8): FAIL test_gap_ratelimiter_memory (compile TIMEOUT after 300s — killed, not rejected)
parity (8): FAIL test_issue_1193_cheerio_chain (output mismatch)
parity (8): FAIL test_issue_339_sqlite_stmt_all_bound_params (compile TIMEOUT after 300s — killed, not rejected)
parity (8): FAIL test_issue_358_perry_tui_phase4_widgets (output mismatch)
parity (8): FAIL test_issue_4034_object_literal_semantics (output mismatch)
parity (8): FAIL test_issue_699_runtime_ui_surface (output mismatch)
parity (8): FAIL test_issue_7981_thread_shape_stamp_parent (output mismatch)
parity (8): FAIL test_take_screenshot (compile error)
parity (8): FAIL test_turnloop_p6_smtp (output mismatch)
parity (8): FAIL test_ui_phase4 (compile error)
parity (9): FAIL test_gap_bun_sql_9604 (compile TIMEOUT after 300s — killed, not rejected)
parity (9): FAIL test_gap_http2_wire_goaway_recv (output mismatch)
parity (9): FAIL test_gap_intl_locale_separators_7429_7428 (compile TIMEOUT after 300s — killed, not rejected)
parity (9): FAIL test_issue_341_typed_field_native (compile TIMEOUT after 300s — killed, not rejected)
parity (9): FAIL test_issue_449_new_target (output mismatch)
parity (9): FAIL test_issue_4977_gc_toplevel_locals (output mismatch)
parity (9): FAIL test_parity_nanoid (compile error)
parity (9): FAIL test_parity_validator (output mismatch)
parity (9): FAIL test_perry_gc_module (output mismatch)
parity (9): FAIL test_three_like_native_class_descriptors (output mismatch)
parity (9): FAIL test_ui_text_alignment (compile error)
Activity
proggeramlug commentedon Sep 27, 2026
Triage notes so far (hello-world-perf):
test_inherited_store_chain_trapsis a test bug. The repo's package.json is"type": "module", so the file runs as strict ESM, and both node and perry correctly throw. The fix is test: the chain-store traps run as an ES module, so refused stores throw #11561, a test-only change.test_parity_lodash,test_lodash_max_min,test_jose_*andtest_jwt_*fail because node itself crashes innode:internal/modules/package_json_reader:301on the runner and can't load those packages. That points at the runner's npm install or package inventory. A suspect is 5289d6e ("re-derive inventories after the mongodb binding removal"), which touched package.json. Not yet verified.test_issue_449_new_targetis a real perry bug and reproduces on main:new.target?.name ?? "undefined"in a plain (non-new) call throws instead of short-circuiting. hello-world-perf is fixing it. Possibly related: 071643e (super() to a runtime-value parent (ns.Base, local alias) runs the parent with new.target undefined #11147).changelog: fragment for #11563
changelog: fragment for #11564
test-parity: delete five linux allowlist entries the full tier proves…
changelog: note the stale-entry deletions in #11564's fragment
ci(parity): raise the shard cap to 300 min
changelog: note the shard cap in #11563's fragment
proggeramlug commentedon Sep 27, 2026
Triage of the two biggest buckets (run 36310190277)
(a) "compile TIMEOUT after 300s" is the harness budget, not the compiler (verified). PR #11563
parity-compile-errors-Linux-shard-*) ends onauto-optimize: rebuilding runtime+stdlib (...). So the time is the per-feature-set runtime+stdlibcargo buildintotarget/perry-auto-<hash>, which takes about 200 to 330 s on hosted runners. It isn't codegen and it isn't linking.run_parity_tests.shgives the 900 s toolchain budget only to ext-routed fixtures:(auto_optimize_on || force_no_auto) && ext_routed. The full tier runs the whole corpus with auto-optimize on, so plain fixtures also rebuild, but they only get 300 s. None of the 42 is ext-routed.auto_optimize_on. The fast tiers don't change:PERRY_SKIP_BUILD=1implies no-auto, so plain compiles there keep the 300 s bound.tests/test_parity_compile_budget.sh, wired into lint. It fails on the old predicate and passes on the new one, on macOS and Linux.(b) perry/ui "compile error" is almost all pre-existing and allowlisted (verified). PR #11564
perry/ui imported but libperry_ui_gtk4.a not found. The Linux parity job builds no perry-ui-gtk4 archive.ci-envfor linux since parity: 2026-08-17 dark-debt audit — 93 parity + 27 compile failures unlisted after six dark weeks (90.7% aggregate) #8271 (2026-08-17), so the ratchet does not count them.test_issue_10155_textfield_singlelineis a NEW failure. It was added on 09-13 (8f9b407) without an allowlist entry and has been NEW in every full run since. test-parity: allowlist test_issue_10155 (gtk4 family) and delete five stale linux entries #11564 adds the entry.test_guarded_raw_numeric_arrays,test_issue_3558_computed_accessors,test_node_http2_basic,test_node_https_basic: PASS in all six full runs, 09-22 to 09-27.test_issue_3199_3200_tls_server_tlssocket: passing since 09-26.Other buckets (not fixed here)
npm_packages.ts,fastify_json.tsandutilities/snippets.tsfail at link withUndefined symbols: _fastify/_nanoid/_uuidv4, afterWarning: Could not resolve import 'fastify' | 'uuid' | 'nanoid'. That lines up with the native-binding removals (fastify dcb8a76, nanoid 7b47239, uuid refactor(stdlib): remove uuid native binding #10701): the doc-test compile can't resolve the real packages. The 400 s wall time is the auto-optimize rebuild plus the bcrypt ext build before the link fails.changelog: fragment for 11571
changelog: fragment for 11572
changelog: fragment for 11573
changelog: fragment for 11574
changelog: fragment for 11575
changelog: fix a quote in #11575's fragment
changelog: fragment for 11578
34 remaining items
fix(hir): a native factory's result is a tagged receiver when used di…
changelog: fragment for 11584
fix(runtime): an unstamped subclass instance reads the class evaluati…
changelog: fragment for 11587
fix(runtime): a native URL links to %URL.prototype%, not to whatever …
changelog: fragment for 11592
fix(http): server.close() keeps a connection with a half-received req…
changelog: fragment for 11596
changelog: fragment for 11600
changelog: fragment for 11602
proggeramlug commentedon Sep 28, 2026
Round 4
newof the class expression itself stamps. Every other instance was treated as belonging to the first evaluation, so instances of a declared subclass (class B extends make("second")) read the first evaluation's captures. The fix falls back to the instance class's heritage walk, the same walk inherited statics already use. The fixture now matches node byte for byte. New test fails without the fix.function URLvsglobalThis.URL(test_issue_5912)URL.prototype).create_url_objectlooked the prototype up through the current global binding. It now uses a recorded %URL.prototype%. New test fails without the fix. The pre-existing exposure of module functions onglobalThisis split out as #11591.server.close()kills a mid-request socket (test_issue_4971)is_busydropped tokio'sread_activecase, so bytes of an unfinished request head no longer counted as busy. New gap testtest_gap_11586_close_keeps_half_receivedfails without the fix. The 39 http/net/https/tls/http2 gap tests give byte-identical output to main.initreceived a pointer-tagged raw timer id while the call returned a separately minted handle. The handle is now built first and passed as the resource. New test fails without the fix (false false). All 186 async_hooks node-suite fixtures are unchanged where they differ from node.100 Continuewas sent only after the request was dispatched at END, which deadlocks a client waiting to send its body. It is now sent when the head arrives. The fixture matches node; without the fix it times out, exactly as in CI.fix(runtime): a timer's handle is the resource its async_hooks init r…
changelog: fragment for 11600
fix(http): answer Expect: 100-continue when the head arrives, not aft…
changelog: fragment for 11602