Skip to content

soc: apple: vote GPU activity to the PMP so memory leaves idle DVFS on t600x - #334

Open
joshuaswarren wants to merge 19 commits into
omacom:integration/aurorafrom
joshuaswarren:w72/t600x-agx-vote
Open

joshuaswarren wants to merge 19 commits into
omacom:integration/aurorafrom
joshuaswarren:w72/t600x-agx-vote

Conversation

@joshuaswarren

@joshuaswarren joshuaswarren commented Nov 3, 2026 •

Copy link
Copy Markdown

GPU reads on the M1 Max (t6001) top out at 241 GB/s on Linux. The same chip reads 361 GB/s on macOS. One bit in the PMP report closes that gap: with it set, the same read microbenchmark reaches 368 GB/s, above the macOS figure.

The bit. AGX is soc-device record 9 on t6001. Bit 9 of the report's PS-REQ word is the AGX power request. With it clear, the PMP ignores the GPU's bandwidth demand and keeps memory at its idle floor. This is the t6030 mechanism from 9bb74767dcd4, on the t600x report layout.

What was measured. A test module sets and clears bit 9 by hand. It uses the same register sequence as apple_pmp_report_set_state(): read +0xf80, write +0x107c0, acknowledge +0x1000, status +0x10. On a 16-inch M1 Max it moved a GPU read microbenchmark from 241 to 368 GB/s. The same microbenchmark reads 361 GB/s on macOS. A dependent FMA kernel runs at 4.22 TFLOPS on Linux and 4.28 on macOS. The shader clock is not what differs. The gap is memory.

Decode effect, measured with this patch. A 16-inch M1 Max booted the same code in its lab-kernel form (the 7.1.12 tree this branch was first written against), with pmp_report.agx_vote 0 against 1 and a 250 ms hold set at run time (the image's compiled default was 100 ms). Each arm is a fresh process, alternating off and on. A 2B 4-bit model decodes 93.88 to 117.54 tok/s (+25.2%) at 512 tokens of context and 81.94 to 105.22 tok/s (+28.4%) at 8192. The 8192-token prefill takes 6.30 s instead of 6.78 s (7.5% faster). The read microbenchmark goes from 242 to 249 GB/s up to 368 GB/s. The spread inside an arm is under 1% for decode. Holding bit 9 by hand for the whole process gave the same decode rates (117.59 and 105.21 tok/s), so the vote and its hold cost nothing in 2B decode. The longest gap between two tokens was 65.8 ms.

The same A/B on a 9B 4-bit model: decode 33.81 to 42.82 tok/s (+26.7%) at 512 tokens and 29.01 to 37.55 tok/s (+29.4%) at 8192. The 8192-token prefill takes 26.5 s instead of 29.5 s (10.3% faster), and the read microbenchmark reaches 366 GB/s. The vote again matches the hand-held bit (42.82 against 42.92 and 37.55 against 37.61 tok/s). The longest gap between two tokens was 118 ms, once per run at a fixed token index, and it also appears in the runs with the vote off. A 100 ms hold would have dropped the request inside that gap. The 250 ms default covers it, and no run with the vote on shows a longer token interval.

Writes too. The vote lifts the write roof as well. On the same machine and kernel, a write bandwidth microbenchmark reads 193.1 GB/s with the vote off (two cells). With the vote on it reads 343.2 GB/s, which is 77.7 percent more. macOS writes 384.9 GB/s on that chip, so the remaining gap is 12 percent, not a factor of two. The read roof measured again in the same session was 247.8 GB/s off and 369.2 on, against 362.0 on macOS.

Design. The GPU driver reports operations in flight. It calls apple_pmp_agx_activity_begin() when a submission starts and apple_pmp_agx_activity_end() when it finishes. pmp-report sets bit 9 when the first operation starts. It clears the bit 250 ms after the last one ends. A decode loop with a submission every 10 ms keeps one vote. The bit is also cleared at once at GPU unbind and when the report goes away. The request is never left set past the GPU's power-domain release, which matches the t6030 fix. The first operation comes after firmware start. The t6030 note says an earlier report is acknowledged without scaling.

  • Off by default. pmp_report.agx_vote=1 turns it on. pmp_report.agx_vote_idle_ms (default 250, writable at run time) sets the hold. With it off nothing is written, so one boot can run an A/B.
  • apple,t6001 only. t6000 is untested.
  • A set that is not acknowledged is cleared again and not retried for a second. A PMP that is not running gets no write, and the vote retries after a second.
  • The Rust side is one unsafe call that takes no arguments, in start_op(), in end_op() and in unbind. The begin call sits before the guard exists, so a failed kick still pairs with the guard's drop.
  • The decisions are plain functions in pmp-agx-vote.h with no kernel dependency. A userspace test runs thirteen cases with a fake clock and a fake PMP bit under ASan and UBSan. One case is the 2^32 ms wrap of jiffies_to_msecs(). A reviewer showed that a 64-bit compare never decided the idle clear across that wrap, so the clock is 32-bit.

The branch has two more commits on top of the vote. apple_pmp_report_set_request() is a locked, acknowledged update of one PS-REQ bit for clients outside this driver, such as the ANE driver, so they stop writing the word themselves. The next commit makes it refuse the AGX bit on t600x with -EBUSY, so the vote stays the only writer of bit 9, and documents that a caller clears its request after -ETIMEDOUT.

Checked so far:

  • This branch compiles on integration/aurora with the lab kernel config, clang 22 and rustc 1.93.1, at each commit. pmp-report.o defines apple_pmp_agx_activity_begin, _end, _release and apple_pmp_report_set_request. The GPU crate references the three activity calls, and the generated Rust bindings declare them. No warning comes from the new code.
  • The same vote logic was linked into a full 7.1.12 lab kernel image. Nothing was lost from the exported symbol list, and three exports were added.

Not covered:

  • A full image link on this exact tree. The compile check does not link, and the lab-kernel form of the code was linked into a full image.
  • A KUnit version of the policy test. It lives in a userspace harness for now, and will follow as a separate pull request.
  • The A/B ran two vote-on and three vote-off processes per decode configuration. The spread inside an arm is under 1%, and the effect is 25 to 30 times that.
  • A hold between 118 ms and 250 ms was not tried. A drop inside the 118 ms gap would show as a longer token interval in a vote-on run, and none does.

…n t600x

On the M1 Max (t6001) the PMP keeps memory at its idle floor for a busy
GPU until the AGX power request, bit 9 of the PMP report's PS-REQ word, is
set. Nothing sets it on Linux, so GPU reads top out at 241 GB/s. The same
chip reads 361 GB/s on macOS.

Report GPU activity to the PMP. The GPU driver calls
apple_pmp_agx_activity_begin() when a submission starts and
apple_pmp_agx_activity_end() when it finishes. pmp-report sets bit 9 when
the first operation starts and clears it 100 ms after the last one ends.
It also clears it at once when the GPU driver unbinds or the report goes
away, so the request is never left set past the GPU's power-domain
release. The register sequence is the one apple_pmp_report_set_state()
already uses for report entries: read 0xf80, write 0x107c0, poll 0x1000,
status 0x10, 50 ms acknowledge. A set that is not acknowledged is cleared
again and not retried for a second.

The vote is off by default. pmp_report.agx_vote=1 turns it on, and
pmp_report.agx_vote_idle_ms (default 100) sets the hold. With it off
nothing is written. It applies to apple,t6001 only; t6000 is untested.

The decisions (set, clear, hold, retry, wrap of the 32-bit millisecond
clock) are plain functions in pmp-agx-vote.h with no kernel dependency.

Setting the bit by hand with a test module on a 16-inch M1 Max moved a GPU
read microbenchmark from 241 to 368 GB/s. This patch has not yet been run
on hardware.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…one power request bit

The PS-REQ word of the PMP report is shared by every client of the PMP:
the report entries (display, ISP, video decode, the AFNC IOAs), the T6001
AGX vote (bit 9) and, out of tree, the ANE clock level of omarchy-ane
(bit 10). The report entries and the AGX vote read-modify-write the word
under rep->lock. A module outside this driver could only map the window
and read-modify-write it without that lock, so a concurrent write of
another bit could be lost.

apple_pmp_report_set_request(report_np, id, enable, no_ack) is the one
writer a client outside this file may use. It calls the same
apple_pmp_report_set_state() the entries use, so it takes the same lock
and does the same acknowledge poll (at most 50 ms). The caller owns the
bit it asks for.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

Copy link
Copy Markdown
Author

GREEN on f7ad8cf

Source review of the combined head: 689f2f9 (the AGX vote port) plus f7ad8cf (export of apple_pmp_report_set_request), against merge-base 2a76ae3 on integration/aurora. Six files, +378 lines.

Reviewer identity: routed by Main to Sonnet (Claude); the model id is unverifiable from inside the session and Main's routing notice is the source. The implementer is a different party; implementer model unverified. I did not edit the PR and I did not build it; the compile check on the build host is a separate step.

Main's three questions, answered from the code at this head:

  1. Lock. The export calls apple_pmp_report_set_state(), the one function the report entries (three call sites), the AGX vote (through apple_pmp_agx_write()) and the export all use. It takes rep->lock with spin_lock_irqsave around the single read-modify-write (read tgt_read, clear or set the bit, write tgt_write) and drops it before the status read and the acknowledge poll (readq_poll_timeout_atomic, 100 us step, 50 ms limit). So the poll is outside the lock, and a slow acknowledge of one caller does not hold up another caller's write.
  2. Id range. id > 63 returns -EINVAL before anything is touched; BIT_ULL(id) is then defined for the whole 64-bit word. The T6001 soc-device table has 34 records, so 34 to 63 are accepted but name no device; harmless writes, and the kernel-doc says 0 to 63, which is accurate.
  3. Another bit cannot disturb the vote. Every writer does a full read-modify-write under the same lock, so a write of bit N keeps the other 63 bits as read inside the lock, bit 9 included, and the vote's own write does the same for theirs. The vote does not read the word back to decide anything (its state is voted, users, idle_at), so another caller's change of another bit changes none of its decisions. The acknowledge poll compares only the caller's own bit.

Also checked:

  • The export's list walk is rcu_read_lock plus list_for_each_entry_rcu; the remove path does list_del_rcu then synchronize_rcu before the device resources go, so a caller inside the walk cannot see a freed report. The export keeps no pointer to the report after returning.
  • 689f2f91: pmp-agx-vote.h is byte-identical to the header I reviewed (sha256 fc44668d, 32-bit clock fix included). The glue is the same logic as the reviewed lab patch with the port's renames; the remove path clears the vote before unlinking the report (stop, then list_del_rcu), and cancel_delayed_work_sync is called outside the mutex.
  • gpu.rs: in this tree start_op creates the OpGuard before kick_firmware()? and begin sits directly before it with no early return between, so every path pairs begin with end (this also fixes the counter leak on a failed kick that I noted in the lab tree). One OpGuard construction site, two start_op callers (queue/mod.rs 376 and 701).

Notes, none blocking:
N1. "On the T6001 bit 9 belongs to the AGX vote; other callers must not use it" is documentation only. A caller that passes id 9 can clear the vote's bit while the vote still believes it has voted, and nothing detects it. An -EBUSY for id 9 on the t600x report, when called from outside the vote, would enforce it.
N2. On -ETIMEDOUT the request stays written (the vote clears it itself; the export does not). The kernel-doc should say that the caller must clear it.
N3. Like the entries and the vote, the export takes no runtime-PM reference on the report device. The report's PMS SRAM domain stays on while the driver is bound (measured on this hardware: runtime PM is disabled on the report device), and only bound reports are on the list, so this holds today; it is the same assumption the existing paths make.

Not covered: no build, no boot, no hardware run from me. Measured elsewhere on the same hardware: setting bit 9 raised the Linux read roof from 241 to 368 GB/s and 8192-context prefill on the 2B by 6.5%; the decode result with the vote is pending.

…ment clearing

apple_pmp_report_set_request() lets a client outside pmp-report update one
bit of the PS-REQ word. On the T6001 bit 9 is the AGX request, which the
GPU vote owns: a second writer could clear it while the GPU works, or set
it and leave it set. Refuse it from the export with -EBUSY on a t600x
report. The vote itself writes through apple_pmp_report_set_state() and is
not affected.

Document what a caller must do after -ETIMEDOUT: the request was written
but not acknowledged and stays in the word, so the caller clears it with
@enable false and @no_ack true before giving up.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

Copy link
Copy Markdown
Author

GREEN on e033631

Source re-review of the combined head, one commit (e033631, "keep the AGX bit out of the export, document clearing") over f7ad8cf, which I reviewed and approved earlier. It closes my notes N1 and N2 from that review.

Reviewer identity: routed by Main to Sonnet (Claude); the model id is unverifiable from inside the session and Main's routing notice is the source. The implementer is a different party; implementer model unverified. I did not edit the PR and I did not build it; the compile check is a separate step.

Delta checked at the exact head (fetched refs/pull/334/head, which resolved to the sha above, and confirmed again just before posting):

  1. Scope: one commit, one file (drivers/pmdomain/apple/pmp-report.c, +8 -4). The vote commit, pmp-agx-vote.h, gpu.rs, driver.rs, the bindings include and the header are untouched, so every check in my earlier review still stands.
  2. N1 (enforce bit 9): inside the report match, if (id == APPLE_PMP_T6001_AGX_ID && rep->offsets == &apple_pmp_offsets_t600x) ret = -EBUSY; else ret = apple_pmp_report_set_state(...). APPLE_PMP_T6001_AGX_ID is defined at line 87 and apple_pmp_offsets_t600x at line 1167, both before the export (line 1550). The id above 63 check still returns -EINVAL first. The vote is unaffected: it reaches apple_pmp_report_set_state() directly through apple_pmp_agx_write() (line 1368), not through the export, and there is no in-tree caller of the export. A request for another bit still goes through the same locked read-modify-write, so it cannot disturb the vote.
  3. N2 (document clearing): the kernel-doc now says that after -ETIMEDOUT the request was written but not acknowledged and stays in the word, and the caller must clear it with @enable false and @no_ack true; the Return line lists -EBUSY. Both statements match the code.

Observation, no action: the guard keys on the t600x offsets, so it also refuses id 9 on a T6000 report; the comment names the T6001. The refusal is conservative and the T6000 AGX bit would be the same index, so I see no harm.

Not covered: no build, boot or hardware run from me. The lab-tree 0003 that Main mentions (main ea079b221) is a separate artifact and is not part of this review.

…st operation

A 9B model at 8192 tokens of context had one gap of up to 111.5 ms between
two decoded tokens in every process, once per run, with the GPU idle for a
good part of it. The 100 ms default hold dropped the request in that gap,
which costs one slow token while the PMP raises memory again. The longest
gap in any other configuration measured was 64 ms.

Raise the default to 250 ms, about 2.2 times the longest gap seen. The cost
is a few hundred milliseconds of the higher memory state after the last
job. pmp_report.agx_vote_idle_ms still overrides it at run time.

The default is one constant, AGX_VOTE_IDLE_MS_DEFAULT, used for both the
parameter and the initial state. The userspace test has a case that rides
through a 112 ms gap with the default and still drops the request after a
long idle.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

Copy link
Copy Markdown
Author

GREEN on 9f2a319

Source re-review of the combined head: one commit (9f2a319, "hold the AGX request 250 ms after the last operation") over e033631, which I approved earlier. This is the change I suggested after the 9B result (hold of 100 ms is below the largest token interval seen).

Reviewer identity: routed by Main to Sonnet (Claude); the model id is unverifiable from inside the session and Main's routing notice is the source. The implementer is a different party; implementer model unverified. I did not edit the PR and I did not build it; the compile check is a separate step.

Delta checked at the exact head (fetched refs/pull/334/head, which resolved to the sha above, and confirmed again just before posting):

  1. Scope: one commit, two files (pmp-agx-vote.h +4, pmp-report.c +3 -3). Everything else is unchanged from the head I approved.
  2. AGX_VOTE_IDLE_MS_DEFAULT (250) is defined in the header, which pmp-report.c includes at line 34, before both uses: the initial state apple_pmp_agx_vote_state = {.idle_ms = AGX_VOTE_IDLE_MS_DEFAULT} (line 89) and the parameter default agx_vote_idle_ms (line 1323). The parameter is still module_param(..., uint, 0644) and is still read again at every operation end and in the work function, so a runtime write takes effect without a reboot. The parameter description now says 250. No stale 100 ms default remains in the AGX code or the header comments of the PR tree.
  3. The unit test lives in the private repo (tools/ptd-vote, main f96602c64), not in this PR; its header copy is byte-identical to this PR's pmp-agx-vote.h (sha256 7ac8acb723fa), so it exercises this code. I ran it at that commit: AGX-VOTE-SELFCHECK-OK. Case 13 rides through a 112 ms gap on one vote (one set, no clear), keeps the bit for 249 ms after the last operation, and drops it at 250 ms. I mutated the constant back to 100 in a scratch copy and the suite fails at line 204 (AGX_VOTE_IDLE_MS_DEFAULT >= 250), so the test does catch a regression.

Note, no action needed for the GREEN: the new comment says "w7X measured idle gaps of up to 111.5 ms inside one 9B generation". What I measured is the largest token interval (111.5 ms at 9B, context 8192, once per process); in the one case I inspected in the 20 Hz firmware stats the GPU was idle for at least 50 ms of that interval. "Token intervals of up to 111.5 ms" is the accurate wording. The 250 ms figure (about 2.2 times that interval) stands either way.

Not covered: no build, boot or hardware run from me. The decode A/B through this parameter on the lab3 kernel is held for the one-shot slot; its rows write 250 to the parameter at run time, which is this default.

@scottjones scottjones left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We reviewed 9f2a319 against 2a76ae3. The vote logic is small and easy to follow, and the begin/OpGuard pairing in start_op() holds on every path, including a failed kick. Findings 1 and 2 are the blockers. Finding 3 is a should-fix, and finding 4 is a question.

1. pmp_report.agx_vote=1 on the kernel command line panics at boot. drivers/pmdomain/apple/pmp-report.c:1340

APPLE_PMP_REPORT is a bool, so this setter runs from parse_args() at init/main.c:1016. workqueue_init_early() only runs at init/main.c:1070, so system_wq is still NULL. kick(0) goes mod_delayed_work_on() -> __queue_delayed_work() (delay 0) -> __queue_work(), which reads wq->flags at kernel/workqueue.c:2286. Nothing checks for a NULL wq on the way. This runs in start_kernel on the boot CPU, so the fault kills the idle task and the machine never boots, probably before the console is up. The A/B runs flipped the param at run time, so this path was never hit. The PR description says "pmp_report.agx_vote=1 turns it on", and putting that on the command line is how most people will do it. Since begin() and the work fn already read agx_vote themselves, skipping the kick before the workqueues exist is enough (for example if (system_wq), or a check on system_state).

Related, minor: system_wq is __WQ_DEPRECATED in this tree (kernel/workqueue.c:8088), so the first kick also prints the "enqueued on deprecated workqueue" warning. system_dfl_wq avoids it. pmp-report.c:1332

2. A begin during the clear write leaves the GPU running with no vote and nothing scheduled. pmp-report.c:1416, drivers/pmdomain/apple/pmp-agx-vote.h:86

The work fn picks CLEAR under the spinlock, drops it, then does the write and the ack poll (100 us to 50 ms). If an operation begins in that window, agx_vote_begin() sees voted still true and does not kick. agx_vote_applied() then sets voted = false, and agx_vote_next() has no case for "users > 0, not voted, no failures", so it returns 0 and no work is queued. Bit 9 stays clear until some later begin happens. The likeliest time is when work resumes after more than 250 ms idle, such as a new prompt after a pause. In a decode stream that costs one operation. A long single submission runs at the idle floor for its whole length. We reproduced it with the header alone: decide (CLEAR), begin, applied, next gives users=1 voted=0 next=0, with decide() now returning SET. Re-running decide() after applied(), and kicking 0 when it is not NONE, closes it. The userspace test should get this case too.

3. release() does not stop a later begin from setting the bit again. drivers/gpu/drm/asahi/driver.rs:236, pmp-report.c:1462

unbind() calls release() first, but the DRM registration (_reg) is part of the driver data and is only dropped after unbind() returns (driver.rs:51). A submission in that window calls begin(), sees !voted, kicks the work, and the work sets bit 9 again. The bit can then stay set past the GPU's power-domain release, which the commit message says cannot happen. The report side has the same shape: pmp-report.c:187 clears the vote before list_del_rcu() at :191, so work that a begin queues between the two can set the bit on a report that is going away. A "stopped" flag under the vote spinlock, checked in begin(), would fix both. Clearing it needs a new call at GPU probe, so a rebind can vote again. Moving release() later does not help. drm_dev_unplug() only runs in Registration::drop after unbind returns (rust/kernel/drm/driver.rs:242). Even then, drm_ioctl() checks for unplug only on entry (drivers/gpu/drm/drm_ioctl.c:810), and the asahi submit path has no drm_dev_enter() section, so an ioctl already past that check can still reach start_op(). We think this one is rare, because it needs a GPU unbind or report unbind with work in flight.

4. Question: what should happen to bit 9 over system sleep? pmp-report.c:1593

The report entries already withdraw requests in suspend_noirq. The comment there says a held external display request cost about 0.25 W in s2idle on J514S, which is this SoC family. We don't know what holding the AGX bit costs. The AGX vote has no equivalent. If suspend starts within the 250 ms hold, or with an operation in flight (asahi has no sleep hooks), bit 9 can be held through s2idle. If the PMP drops the word over sleep, voted stays true after resume and the first busy period gets no SET. We have not measured either case. A clear in suspend_noirq (or a PM notifier) and a re-decide on resume look cheap. If you know the bit is harmless in s2idle, a comment saying so would be enough.

Small things

  • The Signed-off-by on 9f2a319 says joshuawarren, while the author is joshuaswarren. checkpatch flags the mismatch.
  • The vote test lives outside this tree, so nothing here fails if the logic regresses. The KUnit follow-up would cover that.

What we checked: we built pmp-report.o with W=1 on aarch64 (clean), and the four exports are present and GPL. We confirmed the boot order in init/main.c and the NULL deref path in kernel/workqueue.c. We reproduced finding 2 against pmp-agx-vote.h under ASan/UBSan, and ran checkpatch. We did not build the Rust side, boot anything, or run on hardware.

… it across unbind and sleep

Review of the AGX vote found four problems.

1. agx_vote=1 on the kernel command line runs the parameter setter from
   parse_args(), before workqueue_init_early() has created the workqueues,
   and the setter queued work on a NULL workqueue. kick() now returns while
   system_dfl_wq does not exist; begin() and the work function read the
   parameter themselves, so the first GPU operation votes. It also uses
   system_dfl_wq instead of the deprecated system_wq.

2. An operation that began while the clear was being written saw voted
   still true and did not kick; after the clear nothing scheduled a set, so
   the GPU ran at the idle floor until the next begin. agx_vote_next()
   now reports a set that is owed (1 ms) when users > 0, the request is not
   voted and no set has failed.

3. release() did not stop a later begin() from setting the bit again, and
   the report's removal had the same shape. A stopped flag, set under the
   vote lock at release and at report removal, is checked by begin(),
   decide() and next(); it is cleared by a new
   apple_pmp_agx_activity_resume() called at GPU probe and at report probe.

4. The request is withdrawn on PM_SUSPEND_PREPARE and PM_HIBERNATION_PREPARE
   and decided again on PM_POST_SUSPEND and PM_POST_HIBERNATION: holding the
   bit through s2idle may cost power (a held display request cost about
   0.25 W on J514S), and the PMP may drop the word across sleep, which would
   leave voted true with nothing on the wire.

Each case has a test in tools/ptd-vote of the research repository that
carries the policy header (cases 14 to 17 of the vote simulator, and a source
check for the boot-time kick); they were written to fail first.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

Copy link
Copy Markdown
Author

GREEN on 778b924

Source re-review of one commit (778b924, "close the races in the AGX vote and stop it across unbind and sleep") over 9f2a319, which I approved earlier. The delta is four files (driver.rs +3, pmp-agx-vote.h +28 -5, pmp-report.c +71 -1, pmp-report.h +5), read in full, plus the vote code around it at the new head. Not built or booted by me: the build result is the author's statement; my checks are reading, the simulator and the source check below.

Reviewer identity: routed by Main to Sonnet (Claude); the model id is unverifiable from inside the session and Main's routing notice is the source. The implementer's model is unverified.

The four findings, each against the code:

  1. Boot-time kick. kick() now reads system_dfl_wq with READ_ONCE and returns when it is NULL, and queues on system_dfl_wq instead of system_wq. begin(), end() and the work function read the parameter themselves, so with agx_vote=1 on the command line the first GPU operation votes. I could not boot-test the parse_args ordering claim; the kick guard is a source assert, as the commit says. check-kick-guard.py at research-repo commit fec36d0f1: its selfcheck prints KICK-GUARD-SELFCHECK-OK, and run against pmp-report.c at this exact head it prints KICK-GUARD-OK (no system_wq left in the file).
  2. Begin racing the clear. agx_vote_next() returns 1 when users > 0, not voted, no failed set. In the work function the owed set then runs: decide gives SET, the write, applied. It cannot loop: with failures == 0 decide returns SET whenever next returns 1, and a failed set makes failures nonzero, which routes to the retry deadline.
  3. Release then begin. release() and report removal first call stop_new() (stopped set under the spinlock), then cancel_delayed_work_sync() with no mutex held, then take the mutex and stop() (clear, stopped stays set). begin() and decide() and next() all check stopped; an operation after release counts as a user and never votes; a late end() kick (voted still true in the window between stop_new and stop) finds a stopped state in the work function and writes nothing. Resume at GPU probe and at report probe (t600x only) clears it and kicks when work is already in flight. No lock-order problem: the mutex is never held across cancel_delayed_work_sync.
  4. Sleep. PM_SUSPEND_PREPARE and PM_HIBERNATION_PREPARE call release(), POST_SUSPEND and POST_HIBERNATION call resume(); the notifier is registered in init and removed in exit, returns NOTIFY_DONE, runs in process context. POST_SUSPEND is also sent after a failed suspend, so an aborted suspend re-votes.

The check you asked for, that I did not weaken the earlier behaviour (with the vote on, decode must still set once and hold): by reading, the new conditions only add false/NONE when stopped is set, and the only new non-zero timer is the owed set (users, not voted, no failures), which never occurs while voted is true. With voted true and bursts every ~8 ms the path is the old one: end() queues the idle timer, begin() does not kick, the work function finds users or an unexpired hold and decides NONE. Cases 1 to 13 of the simulator (including case 13, the 111.5 ms gap with one vote and no clear) pass unchanged against the new header. I did not run decode on hardware for this commit; the kernel with this commit has not been built into a lab image that I have seen, so the +25 to +29 percent decode result belongs to the earlier commits.

Tests, run by me: the header copy in the research repo at fec36d0f1 is byte-identical to the PR's pmp-agx-vote.h (sha256 b6788472ea0d7eac...). The simulator (cases 1 to 17) prints AGX-VOTE-SELFCHECK-OK with -fsanitize=address,undefined (ASan needs setarch -R on this host; with ASLR on it dies at startup, not a test failure) and with UBSan alone. Mutation of the header, ten mutants: killed seven (decide ignoring stopped, next ignoring stopped, no owed set, release not setting stopped, resume not clearing it, resume never asking to kick, resume kicking when idle). Three survive, none blocking:
a. begin() ignoring stopped survives: decide() already blocks the set, so it only costs a spurious work item. Assert agx_vote_begin() returns false directly in case 15.
b. the "&& !failures" in the new next() branch survives: without it a failed set makes next() return 1 ms during the whole 1000 ms retry window, a 1 kHz spin of the work item that decides NONE every time (by reading, not run). The code is right today but nothing pins it. One assertion after a failed set, agx_vote_next(...) == retry_at - now, closes it.
c. agx_vote_refuse() being a no-op survives: the simulator never calls it; only the kernel glue does.

Not blocking, for the record:

  • One stopped flag stands for three causes (GPU unbind, report removal, sleep), and resume from any of them clears it. After an unbind, a suspend/resume cycle clears stopped while the GPU is still unbound; only an operation already in flight from before the unbind could then vote, and it would drop 250 ms after it ends. A per-cause bit would remove that; I would not hold the PR for it.
  • Module exit does not cancel the vote work; report removal does. A late end() kick in the window between cancel and stop in report removal could leave one queued item that runs harmlessly (stopped), but if the module is unloaded in that window it would run freed text. The window is nanoseconds wide and needs a vote held at that moment; mention only.
  • Not checked: how the Rust bindings treat the new header's inline stub when the PMP report is built out; release, begin and end have the same shape today.
  • driver.rs: resume() runs at probe before dev_info, for every Asahi GPU; it is a no-op off T6001.

@joshuaswarren

Copy link
Copy Markdown
Author

Thank you for the review. All four points were real. The fix is one commit, 778b924bc058, on top of 9f2a319c.

Point 1, the boot-time panic: apple_pmp_agx_vote_kick() now returns while system_dfl_wq does not exist yet. Nothing is owed at that point. begin() and the work function read agx_vote themselves, so the first GPU operation votes. The kick also moved from system_wq to system_dfl_wq. I have no boot test for this. A source check in the test tree fails on the old revision with four findings and passes now. It asserts the text, not the behavior.

Point 2, a begin during the clear write: agx_vote_next() now returns 1 ms when a set is owed: users above zero, not voted, no failed set. The work function already queues the next run from that value. The test reproduces your sequence on the header alone (decide returns CLEAR, begin, applied, next) and expects 1 and then SET. A second case runs it through the simulator with the begin placed inside the write.

Point 3, release and report removal: A stopped flag lives in the vote state under the vote lock. release() sets it, begin(), decide() and next() honor it, and report removal sets it before the work is waited out. Votes are allowed again by a new apple_pmp_agx_activity_resume(), called at GPU probe and at report probe. A work item that was queued before the release finds nothing to set. I did not add a drm_dev_enter() section in the submit path. The flag only makes the late begin harmless.

Point 4, system sleep: A PM notifier withdraws the request on PM_SUSPEND_PREPARE and PM_HIBERNATION_PREPARE and decides again on PM_POST_SUSPEND and PM_POST_HIBERNATION. With work in flight the request is set again after resume. With an idle GPU nothing is written. I did not measure what the bit costs in s2idle. Withdrawing it avoids the question.

Signed-off-by: The wrong address is on 9f2a319c. It needs a rewrite of that commit and I have not pushed one. The new commit carries the right address.

Checked: pmp-report.o and the Rust GPU driver build clean against integration/aurora. All four exports are present, with no warning from the new code. The vote simulator now has 17 cases and passes under ASan and UBSan. Not checked: a boot with agx_vote=1 on the command line, a suspend and resume on hardware, and a GPU unbind with work in flight. The KUnit test you suggested is still to do.

@joshuaswarren

Copy link
Copy Markdown
Author

Update: I fixed the Signed-off-by address on 9f2a319c and force-pushed the branch with a lease. The commits that carried the wrong address have new hashes: 5b54de318780 and 6ece0825c8eb. The fix commit I described above, 778b924bc058, is now 6ece0825c8eb. The tree is identical to the one I described (git diff 778b924bc058 6ece0825c8eb is empty). Please re-review at 6ece0825c8eb.

…d report probe now uses

The report probe tells the AGX vote that a T6001 report came back
(apple_pmp_agx_activity_resume) and compares rep->offsets with the T6001
table. The harness extracts the probe verbatim and compiles it standalone, so
production.h failed with an undeclared apple_pmp_offsets_t600x and an
implicit declaration of apple_pmp_agx_activity_resume.

An empty T6001 table that never matches and an empty resume function are
enough: the harness models the T6030 graph. All 21 graph cases pass, and the
supplier harness passes (ASan needs ASLR off on this host: setarch -R).

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

Copy link
Copy Markdown
Author

Head is now e8ddbd9fff02. It adds one commit on top of 6ece0825c8eb.

The t6030-pmp-graph-tests host test failed at the previous head. That harness compiles the report probe verbatim, and the probe now tells the vote that a report came back. The harness defined neither apple_pmp_offsets_t600x nor apple_pmp_agx_activity_resume.

The fix is four lines in tools/asahi/t6030-pmp-graph-tests.c: an empty T6001 offsets table that never matches, and an empty resume function. No driver code changed, so the tree for the four earlier findings is the same.

Run locally: the graph harness passes 21 of 21 cases, and the mutation controls still abort. The supplier harness passes.

The four findings from the review are fixed at this head. A re-check is welcome.

@joshuaswarren

Copy link
Copy Markdown
Author

GREEN at e8ddbd9

Reviewer identity: routed by Main to Sonnet (Claude); the model id is unverifiable from inside the session and Main's routing notice is the source. The implementer's model is unverified.

PR head checked after the push: e8ddbd9 has tree c80c04833abbe7d179acb3afaf943697e6bd5430 (the tree I reviewed from w72's branch) and parent 6ece082, which has the same tree (06cb58b9d15f...) as 778b924, the head I approved earlier. The only change since that approval is the 4 lines in tools/asahi/t6030-pmp-graph-tests.c. My earlier remark about pushing on top of 778b924 does not apply: the PR chain had already been re-signed to 6ece082 with identical content, and this commit is a fast-forward of it.

Re-run by me from git archive exports: at 6ece082 the T6030 graph harness fails to compile (undeclared apple_pmp_offsets_t600x, implicit apple_pmp_agx_activity_resume); at e8ddbd9 it prints PASS: 21 actual overlay/probe/entry/DCP7/managed-start/supplier graph cases, all of its mutation controls still abort, and the supplier harness passes (setarch -R, ASan needs ASLR off on this host).

Follow-up, not a blocker: make the empty resume stub count its calls and assert 0 at the end of the 21 cases, so the harness can notice if the T6030 probe starts calling the vote's resume.

@scottjones scottjones left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We re-reviewed at 6ece082 and checked e8ddbd9 on top, which only adds the 4 lines in tools/asahi/t6030-pmp-graph-tests.c. git diff 778b924bc058 6ece0825c8eb is empty, and 9f2a319 and 5b54de3 have the same tree, so the force-push only changed the Signed-off-by. All five commits now sign off as joshuaswarren. Thanks for the quick turnaround. All four points are fixed. The one thing we'd still like before this merges is the vote test in the tree (see Tests).

1. Boot panic with agx_vote=1 on the command line. Fixed. The setter runs from parse_args() on the boot CPU, before anything else can run. system_dfl_wq is written once, in workqueue_init_early(), so the NULL check in kick() can't race. The value still applies later: begin() reads agx_vote itself and kicks on the first GPU operation once the workqueue exists. The deprecated system_wq is gone too.

2. Lost SET when begin() races the CLEAR write. Fixed. After the clear, agx_vote_next() returns 1 and the work runs a SET. We checked that it can't spin. Every state where next() returns 1 through the new branch is one where decide() returns SET. After a failed set, next() returns the retry delay, not 1. We checked this by hand and with a sweep over all the flag combinations.

3. Teardown race. Fixed. release() and report removal set stopped under the spinlock before cancel_delayed_work_sync(). The mutex is not held across the cancel, so it can't deadlock with the work fn. A begin() after the stop still counts a user and still ends cleanly, so the count stays balanced and nothing leaks. It just doesn't vote. An end() that queues the idle timer between the cancel and the stop finds stopped in the work fn and writes nothing. Rebind works. The GPU probe and the report probe both call resume(), and resume() kicks when work is already in flight. The report probe calls it after list_add_tail_rcu(), so that kick can find the report.

One thing we agree with from your second reviewer, not a blocker: one stopped flag stands for unbind, report removal and sleep. A suspend and resume after a GPU unbind clears it, and so does a report rebind. Only an operation still in flight from before the unbind could then vote. The vote drops 250 ms after that operation ends, so if it never ends, the bit stays set until it does. That costs power, not correctness. Separately, the module exit note in that review doesn't apply. APPLE_PMP_REPORT is a bool, so apple_pmp_report_exit() never runs.

4. Sleep. Fixed. The PM notifier clears on PM_SUSPEND_PREPARE and PM_HIBERNATION_PREPARE and decides again on PM_POST_SUSPEND and PM_POST_HIBERNATION, all in process context. A work item already running when suspend starts is waited out, and its set is cleared by the stop().

Tests. tools/ptd-vote isn't in a repo we can reach, so we couldn't run cases 14 to 17 or check that they fail. We wrote our own harness against the header. Removing the owed-set branch from next() fails it, and so does dropping stopped from begin() or decide(), dropping !failures from the new branch, or making resume() leave stopped set. Your mutation run says the begin() and !failures mutants survive in your suite. Asserting !agx_vote_begin() after a refuse, and next == retry_at - now after a failed set, would kill both. Nothing in this tree fails yet if the policy regresses, and that's the reason we're requesting changes. The simulator is already plain C against pmp-agx-vote.h with no kernel dependency, so it fits the pattern tools/asahi already uses: a .c harness with a Python wrapper, registered in tools/aurora-ci/host-tests.json like t6030-pmp-graph-tests, so the host tests (tools) job runs it on every PR. Please add it to this PR, including the 17 cases and the two asserts above. KUnit can still come later.

What we ran: pmp-report.o with W=1 on aarch64 (clang, LLVM=1), clean. drivers/gpu/drm/asahi/ with Rust 1.85, which builds; its four dead-code warnings are in code this PR doesn't touch. checkpatch shows only long lines. We did not boot anything, suspend, unbind, or run on hardware.

@chriskearney

Copy link
Copy Markdown

For reproducing the benchmark, could you link the harness/commands and pin the tested kernel commit and config? The description identifies a 7.1.12 lab-kernel form, while current head e8ddbd9fff0252ef8aeb95a3463cf1b8b1ce24c1 also includes the later boot-time, clear-race, teardown and PM fixes. Please distinguish the earlier performance measurements from any run of that current source.

Scott’s request to put the vote-policy suite and its regression assertions into this tree’s host CI is still outstanding; we’ll check the successor when that lands.

@chriskearney chriskearney left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At e8ddbd9fff0252ef8aeb95a3463cf1b8b1ce24c1, two nonblocking retry-policy issues reproduce against the exact pmp-agx-vote.h (SHA256 b6788472ea0d7eac557faf0850a7b699fdf30ab2fb83cb5773cc9a494e32519d):

  1. Backoff limits writes, but not immediate reconciliation requests. After a failed SET, every agx_vote_begin() still returns true while voted is false. agx_vote_decide() correctly returns NONE before retry_at, but the glue keeps asking the workqueue to run immediately. In a sequential host model, 99 operations inside the backoff requested 99 immediate reconciliations and made zero SET attempts. Kernel workqueue coalescing can reduce the number of actual executions. This also occurs when opt-in is enabled on a non-T6001 machine and writes return -ENODEV.

  2. An idle interval beyond half the 32-bit clock range can revive an expired backoff. failures and retry_at survive the last end indefinitely. After one failed SET, ending the work and starting again at retry_at + 0x80000005U gives NONE and agx_vote_next() == 2147483643 ms, although the one-second retry delay expired long ago. The control at retry_at + 1000 correctly gives SET. A bounded expiry of stored retry state, or an appropriately sourced wider clock, would avoid this. Please include a regression case when addressing it.

These are opt-in performance limitations, not new hardware-fault findings. Independent host policy tests pass under ASan+UBSan and UBSan; these are local checks, not tests registered in this PR. Scott’s request to land the vote suite and regression assertions in the repository’s host CI remains the merge blocker. The full package job is still running, and we have not booted this revision or measured its performance.

@chriskearney chriskearney left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking finding at e8ddbd9fff0252ef8aeb95a3463cf1b8b1ce24c1: the final operation can end during a SET write and leave the AGX vote set with no clear scheduled.

In apple_pmp_agx_vote_fn(), the spinlock is dropped between agx_vote_decide() and agx_vote_applied() while the SET and acknowledgement happen. If the last end() runs in that window, it records idle_at but sees voted == false and requests no timer. The worker then marks the successful SET as voted. If the idle deadline has already elapsed, agx_vote_next() returns 0, which this caller interprets as no work to schedule. agx_vote_decide() would now return CLEAR, but nothing invokes it.

Minimal sequence against the exact policy header, following the actual work function’s ordering:

struct agx_vote v = { .enabled = true, .idle_ms = 0 };
agx_vote_begin(&v);
enum agx_vote_action act = agx_vote_decide(&v, 5000); /* SET */
agx_ms_t timer = agx_vote_end(&v, 5001);             /* 0: not yet voted */
agx_vote_applied(&v, act, 0, 5000);                 /* SET acknowledged */
/* At 5001: users=0, voted=true, timer=0, next=0, decide=CLEAR. */

This reproduces with the accepted runtime parameter agx_vote_idle_ms=0, or a 10 ms hold and an 11 ms write/worker interval. With the default 250 ms hold, the model’s 50 ms acknowledgement control correctly schedules a timer; a worker delayed until 251 ms reproduces the problem, including across clock wrap. That longer delay is a scheduling model, not a measured hardware acknowledgement time.

Please schedule another reconciliation when a CLEAR is already owed after the write, and add the end-during-SET interleaving to the in-tree suite. In a scratch control, returning 1 rather than 0 for an already-reached next deadline removes all four stuck cases and preserves the unit cases; it still needs normal review and CI.

Validation: independent host interleaving model plus deterministic header-only reproducer; I reran the reproducer and its control and checked the actual C work/end paths. No MMIO, boot, suspend or hardware test was performed. The two retry-policy notes in my earlier review remain nonblocking; this owed-CLEAR race and the missing in-tree regression coverage need resolving before merge.

Follow-up validation: the same failure also reproduces through the work/begin/end functions extracted from this exact PR, with register access and scheduling stubbed on the host. The locally corrected header clears those cases and retains the begin-during-CLEAR control. CI run 41648287024 has now passed the full package build and QEMU boot smoke check; neither exercises the Apple PMP hardware. These green checks do not resolve the reproduced race.

…ontrols

The vote policy is a header of static inline transition functions, so the
suite compiles pmp-agx-vote-tests.c against a copy of it and drives the same
begin, decide, next and applied calls the kernel glue makes, with a fake
clock, a fake PMP bit and a fake timer. 18 cases: the parameter off, the set
and idle drop, decode-like bursts, a drop cancelled by new work, the failed
set backoff, unbalanced ends, the parameter switched off mid-flight, the zero
idle tick, millisecond clock wraps, release and refuse, resume with and
without work in flight, sleep and resume, and the two asserts from review
(the begin inside a clear write must be re-set, and begin must refuse while
stopped).

--mutation and --mutations remove one guard each (the owed-set branch, the
!failures in it, and the stopped check in begin and decide) and require the
suite to fail. The two mutants that survived before the races fix (begin not
refusing while stopped, and the retry deadline indistinguishable from the 1 ms
owed-set kick) are the stopped-in-begin and owed-set-branch controls: at
5b54de3 (the pre-fix header) both survive because the guarded lines do not
exist; at this head each is caught at the case that pins it.

Registered in tools/aurora-ci/host-tests.json (base, --mutations, four
single-mutant entries, the .c as a helper).

Proof: plain run passes (18 cases, AGX-VOTE-SELFCHECK-OK); --mutations exits 0
with all four caught; --ref 5b54de3 --mutation stopped-in-begin and
--mutation owed-set-branch print SURVIVED there.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…he mutation modes

Review of 1142df9 by w7X, two non-blocking items folded in before push.

Case 19 pins the signed 32-bit compare in agx_vote_reached: with idle_at
0xFFFFFFF0, idle_ms 100 and voted, decide at 0xFFFFFFF8 must be NONE and next
must return the 92 ms to the wrapped deadline, and a retry_at that has wrapped
must not read as reached either. The wrap-reached mutant (now >= t) survives
the old cases because they jump straight to the wrapped deadline; it dies at
line 306. Registered as a fifth mutation entry.

The mutation modes now assert the unmutated suite passed first, so a base
failure cannot make every mutant read as caught. The negative control (--ref)
is exempt: the pre-fix header cannot compile this suite, which is the point.

The suite runs under setarch -R when available; ASan startup crashes under
ASLR on a loaded host otherwise.

Proof: plain run passes; --mutations exits 0 with all five mutants caught;
wrap-reached CAUGHT at line 306; at 5b54de3 the owed-set-branch and
stopped-in-begin mutants still print SURVIVED.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

joshuaswarren commented Nov 3, 2026 •

Copy link
Copy Markdown
Author

The vote simulator is in the tree. Head is db48e985728e23e35119de564cfc9bd0f0f266f2, two commits on top of the four-fix head.

  • tools/asahi/pmp-agx-vote-tests.c drives the real pmp-agx-vote.h policy functions. A fake clock, a fake PMP bit and a fake timer replace the hardware: 19 cases. The two asserts this review asked for are in. One: an operation that begins while a clear is being written is re-set without waiting. Two: begin refuses while stopped.
  • tools/asahi/pmp-agx-vote-tests.py compiles the C against a copy of the header. --mutation NAME and --mutations remove guards from that copy and require the suite to fail. They first assert the unmutated suite passes, so a base failure cannot read as a caught mutant. --ref reads the header at an old revision.
  • tools/aurora-ci/host-tests.json registers the plain suite, the aggregate, five single-mutant entries, and the .c as a helper.

The mutant matrix:

  • Removing the owed-set branch dies where the vote is re-set after a begin inside a clear write.
  • Removing its !failures guard dies at the retry assert: the next step must be the 1 s backoff, not the 1 ms kick.
  • Removing the stopped check in begin dies at the refuse assert.
  • Removing it in decide dies at the post-release decide case.
  • Replacing the signed 32-bit deadline compare with an unsigned one dies at the clock-wrap case.

The two mutants that survived before the review fixes were begin without the stopped refusal, and the retry deadline indistinguishable from the 1 ms kick. Against the header before the fix commit, both survive: the guarded lines do not exist there, so there is nothing to remove. --ref 5b54de31 --mutation stopped-in-begin reports SURVIVED. At this head each dies at its pinning assert. All five abort together under --mutations, and the CI runner passes the registered entries.

Two review rounds on this suite are folded in: the clock-wrap case and the wrap mutant came from the first, the base-pass assert and a setarch -R prefix for the sanitizer runs from the second.

…ff kick storm and the revived retry

Three cases for chriskearney's findings on AsahiLinux#334, written first and red at
this commit:

- Case 20: an end lands between decide (SET) and applied. The end sees
  voted false, arms nothing, and applied then sets the vote with no user and
  an already-reached idle deadline. next() must return 1 there, not 0: 0 is
  the suite's "nothing pending", so the caller would never run the reconcile
  that clears the vote.
- Case 21: an idle gap beyond half the 32-bit millisecond range makes the
  wrapped arithmetic read an expired retry deadline as future, which would
  keep the vote off for days. A retry deadline more than one backoff window
  in the future cannot be real and counts as reached.
- Case 22: during backoff, begin must not ask for a reconciliation decide
  would refuse; the retry timer already drives the set.

Three mutants pin them (set-write-ends, retry-revival, begin-backoff-kick),
and stopped-in-begin now matches the begin line the fix will produce. Red
here: case 20 fails at `agx_vote_next(&v, 5001) == 1`.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…try deadline and quiet the backoff kicks

Three findings from chriskearney's review of AsahiLinux#334; the suite cases landed
first (825e9ac, red) and this is the smallest change that turns them green.

- end() during the SET write (between decide and applied) sees voted false
  and arms nothing; applied then leaves the vote set with no user and an
  already-reached idle deadline, and next() answered 0, which the caller
  reads as nothing pending. An already-reached deadline now returns 1, so
  the caller runs the reconcile that clears the vote. It settles one step
  later: decide CLEAR, applied, next 0.
- A retry deadline more than one backoff window in the future cannot be
  real (an idle gap beyond half the 32-bit millisecond range reads an
  expired deadline as future through the wrapped compare), so the new
  agx_vote_retry_reached() treats it as expired instead of keeping the vote
  off for days.
- begin() no longer asks for a reconciliation while a failed set is in
  backoff: decide would return NONE, and the retry timer already drives the
  set. One kick per operation during the window was pure waste.

The mutants in tools/asahi/pmp-agx-vote-tests.py pin each: set-write-ends
dies at the race assert, retry-revival at the bound assert,
begin-backoff-kick at the kick-count assert, and the five earlier mutants
still die. Plain suite: 22 cases, selfcheck ok.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…ot strand or spam the vote

w7X's review of the first cut (a7330a5) found a blocking regression and two
correct behaviours to keep. The gate on begin had to know the time, so
agx_vote_begin() now takes now (the caller reads it under its lock; the glue
passes apple_pmp_agx_now_ms() beside its existing end and next reads):

- begin returns false during a failed-set backoff before the deadline (decide
  would return NONE; the retry timer drives the set), and true again once the
  deadline has passed. The first cut's unconditional !v->failures stranded the
  vote after one failed set whose retry timer fired idle: failures is only
  cleared by a successful SET, so every later begin refused forever. Case 23
  is that sequence (failed set, end before the retry, timer fires idle, a
  later begin must kick and the set must land); the begin-ignores-retry
  mutant is exactly the first cut's expression and dies at it (line 363).
- The other direction is pinned too: begin-backoff-kick (kick during the
  window regardless of the deadline) dies at the kick-storm case (line 357),
  so the gate cannot silently decay into "always kick" again.

Case 23 exists because the simulator cases kept the operation alive through
the backoff; this one ends it first. The begin signature change touches the
suite's helper, four header-only call sites, and the one glue call.

Proof: plain suite 23 cases selfcheck ok; --mutations catches all nine
mutants (owed-set-branch, owed-failures, stopped-in-begin, stopped-in-decide,
wrap-reached, set-write-ends, begin-backoff-kick, begin-ignores-retry,
retry-revival); check-kick-guard still passes on pmp-report.c; the tree
runner passes the registered vote entries.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
w7X's re-review nit: next() takes the first match, so the second entry could
never be selected. Nine distinct mutants; --mutations still passes.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

Copy link
Copy Markdown
Author

All three findings from this review are addressed, with the suite cases landed first. Head is efff25c15c4f.

  • The race: an end lands between decide (SET) and applied. It sees voted false and arms nothing; applied then leaves the vote set with an already-reached idle deadline. agx_vote_next() now returns 1 for an already-reached deadline. The caller runs the reconcile that clears the vote, and it settles one step later. Case 20 in tools/asahi/pmp-agx-vote-tests.c is exactly that interleaving. The set-write-ends mutant removes the fix and dies at that case.
  • The backoff kick storm: agx_vote_begin() now takes the clock and asks for a reconcile only when decide would act (!failures, or the retry deadline reached). Inside the window it returns false, and the retry timer still drives the set. Case 22 pins it (begin-backoff-kick mutant).
  • The revived retry: a retry deadline more than one backoff window in the future cannot be real, so the new agx_vote_retry_reached() treats it as expired (used in decide and next). Case 21 pins it (retry-revival mutant).

One more round happened on the first cut of the begin gate: an unconditional !failures stranded the vote after a failed set whose retry timer fired idle. The clock form above replaced it, case 23 is that sequence, and the begin-ignores-retry mutant is the first cut's expression and dies there. The suite now holds 23 cases and nine mutants; --mutations catches all nine, and the registered entries pass the tree's CI runner. Review history for the suite and the vote header is in the commit messages.

On the benchmark: the decode figures in the description (+26.7% and +29.4% on the 9B 4-bit model, the +25 to 29% range) came from the earlier lab-kernel form of this branch, not from the current source. Machine and stack: a 16-inch M1 Max, lab kernel 7.1.12-2-20261103.2-lab-ARCH (the 7.1.12 base plus this branch's predecessor patches 0001 to 0004 and the recorder diff, and the vote wrapped off and on through /sys/module/pmp_report/parameters/agx_vote), Mesa 26.2.4.git69bde15340-2, the downstream Aurora lab config as packaged. The harness is the decode A/B row in the working tree this branch was written against (tools/push-desc-ab: pd-ab.sh and the decode runner, with the wrap script at vote-wrap.py), two processes per arm at 512 tokens, greedy, fixed seed, generated ids compared. Head at e8ddbd9 and later adds the boot-time, clear-race, teardown and PM fixes; none of the current source has been re-measured for performance. The fix-level source (this head plus the clock fix) is being rebuilt into the next lab kernel and the same A/B re-runs on it; those numbers will replace the description's when they land.

@scottjones scottjones left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We re-reviewed at efff25c. Thanks for the quick turnaround on both the suite and Chris's findings.

Tests. The test request is satisfied. tools/asahi/pmp-agx-vote-tests.c and .py are in the tree, and host-tests.json registers the base run, the --mutations self-check and each of the nine mutants in the tools group. From a fresh worktree, python3 tools/asahi/pmp-agx-vote-tests.py --output <dir> passes, --mutations catches all nine, and host-tests.py runs all 11 entries as passing. The host tests (tools) job at this head ran all 11 and passed. It includes the 17 earlier cases and the two asserts we asked for (lines 295 and 302).

We also reverted the fixes by hand instead of trusting the mutant list. Putting back the e8ddbd9 header (with only the begin() signature adapted) fails case 20. Reverting just the ? 1 in next() fails case 20. Reverting just the backoff gate in begin() fails case 22. Dropping the wrap clause in retry_reached() fails case 21, and so does dropping the ? now : v->retry_at in next(). Our four earlier points are still covered: the owed set after a CLEAR (case 14), stopped in begin() and decide() (cases 15 and 18), and sleep (case 16). Making resume() leave stopped set fails case 15. The boot kick guard and the PM notifier live in pmp-report.c, outside what the suite tests, and neither changed since e8ddbd9.

Chris's three findings. All fixed.

  1. End during the SET write. end() sees voted false and arms nothing, applied() then sets voted, and next() now returns 1 for the reached idle deadline, so the work runs the CLEAR. Correct.
  2. Kick storm in backoff. begin() now returns false before the retry deadline, so a busy GPU no longer queues a NONE reconcile per operation.
  3. Revived retry deadline. retry_reached() counts a deadline more than one backoff window ahead as expired, which covers the wrap after a long idle gap.

next() returning 1 can't spin. decide() and next() use the same predicates, so on the NONE path, where next() gets the same now, next() never returns 1. On the write path, a 1 always means the next run takes an action. A failing SET re-arms at about 1000 ms. The ack poll is capped at 50 ms, so the retry is never back to back. begin() has one kernel caller, apple_pmp_agx_activity_begin(), which reads jiffies_to_msecs(jiffies) under the vote spinlock. That is the same clock applied() uses, and it can't be earlier than the now retry_at was computed from.

We walked both directions of each window. Begin during SET: begin kicks, or the backoff re-arms. End during SET: fixed as above. Begin during CLEAR: next() returns 1 and the SET runs. End during CLEAR: that only happens for a disabled or stopped clear, where end's extra kick finds nothing to do. Resume during CLEAR: the same path as begin, which next() covers. We found nothing left open in these windows.

One small gap, not a blocker. The new backoff gate in begin() relies on the retry timer staying armed. If the GPU goes idle during a backoff and something else consumes that timer, the next begin() inside the window doesn't kick, and nothing is scheduled until a begin() after retry_at. Two things can consume it: a write to the agx_vote parameter (mod_delayed_work(0) replaces the timer, and the reconcile sees no users and re-arms nothing), or a release and resume, from suspend, GPU unbind and rebind, or report removal and reprobe (cancel_delayed_work_sync() drops it, and resume() won't kick with no users). We reproduced it on the header: fail a set at 0, end at 100, run the reconcile at 200, then begin at 300. begin returns false, decide would return SET at 1500, and nothing is armed. It recovers at the next begin after the deadline, so a steady GPU loses at most about a second. A single long job started in the window runs without the vote. A fix would be for next() to keep re-arming the retry deadline while idle in backoff, returning 0 once it passes so begin()'s gate takes over (returning 1 there would requeue every 1 ms while idle), and for resume() to kick when failures is set. Fine as a follow-up.

What we ran: the suite, the mutants and the hand reverts above. pmp-report.o with W=1 on aarch64 (clang 23, LLVM=1) builds clean. drivers/gpu/drm/asahi/ builds with Rust 1.85. Its dead-code warnings are in code this PR doesn't touch, and no Rust file changed since e8ddbd9. checkpatch on the six new commits shows long lines, block comment style and missing blank lines in the test file, nothing functional. We did not boot, suspend, unbind or run anything on hardware.

This looks right to us. Could it also cover t6000 (M1 Pro)? If AGX is soc-device record 9 there too, Marcelo has an M1 Pro that could test it, which would give us a second chip before it lands. Since this is new power behaviour that hasn't run on hardware yet, we'd like to approve after 4.5 ships.

@chriskearney chriskearney left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at efff25c15c4f1abb1fb141d87ecfe07432bdb3c4: the original owed-CLEAR blocker is resolved, and the in-tree coverage request is resolved. The unchanged extracted-C reproducer now clears all five previously stuck cases; the new suite catches the old header at case 20. The registered suite and all nine mutants pass locally and in CI.

There is a new blocking regression in the quiet-begin gate: a job starting during backoff can have no timer to retry its vote. The gate assumes that a retry timer already exists, but that is not guaranteed. Reproduced ordering through this revision’s actual work/begin/end functions, with register access and scheduling stubbed on the host:

  1. The worker decides SET with one operation active.
  2. The last operation ends during the acknowledgement poll. It sees voted == false and queues no timer.
  3. The SET times out; the glue clears the request. applied() records a failure and retry_at, but next() sees no users and no vote and returns 0. Nothing is pending.
  4. A new sole operation begins before retry_at. The new agx_vote_begin(v, now) returns false, so the glue queues nothing.
  5. At the retry deadline and ten seconds into that operation, the host harness still has users=1, voted=0, pending=0, although decide() returns SET. No caller runs it.

This reproduces with the new operation starting either 500 ms or 1 ms before the deadline, and across clock wrap. The same controls recover at e8ddbd9f: begin kicks the worker, which sees backoff and arms the remaining delay. At the new head, starting exactly at the deadline or retaining an active operation through the first failure works. A later begin after the deadline also recovers. Case 23 covers the later begin, not the missing-timer variant. Idle release/resume can likewise cancel the existing retry timer while retaining the failure state.

Please ensure a begin during backoff arranges the future retry when no timer exists, and add this end-during-failed-SET / begin-before-deadline regression case, including timer cancellation on idle release/resume. The effect is a lost performance vote for the active job. This is a host fault-injection result, not an observed hardware failure: the PMP acknowledgement is made to exceed its 50 ms poll limit for the first SET and succeeds thereafter. Two independent harnesses reproduce the sequence; I reran both and checked the production control flow.

A separate nonblocking boundary remains in agx_vote_retry_reached(): at exactly now = retry_at + 0x80000000U, both signed differences are INT_MIN, so it reports not reached and can return a 0x80000000 delay. The adjacent millisecond controls expire correctly. Comparing the future distance as unsigned would avoid that exact half-range hole; it should have its own boundary assertion.

The original change-request reasons are closed. Changes are still requested for the new missing-retry-timer regression. Full build and QEMU CI are green; no current-head Apple hardware boot or benchmark was performed by us.

Host-only fix control: the idle-rearm plus resume approach described in Scott’s review closes the failed-SET, idle release/resume and idle parameter-write timer-loss cases in our harnesses. It preserves the existing 23-case suite, all nine mutation checks and the old idle-clear controls. Both the idle-backoff rearm and the resume kick are needed; the resume kick alone still leaves reconciliation with no timer to arm. This is validation of a local candidate, not qualification of a pushed fix or hardware. The new in-tree regression should include end during a failed SET, a begin inside the remaining backoff, and cancellation/replacement of the idle retry timer.

…ommit)

chriskearney's second blocker at efff25c: the last operation ends during a
SET ack that then times out. applied records the failure with no user and no
timer (next returns 0 on that path), and a begin before the retry deadline
must arm the remaining delay instead of reporting no kick, or the retry never
runs. The case also walks the release/resume cycle: an idle release does not
write a clear, an idle resume does not kick, and a begin past the deadline
still kicks.

Red here: the begin result is 0 where the remaining 998 ms delay is required
(FAIL line 397).

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…e path could not

chriskearney's second blocker at efff25c: when the last operation ends during
a SET ack that then times out, applied records the failure with users already
0, so next() returns 0 on that path and no timer is armed. A begin before the
retry deadline reported "no kick" (the clock gate from 2ec389c), so nothing
ever ran the retry: the vote stayed off until some unrelated reconcile.

agx_vote_begin() now returns the same delay contract as agx_vote_end(): 1 run
the reconcile now, N arm it N ms out, 0 nothing pending. A begin inside a
failed-set backoff arms the remaining delay; past the deadline it returns 1 as
before; a voted overlap, stopped or disabled still return 0. The glue passes
the delay to the kicker instead of a boolean, and an idle release/resume cycle
does not clear the retry state, so the next begin still arms it.

Case 24 (fb8cba3, red at line 397) walks the whole sequence including the
release/resume variant. Controls re-pointed at the new return: begin-no-retry-
arm (returns 0 where the delay belongs) and begin-backoff-kick (returns 1,
the kick storm) both die; begin-ignores-retry is gone because the delay branch
subsumes it.

Proof: plain suite 24 cases selfcheck ok; --mutations catches all nine
mutants; the tree runner passes the registered entries.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
The residual w7X flagged on the delay contract: the end lands inside a failed
SET write and no new operation starts. The write path arms nothing (users is
already 0 when applied runs), so failures lingers with no timer, and a begin
before the deadline reports "no kick". Case 25 walks it: the retry must be
armed from the failure state itself, and when the retry fires with nobody
waiting the backoff must be cleared once instead of re-arming forever. The
header-alone asserts pin next(): 500 at 500 ms before the deadline, 1 at it.

Red here: FAIL line 424, s.armed is false after the race.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…the operation

The residual from the delay-contract review: when the last operation ends
inside a failed SET write, applied records the failure with users already 0,
and next() returned 0 on that path, so no timer existed and a begin before the
deadline reported "no kick" (2ec389c's clock gate) - the retry never ran.

next() now arms the retry from the failure state itself (users 0, not voted,
failures set): the remaining delay, or 1 when the deadline is already reached.
That alone would re-arm forever once the retry fired idle, so the work
function also clears a spent backoff: a reconcile that finds nobody waiting
and an expired deadline (decide returned NONE) sets failures to 0, and the
state settles. A later operation then starts a fresh attempt, exactly like
the first.

Case 25 (4574c14, red at s.armed) walks the sequence through the
release/resume cycle; the retry-dropped-when-idle mutant removes the new
branch and dies there. The sim's begin helper now arms the returned delay
(delay 1 is "due now" within a tick), case 23's spent-backoff assert and
case 24's next() assert updated to the new contract. Plain suite 25 cases,
ten mutants, all caught; the tree runner passes the registered entries.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

joshuaswarren commented Nov 3, 2026 •

Copy link
Copy Markdown
Author

Both rounds of your review are addressed. Head is 85a68bf69d05cdbdd4ba89b0764f1f063da87cef, four commits past the state you reviewed.

The in-tree suite you asked Scott for is in (tools/asahi/pmp-agx-vote-tests.c plus its Python wrapper, registered in the host-test manifest). It now holds 25 cases and ten mutants, and every mutation entry passes the tree's CI runner.

Your findings, each with its case:

  • The SET-write race: an end lands between decide and applied. The vote ends up set with an expired deadline and nobody to reconcile. next() returns 1 for an already-reached deadline now. Case 20 is your interleaving (set-write-ends mutant).
  • The end-during-failed-SET blocker you reported at efff25c1: failure and retry_at were recorded with no timer, a begin before the deadline refused, and the retry never ran. begin() now reports the same delay contract as the end path, so a begin inside the backoff arms the remaining delay. next() arms the retry from the failure state itself when the write outlasts the operation. Cases 24 and 25 walk both shapes, including your cancel-on-release/resume variant (begin-no-retry-arm and retry-dropped-when-idle mutants).
  • The kick storm during backoff: begin asks for a reconcile only when decide would act. In-window it arms the remaining delay once, and the retry timer does the rest (case 22, begin-backoff-kick mutant).
  • The revived retry past a 2^31 ms idle gap: a deadline more than one backoff window in the future counts as expired. That is agx_vote_retry_reached, case 21, mutant retry-revival. When the retry fires with nobody waiting, the work function clears the spent backoff instead of re-arming forever (case 25).

One intermediate gate on begin was itself reverted. Review found it stranded the vote; the history shows both rounds.

On the benchmark: the +25 to 29% decode figures in the description came from the earlier lab-kernel form of this branch. The current source, including your two races and the clock fix, has not been re-measured. It is being built into the next lab kernel, and the same A/B re-runs on it. Those numbers will replace the description's.

The earlier stack, for reproducing: a 16-inch M1 Max; lab kernel 7.1.12-2-20261103.2-lab-ARCH; the 7.1.12 base plus this branch's predecessor patches 0001 to 0004 and the recorder diff; the vote wrapped off and on through the pmp_report.agx_vote parameter; Mesa 26.2.4.git69bde15340-2; the downstream Aurora lab config as packaged. The harness is the decode A/B row from the tree this branch was written against (tools/push-desc-ab): two processes per arm at 512 tokens, greedy, fixed seed, generated ids compared.

Treat retry deadlines beyond the backoff window as expired using unsigned distance. Cover the exact half-range boundary and keep the mutation control aligned.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

Copy link
Copy Markdown
Author

Note

🤖 GPT-6.1 Terra responding on behalf of Joshua Warren

The retry rearm change is on the branch. Commit c3487a528f74dfc036ffba7d3a4e5fa09cf2b52b also fixes the exact half-range boundary by using unsigned distance. The host test now covers that boundary, and all 10 mutation controls were caught. These are host tests; no device run was performed for this commit.

@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Chris, this answers your 14:31Z request: the pinned kernel and config, the harness, and the commands. The next comments hold the harness files (2 and 3) and the config (4 and 5). Two corrections to the PR come first.

Corrections

  1. The PR description says "the 250 ms default hold" next to the decode numbers. The 250 ms was set at run time, not the image default. On the measured image the compiled default is agx_vote_idle_ms = 100 (patch 0001: static unsigned int agx_vote_idle_ms = 100;). The decode run wrote 250 into the writable agx_vote_idle_ms for its duration and restored 100 afterwards. Its env.txt records agx_vote_idle_ms 250. The default of 250 belongs to the current PR source. I am editing that sentence in the description now.
  2. The current PR head c3487a528f74dfc036ffba7d3a4e5fa09cf2b52b has not been measured on hardware. Every number here comes from the earlier lab-kernel form of the change (patches 0001 to 0003 below). The PR source differs from it: the retry rearm, the half-range boundary fix, and the 250 ms default. I will build the current head into a lab kernel and re-run the same A/B. I will post that result as a new comment with its own kernel pin.

Pinned kernel (both data sets below ran on this image)

Item Value
Release 7.1.12-2-20261103.2-lab-ARCH
Package linux-aurora-lab-7.1.12.20261103.2-1-aarch64.pkg.tar.zst, 65,637,832 B
Package sha256 9d35b354e6272abc9d3f52b06a15d32f2eaa75602c344d2b7b4c69e89d769a0c
vmlinuz sha256 0ddbd99b8c2f3c15808de7ff5f535f03f000687d4eebdc2da26bd747ace406b9
Config 9,418 lines, sha256 443233d9b3d34544afc2b119809dc0a44c3c24c78f7e36f7f7b174b50c239fd5 (extracted from the package; comments 4 and 5)
Builder source commit e8265597f8d0 (the lab-kernel tree the patches sit on)
Patches on top 0001 d4b1db41: automatic AGX request for T6001. 0002 e7d473a8: report GPU operations to the vote. 0003 6bfc98e5: locked, acknowledged update of one power-request bit
Not in the image patch 0004, which raises the default hold to 250 ms
Machine 16-inch M1 Max (T6001)
Identity gate GATE PASS 7.1.12-2-20261103.2-lab-ARCH profile lab-full soc t6001 scope stage

The patch ids are the first 8 or 12 hex digits of each patch file's sha256. The patch bodies are not inline. Say so if you want them attached. The config has CONFIG_RUST=y, CONFIG_DRM_ASAHI=y, CONFIG_APPLE_PMP_REPORT=y.

Pinned user space. mlx-omarchy wheel v0.7.36 (mlx_omarchy-0.32.4.dev202611030221+886ff2d1-cp314-cp314-linux_aarch64.whl, sha256 ea5e2ce19f11d356aaa2b3064345d1aa38848a26637e2a1711a9140dba60977d), mlx-lm 0.31.3, HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1, no MLX_OMARCHY_QMM_* variable. Models: a 2B and a 9B 4-bit MLX snapshot, model.safetensors of 1,059,404,429 B and 5,038,161,163 B. I pinned the weights by file size only. I did not record a sha256 for them.

Data set A: the numbers in the PR description. Hold 250 ms (set at run time). Script decode_b.py (comment 2), 2B and 9B, one fresh process per sample, C T C T C for decode and prefill, C T C for the read roof. C is agx_vote 0 and T is agx_vote 1, written before the process and reset to 0 after it. Decode runs 1000 tokens at 512 context and 800 at 8192 for 2B, 600 and 500 for 9B.

2B, tok/s C (vote off) T (vote on) Change
decode, 512 context 93.80 / 93.58 / 94.27 (mean 93.88) 117.65 / 117.43 (mean 117.54) +25.2%
decode, 8192 context 81.76 / 82.14 / 81.92 (mean 81.94) 105.16 / 105.28 (mean 105.22) +28.4%

Only 2 T processes per decode row and 3 C processes. The 9B pair in the description is 33.81 to 42.82 (+26.7%) at 512 and 29.01 to 37.55 (+29.4%) at 8192, from the same script.

Data set B: a later matrix on the same image at the 100 ms compiled default. Script agx_vote_wrap.py around matrix_cell.py (comment 3), 64 decode tokens, one prompt per cell, C T C T C per cell. Decode tok/s, vote off to vote on:

Cell Off On Change
2B, 512 94.6 117.9 +24.6%
2B, 8192 82.7 107.2 +29.7%
9B, 512 34.3 43.2 +26.1%
9B, 8192 29.8 38.7 +29.8%

The 9B 8192 decode has a 111.5 ms token interval, longer than a 100 ms hold. The 9B 8192 vote-on number can be too low for that reason.

Commands.

# preflight (both data sets)
test "$(uname -r)" = "7.1.12-2-20261103.2-lab-ARCH"
test "$(cat /sys/module/pmp_report/parameters/agx_vote)" = N
cat /sys/module/pmp_report/parameters/agx_vote_idle_ms         # 100 on this image

# data set A: hold 250 ms for the run, then restore
echo 250 | sudo tee /sys/module/pmp_report/parameters/agx_vote_idle_ms
python3 decode_b.py --model 2b --out-dir results/decode-b-2b     # then --model 9b
echo 100 | sudo tee /sys/module/pmp_report/parameters/agx_vote_idle_ms

# data set B, one cell (2B, 512): C T C T C
for s in "C1 0" "T1 1" "C2 0" "T2 1" "C3 0"; do set -- $s; l=$1; v=$2
  python3 agx_vote_wrap.py --vote $v -- \
    env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
    venv/bin/python matrix_cell.py cell --python venv/bin/python \
      --model models/2b --size 512 --decode-tokens 64 --processes 1 \
      --out-dir results/2b-512-$l --deadline 360
done

Both wrappers need sudo -n for the agx_vote write. They set the parameter, run the process, and always put it back to 0. Both stop if the kernel logs "AGX request ... not acknowledged". To A/B without the scripts, write 0 or 1 to /sys/module/pmp_report/parameters/agx_vote before each fresh process and write 0 after it.

Not covered. No t6000 run (Scott's question stands). No run at the current PR head. The read-roof and power columns in data set A need clock_probe.cpp and the q4-bw-bench sources, which are not in these comments. The decode and prefill rows do not depend on them.

@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Part 2 of 5: the harness for data set A (decode_b.py and what it calls). Each block shows the sha256 of the original file. The only edit is the host name in the docstrings. Run order: decode_b.py imports decode_aba.py (for plan()) and starts decode_gap_run_a3c142f9.py worker once per process. It also starts pstate_sampler.py, clock_probe.cpp (the roof probe, 7,972 B, sha256 2d80f399744f4fabb7da77643b4cfb43d35e4cb60f084e716da3e738626053bf, not inline) and needs sudo -n.

decode_b.py sha256 e7a9e06e257b1ebd931056e1cbd3c95903f23fff8de9c6b3fa2d766e480d0886 (full file)

Edit: the host name in the docstring is written as "the 16-inch M1 Max".

#!/usr/bin/env python3
"""Decode, prefill and read-roof A/B with the kernel's own pmp_report.agx_vote parameter (lab3 kernel, AGX vote patches 0001+0002).

  decode_b.py --model 2b|9b --out-dir DIR      (DIR must not exist; runs ON the 16-inch M1 Max inside one queued ticket)

Per config, FRESH processes alternate C T C (roof) or C T C T C (decode, prefill): C = agx_vote 0, T = agx_vote 1, set before the
process starts and reset to 0 after it exits (the vote itself follows GPU activity: set at the first operation, cleared 100 ms after
the last). The roof config is a 20 s sustained GPU read per process: it shows whether the product vote engages (the T2 test on the
same hardware gave 241 -> 368 GB/s) before the decode numbers are read. Every parameter write is read back; the run stops if the
kernel logs "AGX request ... not acknowledged" and the parameter is always left at 0. No go files, no w73 modules. Uses sudo -n tee on
/sys/module/pmp_report/parameters/agx_vote, the product code path reviewed in omarchy-mplus-private tools/ptd-vote.
"""
import argparse
import json
import os
import subprocess
import sys
import time
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent))
import decode_aba  # noqa: E402  (plan() only; decode_aba.py itself is not modified, running tickets pin its hash)

H = Path("/home/joshuawarren/scratch/w7x-decode-gap")
P = Path("/home/joshuawarren/scratch/w72/linux-parity")
PARAM = "/sys/module/pmp_report/parameters/agx_vote"


class T2Error(Exception):
    pass


def _sudo_write(path, value):
    subprocess.run(["sudo", "-n", "tee", path], input=value + "\n", text=True, stdout=subprocess.DEVNULL, check=True)


def _dmesg_vote_errors():
    out = subprocess.run(["sudo", "-n", "dmesg"], capture_output=True, text=True, check=True).stdout
    return sum(1 for line in out.splitlines() if "AGX request" in line and "not acknowledged" in line)


class ParamCtl:
    def __init__(self, path=PARAM, writer=_sudo_write, vote_errors=_dmesg_vote_errors):
        self.path, self.writer, self.vote_errors = path, writer, vote_errors

    def get(self):
        """'1' or '0'. The kernel reads a bool parameter back as Y or N."""
        try:
            raw = Path(self.path).read_text().strip()
        except OSError as e:
            raise T2Error(f"cannot read {self.path}: {e}")
        return {"Y": "1", "N": "0"}.get(raw, raw)

    def set(self, v):
        self.writer(self.path, str(v))
        if self.get() != str(v):
            raise T2Error(f"{self.path} reads {self.get()!r} after writing {v}")


def plan(model):
    roof = dict(name="roof-read-20s", model=model, size=0, mode="roof", pattern="CTC", tokens=0)
    return [roof] + decode_aba.plan(model)

# The roof probe (clock_probe) compiles its shader by a path relative to the build tree, so it must run with that tree as its working
# directory; these inputs are checked before any GPU work (the first lab3 attempt failed mid-run on exactly this).
PROBE_INPUTS = ("clock_probe.cpp", "tools/q4-bw-bench/bench.cpp", "tools/q4-bw-bench/shaders/peak_read.comp")


def probe_prereqs(bw):
    return [str(Path(bw) / rel) for rel in PROBE_INPUTS if not (Path(bw) / rel).exists()]



def _run_proc(cfg, label, k, ctl, out_dir, py, run_script, model_dir, probe, probe_cwd=None):
    d = Path(out_dir) / f"run-{k}"
    d.mkdir()
    rec = dict(k=k, label=label, cfg=cfg["name"], model=cfg["model"], size=cfg["size"], mode=cfg["mode"], tokens=cfg["tokens"],
               n=None, start_rt=time.time_ns())
    if cfg["mode"] == "roof":
        cmd = list(probe) + ["--burn", "read", "--seconds", "20", "--reps", "4"]
        rec["probe"], rec["out"], rec["markers"] = str(d / "probe.jsonl"), str(d / "probe.jsonl"), str(d / "probe.jsonl")
        env = dict(os.environ, VK_DRIVER_FILES="/usr/share/vulkan/icd.d/asahi_icd.json", W7X_EXPECT_DEVICE="Apple M1 Max")
        sink = open(d / "probe.jsonl", "w")
    else:
        rec["out"], rec["markers"] = str(d / "run.json"), str(d / "markers.jsonl")
        cmd = [str(py), str(run_script), "worker", "--model", str(Path(model_dir) / cfg["model"]), "--size", str(cfg["size"]),
               "--decode-tokens", str(cfg["tokens"]), "--out", rec["out"], "--markers", rec["markers"]]
        env = dict(os.environ, HF_HUB_OFFLINE="1", TRANSFORMERS_OFFLINE="1")
        sink = open(d / "stdout.log", "w")
    base_errors = ctl.vote_errors()
    try:
        ctl.set(1 if label == "T" else 0)
        rec["param"] = ctl.get()
        rec["param_rt"] = time.time_ns()
        with sink, open(d / "stderr.log", "w") as se:
            rc = subprocess.run(cmd, stdout=sink, stderr=se, env=env, cwd=probe_cwd if cfg["mode"] == "roof" else None).returncode
    finally:
        ctl.set(0)  # always left off, also after a failed worker
    rec["end_rt"] = time.time_ns()
    if rc != 0 or not Path(rec["out"]).exists():
        raise T2Error(f"run-{k} failed rc={rc}; see {d}/stderr.log")
    if ctl.vote_errors() > base_errors:
        raise T2Error(f'kernel logged "AGX request ... not acknowledged" during run-{k}; stopping with agx_vote 0')
    return rec


def run_config(cfg, out_dir, ctl, py, run_script, model_dir, probe, probe_cwd=None):
    existing = len([p for p in Path(out_dir).glob("run-*") if p.is_dir()])
    recs = []
    for i, label in enumerate(cfg["pattern"]):
        rec = _run_proc(cfg, label, existing + i + 1, ctl, out_dir, py, run_script, model_dir, probe, probe_cwd)
        recs.append(rec)
        with open(Path(out_dir) / "runs.jsonl", "a") as f:
            f.write(json.dumps(rec) + "\n")
    return recs


def main(argv=None):
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--model", required=True)
    ap.add_argument("--out-dir", required=True)
    ap.add_argument("--py", default=str(P / "venv-0736/bin/python"))
    ap.add_argument("--run", default=str(H / "decode_gap_run_a3c142f9.py"))
    ap.add_argument("--models", default=str(P / "models"))
    a = ap.parse_args(argv)
    out = Path(a.out_dir)
    if out.exists():
        print(f"STOP: {out} exists")
        return 2
    cfgs = plan(a.model)
    bw = H / "bw"
    missing = probe_prereqs(bw)
    if missing:
        print("STOP: probe inputs missing: " + ", ".join(missing))
        return 3
    out.mkdir(parents=True)
    ctl = ParamCtl()
    (out / "env.txt").write_text(f"kernel {os.uname().release}\nagx_vote {ctl.get()}\n"
                                 f"agx_vote_idle_ms {Path(PARAM.replace('agx_vote', 'agx_vote_idle_ms')).read_text().strip()}\n")
    if ctl.get() != "0":
        print("STOP: agx_vote is not 0 at the start")
        return 2
    built = subprocess.run(["g++", "-std=c++17", "-O2", "-o", str(out / "clock_probe"), "clock_probe.cpp"], cwd=bw, capture_output=True, text=True)
    if built.returncode != 0:
        print("STOP: probe build failed\n" + built.stderr[-600:])
        return 3
    samplers = [subprocess.Popen(["sudo", "-n", "python3", str(bw / "pstate_sampler.py"), str(out / "stats.jsonl"), str(out / "stop"), "0.05"])]
    time.sleep(2)
    rc = 0
    try:
        for cfg in cfgs:
            print(f"CONFIG {cfg['name']} {cfg['pattern']} {time.strftime('%T', time.gmtime())}Z", flush=True)
            run_config(cfg, out, ctl, a.py, a.run, a.models, [str(out / "clock_probe")], probe_cwd=bw)
            time.sleep(5)
    except (T2Error, subprocess.CalledProcessError) as e:
        print(f"STOP: {e}", flush=True)
        rc = 4
    finally:
        try:
            ctl.set(0)
        except T2Error as e:
            print(f"WARNING: could not confirm agx_vote 0: {e}", flush=True)
        (out / "stop").write_text("")
        for s in samplers:
            s.wait()
        subprocess.call(["sudo", "-n", "chown", "-R", f"{os.getuid()}:{os.getgid()}", str(out)])
    print(f"DECODE_B_DONE {out} rc={rc} agx_vote={ctl.get()}", flush=True)
    return rc


if __name__ == "__main__":
    sys.exit(main())
decode_aba.py sha256 44989247fca196676972e8fada5b9dd711680093d554b5e5cd79ddfc4661e1d9 (full file)

Edit: the host name in the docstring is written as "the 16-inch M1 Max".

#!/usr/bin/env python3
"""Decode and prefill A/B/A/B/A over whole processes with PMP PS-REQ bit 9 (AGX). Runs ON the 16-inch M1 Max inside one queued ticket.

  decode_aba.py --model 2b|9b --out-dir DIR     (DIR must not exist; w73's t2-follow.sh watches it)

Each config runs FRESH worker processes (decode_gap_run.py worker) in the pattern C T C T C. In a T process this script touches
DIR/set-go.N when the worker's flushed `timed_start` mark appears (model loaded and warmed up, GPU active) and DIR/clear-go.N when
the flushed `timed_done` mark appears, so bit 9 is set only while that process is generating and never across the model load. The
follower does the register writes; this script writes none. No token-indexed trigger exists (the worker does not flush token marks).
It waits for the follower's ack of every clear before the next process starts and stops if DIR/abort appears.
Follower log format (t2-follow.sh): "<ns> N=<N> SET|CLEAR <PS-REQ word>[ rc=<rc>]"; other tokens are not acks.
"""
import argparse
import json
import os
import subprocess
import sys
import time
from pathlib import Path

H = Path("/home/joshuawarren/scratch/w7x-decode-gap")
P = Path("/home/joshuawarren/scratch/w72/linux-parity")

TOKENS = {"2b": (1000, 800), "9b": (600, 500)}  # decode tokens at context 512 and 8192
CLEAR_VERIFY_S = 0.4


class T2Error(Exception):
    pass


def plan(model):
    if model not in TOKENS:
        raise T2Error(f"unknown model {model!r}")
    t512, t8k = TOKENS[model]
    return [
        dict(name="decode-512", model=model, size=512, mode="decode", pattern="CTCTC", tokens=t512),
        dict(name="decode-8192", model=model, size=8192, mode="decode", pattern="CTCTC", tokens=t8k),
        dict(name="prefill-8192", model=model, size=8192, mode="prefill", pattern="CTCTC", tokens=8),
    ]


def _touch(path):
    Path(path).write_text("")


def _acked(out_dir, n, kind):
    log = Path(out_dir) / "t2-follow.log"
    if not log.exists():
        return False
    return any(len(p) >= 3 and p[1] == f"N={n}" and p[2] == kind.upper() and (kind != "clear" or "rc=0" in p)
               for p in (l.split() for l in log.read_text().splitlines()))


def _wait_ack(out_dir, n, kind, timeout_s):
    end = time.monotonic() + timeout_s
    while time.monotonic() < end:
        if (Path(out_dir) / "abort").exists():
            raise T2Error("follower wrote abort")
        if _acked(out_dir, n, kind):
            return
        time.sleep(0.02)
    raise T2Error(f"no {kind} ack for N={n} within {timeout_s} s")


def _run_one(cfg, label, k, n, out_dir, py, run_script, model_dir):
    d = Path(out_dir) / f"run-{k}"
    d.mkdir()
    out, markers = d / "run.json", d / "markers.jsonl"
    cmd = [str(py), str(run_script), "worker", "--model", str(Path(model_dir) / cfg["model"]), "--size", str(cfg["size"]),
           "--decode-tokens", str(cfg["tokens"]), "--out", str(out), "--markers", str(markers)]
    env = dict(os.environ, HF_HUB_OFFLINE="1", TRANSFORMERS_OFFLINE="1")
    rec = dict(k=k, label=label, n=n if label == "T" else None, cfg=cfg["name"], model=cfg["model"], size=cfg["size"], mode=cfg["mode"],
               tokens=cfg["tokens"], out=str(out), markers=str(markers), start_rt=time.time_ns())
    set_done = clr_done = False
    pos = 0
    with open(d / "stdout.log", "w") as so, open(d / "stderr.log", "w") as se:
        proc = subprocess.Popen(cmd, stdout=so, stderr=se, env=env)
        try:
            while True:
                if (Path(out_dir) / "abort").exists():
                    raise T2Error("follower wrote abort")
                alive = proc.poll() is None
                if markers.exists():
                    with open(markers, "rb") as f:
                        f.seek(pos)
                        chunk = f.read()
                    cut = chunk.rfind(b"\n") + 1
                    pos += cut
                    for line in chunk[:cut].decode().splitlines():
                        p = json.loads(line)["p"]
                        if label == "T" and p == "timed_start" and not set_done:
                            _touch(Path(out_dir) / f"set-go.{n}")
                            set_done = True
                        elif label == "T" and p == "timed_done" and set_done and not clr_done:
                            _touch(Path(out_dir) / f"clear-go.{n}")
                            clr_done = True
                if not alive and not (markers.exists() and markers.stat().st_size > pos):
                    break
                time.sleep(0.005)
        finally:
            if proc.poll() is None:
                proc.kill()
            proc.wait()
            if set_done and not clr_done:
                _touch(Path(out_dir) / f"clear-go.{n}")  # never leave bit 9 set after a worker dies or an abort
                clr_done = True
    if proc.returncode != 0 or not out.exists():
        raise T2Error(f"worker run-{k} failed rc={proc.returncode}; see {d}/stderr.log")
    rec["end_rt"] = time.time_ns()
    return rec


def run_config(cfg, out_dir, py, run_script, model_dir, ack_timeout_s=10.0, start_n=1):
    existing = len([p for p in Path(out_dir).glob("run-*") if p.is_dir()])
    recs, n = [], start_n
    for i, label in enumerate(cfg["pattern"]):
        k = existing + i + 1
        rec = _run_one(cfg, label, k, n, out_dir, py, run_script, model_dir)
        recs.append(rec)
        with open(Path(out_dir) / "runs.jsonl", "a") as f:
            f.write(json.dumps(rec) + "\n")
        if label == "T":
            _wait_ack(out_dir, n, "set", ack_timeout_s)
            _wait_ack(out_dir, n, "clear", ack_timeout_s)
            time.sleep(CLEAR_VERIFY_S)  # the follower logs CLEAR before its verifying read; a CLEAR-FAILED abort follows within this time
            if (Path(out_dir) / "abort").exists():
                raise T2Error("follower wrote abort after the clear")
            n += 1
    return recs


def main(argv=None):
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--model", required=True)
    ap.add_argument("--out-dir", required=True)
    ap.add_argument("--py", default=str(P / "venv-0736/bin/python"))
    ap.add_argument("--run", default=str(H / "decode_gap_run_a3c142f9.py"))
    ap.add_argument("--models", default=str(P / "models"))
    a = ap.parse_args(argv)
    out = Path(a.out_dir)
    if out.exists():
        print(f"STOP: {out} exists")
        return 2
    cfgs = plan(a.model)
    out.mkdir(parents=True)
    bw = H / "bw"
    samplers = [subprocess.Popen(["sudo", "-n", "python3", str(bw / "pstate_sampler.py"), str(out / "stats.jsonl"), str(out / "stop"), "0.05"]),
                subprocess.Popen(["sudo", "-n", "python3", str(bw / "dcsfab_sampler.py"), str(out / "dcsfab.jsonl"), str(out / "stop"), "0.05"])]
    time.sleep(2)
    rc, n = 0, 1
    try:
        for cfg in cfgs:
            print(f"CONFIG {cfg['name']} {cfg['pattern']} {time.strftime('%T', time.gmtime())}Z", flush=True)
            recs = run_config(cfg, out, a.py, a.run, a.models, start_n=n)
            n = max([r["n"] for r in recs if r["n"]] + [n - 1]) + 1
            time.sleep(5)
    except T2Error as e:
        print(f"STOP: {e}", flush=True)
        rc = 4
    finally:
        _touch(out / "done")
        _touch(out / "stop")
        for s in samplers:
            s.wait()
        subprocess.call(["sudo", "-n", "chown", "-R", f"{os.getuid()}:{os.getgid()}", str(out)])
    print(f"DECODE_ABA_DONE {out} rc={rc}", flush=True)
    return rc


if __name__ == "__main__":
    sys.exit(main())
decode_gap_run_a3c142f9.py sha256 a3c142f9467a2043c03e6b832224568dc0dba76ca83e1abc889cbb4cae28691b (full file; the file name carries the first 8 digits of its sha256)

Edit: the host name in the docstring is written as "the 16-inch M1 Max".

#!/usr/bin/env python3
"""Linux MLX decode-gap measurement on the 16-inch M1 Max (class=perf). Runs ON the 16-inch M1 Max, inside one queued ticket.

Arms (every process is fresh; same prompt ids as matrix_cell.py so rates are comparable to the parity matrix):
  R   release wheel (venv-0736), defaults                       -> the 94.6 / 83.2 control
  D0  diag wheel, defaults, profiler compiled in but OFF        -> how far the diag build is from the release
  D1  diag wheel, MLX_DISABLE_COMPILE=1, profiler OFF           -> unprofiled baseline for the profiled arm
  D2  diag wheel, MLX_DISABLE_COMPILE=1, MLX_OMARCHY_GPU_PROFILE=<file>  -> the breakdown (perturbs timing)
  D3  diag wheel, defaults (compile as shipped), MLX_OMARCHY_GPU_PROFILE=<file> -> dispatch counts per token in the
      configuration users run, comparable to the macOS counts; its stream may omit compiled-tape work (counts show it)
Per process the worker records one CLOCK_MONOTONIC ns stamp per token; the profiled arm also leaves the
profile stream, which decode_breakdown.py turns into GPU busy, gaps, dispatches and submits per token.

  decode_gap_run.py run --out-dir DIR [--ctx 512 8192] [--reps 5] [--prof-reps 2] [--decode-tokens 64]
  decode_gap_run.py worker --model M --size N --decode-tokens K --out run.json --markers m.jsonl
"""
import argparse
import hashlib
import json
import os
import statistics
import subprocess
import sys
from pathlib import Path

HOME = Path(os.path.expanduser("~"))
PARITY = HOME / "scratch/w72/linux-parity"
REL_PY = PARITY / "venv-0736/bin/python"
DIAG_PY = HOME / "scratch/w7x-decode-gap/venv-diag/bin/python"
MODEL = PARITY / "models/2b"
SPREAD = 0.05
# order matters: the profiled arm goes first so a build without the profiler fails in the first minute
ARMS = {
    "D2": {"py": DIAG_PY, "env": {"MLX_DISABLE_COMPILE": "1"}, "profile": True},
    "D3": {"py": DIAG_PY, "env": {}, "profile": True},
    "R": {"py": REL_PY, "env": {}, "profile": False},
    "D0": {"py": DIAG_PY, "env": {}, "profile": False},
    "D1": {"py": DIAG_PY, "env": {"MLX_DISABLE_COMPILE": "1"}, "profile": False},
    # second experiment: release wheel only, profiler absent. B0 restores the historic unconditional pre+post
    # barriers (about 2 per dispatch instead of 223 per token); W groups hazard-free dispatches into waves;
    # H leaves the engine untouched and times the host side of each token (graph build, async_eval) instead.
    "B0": {"py": REL_PY, "env": {"MLX_OMARCHY_GATED_BARRIERS": "0"}, "profile": False},
    "W": {"py": REL_PY, "env": {"MLX_OMARCHY_WAVE_SCHED": "1"}, "profile": False},
    "H": {"py": REL_PY, "env": {}, "profile": False, "hostsplit": True, "max_reps": 3},
    # third experiment: HC adds CPU time (all threads, per thread) over the decode window to the host split, to
    # tell a host-bound token from a token the host spends blocked on the GPU. WD verifies WAVE_SCHED takes
    # effect: the engine's own census prints each flush's wave structure to stderr.
    "HC": {"py": REL_PY, "env": {}, "profile": False, "hostsplit": True, "cpu": True, "max_reps": 3},
    "WD": {"py": REL_PY, "env": {"MLX_OMARCHY_WAVE_SCHED": "1", "MLX_OMARCHY_WAVE_DIAG": "1"}, "profile": False,
           "max_reps": 1},
    # fourth experiment: ST traces the ioctls (count and duration) of one decode process; --seccomp-bpf stops the
    # process only on the traced calls, so the few ioctls per token barely perturb it.
    "ST": {"py": REL_PY, "env": {}, "profile": False, "strace": True, "max_reps": 1},
    # fifth experiment: the per-board submission work cap (MLX_OMARCHY_BATCH_WORK; default 40000 on G13C gives 3 submits
    # per decode token, 0 = off gives one). BW* record CPU and wall per token; STW* count the real submit ioctls.
    "BW0": {"py": REL_PY, "env": {"MLX_OMARCHY_BATCH_WORK": "0"}, "profile": False, "hostsplit": True, "cpu": True, "max_reps": 3},
    "BW5": {"py": REL_PY, "env": {"MLX_OMARCHY_BATCH_WORK": "5000"}, "profile": False, "hostsplit": True, "cpu": True, "max_reps": 3},
    "STW0": {"py": REL_PY, "env": {"MLX_OMARCHY_BATCH_WORK": "0"}, "profile": False, "strace": True, "max_reps": 1},
    "STW5": {"py": REL_PY, "env": {"MLX_OMARCHY_BATCH_WORK": "5000"}, "profile": False, "strace": True, "max_reps": 1},
}
DEFAULT_ARMS = ["D2", "D3", "R", "D0", "D1"]


class RunError(Exception):
    pass


def plan(ctxs, reps, prof_reps, arms=None):
    """Interleaved: rep-major, then context, then arm. The profiled arm runs only in the first prof_reps reps;
    an arm may cap its own reps (max_reps)."""
    arms = arms or DEFAULT_ARMS
    unknown = [a for a in arms if a not in ARMS]
    if unknown:
        raise RunError(f"unknown arm(s) {unknown}; known: {sorted(ARMS)}")
    out = []
    for rep in range(reps):
        for ctx in ctxs:
            for arm in arms:
                spec = ARMS[arm]
                if (spec["profile"] and rep >= prof_reps) or rep >= spec.get("max_reps", reps):
                    continue
                out.append((arm, ctx, rep))
    return out


def decode_tps(run):
    t = run["tok_ns"]
    return (len(t) - 1) / ((t[-1] - t[0]) / 1e9)


def summarize_host(run):
    """Host-side ns per decoded token from the wrapped calls that ended inside the decode window; None if absent."""
    h = run.get("host")
    if not h:
        return None
    lo, hi, n = run["tok_ns"][0], run["tok_ns"][-1], len(run["tok_ns"]) - 1
    inside = {k: [x for x in h[k] if lo < x[0] <= hi] for k in ("model", "async")}
    out = {"calls_model": len(inside["model"]), "calls_async": len(inside["async"]),
           "model_ns_per_token": sum(x[1] for x in inside["model"]) / n,
           "async_ns_per_token": sum(x[1] for x in inside["async"]) / n}
    for k in ("model", "async"):  # third column, when present: CPU time of the calling thread during the call
        if inside[k] and all(len(x) > 2 for x in inside[k]):
            out[f"{k}_cpu_ns_per_token"] = sum(x[2] for x in inside[k]) / n
    return out


def summarize_cpu(run):
    """CPU ms per decoded token over the first-to-last-token window: whole process and summed by thread name."""
    c = run.get("cpu")
    if not c:
        return None
    n = len(run["tok_ns"]) - 1
    by = {}
    by, user, system, split = {}, 0.0, 0.0, True
    for rec in c["threads"].values():
        comm, ticks = rec[0], rec[1]
        by[comm] = by.get(comm, 0.0) + ticks / c["tick_hz"] * 1000.0 / n
        if len(rec) > 3:
            user += rec[2]
            system += rec[3]
        else:
            split = False
    out = {"process_ms_per_token": c["process_ns"] / 1e6 / n, "by_thread_ms_per_token": by}
    if split and c["threads"]:  # user versus kernel CPU, when the worker recorded utime and stime separately
        out["user_ms_per_token"] = user / c["tick_hz"] * 1000.0 / n
        out["sys_ms_per_token"] = system / c["tick_hz"] * 1000.0 / n
    return out


def _median_cpu(runs):
    rows = [summarize_cpu(r) for r in runs]
    if not rows or rows[0] is None:
        return None
    comms = {c for r in rows for c in r["by_thread_ms_per_token"]}
    out = {"process_ms_per_token": statistics.median(r["process_ms_per_token"] for r in rows),
           "by_thread_ms_per_token": {c: statistics.median(r["by_thread_ms_per_token"].get(c, 0.0) for r in rows)
                                      for c in comms}}
    for k in ("user_ms_per_token", "sys_ms_per_token"):
        if all(k in r for r in rows):
            out[k] = statistics.median(r[k] for r in rows)
    return out


def _median_host(runs):
    rows = [summarize_host(r) for r in runs]
    if not rows or rows[0] is None:
        return None
    return {k: statistics.median(r[k] for r in rows) for k in rows[0]}


def summarize(runs_by_key, decode_tokens):
    """runs_by_key: {(arm, ctx): [run, ...]} -> list of rows. Spread is (max-min)/median like matrix_cell."""
    rows = []
    for (arm, ctx), runs in sorted(runs_by_key.items()):
        rates = [decode_tps(r) for r in runs]
        pre = [ctx / (r["ttft_ns"] / 1e9) for r in runs]
        gaps = sorted(b - a for r in runs for a, b in zip(r["tok_ns"], r["tok_ns"][1:]))
        med = statistics.median(rates)
        rows.append({
            "arm": arm, "ctx": ctx, "n": len(runs), "decode_tps_median": med,
            "decode_tps": rates, "spread": (max(rates) - min(rates)) / med,
            "prefill_tps_median": statistics.median(pre),
            "tok_wall_ms_p50": gaps[len(gaps) // 2] / 1e6, "tok_wall_ms_p99": gaps[int(0.99 * (len(gaps) - 1))] / 1e6,
            "generated_digests": sorted({r["generated_digest"] for r in runs}),
            "prompt_digests": sorted({r["prompt_digest"] for r in runs}),
            "versions": sorted({r["version"] for r in runs}),
            "host": _median_host(runs),
            "cpu": _median_cpu(runs),
        })
    return rows


def check_profile(path):
    """A profiled arm that wrote nothing means the wheel has no profiler: stop before spending more GPU time."""
    p = Path(path)
    if not p.is_file() or p.stat().st_size == 0:
        raise RunError(f"profile stream missing or empty: {p}")
    kinds = {"d": 0, "meta": 0, "end": 0}
    with p.open(encoding="utf-8") as f:
        for line in f:
            k = json.loads(line).get("k")
            if k in kinds:
                kinds[k] += 1
    if kinds["meta"] != 1 or kinds["d"] < 100 or kinds["end"] != 1:
        raise RunError(f"profile stream incomplete: {kinds} in {p}")


def run_all(steps, run_proc, out_dir, decode_tokens):
    out = Path(out_dir)
    if out.exists():
        raise RunError(f"out dir must be fresh: {out}")
    out.mkdir(parents=True)
    by_key = {}
    for arm, ctx, rep in steps:
        d = out / f"{arm}-ctx{ctx}-r{rep}"
        d.mkdir()
        run = run_proc(arm, ctx, d)  # any failure raises: no later launch
        if ARMS[arm]["profile"]:
            check_profile(d / "profile.jsonl")
        by_key.setdefault((arm, ctx), []).append(run)
        print(f"DONE {arm} ctx={ctx} r{rep} decode_tps={decode_tps(run):.2f}", flush=True)
    rows = summarize(by_key, decode_tokens)
    (out / "summary.json").write_text(json.dumps(rows, indent=1))
    return rows


def worker_cmd(arm, ctx, d, model, decode_tokens):
    spec = ARMS[arm]
    cmd = [str(spec["py"]), str(Path(__file__).resolve()), "worker", "--model", str(model), "--size", str(ctx),
           "--decode-tokens", str(decode_tokens), "--out", str(d / "run.json"), "--markers", str(d / "markers.jsonl")]
    if spec.get("hostsplit"):
        cmd.append("--hostsplit")
    if spec.get("cpu"):
        cmd.append("--cpu")
    if spec.get("strace"):
        cmd = ["strace", "--seccomp-bpf", "-f", "-ttt", "-T", "-e", "trace=ioctl", "-o", str(d / "strace.log")] + cmd
    return cmd


def real_run_proc(model, decode_tokens, deadline):
    def run_proc(arm, ctx, d):
        spec = ARMS[arm]
        env = {k: v for k, v in os.environ.items() if not k.startswith(("MLX_OMARCHY_QMM_", "MLX_DISABLE_COMPILE", "MLX_OMARCHY_GPU_PROFILE"))}
        env.update({"HF_HUB_OFFLINE": "1", "TRANSFORMERS_OFFLINE": "1"}, **spec["env"])
        if spec["profile"]:
            env["MLX_OMARCHY_GPU_PROFILE"] = str(d / "profile.jsonl")
        cmd = worker_cmd(arm, ctx, d, model, decode_tokens)
        with (d / "stdout.log").open("x") as so, (d / "stderr.log").open("x") as se:
            try:
                p = subprocess.run(cmd, stdout=so, stderr=se, env=env, timeout=deadline)
            except subprocess.TimeoutExpired:
                raise RunError(f"{d.name}: deadline {deadline}s; no later launch")
        if p.returncode != 0:
            raise RunError(f"{d.name}: child exit {p.returncode}; no later launch")
        return json.loads((d / "run.json").read_text())
    return run_proc


def worker(model_path, size, decode_tokens, out_path, markers_path, hostsplit=False, cpu=False):
    import importlib.metadata as md
    import time
    sys.path.insert(0, str(PARITY))
    import matrix_cell as mc  # the pinned parity runner: same seed text, same id builder, same digest
    import mlx.core as mx
    from mlx_lm.generate import stream_generate
    from mlx_lm.sample_utils import make_sampler
    from mlx_lm.utils import load

    model, tokenizer = load(str(model_path))
    tokenizer.eos_token_ids = set()
    ids = mc.build_ids(tokenizer.encode(mc.SEED_TEXT, add_special_tokens=False), size)
    mx.random.seed(0)
    sampler = make_sampler(temp=0.0)
    host = {"model": [], "async": []} if hostsplit else None
    if hostsplit:
        def timed(store, fn):
            def wrapper(*a, **k):
                t0 = time.perf_counter_ns()
                c0 = time.thread_time_ns()
                out = fn(*a, **k)
                store.append([time.monotonic_ns(), time.perf_counter_ns() - t0, time.thread_time_ns() - c0])
                return out
            return wrapper
        cls = type(model)
        cls.__call__ = timed(host["model"], cls.__call__)  # graph construction (lazy ops)
        mx.async_eval = timed(host["async"], mx.async_eval)  # graph walk, encode, vkQueueSubmit
    mk = open(markers_path, "w", encoding="utf-8")

    def mark(p):
        mk.write(json.dumps({"t": time.monotonic_ns(), "rt": time.time_ns(), "p": p}) + "\n")
        mk.flush()

    snaps = {}

    def cpu_snapshot():
        s = {"process_ns": time.process_time_ns(), "threads": {}}
        for tid in os.listdir("/proc/self/task"):
            try:
                comm = open(f"/proc/self/task/{tid}/comm").read().strip()
                fields = open(f"/proc/self/task/{tid}/stat").read().rsplit(")", 1)[1].split()
            except OSError:  # thread ended between listdir and open
                continue
            s["threads"][tid] = [comm, int(fields[11]) + int(fields[12]), int(fields[11]), int(fields[12])]  # total, utime, stime ticks
        return s

    def generate(prompt, count, stamp):
        toks, stamps = [], []
        for resp in stream_generate(model, tokenizer, mx.array(prompt), max_tokens=count, sampler=sampler):
            now = time.monotonic_ns()
            toks.append(int(resp.token))
            stamps.append(now)
            if stamp and cpu and len(toks) == 1:
                snaps["first"] = cpu_snapshot()  # the decode window starts at the first token, as decode_tps does
            if stamp:
                mk.write(json.dumps({"t": now, "rt": time.time_ns(), "p": "tok"}) + "\n")
        if stamp and cpu:
            snaps["last"] = cpu_snapshot()
        mx.synchronize()
        return toks, stamps

    generate(ids[:512], 8, False)  # untimed warmup, as in matrix_cell
    mx.synchronize()
    mark("timed_start")
    t0 = time.monotonic_ns()
    toks, stamps = generate(ids, decode_tokens, True)
    mark("timed_done")
    mk.close()
    dist = next(d for d in ("mlx-omarchy", "mlx") if _installed(md, d))
    if hostsplit and not host["async"]:
        raise RunError("hostsplit: mx.async_eval was never called through the wrapper; the split would be empty")
    out = {
        "size": size, "ttft_ns": stamps[0] - t0, "tok_ns": stamps, "generated_n": len(toks),
        "prompt_digest": mc.ids_digest(ids), "generated_digest": mc.ids_digest(toks),
        "version": md.version(dist),
    }
    if hostsplit:
        out["host"] = host
    if cpu:
        a, b = snaps["first"], snaps["last"]
        out["cpu"] = {"process_ns": b["process_ns"] - a["process_ns"], "tick_hz": os.sysconf("SC_CLK_TCK"),
                      "threads": {tid: [rec[0]] + [x - y for x, y in zip(rec[1:], a["threads"].get(tid, [rec[0], 0, 0, 0])[1:])]
                                  for tid, rec in b["threads"].items()}}
    Path(out_path).write_text(json.dumps(out))


def _installed(md, name):
    try:
        md.version(name)
        return True
    except md.PackageNotFoundError:
        return False


def main(argv=None):
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    sub = ap.add_subparsers(dest="cmd", required=True)
    w = sub.add_parser("worker")
    w.add_argument("--model", required=True)
    w.add_argument("--size", type=int, required=True)
    w.add_argument("--decode-tokens", type=int, required=True)
    w.add_argument("--out", required=True)
    w.add_argument("--markers", required=True)
    w.add_argument("--hostsplit", action="store_true")
    w.add_argument("--cpu", action="store_true")
    r = sub.add_parser("run")
    r.add_argument("--out-dir", required=True)
    r.add_argument("--ctx", type=int, nargs="+", default=[512, 8192])
    r.add_argument("--reps", type=int, default=5)
    r.add_argument("--prof-reps", type=int, default=2)
    r.add_argument("--decode-tokens", type=int, default=64)
    r.add_argument("--deadline", type=int, default=600)
    r.add_argument("--arms", nargs="+", default=None, help="default: the first-pass arms " + " ".join(DEFAULT_ARMS))
    a = ap.parse_args(argv)
    try:
        if a.cmd == "worker":
            worker(a.model, a.size, a.decode_tokens, a.out, a.markers, a.hostsplit, a.cpu)
            return 0
        rows = run_all(plan(a.ctx, a.reps, a.prof_reps, a.arms), real_run_proc(MODEL, a.decode_tokens, a.deadline),
                       a.out_dir, a.decode_tokens)
    except RunError as exc:
        print(f"STOP: {exc}", file=sys.stderr)
        return 1
    for row in rows:
        print(json.dumps({k: row[k] for k in ("arm", "ctx", "n", "decode_tps_median", "spread", "tok_wall_ms_p50")}))
    return 0


if __name__ == "__main__":
    sys.exit(main())
pstate_sampler.py sha256 619fff312be62d8b70ba564659949c174a7ad6fb6f5bbc98be2764e7051ccb76 (full file)
#!/usr/bin/env python3
"""Sample the asahi driver's debugfs stats file until a stop file appears (read-only). Run under one `sudo -n`.

  pstate_sampler.py OUT.jsonl STOPFILE [PERIOD_S]     (path override for tests: W7X_STATS_PATH)

Each line: {"t": realtime ns, "mono": monotonic ns, "stats": <raw file text>}. Raw text is kept so nothing is
parsed away; pstate_analyze.py reads it offline. Realtime stamps let phase marks written by a shell line up.
"""
import json
import os
import sys
import time

PATH = os.environ.get("W7X_STATS_PATH", "/sys/kernel/debug/dri/406400000.gpu/asahi_stats")


def main(argv):
    out_path, stop = argv[1], argv[2]
    period = float(argv[3]) if len(argv) > 3 else 0.05
    with open(out_path, "w", encoding="utf-8") as out:
        while not os.path.exists(stop):
            t, mono = time.time_ns(), time.monotonic_ns()
            with open(PATH, encoding="utf-8") as f:
                text = f.read()
            out.write(json.dumps({"t": t, "mono": mono, "stats": text}) + "\n")
            out.flush()
            time.sleep(period)
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv))

@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Part 3 of 5: the harness for data set B. The wrapper is the full file. decode_b.py supplies PARAM, T2Error and ParamCtl to the wrapper (the excerpt below is lines 26 to 58 of the file in comment 2). matrix_cell.py is the full file.

agx_vote_wrap.py sha256 67d01576b68e52e75274a4b1a647c42f77f381329644ddf5642bd6b6e3214a31 (full file)
#!/usr/bin/env python3
"""Run ONE command with the kernel's pmp_report.agx_vote set to 0 or 1, then always put it back to 0 (lab3 / shipping kernel).

  agx_vote_wrap.py --vote 0|1 -- COMMAND [ARGS...]

For per-cell A/B in a benchmark matrix: wrap each fresh benchmark process, alternating --vote 0 and --vote 1 (the vote itself follows
GPU activity: set at the first GPU operation, cleared agx_vote_idle_ms after the last). The wrapper refuses to start unless the
parameter reads 0, reads the value back after writing it, always resets to 0 (command failure, a command that cannot start, Ctrl-C),
passes the command's exit code through, and returns 97 instead if the kernel logged "AGX request ... not acknowledged" while the
command ran. Uses `sudo -n tee` on /sys/module/pmp_report/parameters/agx_vote; writes nothing else.
"""
import argparse
import os
import subprocess
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent))
from decode_b import PARAM, ParamCtl, T2Error  # noqa: E402,F401  (tested there: Y/N readback, dmesg error counter)

EXIT_VOTE_ERROR = 97


def run(ctl, vote, cmd, stdout=None, before_exit=None):
    if ctl.get() != "0":
        raise T2Error(f"{ctl.path} is not 0 at the start; refusing to touch it")
    base = ctl.vote_errors()
    try:
        ctl.set(vote)
        rc = subprocess.run(cmd, stdout=stdout).returncode
    finally:
        ctl.set(0)  # always left off
    if before_exit:
        before_exit()
    if ctl.vote_errors() > base:
        print('agx_vote_wrap: the kernel logged "AGX request ... not acknowledged" during the command; agx_vote is back at 0', file=sys.stderr)
        return EXIT_VOTE_ERROR
    return rc


def main(argv=None):
    a = list(sys.argv[1:] if argv is None else argv)
    if "--" not in a:
        print(__doc__)
        return 2
    i = a.index("--")
    ap = argparse.ArgumentParser(add_help=False)
    ap.add_argument("--vote", type=int, choices=(0, 1), required=True)
    ap.add_argument("--param", default=PARAM)
    opts = ap.parse_args(a[:i])
    cmd = a[i + 1:]
    if not cmd:
        print("no command after --")
        return 2
    if os.environ.get("W7X_AGX_VOTE_TEST_PLAIN"):  # tests only: a plain file that behaves like the kernel bool parameter
        ctl = ParamCtl(opts.param, writer=lambda p, v: Path(p).write_text(("Y" if v == "1" else "N") + "\n"), vote_errors=lambda: 0)
    else:
        ctl = ParamCtl(opts.param)
    try:
        return run(ctl, opts.vote, cmd)
    except T2Error as e:
        print(f"agx_vote_wrap: {e}", file=sys.stderr)
        return 96


if __name__ == "__main__":
    sys.exit(main())
decode_b.py lines 26 to 58 (excerpt: PARAM, T2Error, ParamCtl; imports needed: os, subprocess, sys, Path)
import os, subprocess, sys
from pathlib import Path
PARAM = "/sys/module/pmp_report/parameters/agx_vote"


class T2Error(Exception):
    pass


def _sudo_write(path, value):
    subprocess.run(["sudo", "-n", "tee", path], input=value + "\n", text=True, stdout=subprocess.DEVNULL, check=True)


def _dmesg_vote_errors():
    out = subprocess.run(["sudo", "-n", "dmesg"], capture_output=True, text=True, check=True).stdout
    return sum(1 for line in out.splitlines() if "AGX request" in line and "not acknowledged" in line)


class ParamCtl:
    def __init__(self, path=PARAM, writer=_sudo_write, vote_errors=_dmesg_vote_errors):
        self.path, self.writer, self.vote_errors = path, writer, vote_errors

    def get(self):
        """'1' or '0'. The kernel reads a bool parameter back as Y or N."""
        try:
            raw = Path(self.path).read_text().strip()
        except OSError as e:
            raise T2Error(f"cannot read {self.path}: {e}")
        return {"Y": "1", "N": "0"}.get(raw, raw)

    def set(self, v):
        self.writer(self.path, str(v))
        if self.get() != str(v):
            raise T2Error(f"{self.path} reads {self.get()!r} after writing {v}")
matrix_cell.py sha256 f9c217b49821818760f8f9bbe52c927a32d4fec716a33f74e354fdeb83f83fec (full file)
#!/usr/bin/env python3
"""One (model, prompt length) cell of the Linux/macOS parity matrix.

Each process is fresh: load, one untimed 512-token warmup, then one timed
stream_generate over `size` ids for `decode_tokens` tokens. ttft_s is the
first-token-inclusive stream prefill (it may include one S1 lookahead step);
decode_s is first token to last token. Prompt ids come from the contract
seed text, encoded once by the model's own tokenizer, repeated and truncated;
the ids digest is recorded so another platform can prove the same input.
A cell is VALID only with >= 5 processes, identical prompt and generated-id
digests in all of them, and spread <= 5% for both rates.
"""
import argparse
import hashlib
import json
import math
import os
import statistics
import struct
import subprocess
import sys
import time
from pathlib import Path

SEED_TEXT = 'Explain how a computer stores and retrieves information. '
MIN_PROCESSES, SPREAD = 5, 0.05


class CellError(Exception):
    pass


def build_ids(seed_ids, n):
    if not seed_ids or not isinstance(n, int) or n < 1:
        raise CellError(f'need a non-empty seed and n >= 1, got {len(seed_ids or [])} ids, n={n!r}')
    return (list(seed_ids) * (-(-n // len(seed_ids))))[:n]


def ids_digest(ids):
    return hashlib.sha256(struct.pack('<I', len(ids)) + struct.pack(f'<{len(ids)}I', *ids)).hexdigest()


def refuse_route_env(env):
    knobs = sorted(k for k in env if k.startswith('MLX_OMARCHY_QMM_'))
    if knobs:
        raise CellError(f'inherited route knobs would change the measured route: {knobs}')


def _positive(v):
    return type(v) in (int, float) and math.isfinite(v) and v > 0


def _check_run(r, size, decode_tokens):
    if r.get('size') != size or r.get('generated_n') != decode_tokens:
        raise CellError(f'wrong size or token count: {r.get("size")!r}, {r.get("generated_n")!r}')
    if not (_positive(r.get('ttft_s')) and _positive(r.get('decode_s'))):
        raise CellError(f'invalid timings: {r.get("ttft_s")!r}, {r.get("decode_s")!r}')


def _spread(values):
    return (max(values) - min(values)) / statistics.median(values)


def summarize_cell(runs, size, decode_tokens):
    for r in runs:
        _check_run(r, size, decode_tokens)
    if len({r['prompt_digest'] for r in runs}) > 1 or len({r['generated_digest'] for r in runs}) > 1:
        raise CellError('prompt or generated ids differ across processes; no rate is reported')
    prefill = [size / r['ttft_s'] for r in runs]
    decode = [(decode_tokens - 1) / r['decode_s'] for r in runs]
    spread = {'prefill': _spread(prefill), 'decode': _spread(decode)}
    if len(runs) < MIN_PROCESSES:
        status = 'INCOMPLETE'
    else:
        status = 'VALID' if max(spread.values()) <= SPREAD else 'INVALID_NOISY'
    return {'status': status, 'n': len(runs), 'size': size, 'decode_tokens': decode_tokens,
            'prefill_tps_median': statistics.median(prefill),
            'decode_tps_median': statistics.median(decode), 'spread': spread,
            'prefill_tps': prefill, 'decode_tps': decode,
            'prompt_digest': runs[0]['prompt_digest'],
            'generated_digest': runs[0]['generated_digest']}


def run_cell(args, run_worker):
    out = Path(args['out_dir'])
    if out.exists():
        raise CellError(f'out dir must be fresh: {out}')
    out.mkdir(parents=True)
    runs = [run_worker(i, out) for i in range(args['processes'])]  # first failure raises
    result = summarize_cell(runs, args['size'], args['decode_tokens'])
    result['runs'] = runs
    with (out / 'cell.json').open('x') as f:
        json.dump(result, f, indent=1, sort_keys=True)
    return result


def make_real_worker(python, script, model, size, decode_tokens, deadline):
    def run_worker(i, out):
        rundir = out / f'proc{i:02d}'
        rundir.mkdir()
        cmd = [str(python), str(script), 'worker', '--model', str(model), '--size', str(size),
               '--decode-tokens', str(decode_tokens), '--out', str(rundir / 'run.json')]
        with (rundir / 'stdout.log').open('x') as so, (rundir / 'stderr.log').open('x') as se:
            try:
                p = subprocess.run(cmd, stdout=so, stderr=se, timeout=deadline)
            except subprocess.TimeoutExpired:
                raise CellError(f'{rundir.name}: deadline {deadline}s; no later launch')
        if p.returncode != 0:
            raise CellError(f'{rundir.name}: child exit {p.returncode}; no later launch')
        return json.loads((rundir / 'run.json').read_text())
    return run_worker


def worker(model_path, size, decode_tokens, out_path):
    import importlib.metadata as md
    import mlx.core as mx
    from mlx_lm.generate import stream_generate
    from mlx_lm.sample_utils import make_sampler
    from mlx_lm.utils import load

    model, tokenizer = load(str(model_path))
    tokenizer.eos_token_ids = set()
    ids = build_ids(tokenizer.encode(SEED_TEXT, add_special_tokens=False), size)
    vocab = getattr(model, 'language_model', model).args.vocab_size
    if any(i < 0 or i >= vocab for i in ids):
        raise CellError('prompt ids outside vocabulary')
    mx.random.seed(0)
    sampler = make_sampler(temp=0.0)

    def generate(prompt, count):
        times, toks = [], []
        for resp in stream_generate(model, tokenizer, mx.array(prompt), max_tokens=count, sampler=sampler):
            times.append(time.monotonic())
            toks.append(int(resp.token))
        mx.synchronize()
        return times, toks

    generate(ids[:512], 8)  # untimed: kernels and pipelines
    mx.synchronize()
    t0 = time.monotonic()
    times, toks = generate(ids, decode_tokens)
    dist = next(d for d in ('mlx-omarchy', 'mlx') if _installed(md, d))
    peak = mx.get_peak_memory() if hasattr(mx, 'get_peak_memory') else None
    Path(out_path).write_text(json.dumps({
        'size': size, 'ttft_s': times[0] - t0, 'decode_s': times[-1] - times[0],
        'generated_n': len(toks), 'prompt_digest': ids_digest(ids),
        'generated_digest': ids_digest(toks), 'generated_ids': toks,
        'distribution': dist, 'version': md.version(dist), 'peak_memory_bytes': peak,
        'window': 'ttft_s is first-token-inclusive stream prefill (may include one S1 lookahead step)',
    }))


def _installed(md, name):
    try:
        md.version(name)
        return True
    except md.PackageNotFoundError:
        return False


def main(argv=None):
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    sub = ap.add_subparsers(dest='cmd', required=True)
    w = sub.add_parser('worker')
    c = sub.add_parser('cell')
    for p in (w, c):
        p.add_argument('--model', required=True)
        p.add_argument('--size', type=int, required=True)
        p.add_argument('--decode-tokens', type=int, default=64)
    w.add_argument('--out', required=True)
    c.add_argument('--python', required=True)
    c.add_argument('--out-dir', required=True)
    c.add_argument('--processes', type=int, default=5)
    c.add_argument('--deadline', type=int, default=1500)
    a = ap.parse_args(argv)
    try:
        refuse_route_env(os.environ)
        if a.cmd == 'worker':
            worker(a.model, a.size, a.decode_tokens, a.out)
            return 0
        real = make_real_worker(a.python, Path(__file__).resolve(), a.model, a.size,
                                a.decode_tokens, a.deadline)
        res = run_cell({'processes': a.processes, 'size': a.size,
                        'decode_tokens': a.decode_tokens, 'out_dir': a.out_dir}, real)
    except CellError as exc:
        print(f'STOP: {exc}', file=sys.stderr)
        return 1
    print(json.dumps({k: v for k, v in res.items() if k != 'runs'}, sort_keys=True))
    return 0


if __name__ == '__main__':
    sys.exit(main())

@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Part 4 of 5: kernel config of 7.1.12-2-20261103.2-lab-ARCH, first half. The file is gzip, then base64. Paste both halves in order into one file cfg.b64, then run:

base64 -d cfg.b64 | gunzip > config
sha256sum config    # 443233d9b3d34544afc2b119809dc0a44c3c24c78f7e36f7f7b174b50c239fd5
base64, lines 1 to 473 of 947
H4sIAAAAAAACA5w8y3bbuJL7+xU66U3PImmJenrmeAGRoISIJBgClGxveNS2kniuY2dkOd3991MF
vgAQlDXTi46FKrwK9UaBv/3rtwF5O7382J8e7/dPT/8Mvh2eD8f96fAw+Pr4dPivQcAHCZcDGjD5
CZCjx+e3v//YH3/MJoP5p9GnkTfYHI7Ph6eB//L89fHbG/R9fHn+12//8nkSslXh+8WWZoLxpJD0
Rl5/uH/aP38b/DocXwFv4HkwyGLw+/fT6efrf/7xx7fH0/e3Pz/dv/z44+npV/m/jz+PL/99uD8N
7vfzq/H4MJk/jPfjyXT0sBgd5t7+fjSfPNz/Od3vvcWf06uH0X98qGdftdNfD7UlMVH4EUlW1/80
jfizwfW84Wi4qGFEYIco2sYtPrS5kaOgOyO0qQGCtn+k4ZkDZLmQ7apHw6vxcKTDcCiyJSwiy4i2
A5bdcJHtuKPuuAohJp95pqFplFnDxoiIixWXvOC5THOpEcmCS0aDDpLkPBKFyNOUZ7LIaJQ5B2BJ
xBLqAIk87jYnvEgzHrKIFmFSECm7g/o8TySsZ3l7BlSkjp5xHkkWsJgmSA8SwWyJkBnT+QOOyydJ
kQtabChNYfkFBwJG5NY+AhxRpICLq3UB8yShPhWCZLeFzEgi4lxS4cJU+90xuS6SPOpFwA0WQt9X
StYcAA0PeZMawrIvxY5nmxZ3mbMokLD5QiJDFQJOTTvMdUYJcG8SctyyJAK7gnj/NlgpTfE0eD2c
3n62As8SJguabAuSAdVYzOT12AP0muQ8TnHRsGM5eHwdPL+ccIQWYUezjGc6qD4B7pOo3tKHjyTz
1x/abjq0ILnkjhHUVgtBIlBDjYoIaEjg/NW6Hc1rLmRCgCM//P788nxoVYu4FVuW+i2pqgb815fa
aaVcsJsi/pLTnLpbO112RPrrwurhZ1wAr9KYA9+ABBB/ramjHBR0+3NNtsAXQKESgOOTKLLQ21Z1
nsAag9e3P1//eT0dfrTnuaIJzZivOAdEcKmtSAeJNd/1Q4qIbmnkhtMwpL5kuOAwBNUkNm68mK0y
IvHwnWCWfMZhdPCaZAGAQBx3oIcETTT9i10CHhOWuNqKNaMZ0u+2Z7KU9QJAndx0gbFgiGBOphp5
HOc9lJNEFiIhKZBQmj1Dnvmg0ErxNBQVqJ4MlFQ5VyMe+rABXearUDgERMfa0Bvqg1mkBCxFjBPu
gJxqzMPzw+Dlq8Uw9hhKpWw7nFeDfRDYDTBFIjXNpzgW9Zpk/qZYZpwEPhHybG8XmmL/UsL8NFdL
yYRSXpbyuwQH/kHPBVW1vzFIbUMKFiiNr8RJPv4AH8clUWpzPKHmqYKFW9+heowVCzcnB40p7JkH
zHfqy7Ifzuw40BIY5voJdFYNNi3TZ3QilBzXvwLH5Gu2WqPcVbTVpyjpTvKb8iSFOW7FXx0KNno8
Da2zptBUfNY1oGKlHUlko8lbFHU+8NN1OIjVYdq2a7ODqqkg0Y7cCjhOl72pcOoVmAeL0DxJM7Zt
EcLQSWFETcGTAkbvhUcidlLR3GfD+BmlcSpLk+gQwBoekTtNBVatxklqiM611QiZdFCoBioHrTOP
ocxr1OAWzDHz9TUIfw260OeZmz8rcASuZQF8reMoTgDx/0PuX/89OAHDDfZAttfT/vQ62N/fv7w9
nx6fv7XssWXg06K+IL5as6EPHEDUWKbmVnrR1VsxcrlWsl2ZxqJsXu+QCuAgZ5oRXooA7TI6lDis
7IcU27Hm2IGiQ/sizCZgRXBprYEU4MbRxrhzJ6lgBouAlas5PGACvczAyaoXHESjxIDKTPCodgjU
QWZ+PhAOfQucUwCsy0tlY7NQ+AkMAtrWxapqFGEMo2ioelV2SdPmFK1zDXd0sZtQ2VqrxIHhiKKo
tQoaRM0g6MpfRky3fQrG/SWSyN5c3d4cBwzuFBlE7kp0dUwmmRve3pR/aNy+acjNDXllmzV4LWAS
nF4++umgg9cslNejhd6OK4/JjQ73dEXDEgk8mQT05oxvU2sDMDs134j774eHt6fDcfD1sD+9HQ+v
LfPkoDfitA58zMZlDk6IrMLL66mtcWDbN7cFulFOMjpmNdRwFUFD7JvHpFiSiCS+IWatiVtivAAz
5klMYK3RsgijXKw7gS4QaOQtjGYWpxHzwUiFwAvgTvJ8tYboavf44+fT4/3j6ePX/dPT6fvx5e3b
9w9av1jp0jJiFtDh64/Xj4e/T4dnTOu86gmY0RA9BJJloFeWQKlAUzmYnwFgGWzzVIXUYaRHEiYp
7A2IiCwLvvyMa9G2tYJtpEKPh1e0SgKYno5ChK1veYaGJeaBiych5vJXej/VUGxHLpsfbaph7fUU
u4xJuiT+pgNR/KJRpW2EcBi8KOmGLUkS7FggtYMOCcsK56B+KIwOraxLvYNjS+VkcVz4LOjIgLID
7v2mTD/qqjELYuI4ggBI6pi6PiDQvXc06wy2zlcUuF1rT8FXlYarWTW5T6uenW6ZTzvDQ1fT6NUb
AybtNBpOadUWM+G7NosBmIvQoJMaHCKJZk5ITCGw8/U0UY6qQTff6Fg4GoptILimlVPfwFLOvd4A
x2b8Tqg0fgM3+JuUgyiicy+57oxU7ksuucUSGTUyZSglQHSVdMn0qBx/kxhYVfAc4g0tIZMFxeqO
6UOCAECDZ7REdzExGm7uLDi3fk+M33dCastZco6+aWXWWne7bT4TFulYNF46RcuPA8yCKuu2y0B3
s4CSqIgoJn7bvGxLFvT/aAaOYoyna2hyDg5zzO4oLkmxKM9iMBnUYEALTcAfLiMcFDxL1yQB85Il
RhJUa6+SOh/+2h+f9bSYkcoqf4N+9qly6EtHx4DjadsRV+mGtb9VitYOVAVIP+aL6u6OnYSw2CDq
5N2aiNTwHfSkqGZnaRSq4MJgASLAG4x7Js0hftZ0Mv4s9IxR1QKhH5EULbOmwWnKdUIItkpIFAam
WGd6g8qD6A1iDdpUM6FMF39e5FVmu/XHqrYiRgWEOWXHpkiwZYLWpNZoB3MtwbozXT9v/FiXVEGN
ZBT+Vskt4XY+Eax0ZOOn0gSjBid2iYh9XPJFQCGVaTSbwza4i9vYsBV1G/jVUWhnqG0cM6FaB84t
NWDKxK8Zx+nNxAKFHVYmKd7JpCmcgbCzWHDWhZ0nW+WwNgukPNnqLi49HL++HH/sn+8PA/rr8Awx
FAGn08co6nDUvFtzCIumCogXHlvkc+72Yi+cURN33CxaN7yeYCHziR3n4zWP4ejifYd9+2FeKh7f
Xk/3g9Gnq/Gn0eD34ejr7OHh63w68Ibe7OPQ+zjS7gSXECFAMGCN8Ofj88O3w/Ng+Gk+/uR9aKM+
0FbK2Gk0/vtwr6LT++P+9bsjZtjGeOrq0kTjtzKfaumRjR0eGNjVDZKJjSrBKQl+BoqkCPI47Tmp
nnWrXUWPvw6D/OfD/lRG3iX2Gv7kvw7HMk+iLwL1KnrrzqneG6zGM6+QWiGJZxP7vBuhsaDlvW0D
rbJDRVjG0+rujGS6VM0mSz1PaCTf1dgq/1mkoJjLGHPugsZBBZ0aoh1j/JUl4JkwlG8GZnx2DoHc
XI9HbgS8KAO91Qx0dQEaDNfOB2rP35RkqMijWYEooivQPYqzQVVtSZTT6+HfD4f9w1D7T7t72wQ0
7Q5UR9WGwdQaleOnLneqqUQXbd2Jceosv8hjRyuJ2DJDwxmYTqXQbc6GZgmNwKIF4ANTXcGEYP8p
yaJb+F0YhjJdlVegyq8R1xO31s7VLZh9a4E5DjB7KPfSCEYxwypZAoSQOfhMJAgwhadC3hT8OeLb
7EfAX8qUeWzuEtKn/QnVKwjK0+G+KrBoBLK851N3X26DWiKwjK9JP1zkyQ070z0C6+T0CxQ0TU1N
VbbeMBLw/jGXfnwGSDOwAmfgTOJN3RmEiFjerQXvuYBWQJ/d9APpzW3Cz5B6M+6HRavRph+6ZuLM
GWwoOhe3/QgxDRiwzeYchuBnaAaqE/h4zdIzKPE54BY8h35wcnOuLxlPz3RNzzFLShOBBqkf4wso
yn5oBo7YWbploHwEOXPmQLbNecKBsydldIYhUT9GmAQI09U5QeXpenVmp0LGY+8M+Db5koNRyPpR
JF1l5xaQZsGZzus8Cc6OXiKcWWKesBQv4PsxthAfoho9g8EEWuoz0nSDBqUffAd06vGkHOpY977D
1iFUzaDWB4fjcX/aD/56Of57fwSn+OF18OtxPzh9BwfpCTzk5/0JfKbXwdfj/scBsQwFH4MrTgua
gdXLY1D1w7HKhY8XLn3cQZ4oZG9s6mc0NTXSwpuNR1c9tDAR55ciToazixBHV5O5dwni2BvOpxch
TjxveAniZDy5aDOjoTeZjxaXYI5m06l3yXZGQMfxbH4J5nQ8vPLGTtOLeFitQDJMrIOWSkHJ4AUA
6z9sbzIZDYcXzewtZovLMCezsedNL9rNxLuQ6tPhYjK6hJbecDqazRcXYc7nw+klO/JmU5h9ftns
wHDjizBn86vZ6BJMdUZXF2FO56PZRXufTaajq8UlfCRS6ueV316VKGDtge2pNiNfzWbe1aIPPB6N
5t7VtB98NRkvZn3gCXSfzq56wSBG89GkHzyCY5wYOVCyZQBpFu+Ne9SKjTgG3r0IcT6Zzi5BHA9h
a+cQ5Y3XjtojNGH+GWK/vMEbAuMM3TyG3mXEMOxpZGw2mg2Hi6HnLN/pYnsjxB7p5ETHCi8PNzzT
FObQLQ89yFf9yF+CEEzJsN3ecDa9eGg6Gk7cpEi2DFxloHMWg8/lJ2ndx0GI2rVrz20KanHoGSnN
Ls5i0XcO4OJBrIYlVo0zhol9Jt11UP8//8FWA5ONugYVvfI/mlUYXdMxc3W2cLakTEKMZxcgTea9
y2hQFt111LCp917368nCbE/1YV0QrQe6SxFDZ72gCbCJljxIMnWheD1rkgNrLtMoX9nlFnhp7xbC
O+T4PpA37QWNzV7GcEMta3x3PdIyN1XlUIZFh3YeWlWKxjSu0uvaGDt33lmVEWCV0hri+h1mYqIy
zaRftWcE66C6Lf2VT2Z+ReUXzwKtyvoquoqofxbBHEKw8wsoWIxFAypHfevArm8mLkK2hq5zp31Y
bSK3Z1IXgnpJUNbw63x4Q5O+pI26ulQFNTyDgOx61GQiyzqSKoUGDiWNNAaLmWQrTL+hlyAh1FjC
zGXuQvLstiu1YldIucyGwAlJr9xKslrh1XEQZAUx3dcyxdkp24N+vxafRoP98f774wnisDdMJXez
8eX4wNIkDJaxXsD8zgjaLF7/LNZmU3chQwmEY8NKlaSTcU7S7rq8C9Y1vnD3qczwVn7d0X1Ve3XW
nYKh6mjTjEtMVWIS03gCpR7XhKxIqOKIjDR5S2sz4ws2M7lwMyTOnVTEuidY74p2WRAYKO6zredm
1lY3fX91JT0AfbsoOu7uUjJH0/+B8K4R6DDteOXVMmJJbUjsOpnpBXufX3gyNCVJd4be3hbhYp42
l2xlz6tL2Zvb2j7d0NvSrI8tPUBC2V3i1btEuLqYPVe+sCc409miQSRyptPg6tPswmmrnta8s3fn
LVX0tsMsInbIUSpoHvAiiVmvjqseKOHFS6QuyzIeGXVA3CcdK5WhRVRVKVhY4brIrz0P9DZQKsD0
YP3Krcio/vjFBIMTU1du2hfWoXHCyxeY7uUnpvI0wlbVOVoREgVdJ2SuVZxBi22rMDFJ8c2cKt51
bCaIjZMyJi/ve17+OhwHP/bP+2+HH4dnx9JEDtZXf89UNWilcm24UYHEBuITLEroCQ6XoIrUZbyr
UD8uRESp5m/ULdW1W1uGG6v6LwVzPwiIVc2ZKmd7H3VHNrTzRKQFf+FCuetVXQFi58YijQWrugZ7
scEWq6SCblWejoXFIDUhz5CnO0Og9iD9dcBX7o5+tDEWWd9zli/RjGhs9wVU3Q6YmoYh8xltX7e4
h7aGchyhjcFDo5jSICCirirJtitfWubBCivBlg7/u+RCDdwm1Pv4vX6fUWHEDUadW0cYe3g6aEKL
LwOMkq+6pSwiS+GPIGNb4yq6QVnxbRGBE2oUm+rAmCZ5D0hSrdwjkCVA+UOi2w6EV6a7uSGodzII
jo+/jCIdgILi9RnOZu7MhnReNCKCupvV+hp5he6s2mOMkrINncPj4X/eDs/3/wxe7/dPxrMYpANo
ni8mZbBFUQbfa0OEJnvA9vOGBojEs2p5S0AdCGFvrdTRnUtzdkIxEmBRLu/SKK3Lu/AkoLCw4PIe
AMP6TpV6dZlBVx8V4ueSRT3kNWtBnRg1NXrgzdZ74PU+e8+33VQPir6HhuG+2gw3eLBlA9BKeth8
UrWq6qE+8iscTE24KV0PkUZEBnTrkirB/RpTl6weadHBHYXmTgUQPzWvcuC3q1xsG3ekSPlLCGDZ
F0xNuiFFxnPHOzSEBkCZ2yJ7Dwhr+oJ+nfX4yUJSlViqZkMv0sFvUcR6eWozfiyYoxVfO/MkujXr
dhowshQWHN2ou/QMS47NybahPhm2NAYQl4pl2Op7B5UL2UOy5W1KhHAAt2oBeaJq6P11FYvqrwNv
QnRJeVXBaT401peDT9d3BO9xMTVnHIFCU92BCHrZL06Qk4jdWRWVm21s+DwSE0eoW1fUcz0G15Lk
a1hA5PU+WFAbj0CqU/RwTNGu0kTqeZD1xQgtJDl8fDj8BLFwOrlNUa3Il+oJj9J5OuGr3KuIZV8u
rcx/vovgR6An+jNy1RMY337pYIzbM1K3b5MyxmoxdMjPgID2+sWb8eEEiK208reNXZf2GY85Iksl
mm00ADYVBoD4+GyZc6fKrXxn3fideQKctkowZefjC1NtHRmVzs4M9oR5RRQuW3icHXpHCvNElbsV
6lsh7k8/AFoSMyM/6qvPIoQRWYluJaMFh75dnPazGgp7zfnGAgYxUUEvW+U8d9Q6QrBU+n/llxsc
B4tAfAKA/lKeOirMwZZLlWYuX8h0EfD7NPbDmlaUqKxC9B4gKEIgZmHoaW3f5fdghMxyQNqtmaTm
89PmUaL6Zg/GwJk9jogxhqjKr+2jhXAerB5GqxjKV8wF1ta2BtWbAqMp73wDomEU/BhN71jrXbGE
bZWPmlz197goV7t6O1Uu1EzGtwQzhPAMtCmUNUqTC9Cga5XEwLpUjO2dYHyu24eCz3jL1CgNrWf2
1bmXklIIEkKYEKc3/nrl4I0GRa5ZGXPz1CZKrRIq9kCdbWFU45ef6+mBBTw3b6MacgnqY4LlDAhv
fkzta3fpILaasYKoRyPnbV5ZjlmePM2IsFdr5qH68lO9eSugH++/IHPdvrWbsIF9tz/NcBG4I9bH
0HoQ8OgNHevsjE0JT4z8XjsEfovE6WyYs7y7at/Ku4Ws76zwfoIpeiQYBUWYZUT3cGVkAN5FLd8h
2B16vqWi47z7VYjSOL77aYgUoiSJD4bU23XHEDFHrDxwNsdBjxlL8BoYLTo+pnVIbKkA8pV6BpSe
A+IbIdfrdAUvX0yQjHZWF9S30dTH50Ca9uJBHlGhniDhGzzUkE5Dg5eqeI0dEdkRRB5KJC2YAL5L
7Mvq89BG9anJ6wy1iwLGqw5rABPWftCgPFP0avERJlZfLK0H62fbIzRlEp9izibvYXhTxyDl9xMm
/d/LkTwNkCpqAxG55bns2OqI/S9nX/YcuY30+b5/hcIPGzMROztFss6N8AOLBKvg4iUCVUX1C0Pu
1tiKkVodkryf/f31iwR4ACCOmnWEu7syfziJI5FIZAqfceNTE/lhDlsDOnjuDn6jZF2beLEShaxD
OvUVqzrFzZIXjEL9S5loU4rpcqFfsassU5RRdoDhgkKBONSvk1BFmVxHh9v05trKy76VpScXk8GY
3MSaWt873Gu6o4kLXoWicLiWUeUxYYvNJh+bW4rkLjvdOwkjEeFdgbj6QfUEqf6Sn0+yc03SPNTU
wE0SrowBbZa+OM68Azm4cPgxLVS2Z8/qsOwfR7JJZFrxRhifZUIo4y/mTWKXOHOABmR2ihrbDLY4
FHVs+pVGsawqcxRnfH21YfrxlcE9yXikESfypLr849fHj6dvd/8W92I/3t/+9dwrdyfvcQzW5+L6
whw2uOkUz2Onp36OkpRvdeAf+XzA8lCc5HiQMAsYcMu9iz093uov1ElcIhPOMCcObF4zWfDcIHjC
rzTiNgXG/5g8a4Brz1Y+v/abmuJNRMxf+/1TDxD2OroTMBVzLntDXXNiwXYVIhBs4IJ/h0GssSUY
znCuDEmTjG45LaLfgMTO5g9Xqy4MzMgruN0gIGWNXjPAyw1MSnO/cRUDm6/0+PNP//z49fn7P1/f
vrHR+euT5F+ULbUF65wz3Oye4P239RsQ4Soqr6qTfH7f9x6H5J8dvMJMHvgtqGpgDQ4ymGDCVxht
k5w8zLAlv78hmbPqM/gFkrWdg8+NPTnMdMXAw8m9EZzjvclxBzwSwtTg04O7uWF9wV3uztlfmFhg
IIPrIUpVTSfwrns6I3SF4k5AajUGj2Bs/3hweuZhsKQi5n7DVd3Mew1W/IyoVBgJVS2fYnkvcrfG
wx6GNZd/JgD3vLTXTmrCDODx/fMZlpQ7+tcP2eKDSewUi4Wsv71W7wiqppwwphUbtxN/qn9FMhOZ
bZmHWGFMRdG4wc6iijgx5VmQtCImBjitSzE5aSqTApeszlwPPEsCbuAaTLp2uzbleGYpuSrdkG2e
FuaGAcPqpueAzYmYINjIPWtKey5NdTzF7LhpYoDBibEs8Da83prLmk7FvXsZN6rKbIDhel4bh/J4
lw0ExJp2rPKUiZspYqs1kyn1yV7cC0ldzHThebianKFJo5whcSW056AEV92FS8zTw14+dA/kfSav
Z9l9N0zwwTHWNC0Z0+aaafJ2q1RS9yRYVpQdXlWfMrHqwykmZaCMSrEQkBqXfNed22ePl4C0Aqm1
Ka6yQxxwIcMTi9Or3AVsX2BytIXJP5KFN4r83Pd0ymHc+maC2Dl64uZqTjqjy8Js71Wd+0KZHsur
d8/T2Ug4deGuk2q5HZMVFB9g6M+nr398Pv768sSjGtxxVyWf0lDb4zIrwM4500qZGKNlq4WNWiYt
dfUxbYhJ1AeIevExdtehPAMLHB49KF6HVN8x/EqIX92NXnFYnr0nRWmkiyqRpMHyGaon9x7JpCx7
xfU4zm19xTuyeHp9e/9LurY2WKZdZS3Rl/7nONc4QfKIa1gjBYQcG1ye+EJi8p4rQINMWE0w9jfI
ANZ8DUmEUzB/1tul+TWiI+Plf5zgmPxnScBj2X/Q2J9/evnvt58m1Ki249/3y8fr48vL29c7/ufj
59u79IFVBCvy8e7r2+sr2xZ+ffz6bxg+Uyo2au7+++O/Hn/c/c+7/35/fJVPk0OhfEYZq97zkyO4
Ygdl1s/baYA66ii8eb48/mqs/+gzUrbfPF2y3j3s4EVT2WUZlonfpUme5L4nC9Qc0Ni3ql9KyBgu
x3rlkZqvzIfLflQi87mMQ3tPoxYAq6PDmxYvvDsNfco9Ypkdkpo7Tr5bOGdZ3msteXZsZzK/0h/c
0jSm+4fx4ZHutZ5rf9g6iuTzzsS6sD9ADaU/V5ohtFP9gfv4nN1S8jge4K9L3Xt7A2bZu/DolA4u
9+FRbG9u2sSFlqeo2pBHf5uvuQ6VODZtSnHkNYPGCic1kGbe7BwLR0U1Ne2ofV16WJH2sFkGI0fe
ReDzKnK5IJiiMPBvnRhpTLAB15pCDTbMkkBRRDcIhBzl3GcoQ27QcFFhrgvcvXeaZq5OajG7O5AX
+JunyXFTfXwg4t0RNTiHGiU7ackg0lcflmH+rdgphef0cxRu1luL0zfxMbMY52fVF5jKMU4r032N
yfSX905c627Ve/9GXK0IlzI5YtNDvwQ7sNNeCSYiWZUzYVnqW9m1Jvsx2A4p93CMzF06Gp1dxqwv
UEwml1q8ouLYYHqHNhhFcHdNg/2I6qguQ02j3jhzb5nG3hO2FVwva1BszZTN/BpYyOjKvc+EODf8
S4gbgd7BlQEHKg9uGaLcOgzUOSWSfJoei4INKzBz0a476qycy5JEBF1ggI6bsZg8+4Pljyll3f8I
XcxIroWs+tcDPKCGm+VbtyVYk60xgFgf1FWVdxbbpKkt/StELGthuFVcl7CBJovrBDEKFXvOpL6A
Awd0FmgKjTWB+1w+wTPz7pw33Pm797oaXqJAQeoCbKaK6VBwK5gMsz8u8iDpLzsNX1ZYo4j7f/1V
LvjqTfIYm9JxCxFY6sCyf5AEU3huHn/9+vTxcVe8fX9mEoF2KZDGhUVfYEs78O2nCMnnsOqAmM3V
RrkfASIy0FoDjS0Q3KhJGg2nvey+XEGn6KL0L9CSJonCUUounz7hiT0YJM+kS7brnZDy+hl+dymO
Dz+/SksR5FqT2mR5V2L5HjDjhK6q9qpKa8hy2BlyIufPfvZerc37CGPTyrR1tJl8twa/2GZ6qDSS
7omXE/nyl2kG8jKAMEG1ZptK8qBlJzZypFG5pSJhOxHRK3Sc2s0JiNQaBdfc6udV/ohw66cTLEWz
o3Om9GeK9+Z1rE1r7rcbGW8XoBR47c1m5QnJbq7lIYJr8TJFjWHEqOPTJLDRVlRcYL60hzsJ8cCE
zDMD63S+26s8nlOPiOX3wCPvgpp9JRs0jRweQQUr5yPGq8va0nBcY+0L4PoAOgwEEbI0RkchLF5u
wJuyMMR8gob39dQuN0aOCezqrBoXhJ0pAhMxlF1Yl6zM6oSRtpLg+kKxWv1zam5pVp1nhKlXiDp0
lAnACWICqKsLo4GVgvXCbwCxaZuYPiEWTVDnEScKKRL0VedmPwhtawXCZ5HeUJHYRFTXMoFjMruB
DB1oIDfx1bzGApGNOLCoMx3ioRT2z4N8kaKz9jiR1tmBmpz3sp3BSL+ysq5VlRqSHNm/TGRioT/s
89hAv6BDTAz08mIgwoGHHxvnrLw2Zl5WBvIDksfcSMY5k48rbKpNmphblaQHA3W/bybq6Oai72JZ
3OcMJnWb4juOlkt9rj//9PWPX5+/yre4jBNXlnFQpCvFdQebvmv1V78Qg9Y/M3E0l8+cIfztwzbG
JKZUncjr2Uxem6by+qa5vPZN5vV8NkMFC1zrzcTyuBNJrRN6PadCFsrKxymEnaxnlG6thH8Aapli
AlFhUkQfaqQxjWUpmwSnKMvpQDEndmwAUMXzHm7zdfJ8PxmJngzn24coBx3WXX7ta2jgHQvZU/BE
F2FHpA0ZxmKdj3mZhwv/yFVcrI24aRvSb2FrmNPqXsRp2pIsaP38mZ4TIQoPnuAoVsRGxSAXimnd
SxrZg7Kd8rT18YGbQzCpp6gVpRFD6LbfI8mwuu8bnB6QlOq1f3f49v4EAv6/nsGtli2q9ZSz6WDS
s5CVw1/i29nQ8bg8KQJBz8riAucPfe1NaXtA3NQmLg8UCFnbS+34q5ZXO5+HxHHwRdhaByCvDi52
RTKJDaE3ypLraxQqPNUjD8SSF6QRyi9jTh2MP6kPZBYoYYiFB06+MxtTd5alMGHMKga0My4f0RY+
n4a2rKuLxqH8orHiy6KRQRJqKymGZ7yxpdcyWls4xyiMLCzcJBbOJMSb+exz7nHFgxyZAaQsbBWq
a2tdwRLQxsK2RHTWdmqYpjJ5/OQW9hHlNWpcE+GQn9lhRh0zZaxmyH6bvhmQ9RoDTf8YQNMbDTRq
SjxXo/SMIiZs0jdximYsdXBOC+BDyaPjKWX0++WcpJ2XJzojKw+zy4yC4z14GvEq0/iCNe1DjIJa
ekwbJqQapSVAQBi6QebScmOjQgQ2V8jqUgSEOQZ6SqXwTlVJ2jeeH5mApq+ynFTRWM8dXkaaaKJP
tXbBnZ1K47GJFEqG9zOCITOuR1IoQkui0rS1njV1NiyoebCk53r4/ArYRs+uqZnOam+i9700Z4mB
IY7AerMlni3JbAsZ50Sry4szVm8uOqghW26l8QGX8L8+f3/6dvf6BtZJHyYxpaViz3s1smBEO9iE
t1Ip8/Px/benT1tRNG7AiFoEpX91QbhpPIT5cKMGedCNcrdiQHlqlJKkdiOOuYfvrwTcRPE4Xm5Y
jlIPwCz+TABHVdRlxpC2RNxbgxuTeatQZlaJUAJVuohnAIGiFhFPrcddytMv45blxLECPQB9aTJh
+MsoNySpC0K8mKqmPHCMPh1fHz+//u6Y+XD1DRfT/ET96gDBcdLF70ObOiG9AwQnpir46ws3piz3
DxQRL0qcUr0obTs2oxwTZgK5xmqPqs9OPggebgC6+LvasVYJAEpKN5+408P27+83uyg7Qdzfx3Bd
M4cI355uzMU9WvKQukvJUXmgRzfE2x+gqnHzPWNMqJDgnZYLVWa2o/gIUeUrA5+b/7oQ/X2dE3J8
IKqQZMCcqHft0eXXOcK9AfQYFOeFB5H41h5+QnYCdGHWAOH2wj4E1wx7UNwFkwvi3D16CLwhcgHO
UcgPSYOPKpceTLoOI0jR6gr/InH7c7haa1Th/KnD9Qw/cpSJozLV2dDzYHkyZdjT1Xmm8lz5cYO0
2sUtEbUXmphZVgbLzJmni+Hi2ZvImDhTZJKey81R9U8qr6n850yzK6i29zKCS8GMlT+BD/tXH2yx
vvt8f/z+8ePt/ROeg36+fX17uXt5e/x29+vjy+P3r2Bh8fHHD+BPoo3ITmiyVHW0xDinFkYsNj0j
z8qIj2Z6r0ibmvMxvBDRq9s0eh9e56Q8mYHmpKzSKdUlm+W0nycE2qzI9KhTyIxSzDHyuUSQynud
Qq8VGeRU3jnkaO8fNijHAbKV0hSONIVIg8sUteqoevzx4+X5K1+r7n5/evkxT6vot/oWZAmdfWbU
q8f6vP/PDVcDGdxDNjG/ZVkqhTAWrtWjvKwlEHuKlgyUXfyIYaD3yjGg/2XS8GgJhHJkTuUKHEvm
6j1CZsyB6+IBqNNmQLliuqYQlLng/RnPNZQzZS4QVZWz2rvKZ2T0/gB01PEGIVlmNPV4N2TgUprr
DDN8PJiqujiFOddxCbZySFdSmE6wCkA/vmuV0U/JQ9PKQ27L0dBRw9F03hdNfNVJ/FE9vBnW6Gzs
zHW/gz5SHVljAnNXM8bUhOnpnmPe9hP7/65vmNq4Xt8+h9eWOby2zOG1bQ6vjXN4bZzD65mOdEa0
ZDxM2LXcpUpzZwzTrJIY6IzXSwsP1kcLC1QcFtYxtzCg3uIhnAVQ2CppGkgym1oYpJnnaFAP9hxL
GdaFYe1ZGdbmqbo2zLu1beKtDcvL2ru+yIiypuosc00i4/ZonAv93bYyvPvb+gLpFxw9Y37Podwz
qhkOV/9Zh/aWmTyHzHiMATeYZ4qMLDr76ApT6XiJs12EXWTkgLX1wcyRF0iJjm3ktZGu6TUkjnqO
khizU73EI9Rc/CWPS1szGlTnD0ZmauswqFtnZs33OLl6tgwVfbZE1zTdcGASSoLpeRz73aX7Q1ft
f0lK08MFgRis27jJKDfsAat9dmBWXgjaoOBaz2atKRIppjjgekv+0QmrvOl5YGo/uLFDsGz5yH6J
CNPq6YrTuTOLSiOqNYlpofxgnRuut0u5MhOV9a+YP4ZqqUo8+DW+RFKpF+m1CidgPR2SdX3KiD0o
s6qYT7HZIMEH8KpcVpVql9RzYdj3S4JgT94H+rWtMcfA6NlJVlgMplKiKs2AAOG0hBPGV8X+auTF
zS6KAnuOHLRvktGt+KsNYOfM5SMzoMsS4gDBqgChTiwtOaKcnWuRJXi3jDyQK649TYa/XS22djYS
HGPBBfVXrmjzbrsO2/YmJGtKcAP0RL54MVWC8oreCIP9Kbj39OF9Yu2JhubLLvYWxmDbbumHfSmD
Ll7V5OqFstm3iyxBSWUc+SUOgsXK00ImV4BLPVsrLyTZRG077ywjum3IZrHwf8wHuguD1lAzvrTw
sqahOdG6w0U+w0uMQmHkXGM0PWzLE2OQTRrnJxnXWqI+53G9NwfTObJlybQ5rvPqWnPZYMT2JJOT
eg1RHqWdSiJy038zB4RQ9RJS5h6r2sxQxVaZU1R7nCtStswd4iwYmbChzhgHxgDnsmCoZKzOwZUS
J4WxpnKu5s6REepBzYTQRGCMEILhtVI29onalXn/D9TWbC+ELxCbrI+lJPpli8SaBs3kb2osXppE
g88ffty//+Ppj6fn77/9s3foo4Sy6dHdke5nWXTHjCRzKvcYNqPyS7z7Ob3RjDo4kWR7E9GQnKL7
3EDdZ3NisidzIqIGZHEPbehybKgFjc3tOxgbwv5GxZycNo2tVEO7T3szIzlWJzQn35v6KeFhqmZk
cPWkBrAaE8SmvE1ZH4+GLqyxMbWZPliNz3MBdxGGb0b0vYYIYw3rrsE7/RznNQ7dmLkDW2EH//L4
8fH8r16Lrk6PJNceujHCTHXbk/nysJzTs+ucJq4gR//BnMAd7ctehQVVM+AeMmWrwTzb3mZlXmPN
1mXMQrsS53Su9gGfpgoHFWrgk4nWv8COQgMr0V+m9nRu7mLkKD0j0TWFyMQAr+RGRhKXODVycE0Q
tXDovEPiRHuiDe4+uLWA1gSgH2L5GH6IhSn6fp4BuDrQlxWgk7ioc0PGuhGbqAXS7Q5FHljvd049
7c3wRJglKjOGV7HObU99ga3qNwbqbKzxEkz2RoJD+UsyU2WLytA9ODP0jTAbnj97FgWYPpI++li2
vMhZHXvGfJXuGTQRt3Ja99FkeHlv6UC+ZmL5DV2aSKMkLcGNNKlyJXj2PqZFzF1hyuVN1OGfF5NT
UAkle5eX6KniYXWiF/37Y1OJdl++Egi0k2YfJBU78F7YaRXWllcDUX3xJjMurTL8lDSoRBcp2aV/
tj6naIqbkZxXVb1XLN8gtgGTGwxZqQzTmZq/cdDfjek7C1DYabdSMXNhmlPZOmV4Wl3yG+0pzilp
bIOP95P6oABsIiLQhVPuiU5i3TdU2ibgF0w4fcSXCTEFpqjBOQq4vYWgLLKLHvCCA1ME5G7yczA6
ymnkWCtNRri/8Fz2gZqcuqYVzwLA9Y+qhGrl5L2rS/7YTvF1KzGmt/6yH42225/JAw9PJE2Je/lH
DY4SGxRzv6pEe40nDOGFoln1pXH3+fTxOZPD6xMVbznUA3FT1eywVeKZ87H+JmKWp8aQHXeMbZen
Dfuh3qMAYS/ruoBwuMo1A8ovwS7amf2hMS4mFTXpn/ayIhouFVAqr26s3zOYPgpIkDqqOGBmaUtU
q5mV4OMsmYXFGVjCjsXAPeJUzUn2PMZ/dnmYgDO2pFHeVVLTQZ4nSImx8V1BMi6u/KXg44rU5o2C
91FSzJO4YtmByt8Q7UxEf3754+nz7e3z97tvT//3+evTPMwkXJuUVF5ZoBKJ9k2oyhd6MKmd9LTZ
gUSbyEcC6OsE7+mZ7LUeG8giSLI19rCMFIHDlIhyMhtq/Je5DE1HaUBA82a5klSWAAT1HDfURGMf
oFGWUIl1XM4rxhlldcKxtWYcsk9kuy2JEdNjdDJy8txIjq64QUaO+Pp/GUsvjHRDd3E6jApzU+PD
um3dTS2aS25LXeT2D5gU4SJqZ4MuKZbRZjMj13GwmIMzMUIV4oX9r45xqKBO6GajRHwZdbizEQg4
2+xlbGintY0NwVqGQgax5Ve2tZ7fFJHdtiBIV1UZ2w6b2uyslTFPiem+Rt8aJ99/iXFrALOO5qxY
NMEQzZWH41fEn3DJI5eT4KmZtLlnB1CMBcpuynVxAfezBs6/zFtXnxDkFZSDm2IeHoXJF8SNTxAE
KsQirkxXlWejU8AB3aD7M254FAgIIdegQ7qf1547TB7i5QCkU93ZSZUVV6e1mTk4LpzVuUnjedzH
kX0Vm46qawxm2sdABLxJ5lBGBC+RMA5yM3d0KHkL6uefXp+/f3y+P710v3/+NAMWiBwN6WErNpAN
7hzlnMjgONEaEETJiEdKNwcEEChC48EwuhUeFheTgHvCsqpF/Nbq3RNxWcvv6nvqodZPprta/91l
6Ywk1gxd4mQMtvgdrIq0XW29Q09Qfez1qtMM6WlwtckkOPthcQTCqJcPv8Y3y4rhJNy+HzCNc5VY
Jlh9Cs1J4M3d3Lieb1l5ga3sAEAgxzSXXOU9vt9lz08v3+6St9fXP74PZrl/Y9C/98vrh+xWELKg
TbbZbRaxpUiCC7VImK3BYqG3rA+IDC2wti5La0spdblaLrVygNThMNFLYowogjFiLUcgWFInInT1
dIGTpuJRY1+N5L5iMkvZjQdKp4maI12rndzjNAzY37H2pXuqqUcI5QPH1eAe4hx7ba1/PTmDKLs2
5UqrlCAa68RZ23kna9XarY6Z5WB502geT/kmraWiqpt7/BkovWOfQfHF+or7o5UCBjcVWx9yXWEy
HHJ0MvcpSg4zV5eqNxjuHBdcrxpIuqdB8FMMwRqm1iB6pAAcHGMMrkSFDJXqh6oYXDkW+1gP1pbg
eVjkGsZSpbj2VnDATxlfiRomEYeMVGYfP0IhcnfU+7Mhyjik4Gc3BR7LB+ee0EeknnoG6HBSTjQo
kcOnDBTJnfx0az7warieIKyfzJfrCgwEmZvAqIGQeuAhyairwhC+E+nV6VKL9CsS0MKcFxuF2ncD
ue9EtOwdKlTgNiL43uASHE7IluIIPSurHf+KGSdbUohRriSgpLBWJa0svcY+eKx+XjaEL3rWZY1s
PVU3Wk/VseZ/k3c1rY+WDDT3Xby2fQjEKuMu1sgsNzb0QZGJwPeJbTQAxjJIOQ/iRNuHBiAsQ84E
RE0If5gMXw7gUFw6JfQELlEeajaHJ6+U0iSWqyzP7aS2yAkyqEooqkpUe4HkqE4PEdGLlfD17fvn
+9vLy9P7XM8ECY8VoZMPU7VHYtDTxbbPEtfgdo7NLzYmq4Z9+lx4o1/8+S/4b7FYzOHq8tNnwYRn
cFfOlfPacOW1o8dzCTGetKFrB8JMsCIzyv5k4ps7q34Bt4JaANjHUovZltTOr7kfvz19//rUR6Zk
yCfp+yhyKc8liVPEhiyPQsqrZRQTPJn2m+LH82/fr4/vN5cd5/bWxUXaFene2jns+GYJ/sX7FxNs
z/uEG1za2enV0xv+lorQQm+/ssnw/OLqjsF/uAcqplqexrd2LRehkyOu7a1wZjeCzNN7nPro+7cf
b8/fP7UKsF0s5dEDzCXLCcesPv7r+fPr7+bFRMmbXPuLLYoSa/723JTt59oktNE2lJ4oCVjqVtuz
abJbrc32l0mbw1JvGWJJ3KSWwDY1tukL6yRYLGeTnYdvff7aS6J3le67PT7DKhE3D52Q/6YoACLQ
gfBzYXQ8e6FFLSuUBoqIg6n6sO05+iY58Bt8EVGvRGQLEZ10cCYXLCZVInh9gyDb0vdoREUz3BQ8
5OH+jPPR7C17fn/9LzYN+Qtr+UlsduVRZZUAagNJyGGSfuVMqMSN9wRUQlrk6IkPp4eUVSOTAxhA
oLShij//JLnnndLxWAbW/p5wQ4ArU9kDr2u/ONkQQUsBDCctuV7neu58t59Aer+OSlgRTf0ih1kb
RhqPU2vmaVRp4IBWTQwP4xjkbHRpVKFO0OFIOAytBhWV5UjAYTFcIA1gPvwMxQ0Hs/4iGyeyjlpm
gmjOMzGzL+ec/Yi55aziQL1BB+W4J37zY71OIzU20Io5ESJtz3NspDiRPErLMW7EoM1UJT0wM77/
c49Ojl7hsTVYu6u8OjxoI6k/dPNrOuEkZJjdxtFlWbTExeUfHyYVWlG1FJkVHBfUikEvfpv1uEds
XY97XqcLwMPFiVQh+fKbzaeCR5RmQr8j9hUYRLFzsxzrBi4xJteq6jo2aRQ4PW6KO/LXx+fTK7w9
hI3s7pFVSwozgr9/Pr3/6/Hr013de5uQuw30ESQpsLgUT6rc2MT/r1K0QlJkDpYxDZNhARdBwEzT
vcCyeQb7KeIuaYMtyQ7zeExyENlsz+3icpSZpUeob5bFfeSpyqichHLYgVB1XH6oqgPEHe9bMtuM
UYbv/ob+/Hz6/vEMwSbHrzp239/viO77A8LhIiJfzQDlEjfsoEiocgWmMab4kJioYdT6ELtxQbqs
qVhrUy37hu3gbIIOUUaJyh2mPNczSJ0PvC8QQN7YrcBlXQuJupTu+z3J0LcAhFzGTaukTZXrBSVx
TSD++TwbBWYNuMSWBDACpDyQVcFW4kNsDRcMGfWdKDQKRWypNw/blajHWDl137HWUhI2CfvATebA
xZci6/c3ts4aZ+t/MspkvWRN4GB6VYJFDWTus1Lt5f5ZweMLBMS8+9dQiCyS9xWygxQlKF8miiRJ
NOXoSIdOTED9+qDXUbAJWzZxOqsjffrt/VEvW66fBTCTddLZjXh2TajJBOFQyobu8AsMQbB8M8WJ
BT2ZGQQ3mZlz3rcWhmyqUFBFZ8Z+8olAHPHWfzy+f2hnKUgmx9M2XWYDpo+mrRjsAz3OyDydgmBC
B8x0E2oWiHuoIK/hmf2THYq589e7mEEpeFV6EVcT+eNfalRtVtI+PzEphChdJMLw6bUWsfmaatZX
5dvn093n74+fd8/f7z7eXtmJmB2OP+7Oe3z368vb139Dih/vT/96en9/+va/78jT0x1kyPgi0/8t
d25Gc8t9jIWBrZwmS63ZEZKlZg0SKayJ+Eetavt3g0i7VuYYWZ1JlMKMdNaVbOv5J9t4/pm9PH6w
M/jvzz/mikE++jKsf55fUIoSm4gOALY7CRFeT5nABpReeHgSdnazJAf5eB8zYfWKU3rspEhIBm7o
5C5VLpSPAwMtNNDA+Ep50DC2oEjh8DajM7E6nlPPFOd6PzQWxSTnVXYeO/gii8LG8T2Fpuvxxw8w
Qe2JPFYyRz3ykHmzRacC8awdjHrt4xD8MBauoZjHdNbcQaXmqZOIrPz08q9/gI7okbubZnnODSbV
EotktQqsFUpjGmd5TI5WRJEc6zA6hau1E7Lc5uvlwj5BCQ1X9slNcq1XlD6FGL/amGH/u4YNX97C
gqZWBB/QXXqYLQXp88e//1F9/0cCXT+7N1U7r0oOkTmOs/cziRU8LlM9U7ZMAdk+I+Jr5wSwo/gM
IMT9JGH1+43VSPLopySFZB0C0eXKZB12VLeZGWnYfXL044oWm7WfpnqN1/3QQ7yaeZ2mzd3/FH+H
d3VS3L2KIJaWkS8SmAr0Z/U/9P6sGnUt64ncKmjJ4+mwEw7RZZwBRa71IHG7P5uKhcDTFx4CO7dt
LnIqCKat1wBkAVuJxwd2lNobTQJTKkm9lRLeqOLRMKklLmwFnvbZIaZBSM6gDx9sZJ2q/S8Kob8f
VWhwiaboGBlN0RpVGT9BNRfY5OXYoYIBJlsKDeT2PFYkdyYnWE7XTFhTX570hK485zn8UMxeex5X
0wLApGVJGzUa/ZAI7kIIgfUN11FotEX+IpZD6VcHL4IQm2b5F2mL1znHxMJTla8Ki53YcyU0p8Ld
LkNrcZXGYTnxbRRYTa6odmd844jVUcvbUMfkBhz0wA2wvjduQLKesXy5EfPzTy///faP95ennxT2
tcEU7UXkYDXzPsRRSZsHeCtoO7VzbB/H3BFjW8DgfbyozWxow0szyWBcovIY6yK001bnp81ecRMD
vzuh9REGOZpKQ59P+3RepjLeJWJfh8nAQOZxa9ZgHW2Xkt8MPutVj0P9KzmYw7Mu0NoykM/sH2aT
9h7whR81oKvmZ7YL+2wznRpQNfPxsUcuhRzSB4CG0LGc3oeN7RCEKlf2Is4+XguLOomzs8TO0wIa
avmCaUB8piYLGA4YjBy0PLkHO7ONn9xJQl5//vhqUrPH6SpctV1aW3R86bkoHmCjsPhBiUtqOWA0
dWHWxrIJk9VJuAiN3rqyYviOUgJG3LStWQjHCdlFIVkuTJ6gUJnkFTmDST7b2fgFj/yMEq+XYXBZ
LxbWFh7rDufm9TSuU7LbLsI4Nx9nMMnD3cLir0cwQ7O8z05lhA3AjjLQauXG7I/BZuOG8IruLN55
jkWyjlZm/wopCdbb0OIah1LWnUwurSPXPQyxHTLSa9fC2Ylv2FarjsGowK71FTYxHUkzZJ5/YBjC
/oA42TbbZnhhzERG421YSfunaNP1E1z5scM5E1xSRLncap7bFns7OOxxX07rddvZzLWSUJekxAkE
1XCgn/kTF3S2QoeKy5qJbHZw1POLuF1vNyaHUT1gFyXt2pDxLmrb5dqVNU5pt90da0RaFwyhYLFY
OusYLnPzYUTrk1FC3G+CxWwtEVTrQ4eJ28WEnIt6MAqYTaock8hq9q2AbObjwg4ORlJt0w/CCbtA
5jIu4nLtUlhG0AGV13vzzNonRXcx78AFxK2Nc9Zt1qHJIQ0l7Q0IbcYNpijw8FuxZ73UcWmxizvU
UE7QthZfcScm7HVJvbLY3oFVGdk3hd2MX9S2Dsw6JXnfFAokeA3f6yJm8xCYHXi3mC7oY5zCpVQj
aao5So/wDkT1F9xKaZTJKl2m8vvlbLxb5jXsq3b3+dePp7u/fXv++Pf/uvt8/PH0v+6S9B9svvxd
ehjcS0pEqnZybASNziVH0hhwh7nktZeJI1B2B8FrP+7Rsw4qwT5INtfn9Lw6HBTHAJxK4HV63K/V
UzfQITbDh/aR4Nxv+ixZYiRj/qeJQ2Jiped4T2ITg4t8RD41CFZTj3lNqjCtHVpnXHN40ih5i+d0
JW65IPF7GXG/rjh8ha5uD/tIwMzr2QBa+kD7sg0dmD0KHcx+3ERMNGD/8aljL+lYk9jOZXnsbCvG
AGAfws6PrTZ7gh0n7urFONk4KwCAnQewW7oAxcXZguJyLhxfioeTZsPBgWiSgjR2PmLFh2Z+gQ4x
X/lKdGUbkRsjPKi6Me6W1jTyAUIPAEeFo6mkiBta32PXcGSiQu2YGGeI82zZ40QdHpq9k2uuP1tj
MkeupHSVmRZtFOwCxzjOxOMj614vVsfa0be4xLYn7AM/tlnOiyZQ5JgF5KFYRcmWrRehq4KOj3vP
P0wXhNuFCxR3rn4GvmdtzGtXBihLqONDJdFu9adjrkIn7DZLO+KaboKdox/tb5WEjFF4Fry62C4W
gWMCZHoHytz+zfJf+p5zRDnBFUtofKAkKn7UxaZj16RyoKqBys5d5Dono8KAjfNzPNuFNbFPdvpt
sh8q0rlQJNOYaCoigymXDmxBzAXZmCU3RhUHT0NO4AYH6dnhEsWNOTP4pgvpZkZQgjllDlqu1gpt
FjNzL2yQZQUgp7g8iwmAsN5t0AGDZbZH5ZkW3ESc4nLe22khmxfqduA8ZaYOuwHVW40UcRkfUNPB
D7MeGDJhI7RuMJGdg6X8SSJhDQBjdtB0KFrdojuXPAoqSs15cmf0SnakjGtyrKhSf3rE3Djigtk0
KRV3YpBJ/wE0CtvO7hUq15nPvhZjoL15YQdWE9tYSW72789Y4BtRNo1lJAjaAqbypFaiuTEOjDKF
8AU1lZJ20uS+mqid7GZWYRCqtXRiHS2aJgWEq9jcwBTBpZhcaHom6oeEDVFphXiaoZCyPFZ8HjLS
BfXhT+QaCSL/K3voGibic3cUBFuGao/PUKIUp/vp6z8LHxdEqzxFBzUOC3wYUNgrFBE3W51Yl9T0
wHWMfC2fAGnCihqsuSQaLG+yVwyg1fzoPL1H7z389Rnrx6CRKhmOFhU169MH064z0dYgEXIJIXQX
RLvl3d+y5/enK/v/7ybLgAw3CFze2MykObMrK/Jg1EE4i5Gews+eMBQYaz5P1RuSqkzVFQOU/dNP
qNXhzI5BcmeNRMcqXl1q04KN7s9xjr/onmkzaShi3YU0RXExp4CKBUHkmjjlDiwtgKY6l2lT7XFp
RcQlk29tXLgkvPC7RC2GsYQBa+V9nMMjYMWFjupCFQhUDWTGndjnkRLnTU3EfitpNL+Yoy9MyS9n
g86p+RByoGY6k0lri+KM2jzkEJSY3U+CyqbSnEX0NMkoQXY7M3BRy/bJlF/6Hcx5qw4buWdFcBTX
G5Pn8sMv4cNyXF4Uk37G6y58LjQVgaBJpjfZpgtOJUppmQtPoHJTTtjmEfbSZDKWOxS1XSWyQ3dp
WYUgsox40URMRlef78+//vH59O2OiIeQ8fvX358/n75+/vFu8ga4ipShs+LabMPTLAVSpKwnfBgQ
Qz0Y0sR7LwY87RFHFAtQY5MsNEWw0K+OZ4AjbkhyZNJdeWPwFTb1KL6fx1+ZAW8MbVLQzSpaOFpX
XLZbtF6sF6b2TY99T+RLr0r25tW27Tzuy8jqaqr5F1fZveMqU3u369DZVm+gnvsk3p5M7eQMXvTg
0MxZkCHyyQzThyqxe/XRcIXtfe4ABDce1DN6TmS727nrRQqSDCFevJVTwN4aXkCKI2iM6WL1VmTB
g9EteLpwJpqHdplez9+4OEk+eMA7HtWHG5OS06rpoqSyV6XHxGlcU5R4Yexsh7ygPE74Aenousej
xJ9REX+xWZDIqNS2j/QAJkexxUi6G4rv4YysT6AB3vi7geDWi4GPUhF31YRIplnmLZe2W0gQYcwz
Gu4RzBfjmj5ztBE5VGUkB3qD38JkR6kMy7e1vSOkqLAuMSyhjXHwfvje5YYfdsFn79DudWJeGLyt
8oIgSnwZ+2Apq9gNky71Tzh4e3TygRA7ZyPvgEQQOtT2UlDGfYGN0ofKzuUB28yLJZx4EupDgdFJ
zrrMO9mP5/iKsHtS4W24Ylv3XyZW79J3spawadKB4eAYwzEepMMY+zGfTYx4MYdvwe1hj+wcW6rl
AlkZtjS258dFsDgZWyVHlT8kM8EGp5aQNL8UyLMwp4pfJvazDJg84hsDOab+0V7EzQXluRfGMHFZ
te6K8pAr8iPrve2WSE7EZE6U34KqbplwHEhQgW8DVhn1tOnMzqWK5ts2AuVED425/IwJdaX3o5Te
ypcxvaWN7J9NVVaFt8LlBafYu15XJ3OJTLCqvN+5RiUBtYgPd2+/z1QlAtj9vJmdweio8G4xTeot
0SePj7AS2e6EZVilv5M2oUhckHN58MIQuvdiMJvAfpC34qTK4ybLY798C4cJf24JvP9sqRdI+Xz1
ws7+Nj6UVW0zUJBwFB3P1DvD/IgLatg6fIOYc/FPwGtcHlp/E6/4i00AllBzt2/TMpWmxtAsxwdu
JvaqECQng+TKKJKDkuNDV8fJiR2HJC+uKO1ogw9w/aWjwZ+z6sk6wy1KVRx/v9ODzT6WSAa6OWGg
jvEdwzoeY8YkDLie4mi2FYkxyITbbRAcdYxkCwg1fFX3aHuOfANfLOyAIkm6exouwpUd05/WbHVq
t9vNbr2fVWygD71ougHtT1t92ulctVoGy8WMuul7T72RLbZL1me2Ehh7M6aSzEe5/lUMKhO9lsNQ
J5jtALFecn/0sBTcnzjUgnFS5+BOU6blLdVzFnaz7TV+sGSeg7kFDRZBkKiZ9UKWmRgsDnpJA2u7
vQ9dQ3PCtSH7z4FrxWV9d/BDtms2Nu1ZQZBztg124PnY2AtCnps1adQs0oA4UgqMI7G9ZoPgZ0dU
tGp4bCBz+SU3CYhndQdP28me2r8DByxXHQU9YevGeTFJGa4X88VhCKYFo1eaJZOuhm4XUduvlGNu
oOiMHKUBf7H18DetrTJDd0vhCISUpK3YE3F+2OsFJksRICeNQ2Ja4EFPqZRLKDtvtuqVDGpi1lM4
IdbmpfU22jomDvBpwpZ+23IC6ZdbtSqcuN7oo0iQd+6y0pVj7nHEYRPaxsagX9UK7p+yHNhWGDbw
p/nCjiDp0sv0iIEHm76ocb4us9hglzZR41sBsbcvt4b/Gh30wUWz0YsWOGTmGmnlsbBK4LdIe0z3
sRL+jlMTsIzBRZyoRkyMBeEwzMsFcPkznwyxhEZDJ4bQnOlzWnGxvQkSbALP7jEu7BDhS9nOx/X9
chHsnIDtYr20A3r/w7MrP672L/54+Xz+8fL0p+pphXcl2yqLc6t3sKAOwUXUogamsHPJUYsaW2+O
0AJXDToMYlydEIcIx7hdCxCzD9dZ0lHirCVxgv3o9iTlj24V4vDWWiHqYeGAVtS1huIN1sII1HUV
00LBVeqrFSCI8C9sMJPObPyoo0zPHnWMEkhMZ279Gfy8nBhU7aUqDzVK/2pCIXH7Oyob+ZAc1/Iv
OVY98EZXpHIgPc7gttOKZSdQufEE/GtteiBE9n2URH7bPWUIjCSmUtlAOcVX5caeR5FDh5iciUps
aL4NVgsTMVSJ7Ay+2cr3pUBk/ytWAEM1QVYPNq2NseuCzTaec5M04TddRk6H5BDkMqNMDAyhR7bz
gVHssYGTFrv1IpjTSbPbLBZG+tZIZ4vlZtW2Zs7OyDnk63Bh6JkSpPOtoRCQ//dzcpGQzTYy4Jsy
xeK1jblLyHkPTgNFBAYXROXFOe6K1ToK1ZEZlyHb9FXaHuUn2TKV45qCrTXnVqWimlRluN1uVfIp
CYOdlinU7Ut8bvTxzevcbsOICSezGQHMU5wX2NDh90w8vF5lEQE4Rzlm7QBlp6tV0GoDBtfHWXkE
o6aJu9mMueRr0/hJjrvQRI/vkyAITFM26pA81K9gHSUtM/B70swXmkpwlMaPs7i+SkLuJEAywTk6
7N4Yd3Wyc/Q3FBNvd2Li9lQBQdGrJah7mlSolcK7yWXsTvb8qSL9jERTwNFR4GvyXbBZyHKjoMzu
TEaGP7fuWifGpMdrYzSwZlVdn3Kt8ozS6ZEWVa4WIKqnOj8eAOwBreNmtQoj1RaSrV/BwlyHYHFS
vh373SVI+2BAtL2s6dn2VgJ33kqg2t43y2zZ03tP17/qUIMaJaxbclctXTH1rkkZrS3mUJA8sM6Z
IDQNaHV+FvLjx5hu1slq0arOM+UEZgsFs5MGRp+/TRt1b0mh++IDGjGfQICVFfKjlIHShy/es71G
Ms1VmQU5CPfzSlEcYBkeIx86wpCtCAGlZOf8hgBI9wfP15hdR+P6GtpupIEX2njXfLlbr2y8aLe0
8q44wzdUsyFYDgKxWs7WW6Bp0wtItukKPM2wSuKoynK5Mr1uUNK74z1qqPxseKDo1Rnp1pizE8Ji
8jUCbEvGCFDjcI5keIAC/rGN9RqY/txHpOr1Fj4mapVzqiDZmzwAdL8iGlvryuKab31rzaBSlZ17
bpjsfJY9223W2ksGTtKilQJJc++7Wf+5CDtFnz4QZ0juY57NYgN6NooF+ay6fwSS2SEe44RTHmqS
0J5klv8isoGDlSX/YGVLso6EQAl6B1P71tGs/HVk8zG62WkFKbwwcPGIg3e28nZrR56uuqzO5g6B
AOzZOc/FfmIUeN1mGDKyPjco96+ZbP9X1CMNDVs1WimjLBcL2/LIuCsXdx04Um5nKZVclQnS0E2k
ESC9mcT+FUWybZXCWdk5m8jMWVlzW1lyO5ensrqWOkud3KK3+zC6RroROwao+cvA5C8MKyNLC1I8
MQzhlXuubYVXxpC4XlBTs5PcdmH8tMDZyO3ihFnlcjhCpEQdigy6Cy2WYT2XOLmpnbsJo9jJ3Tty
3m6Rs1wHdxsGMbL1FGvt2dAF6dmcAHZBpWuBoI8tQdSDJ/WlzVZiYLRtO6sF0NgoBw9eyBIFi163
Wytn5xtVst8c9qPbBZKqqRmetmujFjwHmEVnYKkdwZ3RoNYsv4G9zl7J+5q13lXX8sZUhlDfkvzl
IY2lmfAlDcIgaK5y/w80x7lXzpGbKqDSYhA8BYa/EsuVCLyV6nQJfCoqNj1SgeeNkC+Tkgd976uB
l8UnlCsym8SM6XbdZGFkOU1MwIKhlr8svbgkCVfhwl3bmGpCpMxLs024DL3FxNsw8Fc6acJF7EPN
vgq/eYHXoi9PHx93rPPlq5jrVbcy669hlASyu7G2RDQyK+uXvZZPtl6C4Ay4sNhsSnGXxUva7z/+
+LR61cJlfZaeR/Of2jYgaFkGjltzRIhqDwI8UscNQSebc3kBKmLa4FYHjQEqXiBK0vMQdeVDqyHE
ySNIuDnW8+05EE77bDLO1WAEHqSVXftzsAiXbszDz5v1VoX8Uj0oF8CCii6ialKH26JaiwQn9LCv
4PnvX7K3NUFjAzyxuGMbAfVqZVnWJxA97VM35J4GC4sPTAWz8WLCYO3B8NcJXYqb9XblRuYnreI6
oL9fNZH5w1WkvKoe+ZStCMvA02tilLoxyTFucoxq20OKqSXFNgojPyaKfJVqN9Fq5wHVDduQ3JgS
XanlRn3EgMdOOGYTN6zGJaqPlSWm7ghzWe9OIFpd42v84EExcd43oiEE/NINoUXY0eqcHBnFjWz9
MyiJ6yBoPcNgnxSeXB5E0D2SuZcvxW4DCGzNCy1mXsAVsY9MB13O5r6PeVdIZ37OATvH3UbxdyoY
yUNcx44SEUg8Vm+QHHIhTGqNXZnYrVFEqx7KuOYmTVpBFpQicI6rOGE85bg00Lq4jGfhwmaYKPUA
LGu4BMBuQFLtLW5gRsghC08eRGNRUygItnZ4QGfMltXC4sh6hHHlb5x4UASn6IpL2/OyEUeL1PR5
p9KyqpE9Q2gMXQ05stmpvcGVp/AiPvDnPp6mgF+dyuJbTkXtY8sLowlGcXnwdsoVp+yHG/TliMrj
2TN4YrJaBIEbA9LOuahdX6Gt49TwDYDMZMZJilQ5+nXYyK0J59t06hOubTwT7P6KsQeSERyvfR8P
lQQdY9+YRojEV+vqTcEGQhKzxW9hsJCgRO5BmYVrUOmYWMe4vCoWfhLvtGc/jJzJcGcyEBVcsVOw
qZFUxdLeDNgrhGysOJGfyGxZI5utxW+1ittsN5vbYLsbYMaHyzIC9rrui+LRQuE37DgQWLcuBQr3
ol1hea+jIM9MIMVtghsvdH9mh9Ugug0X+nsE9NNMOOtwUm4ji9Cr4B+2CS3iwHJ8n0MPtteXKpRS
UvNbmxuxKy8OnN3UlldQMu4YFzU54htKRsjygkkBHeI8bm+EGSQvM7pNItvbYRmXnX/BlJy9uENV
pbi9EdbtETvGlDeNeTWFbXVWup/t8qi+AcYON92x3a5Cf72PEC2W/bm0mTyo4BLDy88uIbT9D/Hb
YO1PgHPM5uJNONubSRlG1uRhsw78n+JcfrlhTJ9oFgahf4VFNiFHBfnnG987uqvVU+kce8vAY6Mj
CLY3ZFkUJAiWfhjKM7hxx/UNWHII15F/7Sz4Dy+srC781VB5YTsp8Q8bJoLiW2Dt+pyzDP19yU7s
LfZ/SHb+LyB4kX9UpLTL6Kpd+Ld7/u8GH470Nig7H/jredvOek0pf2d3y2jjNtBVUVcEU3RbTTEN
b9i22efhe0J1CzK0xeuY41Y34vzrQFPcMigJzlGc3gAjN/U3oUEYhTfAiuyWyp2bG8QXhsqY0B3d
tIuRdrte3fAxarJeLTb+veALousw9I+XL/wI65eZqhzvG9xdspW/4U11LHr50V8BfE9WN2yaDLYJ
A//q/IV7Km9vFweiIAj3dm0YJsn8JoAdFoJl67iCEJJ7Etf2BbsHXrfrnWXzFIg9k5VXC2sFUdQu
WF9Tqtri9LUv4u3S8sEEgquy90x+sqgDJFSKkir1wy5Y0yZpHYN7V62hrio7oQdSx2XPnjfm1NJf
dtaM6+qKmkJ5OSQYDyhWn9ELclIEi928EPCLmscUnPpwpZ61vLO4z5plUCfZdrWxn2zra9H3t14j
4PDOm2fKO7apaNw8QKQVz2dI4024XfQNIK7Bl7Z55BzHuCCsRWcX4p6E613sRqzDtQexCcPQgUiK
2Hp86RvdXML1or2h1Ry5Xt2M3JiQ2oVjQopkvOGc7BMKvJwdSfnN3fHx/dt/Pb4/3eF/VndDZBsp
8pRt/z7EBZp7N+svfU2Zju6HTXezoszfH98fv34+vUuRBYf1kkqvgC+SVmlw+EqbuCR5rDmEuNAB
YKJ1JBcReifO8Toy91i4M55eypS43W27mj4oajxh0s3JJotUwWX1YkcsybAkT3lkrDOtwOPvGGPo
6f358WXuXbVXVfGgvYn6jLpnbUN1cRXBLd++/4MzPkS+PIKTwaF0n0dc7OOuzheBLf6figpcqKGq
HTwVhwdTpnALfYZcTZXyB99xWSKlnR7oDNHfvdno3Tlu5PeMGncaLHp72BYdWaOKyJDWBYHSc03M
VhGqS22J6KjbL5bIgEPjcIYvyIlIkrKt3Yhgjcmmdbau91z5C43Boze9AeqD9U/Ya+JF2rxS9uym
Dl3sjORdXvvK4ChcZjlqfdAEnBvFEKkBHzCTWC1XL0Pv1npsqDFsvbIeaMOiFBHNUs2moqjaWDyd
yK13im0s3tHa/D52xzS3OHHqDsRohlN9qQqsGO1ALG62cBvAx0vSG2fJFefUJLFPDh4F7kxm8wPo
CW14afozNkaCYBolPRk9J3Bf3IaphesCw41DanaKwPaJ3te7XP+B2EEwaLZxFsg8MSfgPl4aPRlP
iD6SzaspdcIabbEzmEAtPKa03KpOqJidhZnkkCTdDVkmpeYLUQpiW+e6a8/BiwW6KHHg2e/Zt0rY
/3VhC8yaP7CPb2fOXC/0k2guVgiTJXZWn5uGhXL0gDDpuLECm/OVSgZzGNUmihEL/gRXOFOY/Cjw
gpLfn38YS2MdvxdCNcsyz1F5QLNMNeOfiSrcMUxjtmfkNFlGC9ND+AFRJ/FutQzmeQrGn2pjOQOX
MMPmKYSzBomYIhU/q16Rt0mdmxc8Z7/JpRxRzg4uXIBSKxvnh2qP6WgJyDIZpdH9Hx/SR+gdTNyR
Aui/v3183n19+/75/vbyAmNl7m5CZI+DVWR5LTbw15Gb3zr4RbpZrV3sbWCRvICPbWpjziQ2BQBj
1hi3Syu35LqZ0M7nfiTZkDxbIQST1Wq3cvHXNoNbwd5ZLiiAbXOi1/O0y7NpAfjr4/Pp9e5XNgD6
D373t1c2El7+unt6/fXp27enb3f/7FH/YNL0VzYe/66PCZc7Zs4H5yZWAx8xZwg+lDwAr/XeEHCo
QBf7R3AWAQ5t7J+vKuIUn6z8amYcJ49JPZw8H05J7PbzDqDmFLWOD17YPK4Du3djY08/d3MoQl7/
yXaC70yoYph/iqn/+O3xx6d9yqe4Aguvc2ivTFKH68A+tJ0Bh3lHVPuKZucvX7qKSev2NscV6dg2
agfgchaWnDen+vxdrKh9k6UBrzdXCBzmyEG2xVT7cvS8t3/WPL4gx0QAh1DOOdCH02bLvAcyExak
VhgqHhk1bBEb/I0qFpBaccvMw5xYXisBr4iJ4jGI07ggJI78Nb4rHj9g/CXT7pPOvw2PMMyPTZaC
hnmok7p7qPGrllMrYhYzgQNb7FmB3XvVsvJdS5/Eh5cgqTW6uYRrnaBjtAys3GkNtUKsCyQw4XTn
Kn1yRsiP4/f/AdYFrcSstfKdawcH5AuLwhK47OC+ZRvrwoFwqAeAXXYZW5VcjW1tIWEZ07RMS+z5
6QaordV1M+fa/dcC+8tDeV/U3cH5gbSgGNNMlORPk6IMmntujUnr97fPt69vL/1slrV3NZ9r4ryg
5DXFlkOWF1y8D3O0DtuF/RPpS6o8K8YwWSOxwOqv+QbOqSAwRGvLM4gjMXomqInqaczgB0BI3TW5
+/ry/PT988PUzZAwyTFEujzxo7S5rAHDdamyw7GRo3ulAxYECWngXmVeK1qzWr19/ff8pMZYXbDa
biEWpRxDUKV3VS08ier+H9dLu3dHLQu2GpsELQ11uhTWSlyKoZLWMlK6DWvLC4w51mLKrwGzJLre
givYH1YchHDrLstu9tUnt36zjyQCltV5Pn/wNT1czKPFJrIbRAz82ZoxhBwyZD92f3/U/UuOpNfg
ZGB0h6Y615LWitEL2VuXhIfzcXYuR8fW0yjiIk2fo2l89GWJHZ+Pe3kmDtzrbmW5CB8gYOm4W1lW
mwFUMIE3IoutGzRugsS68Q7YKkG5xdJ+gBBc2iKljBBaWN7Kjog2WC3m63fz9P3p4/Hj7sfz96+f
7y+KaNyPABtk/IisieKtO1z4TB/3yNbmjhyZWCJea0njoE8yI3QZExl5WNgcF5j+vApCyfdp/221
JLi5h65WGX2583Gkfw8lzQPJiJ4P0GZNE2or8QpQJ3WXYNSFPb2+vf919/r44wc7TfOiDUcPnnKz
bIULUPOVI0DmIrDK3z+YZomKcUmtwsDgGtd7Ozuj8JftnoxDxjntOgMLZGOdHZxvPbgLbn5N7Vwe
UuOS2AHFfrsmFhse8bYwX1jsM4enh3EDIWchooMd5pAxh/FlC27E+S10p4NtlwjFmClStj0dzRpH
++gctUSc+vTnj8fv30yjNk7L2jFY2pxt/1vHYCNL7W24NkCu3UxdqnxDeCm5WHgAoesjg9Y38gE2
CxcAbE0cOdAaJ+FWnzHSiVzrY7FyZOm879WWgbkDX9+GzzlP0ut5se8z7um2dfYBZsIQ+0ewdoKQ
QIVLO6pJk8gW3NBQ0fGI4WkAWz2D9dI5DqJgF7S+oeSY70USRdutayBgUpHGMVUbeG8RmW87a3PT
QaHlaXpRE93/Sp+pITFPfXl+//zj8cW9H8WHQ8M2PVo5WsQWeItaXbCZQK+fGbW6VxAQ2s43KeTU
xbWw7uVC7qibqhWu4ftOMTaet55T0qdf//jtt+fvv9398fn88vz5/DTvGpJXdf0gdhj+fPThC2rc
jTgTi0xvLVTyL2G6JeWWdl2DiBrbUyLbVT46CP5JtVt9A1RacQzcnCYhk7C9BY62fTcgZ7Uy4uw7
pIxy7LNzmCBVWWaLbwmYBu2rinL322ZzAbj09aLkwu9RcRaOX2xHHQ4m6P7MhDhhoaVlNPFE3Ixr
4mqzhObLdxHaCz2z49WDoUBOt6qBFdAsEMYgKsZp0u1jysTbB4uDyxUPCgOZmY2KEaEONtxZQ/Al
8KRhE+r68gF7sD3eHTHXcGG59BggKQk3lt1CgQR+SOiEDAZcxPLCfMSFxdn2gmDE7Imz/2z8gq2C
Lv6Q//4+3NgUugMGngBtFsvFLaDQDwotZmlDkzCpIScnhmW03S0iJyavtxuLCfkAsS7IUzm8G93l
0Gi9CjyQZBmsw9wJShFFCeVeU9tgubZcvEs9sNnsNmt/N23cdWMjYBmsWj9mt/BiwtXGi9lYDBYk
zOqG+qy2/vqsdls/xvaUcZxhxT5abpzD+hCfD0jst0v36nGo8jTD5OgENXS1iNxju6G75Wrlhaw3
XohndDDIxjMSzwkJFpY7nbGj091uZ3nHMzT7fIhs8e2zM8r7Xi6KrfZMe/LgBtbmfMuRNF76BscJ
w9Wu5kNU2Ck/fjIR1CSFgyeCqiHwzCWyvHSUIMtbIFsPpAgW4eoGzPoGzM6PiQIvJrA4DZAwu3C5
8GAo60E/ZnkTJvBj1qEfs7mhrI3nWxypr8b3Z3h5W58pLPgra9DREU8iX71IslmHnj4gZ9I11aFr
Hs6/WB52jtgWd1lcDmG4jPavPZJr9+ui7n5ZB0rsgBFB29pdM5iBSVy7P0/C/oAAmInN64AOrC0v
9QdcStbhwoMIfJ2KVyd4fuDEZJtgu1hlXsw2zA4e0CrarIgbQwlFZxpT5MYd8CHeP1DUXRm0sb9z
HPH5KtharfpHTLjwYTZri6NHCeEeCUKHGbtH8BEf2Xh0f98jiWzP5EcIPXuWQwwqb9hO3CiSh1ur
VcQAolv3svpLsnTnwPe9Lz5IE4Sekc8DcB7cI0JIOqsbMBvrJaeOsxpIyLjd4gZMcAMm9GGYSOzN
ZxkGKz8m9JcVLm/IZ724AXNDnbehY03vhaiZ1DTx2yDw7LiACTdeyHqxXvlBwc6PWW+9mJ23PhE7
mIY3gDwLiwBZvhVnbv19s/ZtTwIT+DHRzo9Z3VDWTf3nmZ1FEi0vzWrhQ9XRwte0pN7ttu6vRROb
34ARUZMw2nqmTNFs2M4WeUTDxPoWrZ94xTryADxiHgN4c1j5ABsfYOsBbH2V3PoqufVVcuurpG8P
KHyLe7HzVRLsUpZ+zDK4AeNub51sN5FnZQfM0rNolBSM0FFTYEKtz/t6aELZihl5MRvPeOox3g2e
4VgTveVttovQi9kt3J+lrHkkeDfmS0u7UxOfUBneCoz8QP6C0Z1hlSRdvfV2GL++tkgxdWF7dTam
piu2a+ZuzLXwStFkT4lbIiPstLvyITwrOUNEf/oQSy8i8ZTieO0znvgKxDZ39yxDRTK/L55jwuAG
TLRY+TBruNNwN6wgyXJT3AbyrIsCto88Wz1Jjqt1eAMmcuukCKVks/LpbpIg3KZbr5qMbLbhDZiN
pzjW5Vvfqb+Mw8XOC/GsQQCJtj5IFPpmT52uAk/n0GTjEYOOReIRAWlRB57VmUMiP2Trgyw9Qx4g
oR+yXvogq8BTXXAz4NniLzheb9dupcaFBqHnxHShENbWCbluo80mOngx2yD1Yna3YMIbMJEfsvJB
8s12RckNqHXpbT1bl47ZDSB0NPl55/JTrIR47ElgCQ7OkswX3T2G0JhiYnWdM8AMQWRnGP5OBy7Q
zcFaBxgqUHNAJfh76Y0jWO55/NAV5OeFDh6O9Bq5ykztTVEWn3PaHaoLaxequyu2RJQypchAB8ut
fW5OAtY4wk23o71yAmHFEOd5lYDpiq8R8G6s0x+PaXi16vOuUjrDzB6LMbOnVpr580YN12j1WRqe
EjFr0P2ck6KLxpgESDAq6enOoXUW/otMHWu3iZaMR+xlXMFGLa2kPhoog8MZKahSzyira/xQnakj
u95BRseNe0TQ89RQBETX4C86WG5slsyL4ubllhBPQ0kNfxLS1Q3qc5qHJnr8/Pr7t7ff7ur3p8/n
1/9H2bV1N4pj67/ip5medaZPCQmBeJQB23TAphAmpF683ImrO2uSuJaT9KqeX38kgW0Quvi85LK/
D122blu3rcPx82O2PP51OL0dx5uKl0CvgYm6YA7Q9K4N2yzqq25flSNvZ8DksUJ4trmJk62Tzb2d
+i3LKnHRwEqSPj5KAvANtDmjdtb5xqaVlDiSLRa6UNvekJ4i1pIGbkfSYS2/fpxnRegBT7hE1Z+q
ChAAKZsbCavSkQfh9BxOIjifOv719/374elal+L96WlUGzmnjK0x8JCV5yrPx2KdgXOOPvBzDRYe
8jeMZXPlBiDTOaKcxwXV0gUwSZ+8S/n98+3x4/n4dr48NXFtViySSR8kZN17T3pbRuJxTSIfUzOB
odBgBAqYFaatzfPHhrXpspDdLU+dYcVYfk9rSEJgvskuSdI1p3AnpXjM0bBWeWx4XUVw5Ns1wDAD
koQkwqFX3DfmaNoSAvMFOVlMzAc5BjdxTHtLPQfmPrKGw8fj1Zouqe7qp8Cn53A7aWK6HCSLLaER
QGYtCRhDa9p7ii3pkoKtsGHP9QIjG+xhc71L8rU56CWt0/tNdcd2S2ZOfhF7SJwztWnhzLGpoShh
YDgAI+By4WFLRlZZwOecstRcHHFbWf8YeRnvMsPZfIGZ3OGIsKWrUHNN+Y2uv+3iYmN8NZxz7tLC
dG9HwISUBQHAgWM7HoDWTiCBXYd8BMZRCGy452PDNkZPCEPse3ZCAIGDgF0EEjgIEbITQosyO4Lh
+NiVgCM7gfjWNJAIhHYcYjseOb6PiBmvA9O2xxkO7bAt8nS9gN68MFc0uQcSmRVsuQGcfpOOyUrz
gGJFm6xMK+nHzZy4uk3NSeezhK3ZjngIuV7NZoj2rtcQr32CPAtsPJjaw56laVUxrjGxfH5HHL14
x7CNBdUa14Fnrncsja0xsMwPg9ZuKbECA4spd/dAeCdlHvhYXZSpwZDo7rGohkReosjSlsU5d2LO
cp3t8sJcZcQR6NBirJU0L6hhuapkgQcMR7U7j/ymJ4ps7vploiXB0st2hAjYCdALrXrhmrNYYD0D
B9gVC3ERIjuBBA5VRB5wEex24IVkaz0dyb8hIN/8eMN97gNkmWxwQgB8x2wkLxC29DR1jDCJLDr7
WrSWytO0xGIT0yr7tllTqxbOHJs27wviW0wqDiPP3t/1FEckyDEL4pTIMtZ1r1IkoUcsvcCVBImL
xCd3bbFdOGkBsXWjtbAUbbjJu0Y3TY2l13ubWuRlPGn16nMkl7Bsb6dJy1x4Nt4VHthxc8NaUF/X
tORTFXNHXLFiax/9BME6ERIXpgNg9mVqWv84D0GXm6Ajb/2Xpx9MNwqvjEXWprx5bfKaLlN9IPLu
beeUmm0Lw9WqK10smcsV81s/6LdqdmmTrg37Slcyn5UsTX3vlSVWd4hhEBiwEowMFu+QVGLTfHzA
ukchNuymDoqDz74NO6EjEjSMHQrJFdKCrjHCzrRLmulW/pVmvHl3pWQsj0ynJEasAIYedSWrjCLk
yqIwtMKbSNBJIiFsbyA59Zl3Y90NrMBwGfDKEvNnTG5ghTdUv26eG+JbaMRwL3BEI4Ef3cIKwA0s
Et2QfmK6Hq+wwhvCggDcolkIfGdL61QGbyonglxdThGXHjeVnRktse85oywJwdENJGefWpRfwwg6
S5JP9N3qEiR8Cym4gRS4ylq4nzE9HjVgLTjJGdSCz2DcIWEEWieJtDeEtP2Wem5aw7vv4CYWuYkV
OVn3hYMhjTXj2qrCE48INqZzklduRSn0MHLqo6KsnKdV9SC8X46eITb6SnV/3Ljao1iKAd4tJOQk
YQ/dQooAdbNCFN1Awm5S4DnrmCBFN5BMp6yHJF4N3Sr/Cj3Dwe8hq2igO1Ff+dB4S4QQuDpwli+x
B5zVlD0QDwTUzSLQb29hhWsHqy4Z9gIE3bQAImdhS5p74Oho4S00bHpnUaXdFpp3U06xR/AtNHch
NEa/gUp/l9N5NtdtkVdxt5t9cc6aZ9XA+y6HC1pe0So+P2Y4cLmYcRO7uzEvkcHzGtXutybWygs+
XbubJ1psneq/qWJskAcGeVFoAUbXDxs9sqJVqUc26wct0BaaD6SemixOR05Rs6r352h4EveM1zr3
0xxcZS1eJXAUfbbcVGW+XfIRTZFv6ZqORYV86VIptYreqyXThTVMlrgOYEpVlSYVrZHyheV1dYGW
mT4wVlcpLb4N6xyX1puH8f81T2I21vf1fcyR+Oyyepg8Lm7nm3aXNIkmFXGqtgkhWW/qbJFJvxHd
gZU01jnuLVLxroj4oNL557/CwruS8pyQjHgVIsNBDpmKttzVfC4eGaxLwTHOm4XRk7by/fSE7lbJ
zr/b8arDxFfGD8ptzlJipVQ0W/OGIw59TWjnN40G2hrr4qyHV1WHPbBbZMKDqlGXbDtPqkY+rMPS
PI3rqxvXp+f9eRnr4+8fw+cZ+2KghXjU7VoSI1T4iNssd3VjIoin0mqaWxgVFZ4VDSBLKhN0dsJq
wqXjrGEFuvgGnWR5oIrH4+kw9RTeZEkq+sLB21C9djrvDvnQVUrSzM9NQ4l0FHjvsPDpcPTz57fP
n7PjD7Gm+K7G2vg5HL9VMZCLgk15wY5fr+gINGmmS40Kp1toLLK1NLHXS62b8o5ab9fDzltGv4KB
r4iaMlIkRVpA4S9OcZUvsUVO2WqX8zTE/C9t5JJ2v+ZDhxIuZQ/reOwAcarMUdFeXuKYqFotTVGI
00qjCaHzU/78x/PH/mVWN4OQr+cleX0oivFe9gBap/W47oiHo2lCy1rYD2SI9H7/u/IaexIXqHyj
i6XSzzjv0hkTjhz1Rzg5fSse3ZxWj7Nv9GmWhh3G5cCfVjhSquj8rs2wO4N7+P1x/zp9EFZQl6yM
ByPyRbSbp+uvOrl8lVELlBn1dEBSxwwgpIPSelMwHbDI1mmZaeP5LRVHen/TQrl42n4eJzrwjgcZ
11pks870SihopU1eUUXCE5D2m/U9AdqEbxo+VzUAyDcCO+03JY0hCA1IiAA0Qp62kFjqAz2wjnhM
kJgxbWa5aZq1cyOiLT7xw8OGqPgPDIAZ8swQNkOBGTInI9DH9TUyxCSA2IAgg/rqO+D5BsTzkD6i
ugEB0etouxZWlQ6qOy9IU/mmrDZ6YMs7yzst1BCMtFWviQGC2qzyGQktdECbVXJLLM60rfZbjFo1
wJwESvYLHKhtQapJ/bS8j4ejZS8y7h2e8e7F8LQZOfCXkUCIB4XUjekcqJtxvB0gR2MjoMQi+3T6
tn85/iHGC+GcedK1d18vNyEYdhJD6W40ORsh4jXtke2hfCYzB7pnPVXWKimbyadcxD+DBrH6WsIY
PFsGelAUT7aIjVaUbC2yTgy09uXpOtpatEe3gAw7o6FUmkRqknuomtiOcQuRN6xxI7H5gx3NGTV9

@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Part 5 of 5: kernel config of 7.1.12-2-20261103.2-lab-ARCH, second half. The file is gzip, then base64. Paste both halves in order into one file cfg.b64, then run:

base64 -d cfg.b64 | gunzip > config
sha256sum config    # 443233d9b3d34544afc2b119809dc0a44c3c24c78f7e36f7f7b174b50c239fd5
base64, lines 474 to 947 of 947
JRTzOjGItPkR9gdl3rAhCNEcxrA/Ol/ulPm1Dje2BmnfiDqrRCBkdBEBD+vlSCdfP7A01ci3QeAB
jfxbMKrjZ3mcBhBp+Pw3u3uYytPYC8hULLoUbyrOixRiXXKKNvc8jy2mSFXnkLTtdop8SzwEppGo
xXFLCVACgF7XAJvkkxWKoWqhTrXckqqWD7taU9xxa0i5BPoJrCUDSTYHoS4PHEAAgCLWQlL1eo2d
wZ28Vvlgj1tStVHweXWjkdc1Nzm3OqARq98a4B5OGqKsaCUp2Z2mQkUjC+ssXm8aPtzuxt3qGdyU
aUU1tbaM68bHUFPUdRqv1hmjphDl4pZGvi3qHdC1g6YMAWz1cn9QFVnBOrVUinZljLv5NlmmtQ5J
0vjc/XXzm3+L7u+X/aiH/5etf+fzZDLtlDtp17/rIV2n3ENdn6xDqktq2fH7h3z88+nw/fnt8DQ7
7Z+ej/qEyiaTVax8GBsHKxrfVQvVYClYBk1+qvpFnDg7Tw516+5yaegyF34dy/k8OZN/jVYtu9Uk
SsMQBCtjmGIJ+04NUArhRJr54dAg7aR1SnGI22nUEjB4TTgzFgEJoMGIS9i8Guv2Lk3XqWJE0kid
XxQVUWcjFRUd6Xqjy5Fq5V9kynO9QmpI6UUx1+tYlyU5CRmeFlYVqiznJVXWjHZOeM3XoddrjUk1
wY2rHjbiIL7LmtU0uq4VXRe1xNHFKqcG5589W/O8JH15OfxxOs6Spp69ygWUHy/7j+/H06u6vEJf
eQ/y/Ohk/fhTnklUxD9+vBzMn47tRXnjNmPlxMaUAG2Si91aFa7kvP848L7EQfp4Pbw4OPH+6fD2
aMmCouvzHeaYdz6Gl6SnxHpaPuKV+vd/dEtb9hRmbx/OXBT701+Hl5f/Ry4WPGXx+JLXdbHt4/Af
V4TPj6ejeNl8Vh8e/3w78ir09795Uh//1/Hh+vOv4wevRw5W8+xWzLotHYyvn/uXx+Prq4NW7d9/
/H44nf6elc83q7AqvV28SI3vAjqiPD7+R6rPTmP71/fPtz9crFoWx+Hl8PhxOr49P7piZ59vP58d
nPrwk+cie3v/OH2+yldXFUJzOD2/P788PzpLs8mog9HycN5+3qx7wzvtRRF/YeJ6Je+Kzy+0j13D
c+tLEIT5pW+6vHOWOzuujt5EUhb8C7G7273wys5LGaJCXjrS6V5B/FBWKWO7RVYV4mXqa2dZiyv9
uzQtq81wAamJ+4h0M2ffm1h8daMGoU5+x+Yqy+iajzHJcD5wlU8m8/PtAip7t1e5xsyUcj7WbUqm
Q5Ki2zjJltrwCumGxfQhW053yUZPV3Wi/dvj88vLnrd/wyYDrWsar84FmFXy9bm+APefH8df32Xb
4wPS73/P/km5pBNMQ/6nOihm3UvxatH126LSH5Jm7y1vsm0hhxbLCEQrgA0nz/slrhggw7mmK8Fw
LP1CwBTaCYQiKwF6hnNh/Qpbhg0vW5xx4rvw0IJnRWtylXwhQKuWBMFwDXhAiBwE7IoidEVBkIMQ
ARchsBOQKxcI+S4CdhCwg+B70EWwhVDUUWHyKX9l2GtEx/DtjNL0TMeFUXseNK7kcrwBcu9q+mED
ELQH3dgzsFkCj3pzO6NJ7QwG8NzaOWwaEYuDAd2M1MmIXQySOBmtb2d4iFibqGAo+lBwGPhgWp5C
jiNrwDAgwEXAdkJorTGSYA/BD4g9kThwKJATsIsQugjEQQhdaQhdaYiQi2APIYD2ptcEPnVoKoTY
s9Sj0FCPQlfuQ9NlpyvBkTJCrGOxJBAHIbDrL4LEXlUjV0WLTI5B+olbQuMCuhm2Uqx+w/7aRmA4
DlGBrIw7VGA7Ac/pws4IqCOO36zJbBKMlw5CsLRF0Szbh8AWQmzwtNkvF9ckvSPT1ZH96+G0n2Ws
1Mz+6lUZItUKOB9yHHyoNanzdM10h9kwsVYJehcia9eR3Eeh1SSQhMhBCIidQEC4a+Jioq/Fy/79
z8F6+iR74nKZrRjFTf4A2AmBamf3Oh9H3q28fYo1/qfD41G8nvzv2Y/TkWPvx9M7nxI9zV6ff2rW
B2OGEJhsQMcMIx/rpDmCk42JgpXIH65U9w1BnByf14tdUbbTrrNOqBdFYWs+nJjQ0EeTDRJxCpj4
k8jqlAbcCI21cqjpuuu8QRDQLIZobk5DQ7fdbsT4jAVPO/InSWtKGLIusyP5fUFCFOmkIR7OlW8r
wG4pK2EXolqkjNIAEzIMeUS/nlIdBjE9aSoe6LK1TslADoZPWgcjAL6DQXzb4DSvibWJc9w6fnI8
sE7BWB5GkTUbdwyYnvXpWwht+RBri6XIScB1EdhCEVtgnsXa7XBdWxP3pEPfVlY9xZbEuimx51sa
rMCxrqmJDVlgnnbV95AAX/PdfaTcAZ3CaNLghTTQB2adjzRli5QXwAYtRTTA/ah9appd6IW6AziY
9Lbj8FCztj0e3qztMXS2R+twKRjId7VYFLkY2Gps0ztCvNZVjaGzoltnV/WKcdtB789DUeVAvc+v
vEv96yBW1mdiI2BShtsy4YYu8iZDXAcQNC3GaZjXwfhLR3k8cg7vyMVWnjZa0WOHGK7YZDQwhtBt
KSTV7OPz7XC6BDvaQRV36ic1og9d/bTfYHp/PLy87N8Ox8/32Z+Hlx+6oPtuEZpPxBUYhpGmI2D1
rsjKLAHQXrQhUm/unleTzQkcFPP74fS8f3n+76EfOJ8OA8k0JwmLwHYeebZq33PwDRyrRScGgsBu
s/aU0FK37RkcLLKL62ZUsycTtwkkRHp/NW/LdEvFzAsCvdmviUK5rtNfL+li/nz/OL7yFIojNLLm
TVb8JX+XclMSqEcJOmgRAzA8Wj7BoAVDJoz3z4b4sprXilaPFYRULOBJrQ24aHwBNoCx7zMC0PT2
V4+zTFkv1ZFqCAJDyiWGzFjbGjEPeUYMBoEBa3MEvGphRDEAmNnQ0KDHr4WXeHOuZ9+C8/L1DWX/
tfZGh8eGWMXHEEO0LIMeDk1YUeapHhPTGgJNFbFHiQ3FwIaGnhHlYGsBI3O4ESGGrHZHnzwbaPpy
S7nxZviyVa57jDFzB9DGuMOuaw+abmXY37wf5Mb04nR8++CfXM9SCKdf7x+899yfnma/vO8/+Ljy
/HH41+z7gDq8wkVIwpA3NDilmFfeVjmDyOo5IFE0EQaTw5rimkAEfmqE3pQZcNv/p07qnftYJVOP
+99fDrP/mfGumRsPH2KEMGYvqVrlYGixJsQPoU6o6EC0lUA5+nuurTFMLieMuPxXdouuuc3se6oG
OmGgE4YaIdR9DpWUJ+KtY4VYzElA1XiKGnmKKhheeb4ai1CFpjoEQFfykBCNcFpxRB3RV4epznlH
6umkJJhISWfPKkIYeDqhkqea8QasRr+l0xretduumJAOyurEm2inSZnXRgr/W86JKNAJo2EVq2e/
3FLj5aUBJQp5gWG3SCdVB07Oi3dCqAq5XQVBpZH6XqqIz6fN53pxPBGHQqyVluqpdC6PgHbu3Df2
ZBxOGk9KrjssPlFGHvgh8SatAyuaWLf1tNLL6wlIK4RaoZjGqFmT7Q5BQ9ZkX6Q2q5KPb+20VaEg
1DUVn2ilSgDdhQpxQ3eTTKskwkhNd29zT9YXRJWN+87aWFm7nkGbXKht72pPJ1oUgddLSjXjca6P
p48/Z/SVTx4e929f7o6nw/5tVl8bz5dYDiFJ3Zib0ZZgqJReJ9sl6m2GXt74uXIOKi7Q5KZLvoRe
MJElNULqNZNeGlCtuJ0M2EE0rawQqMRNhXmXBadC6HmakQBEGuF1aM5YcnPPxBsv0Q9LEDAllgiO
oxiPr/9wxzu+0y6Gdx+1hsZVx+JRDaheCRuEPTu+vfzd22FfyjwfZ4wLdEOAuI8DQmCEoku1ZWl8
vrF/3vuYfT+eOstCqZfc4lV7d3EdbNIuKYs3NUwNWab1nFt+CKhfrdI8XafGo4qrTbVliBoCFY45
fbUWS6HamDuhko+s5ZME3KhJypeMLHNsiFOiqmmSr+criCcBCWlkCmc9L9VUSplqKYr5MZxUY947
BQH+P86urbltXEn/FVcetmYedockeN2qPEAkKMHizQRE0fPC8sSOx3WSOGUn50z+/aJJSuIFIKh9
iGOjPzRuDXQDBLonZqzQRyio72/PZwXdbU24hP32+eHT081vJHMMyzJ/H/prmN0bPK2yxsyEKizJ
nmG2Neiu4L6+fnm/+QHns/9++vL6/ebb03/UEyY6pOl9E5OFA5L5RceWyfbt4fvfcGt45pkEF6Tk
h5KAjBXjZxNwJDOKmnWoI8qKBMuenG2LA7xtiu6HOdo3XVO/zacGlenpuqX4VXTAXz+fn1++PUs9
YJRpcyRlqYioDeT2HWGTpnJnFNICBjVpTwqFgU4HnjRE6j5lfb+MfGUISoEzRfwUoJYKR/2XikZF
kzIOT9LzJN/eNyWJdVmE6ieMcAiCtJf3ZxMmlGR83IYurUnoRpreufOhw5CUIyo/FKdhYp3sPD22
/fnpywvcGR9MCJEtbr3JDIMWzoh5RcruSq8wHMbt7AAJwfum2N1DJGSSKjvl1C7548sL4BR9smUu
bWSP+Pjh819iGn4YHFtLGzzkAA/LGxLR6HKZe1RCP19OQvR1SNuUNNq27rTBjUc+E7FTXiEqYuYp
fUPJoDtcisYq+mUAPsu2nCy0YkYSBZmFi9l3UVhoAClt2g/eehg43FoFOvkP06O1AAgjS6QwztPx
SJKahGOIWA2rdJbEqknGbsH8OmEulZYtvBVLsbRCQGM7eE02pQq7gGsgUesBa9NOSWnJLNwNA3j2
qx9JGnD4ltDtjjd3B1ru2TgbjopJQhqJPhip/3NqowhxP0CEVLXq9QDwUF/w2TTqqXP10W1NwuLm
t+4qRPhanK5A/C7++Pb55fnn2wOoiqkqEiwbyChTNesY9hbt+/cvD79uyDehlp5mRU4KjCYi1qU1
cSFpMDhxAp0uUdcDB00LpZ8K2jEM/CYlF6MD6XNSN5tn1WlJs2f+51eAkitNkG0XJWGklIkUJ1Te
uqH2zlMSYfkoDQqezZWJwF/m0HxSY4Z3dNZiSOyV9jmGsFq8GVfIdacgpDPvRBnJ0cWC3by9PD7P
+zTc0SLPSEPDTHmHtcOVeRtPI9x5qjhTw/XyrMKUSHi2e/aqqLjd0uLEkku5a7lIB3EUAQk6rZxx
eD2TcD8wrc0KXLDUHQOYuxp2qMM19fNMxLTd4arCS7TMqog14BhVXV5KthjGXlhUPCpqiNEoDI+N
7xgVauKjMl92TM6GsRokhrXgGbIVn6I7qS1xRJqC+a7C7eYEZat5MZyyQ7aFOqmNQ0YT8Y/6qgiD
HYYGhlUv0i3Fa5eODh/m+rmoRgn7Hp4IMgaHtWqZ5TuaQUDw0EViRE3DUhfMc7ajG9zfonKttcDV
HL21QH8l0Ftot5CupAiRvbDOCIyQW9PxFxFxYZvGEoJlriOkxkdrQO5iWUVkWsww1c064oq04et1
8tH5j6G1+KV2ke2sAnqqEFEzoLsgcrAoL92walVBlLHF6d8C0l1U+I7i8rBaPY05EZ7hilbKkrYH
tT2wo0yY9FQVavIM2dOSZuoFKBdWYqXWYglN8cKivqVhtbAIb/HWkn8kgVU0P1QED9zu9AmdGfHR
kSafd65oeOQyBKSKgINjlNiZ7JbrdbH1x+bIiQyGwtD14JAm1mLTtad2PxwNKcpsbZypcVM8fHv6
MrNoOnMIb3hzbyCjrg3Xwwv6pQWL3+BNN+Y0ITosO7DmT8PgDU+dwmkyjhwncHW5DnmDRXVueWaY
GuwmFytEbLjGvS9m7q2xAr6jEAvH8oJoBZhHNrLWcOU70zDTDTIPdGsYVmP5eE2uSuQ6ClMmS9xr
4EliaeBks9k2MRhIgQ6Z0Ag3+wg53FxQIh04JrSmWbMX9Whoam2wYelz3GNhb8T3hie0ckQtFyND
1/diyWzCVAMC000DaQe72bXjvRaKTG2jBljTwteAN+vBlrmec2DroPdit9HsQIB2kW5W0URYznvx
X4Asay2WBsg214P9hQ2IBOySRTUqz6EKMijP4ZvhNWiL69BZlieHuiG3YjHL8Ep0YXjBn6EOfYsj
XEbNbRQgYW/hZqebuLcRBIFNTXhO7GvXphbNxYRNieGoQraM4SVYXzph2e9EvVnDmeHoeO5ptj1t
kPeREXiRoZPxhOAIxifhe8F/h0zbPa7PItopZoZv6daJhGZhnmTgIK9JwsgMPF2GbZNsDOTcaZcV
QEai1481WoHc2o63AseONrINXwesoyYNTD/QiUaKt1m7G24iBxGXYD9wXW0esWGvmzTBseF4R+Lo
1ok053mZJxh+0UAhDkuW+Ibt7xLT1IKPlbBjxX4yq5DhmNF1eEu3QIBVBuLUGjqmcQ3avQYdOFeg
XcdCV6BdT6vRJnCyHu4JOze+Au5rLcABOjBMnSTmYg9Capi38Gt2EMZMrstSUkba2Z5zLIZJq5WH
GSAsr3ZK5SyCf0I1c0vs9hsHcZ1dI35i8JseNlVVm0ZsIDvTLtOKGG7aXMJ0EDZnmbrCmDSvQFsr
anRG+1ehA4s4V6CRr+VNMsKEaipdExnearAwkHQra5lnm7wpwYtthHTVOJ3Oif8c36xirfgPMrgo
KO+0C/0pA3OJsAMMyGhbHi4s+/+R0bHIFVXkGUYIbpOvzBBtPHt1jyVRYNirwTwxDXeXaDXyoNEo
xt5qdGS6kbEeLQbAXI/2MDauqXdo1YZ5RYYdtq5CXyFyLro1agNdkyG9oq3+VT0j0I7omXayuWaJ
r+kkPw5Xl5THYpZo0YTu88ZGxyo2tzpsGwAuuRNbhcRkKwa3xwsNU16BZwbyKi86rsfbiJsJWY93
Db6qPhTigQkLknFhQFyHRteg/aDSwcFPAg5rx3XwPl0DhiMkB8EeaRX6kAjTmpeH5L6zCy1tczku
wQDaw84h3sU0WjpY7898omLp8HD2aGpAOaQp5dPTyJ52n+GC0xAUI0KOtufPcB5pO4dH4AUC9kZs
p9UJvADXE4blc2GGmevANko5wavAxdbU2vaDEQy85nhXb3VWYLfRyOtma7m2F29M7QHJOUfpurZ2
T3VBg/kUBFfByeq6VDxqbSgN/kgj0n1TPTLL1s5pYacXRKyldVEYjhNanrX4raQ/7Z5xisucLW4o
+W7hK3yRmAs7zLs6WbjQGJFcbWiLpcfxzEXywgc1WLiQUy7Rg6XvuAweR8x9xUbdBdj2yvb7r/cf
T19vPr89fH366+fnz+ObvMDknsUbxV1PIpakMlWsN91K011mvQyjpujpB8WIqfdLuAyL7UF1RSQX
m6BpfUNalgfW3BHFx5/2Y1pqWgf9l4Hu6/9mYXGjdQFXiKUACMvdfnqrfeR40SIGziRNxbWPIcZS
fNAcYlTnuEOMrRDIEyalYvlFd3wRVBJ94ztbAdzfluqrSvxeLf6HSD0vqy1RCWZFS07zprtyJ02H
681T2alEmrq09LiNa0V59UTd1ufrZ+NPhpAuv1oqKHFehqQpynxDPn740M3jx7evf9RPN9Hp/vb0
G6DIprkbXpOFOJZDgGihHsNKmld6mBhxWt6twi2B+luNDc0qDIfstyTkUtUx66Zz5/3zdFO8vX5+
+fJ0s/358vj0eJN///Hy9eVdctNQlHibb4TwpiQ/cPArANe0DWMJQ7OP5oQONJZQMZZnDnMmA5CE
RVESkhb8VMxH15awmICWSptBJWWSDG8S0l0APOESCvbir3FHa7p0fjUyxpuSTiMej47cSMbbO/On
y7Tze4ZN1r7HmPFWJy/dyMzagLSyW5kxqKqbTlcJeZpEcIlH0cviTa9upDIsqEVqqUjh/YaUlurr
TAyPpcRKLWwaFZ2mjCuJh4owrCIKla4i5cJOg2FStoiZwt6oFRdwBD2raESVJZe0UtKoyjlnDBFb
xDZPWeaCfQb9yO9NxTFfR1U2FSkpuMKKW01ApcreE2KX4q3i7Yyg7+/LXEVDUazsgSrPozw3lWTu
u5ayNbwU1rxaznC5V4u3kmmIS7HGqPsoZeFBqk1BeKNkuEUF2dgIo63mtqOeLqDZD4po8CBBBA4K
8lRZI3j0aanlurOzPXV7PKmz67gPZv1rkNA+07gfJUWkEppglBSKfzFNkrKLNz4mhHlxj0uCZwQq
xItsEjrOIux7OS8gSHkBYchr2FZhphC6zYS+EDM9kzda5IfnFjFEPRg3NBbWCokamo/TU3x5nsEW
SGcOIwzNpaV1yTKmff30JHl58BpNmhmuVUGXCft3O1SbUq3S6pvNw6d/fXl5/vvHzX/dwKX/PrT6
7KkifPTqwof2wjKYIUBL7NgwLNviijOzFpMyYdZvY8XZRAvhFXKMu0oJ6LYh9SJdtbUCOo9yy06V
5Gq7tWxkYVsiV0A/RaCaNh+nDLlBvDXcpdY7hrmPFzqo26spyTlPkdiCya7uXZ4DjQZpMHMuCHwM
Ak9R0QtqzyNLcQo3Bqm+sg5Rf/oqz5sXVHFMNYj+NcuRanB3YZ42x4REOn5RIeplrEF5OhR4+DUC
PchFBl6D0rIqfMepNaC0QMgIwkV52RbjGHyDIirHMryk0BSyiVxToZkGnViGdZhlukZNh6xfvjSL
1OjRtHxthFeFIz3Svxht34/O/WO/fnt/FduLx/5EsI9FM1sUuzfh4g+WJ8O3QsNk8X9ySDP20Tfk
9DI/so/W+ZJxXAp9sznEMTjTmXKWEEWnbXEInxVCkpBy8vRXliGC+wVivpVCw5b3y9gy5yeOl76T
8uwVK8d7Aq+M5e7Dl/v1vNDm29G+Cf5u2gsXDURElK+PF0y1xeOrHHNImBy4ZdmXts9pYLGQjx/6
l/x//BvcMf7x5fW5+9G8fn14+/T3r+bTl58/BPp/vn//+uEcSarEYl8NhzxCkw/V8MwTwGjjCAM4
2UtCUoMzMnu42RHai/u7nMlOKFl+yAZWQ/tnkzM2MQjH6bBRFwqEDl7asREXgYW9eTlOKsJxBjCa
SLaFRzUz0u4YkWEIrAweTN5drIpBeomPqdjOjRPF2t5F8MrjGF6Aj7mfqZPeBNqtWGfmKX049NH7
dNZ1CTylHyemtBZCnw+jq/ftVyYKk+EgeoLNekzWkbuySxwxiu4znNIQDk3ycsIHrvCKvU7EPiJr
4BIWOrULzNrkibBOxldZh/Uo87CJJ0zF9N3krD0KDKe0kCdNjBm8/c/3h9H717auiojSp6xdZ1cY
ro1P1xXAHNr2KxhUqSh42mVCeA4QCbKUyBSss7PkDj0fL8jR9//5bce0pAbksSEVOHr4KqPNU8XG
sCeMh0eQDmmhaGlaHGzDbA64nJQzi6/aJs7bicXwFNMyhdBBYD1V754rOqwIL3A1bVRJcdIcTNcZ
uwAfELckE7+Ei62b147OFPEu+u/2OffI3zHMkwg3ELo1kaqZc65J3ZSzAIKM/5okNKd4AGP5PBE6
SagsNccedxAT6RSvMONlniRTCW4jX2Nz6EHzlBxiiu8Uyd1cmrNyhW1D5nl2NMbjbRtQNmFkyb3H
nfIJpUjcObti6AttkLiTJPM8I7AWzSkVFuJST+QrDyfzTixRvUAtKRaAnZb/6bICNAiJPXsD38b3
HpyzznJhMcq4pg215KdqUxwrIhrLn72Py+rKTyNpoTIfB8PZ3dGbLklemuB8CSF8MVnPRXROA/Ki
mcS9GAO4b9qGHpaGWEnC4WUmnSrX1ukSRFjWByndl3mrg3gub2Gbs3fBbv7T/nJ2qC7+7o433l4f
Hj+9fp2WsAlTF9U1DGtz3FHGE4XJembQcXh5e/v5fpNAROdhgaP07gP40/vL87fjw9vTtOiIMLrN
2s3HTKhOn7AvmfuPAk9P758e2nEcus1ir2EfSqR3q9R+9D6jb8Li8P6/87GPGbyuK8P5kAGF4bmy
6knpHVOsFGe2B7HRqhWMGVUQYM7ISaSrjawuNBQWuiJX3zppI+qwKjWtEFZIwuasaVq3DTzUQ2G+
DE7n7ev19fnLk0yiw12FlEPeZ+u03inw75QLSOuOupZpKPlc8raZb/+0PdsYimqf0ibtX97+9Z/X
18ch/ZzWJiavr9+e3/ua9Ihz2iV+9pDcJYyiXUvnN983Gx5WLFJ7VIOsbd6ixsMi4M/OK9brz2+P
x5e3p5vfWBTi3+eyDsl9iGCcDPdis6x9rOcbVhD8NiztnNYhvr89PTxC1OYR5pKqihnN4PRuzFeB
6QNEP789DOFdwiCes6xXu9DNWm3VwzoLBc4zMeeKqxSTLGr91leqq+CPp2/vD9IK8rjYYqX49hnn
+Vgeg8O2RNjeycLsbZ/Vd7be6RFz555j5kLwlOMI3pJYIVstcBh4C949LqofH0zkWdNAGWqg6mnh
BIj0INWF1jHINdfVDYBr6jaN8SMHmaYvdvvrcNrq7W1T8YRvCDF9HUQVZm4AsR0tF8fR1sU1kR4S
aCG2pqP3DlJcphpAHF2LktBRfaE9Y9q1glK411AvI4++h3wdxEeBo8WYrqvFWK6pxRi+qa+yIn7z
uM7LI7aJrOmn7imCNyzMZatNax7MVpspiCEnGboqHhOQimCrCI6K4CoIttm6s1QR1blgSVYSR0Ha
x0RHSVhi6fQs590MZMfVrjgnnPrUoEPZVmJbk7tFE6qC4JhKgqo/gKhk5yoIHlIQfCXBlhOQIt11
5OlDT8+jdFOVrm64p5Qg26prX0lQckQmklcPyaVYpAdygbL9BFmLQgLRNA15ZgcZ9QphbGE63RzW
lqEKQnzCRNizNKpJbOGWq0OYZyJbB7FsUwMBpxZ6iKuFaFRXC1nBRW+G9DDdOABMY2cAxF9WJjET
G73lwdzy1NVYgLsIL9JpK9lo2cygEfdFZTRjDvHMhKr1V4A0urYFIcfDepRjrKiUwqvaCBNYK0CO
tabiHtJKUgdc01doVV8F7ope0DBiqR+YbnOE+yytH7PV8IhuKVfcfjvhizA1XY0dBhhPY1u3GD/Q
9nCLC+rVON2kBpzvruMHuBX8kOEaa/i1uDX8RAfjVQxb4AqODvh80YOsf9aU2uJWFOpa9rKkFmRj
2ZqtQckdR7MGC4hrarm4yNRCNPUFiBNoIIFpGXoIWgEpNJg7AbJXYJYXJ7blidK9zxnU+enE4mfr
G3wNONXsL1mKTWSgFZjlBV1ylDVFlKFd1/opz1hqIcPRYhzN+QRgXEN/PnTCragXckxtvWxHox8Z
x0hjVgLE0cgCePjES18QOGaWI9vgtARXQfBcW0GQ7T9agi81xjuSqQh4O0A5oyBrQ4Jn1gqCJa+L
4xtOKqW4tmVLaykseFtzcsRjHPjeCkywPKo8qZBl4ChUvrmYIGlo6W2fIVYn6mcsMtfWQIiqvb4O
LXp9LeorONdr+UZhbWpUCGcIW5ZHNKBu06sHaWysQySWT7QCozmebDH+csuOqa/ydjWEaDRfC7G1
EH1dPM2WFCAazQgQpD9y72E6GWlhnhZi66vt6BvvaHvZ8/RcPFcL8S0dxDfsNV0IsBVdGBhID/k/
yq6tt3FcSf8V4zydAbYxtuSLvIvzQFGUxIluESlf+kXIpD3dwbiTIHHjnOyvXxYl2SJFSllgMB3X
V6R4LRbJYtXk+NkuJyu/XX+CZTHNMtk+2810cTeTA1VsaUdZviZi/q6nd6CbCf025Wt3tZhm2Uyz
eFMs64nyZqjy3Am1SPI40zyr5fS3vMUneJzFJ3icT/CMj4qMF2LX7k3yeO543XmBwFEHGi9RUsBj
lD1DYAZkeVuo8u4+z1oePs3Kjayd8Y5yQdwYcZyeIVKm5da42aaA2a3RUuQGazaZ8gYtKlERfwJ1
dN1vwFDjivFcBmUoErtiACaf0jAuzwbPLDvjE626mlko7xt+xfRqW9x7XkED5UftS4OCo9D5S5JF
PFbQEu1vvytIq+Te2vqpGcbS2881Gtrr6RFiUApk2EfAjpbgQ1HNF2FcSc+FOrmsDgZSHYZqEbTQ
O1cSGMl25RuiVAlrK8msMm2CJFSVBCVqLj5J7mim03heQAG1rH0a+SQTgCV/HIMHRzUvHFPxSyfm
JUO01IlVhDQamPcliZa6KPOA3pEj09JLO1iNVjiLhaPXA4tm4HRHaubPNQnb5zo2xp5KhmJ0RXkG
XjT7b/072qBTScoM7UgS48vSBiJiMqnfJEk+yKHxaWTL5KtoHTWPkDv9GLjNREh9WuqzIyy1r0dJ
XtK80tohzhNOeoHem99N/ZWSRrnoLPtWpWWJwDUDR2DHHIZWxh3doSSgllpHfO252vgR7WCYk3dH
ojfo3ZHlGbLkXGEZAULNZI8SnhcqbUfJXro+1bPfldixZH6gKE+p1ubH9qWVQqUYBZqEoFwj/IH8
UpsDfE+zGGV6w2SMCtGpfyPBMlKGRiSBXqGEZPkut/ZUA4ueneaobdIqIZzdEa53HnTFUPp21Dr4
wwKIH0Wvu670/pQFYlmlfkIKJFZAHYqEgj4g7mNCEjaY+dITQirmDdElWgJv9XXiMUwQi/VWhuhO
+7xMArvMLUkjfrQMqVBHWB5yjQyxv0pdNqRVwqlhmmTwBiILeqtzRxnUNuPaCM54SSO9Oo37jJrF
lCSBpT55qcoVkPfg+4yUQg71hFWPOChNQTLR8plW+YJwlBwzbTEuxPKU4MBIbByiGOiGkCl9GPIz
AyRgZgTTUgPEGiG9P2JN8oLbJcY18dAjGmRwcSixdSIKjS/PR9CvxgdTzToMMR7U0pXgiUKXUmWO
MdJ6Q6z/g45uvZJqRJK2nErBGI1SJIe5pXRM0TqkA0t9nLCCkCChmV4MTpDyBF8S7FNQwEICCKWS
MF2RzYpErpxqm6bWJQx8XiPW12KupGHpU1TyP/Jj+4murD3qIIlQejRdWyxhjAzFuyAbIztLLBYL
h6Yk8BhCKbYP6K5InzooSwWKel0wV82pckIx4gYKT5XJl2koETMmsPbEHg2Upz2lac4H671YE8V/
kXVNFgJEzQcKpbZ0RxnU7OsxgN1gpo/jjOVlHVe+kd7ttuQvTdFPisEYSoVmaw45dUWdhZ4ortCe
UOMezbTlaV4L0OCLX4S91+m3xwAa1JUArKSbLdv16y2pzs3K3Q0e6MYmnkanHDy3gidYUP7BQ/rm
gdpAxveEdMvRPNNVMvNfxCeKt5fLy+PLubcbvJYLkt75ZmcQgEkdwNjmE5/Q2a7NLLeoQp821lUA
vb1jHiqPErRE5jTtC+7mndHL49/4x8Pz8+ls/pzcE+X4DgsNE7z+9V6u2NL22zyPMa3Bo0xCWo87
ap8MjgLk+8Q8TXONMQHnAKByKNQqETXzK6anz7LGMYVCRiVoiYjVMVZHhspWYMXVQ0uCR3OGySjz
zTKhjGBwy7ZvH6Kzrn3Tp/fH0/n88Hx6+fUue7qLYK8M3i7QOXidoIzr3w9FxjSjXK6+1OLeTOaj
vPW2lDfnkf4BQZK77grzhFq81HZ8AWXS0x45CKmfCYEtJN5ogpClVhwUJtmvkVhBBAEGg62ZK56z
Sizp8hEc+KZ0TE0o2GKxof3a6ErgzuPmKrCZriC3byLg5f0CjiUuby/nM3g3MgsAvN4c5nMYOZbS
HWCgx1hZaK/0wI8wKsZSwqD7aaJ28R4sKGLNocjHEDW4GrqCjAbER5ldqJG2OvaOPVTOYh4Xo0yU
FYvF+jDKIzRMd+ksxj82VZpqioEl3mL8G6WH1mvwlz3GFOP2QBTGlXF9at7nzvD54f19eLYoxzFO
1c4UanamqFVA3AcaF09xN24zoe/890xWi+cl+PL7dnoVQvl9Bo86MaOzP39dZn5yB9KoZsHs58PH
7OH8/jL78zR7Pp2+nb79jyj1SckiPp1f5fvPny9vp9nT818varFbPn04tWSr54Y+DxxOKnuCliCn
dpFqK0OXMeIoRL7aGB0YCuVZ0Qj7IGXwON2Mib/7G5Y+xIKgnG/t2Gplxv6o0oLFuSVXlKAqQGYs
z4i2O++jd6hMLQnbE8xaNBG2tJCQHnXlr52V1hCVFCnXYUt/PoDj2V5E9/4UDbCnN6TcmSmdKai0
0F4yNLTdTTaa6DUIffYvzwBmQp8W++OFCsU544O8qgBrYxPChBmdpEKFv/16OH/5+fLtNHu8CX+L
9ifj5Laail2OSfkQlNgyCYI9dtUmBIrUYPTVWAJxPrIOS44IBRHhI5+rgwpBDKDkqu4V54eLmOM/
Z9H516ld62bMpE43ZUAFM5RZaOh6kIZboVpPC7D8GODuEsRQG3lIPFZjx1JVcOUv1jNtjnTUGgbG
hxEZLLpXJGWpBckKbINoerAg3ZWUGeUkKrXCw8K5Wc/VcrfEgfp6A8SQUbobBjp0rXkhapytaGJD
ccCiSeQGM3V9CyFaYtANzWB55wptwIjpl1Q9CMfKg5oeso8pJzEZyPEGBZvy1quZ6jykn3cBTyTM
UCtaU88Ik7QgkREJeQD6VW4Ed1TZ/PcQWvQ9tfQBMz8Rc99arw6sOTWX0Vs4/VeAKrRyzU0SSae4
ltLvzfSqMtLhcq9AWV0MlkQFN2MJM9fqLvepGJ7Y3CYp5nVlq7X0r2tGcrZRjC81zFtasENl7Z4M
7VJL5YrEceeuEco5XXsr83C8x6gyd9q9WAJgP20EWYEL77AyYygkVqAuUBCQwCIfSFmiPS3FzGPM
nMUx9XOz5LGMWOlvXHpgM0qCvaU5G88NZijNqNC7rMmwJd0BTvDq1JxwT1ns55ml4Vi1mJtHyz03
j8uqCDZeON+45mQHs2xo4s73tDv1FMK4GJCUrrUyCJKjiWsUVHw40HZMl4UJiXKu3jNK8nB/3MlZ
fNxgSwiRhg0ug1KLBkAD7SJOlhVUjS6MxRVpTS5CKt3Q4RiVkZYuCbSNV4kyTHbULxHXhTfN96gs
aV7q1dIdpWmba0Z4s48M6YFXxpP4ZmGHO6hQk61HkUDrA/JVHhMdHL0cceXDv85qcbAf0sSMSv9W
7mruTjIt1xY7x9bb110t2puUtcVVXKP5oJw1FgzqNj/zc7CKErtW2xkQH5xnyDseuX2yfe0A9jja
7ocgoWw2ufXPRuS2MO1PneLHx/vT48N5ljx8KDuE/gY+7rlBzfKiyQsTutMLC5qaq7641E4zDeeb
YHTQHpIqZ8aWovXTN1sEvRQNdSSuis4knZmyT7Lajh1brp1fsVracK1ucT56eLdxzaq0brzEsn/1
fFNW1xWhcR1r3NwVp7en1x+nN9E2t6M9fVcXlbA1mDwPsjIUB+RsDvYTp6yA/OVGyL5phE84VtgP
8GgZURqsVu56jEUscY5jMVuXrZDfVXZBFTlzPtXtjWefifO9+Wg9Go/Gg4O3/lg39qgitqkvrRSZ
YjIjxeLwyK0ybn4qcIerr1mV7rquydKUPqxzX5fMYZ3pHw9rYiCRAYlVPtMVlLAuM7Gm6cQUrFRv
J3QKFg64K4QXJlrrC9oAOQPaDg/KoHj/bGjKpVxDMp52hjUfbKvln3rhO2rX/h9GEOHUgsgOMkOZ
NREZQ7puMjM0vWVJTGzZKt2pHQ5dmcI6gcAetlXvxhbaCtCOhDGwGxIjPI4VlCPCXnyAnenix/o9
bv/zO2zFuoFmw7ns1euaET18+366zF7fTo8vP19f3k/f4GTwr6fvv94euhs7pSZgHmC9qlIkRiso
28a+ZtIjt81syU5IJU3d47FpZAF5MKiioQBqPjyQC1WGYbdkp8uCfFgwQ3l6qPFAyS612qbhoJ1r
EjAyil7pTveq7SjsRoGDg8bbq2G5iEB81CnTqdKKzkg01b2DsH76GLVCUV1K4YrQdkEY1XviYzTQ
fMG8ZngGrCyb04P6aspzLEivpPKnmCJFb+hcaVjx8NmQS77YLBaxOZziLSGoAzQd4QphHzI3iYUG
rzBTvg6/a4wteiyA0iliNLfmyAsmlDDvoF1dqKBOZ5kzXzhz5zBsCZQdNkvLK8BeW21HGOLAZcx1
LA/62xJw0VKLtcXDS8Ozh8bUAqxcBR7/eD19wbP01/ny9Ho+/ef09ntw6v2asX8/XR5/KIZBSu4p
PHJgaLn0xsoJXGCLUTO/mmArqCt7f2VxJ3PjpHzlWB7M35gyf7Mr7rfzZTr1XeZ57mGKiWfuejWf
KpnslflyPcG2x8HW3R5gHjNmnb7/3y7S+xadL6e354fLaZbCTddg89qUJyhqlPC0serUytpa9Lb4
2Oz2uZlrWB1LqfoiDy4xaranYBx+lUlp2tM5i30JPvCJicgCb+NthmTt4kIkrf0k758pXkldtAQl
LIL0x44sd4CQ1HrqAyAuj4Xqi7qxEErx7yz4HTL/jDEK5GTfuAOKylT8Y7JABZQFcf/W60qqwTMj
xoQxJUjEDa9i5pjSaasBAGKhz2P4y1rGNmnCw9TemDRMaxbY87C8OJFt7W8sjyYB3YGNejBWvmBv
T1wJCWXPu2IxHgGDmK7F4J5begffx8PmjNm9vZFyFlMf6dEPFJ6Um0MggqVIXYXMZvCSkpRxiu8M
ZQVLN9XWXRpyyeBhJlrdvX4YIlJNwnminqFKBr+E488MjofjPZwaZhEZRluCqE+Gi3uZA8qELrHa
ml0QSY6clSiF+Gs1Yu567dk50d6Zzy3OEa4MFtd4TXXAabzjTTCsRhhwOZ8vhG6xtLOQZCGWRtfm
WEfyQLAxdwp3RnGbY9krvnVGWqrAaLsy+jyUsB5TrMm0cLfLkYoDvhorVLEa7T2Brw6H1vZ0jM0a
8+2GLyeqvppom7U7wbAdyyEQG0pnyebq03wli32qBMgCWkmiKoEbjpHxGTjr+Trd7UZZvPlYL3B3
tR2ZIxwjoWRtRhgSvNouRjtSjN7Vf+w4Ze4iTNzF9jDJo/lC0cSNtJb78/z0/Pc/F79JBaeM/Fkb
hO7X8zfQz4ZW97N/3t5Y/DYQWD7coKS2fmNHhtWgR5KcJgfRe7ZEEIV8MJvAItw/cjIxiUc6gnHY
/wwDUJ8f3n/MHoTSx1/ehGo6Kp2RyGM7MplAJC9XI9IbAkiutxOiaL4Y6ekSwvKuxnDP5gKraYYo
dW3Od9o+24zOCNhhLtzFcKB1g8nWelGZV0Vv+RxvpbkzLvu8xUgZM1xYHXx3AuVTJZGhNZMiMs4q
/vb0/buyP+lbh+t6RWc0roVBU7Bc6A5gkWlGUx5YkJgILd8HyyIzbngeqOC4qCwIwpzuKD8O5nDH
AGufbSZ3PJ2puxQFsvmeXi8Pf55P77NL04Y3CZSdLn89wYarPfiZ/ROa+vLw9v10+c3c0vLGm0FQ
e1v1kGhyZAELlPXP+RQsI1wJlKclBJ8PmaYkXsvEjwMh1mFNVDgZVo0NRhZ7+vlweXrUplJnDSB3
PNSnSdMnHXmxOArtE9FEPpfpomh2viMe/v71Cs0pg1G+v55Ojz/673MIgsh2PzWCEAQZj8UXRSGR
FS3yJMmtaBUUvLShfsZsUEAwT+5GUHLgNjQZSak+QNaw4i6vrCg/FKVyL60VCS4Ebe/YTM3fe0Ym
/p9R/XFDCxKhG/XeHF0TAd3AXnIsvdV/9AnaJgdIMRb7sKOZ2IWm/Mfb5XH+j9sngUXAPLfsFwG3
GdUDlu2a6KDNWsFFJs9ilv/1oFhvAyPNeAhfCrXySTrEaDSQlVnap9YVJXUb2FItarkbHH80AX4c
LItnWMa6dMj3V18Jc+3tIJlI/nU7wXLwLOehHUvArH49+ywW31g9lvXGGWWJj6m3Wo9XKUWH9day
UevxuAvX4k2pY8owd+cL7zjKVLIVdidKTVkitDrvEzzOZzKyeE3vmA4L2yFux1Hg0Oq6T+Fx159g
+gyPN9FlywX3xnvMv3edu1EOJrbXW0sc744nTCHAwHiHitG+mGRZWK4ieiwrb5Jl7ox3FEnduTM+
r8qdYPEmWVxngsXz5uN9xAJnOV9Z3InfmIQc8IbaQkGnpBUMge30KNkupwWRM82ymmRZutMs0xJv
O58WVYvx6VxubeGkbgNpOTnWQEotvU/ITWdqMjuLCcmR4mKzXVnWV0MoMBgcsLP9xJIWMNdxP1PC
zbRImk/ILTmxtnhi3hzWC1VYqA+CJuuzcLz11ChaTcgaYFlNDte1t6pDlNLkOMW5WU7NICEJxuch
Cem4nOB3iw1HEwNy6fGJ1gGWiVUcWFbjGk7KwLv9RE8XKzwxD78es3s1eLTs85fnL2LnOjUUimQ+
sTABx3gBwNFVhsn4CsjFX5ML3MadqGuUJ0FIWWyZ5o08wSy9OgeEszp2EruLt6mW6Hn3gFMIwxeC
FN0cAlzT36iWB4qCoRdU+JaqboIX31RzoIGHxUo+lJIeGpiK5oqzIHhuW8Lztgg+YWo2lMJtUjL3
DobqtFF/RdrepqEhMvXIHpUpfN7m0z3Yy2jHkJERl1FYbUUE8N4GyrDe8CAAWU7hU6F3ig7bmSdR
kbjuvLZlXiQHK9bcVnOCrRxtCMwJ+DCBN3O3/iRXYe/nwN6I8pguhkas0yjlEzy2Hrb2botZLy0F
bu18JvasGnadMvj8dHq+KFMVsWOGwX+rtZwpMm5bBd2vwqG3DpkfmMMrMcL3km62QWpzsnxcQCP3
+S0DI0kIpWRjTDFBhSUetFqXXtmMZ427sG+8CL+as+bblJc0iEePhDQqNTrN07SSViGKfyKJZblE
LR8FUXYfKn6FJFm2Tzg82gOws32RlzEwoAYv2GUWYHqm2nddyUJiHLQqaDYNjQ/bK2TshGwXlQiT
OiqqccYDseBtZ41Wylzt106Js9S99QRS+4gNGsDgJUQCsA6YG/z6tbeTAEzro/pVjJIDCdAhgilf
EkbM8kRNJMTTIfKJib/fTrayWBtz0FRXn1blfe0fpbfnFGUoksNaGQFiyZcuykyDtxkgKMNx3p8O
je9j7bccdEoMTJU+sMJv4ZRklSkvcz6a2XYL+ShJ8v7DxpYuT9CHX0zVRb1HrnEKjvVI5+/I3iiw
iLAYlSRon/MYcrQLwJZBnm+a4KAw+XTdyZfwNOdJv5UlkSN212sW1ZVEw6I1tKQpL0UbEsP9d8gN
bccac7JbASUZfHWy1v+V3dS74SWZ9K4kZsFdne8zuG7JwVFkcnMp9fj28v7y12UWf7ye3r7sZt9/
nd4vJjvNWAjhcmcOkD6RS89MvSRHvzIvPUIHIIFZzkUF9U1hcDgSKmykHh8n3sZx/aGApzB7L61n
kquckRB6fDydT28vP08XTfqggIo9pKMfErX11tI1eT0/nF++zy4vs29P358uD2e4XRAfG+a88fQz
kC7bsSz6H+ngP5++fHt6Oz3Ckmz9HN+449+byq1RZSQi/ml42cfz5cfp/Un71taznFdIaGlWLGw5
N26KTpd/v7z9Lcv68b+nt/+a/R9r19bcto6k/4prnmYeZg/vl4fzQJGUxJgUGYKSlbywPI6SaDe2
vL5Uncyv3wZASiCJj/JUbZ2qEwtf40JcuxuN7uzx+fDt/on7RwJf7Ia2ftw+WlhPr5s6igixaFkx
CmgynKBt71frwmbtFm20800gke6yJC1FLAvPB3fpg6Jp06vqNCbuU299R01slg3i8bn4xIgPXeo9
u+6KtEVrVicQXxb0uqYlfT6HGTAMzPNoU+7PZPqv3dZLzhFpy7qIiZ5zOfUvsnK/gUW7tI3z22lK
S51XRbUiC0vLrY5azqpfp4f/UY11qLqb+vD98HLg0+YbTdUfw3c9vOAs1j5c5hCrAtNQn+F+sAbl
3rq4NZxAa/p2aX8nUm/iUp1/Cs5nmQMiryhkLHNtENdnROV+hAqYHQ6JnI8QgRhAClGcxKlveB8h
C62rHREzyzCMNq6uEXKlBP27SjfXKPf8RdM1ouquuEZSs5AzbOkHes1CZm8K2S62r5O410n8ayTL
jJh6wQrOqSF28fZqR+ZER3/cfa31G+H6jlXZhhvlT/gDufzY6f3lQfe6gVsGSQXYIKWqy4Wyb3BV
lXANOrIkGqa36a7h0UiigeWk0FPVddRsidYwAhcsynXGqEPox5na9ExD/Kc/c8W+2NNSsaEFCbv2
0d5k6E9pbS8pu3mU5YtSp+6TknxW7hRbH5kWqY4KZNLFqmIQAkfK+9X9j4OwUJo6G+sraatVI3xH
/UYID8YzB1Nf7PyBohWQnPUpM998zpJEQ88L1z5sXLtGeTHCpTGXkJOaOoub6ReoNHn0VX8vMiTl
kmxDh/l2pdN8l0tJPu5NqmCgLqbZxUb6mgnY7mzcnmUC1cjnjh2JomMFkTRwOTye3g7PL6cHrUI+
5S7AuS2LdgFoMstCnx9ff2jLqwrWN0hf4iCnwvOU203CXR9NL5apbX9nv1/fDo835dNN/PP4/A9u
vPRw/E6TKRkJN4/E3FMyOw3vH3rmXwPLfM/PfCYOcg37XPHkPFLgd9vAOLkqVnVbZGyi8K+b26zR
JrbrNK8GO6m8IYjqCX2z3QzXPPcnjxqkYNrKIzVaS1dnWrXLnbgUEMzopE3pJEtVVGOqT55pt50n
oXZFHPvkMzyD3xoMKS7DNRmVUc/Hk86N8+rs7XXxcrr/9nB6RLNBi0sBbF/9sXw5HF4f7qkBn08v
2edJIb02f5vFcXe9BC9d6rgq9MLZlYpETd/f//v49vqOPkMHS9vW/yr2KNMEE2D6JHbk/Ph2kOji
/fiLG8Oe15vOpjpr0r2I+dbpW/JxuOmuzo+Xfin8rooLxzV48ZOd4fP7/S8avkc4NPyipVroVfva
zOp+xN/RTarcH38dn/5CnapDz+aXH9rGpGlBcZOcHu+PT5NtbYBMdjUF1WxF/C1no1mCtGzpGBj6
ku+RGBO3m7TsWZbzQtK3TwdfJqg+zxgaDrg+jxbuQU3nVEJuXtbp57NTf/nzZnWi0Xg6qcdAB7Wr
ctdHGiw3SVpEG8UWXiWijZyL5twFGiDgbt4YSeV6mL8+YVUEcxObku3SccuT8fF1+Uhp530pLd03
8cUQPv3r7eH01Hvh1ryfleRtlMQtdyeo3es6miWLwtHb9iFBF2JgnC+PA8uxDG8fzZXODTtt171C
YjouMKLsaKpm45rglr0jqZsg9O3ZxrDCdbVOFzpcuKHQfy1BsXj1bgN3BQUxZ7VO350Ny8v4/YS4
IdCzk/LKUXhvgQTdiwCIsz5AwxzFtUI6PE3kVJzsrtdV00CW69AQoPvcNnF09B5Hz6IF7rjz+Qmf
ze9eye9eye/5V/HZ8n1rPj/hs/kDYz4/4Sj/oohMYPlLEPLVQZBjICimhStdFAOCzAiCGYIkQp7d
ksgGpngk4dUJUKxJLMQYUH0v9znjIaejJew9hWQuUmRnTSU+ubUTvEqbnobrwSHZ7d6wrAjWqJCg
dt/uWRICJP50a6IH30VsWzbyEhD5zsxC6nHs3yDyUYhqwgIHvIImLHSBoldi4FMYnEzFPqbZ7SLM
s8D5xuIIvlDnGHKwwJrbwIaXK7fBIhqfhP8vt34OaiuHbAwFEDLtGciBkG+0EPRCF0GBiw4c3wjN
2p0DAwSalgMh34ZQaGII3T/6ludBCHayb+G6rJm6QvjJNq7L8WELPcNrM3kTFtURiZX5dUp8SPme
BcfLx93ke0ELe8P3QwjhT/Zx9/ohnABB4CMotGAu5PlBQLDxIXjiHyWh46FmpCFa7XFsEmRy2UHP
S292aV5W3CamSWPkSWGV8ogn3E9PgmJ2Bg6wFF/vfbATZ5vI2u9hmXkT03EUQszxTYyBOw2Bhd4M
pu9iLtWgt0IcM03kXkeAAcbQazIBOiYu1QK3pBxDL8uEcGYYMKPtOAZsa+ihEzWuSI7aQ8wB7+44
5iKLZ8JCUN8m2kL3BJu48ZGsQhK9ie6kBAZOMaHMYlWRtRmaqxeS3XUSonA/QML9T4J3J+W6aBe0
V5iAr7gQgAOvzrjZ9y2J2Z89w9WvB5YIjUNRJjM+VhrRViMw41kYOcfrYIcZYFwkhWkEzDTmyjAt
0w5m8YAZ7mwJnsk8oLcQFNQC4FTjDAdzMNoiehh5V5SwcI4zSxDOwraZGlcIirkeZv5sBzM6CZ1Z
2J37+sB25nIHXgDrzosqtAwPZm/y2HEdOLsaFlvI9eFu6ZkGPJ/6tw8T/D81r1Mo+Eu9/m9KvpcE
0LLPMU3IWDnEldcfbNjVatXmLl9OT2836dO3QWu4XFqnJAjl6VylSuZOvf786/j9OPm0wIZ8YeRY
LmLweMRE/Z65LmJnzIWelfjnNshG3D/fP1BfcGuoj9hXmi7mNx29KeT1KmQdPw+PIlqCfNs1rLhk
5Yb+WdHeid7LNHnE2mrdqShmaNKv5RzRoki9ADGYLADnNJVnApk/iz7zlw36o3/91Qz1gT+qgvkG
eEnN4sQ2WlgqwY6eRaHvNmzAovA+yWoelJWtKiD/DmiAVyVWMRur6SQqXUTDKtIoq3nw9jpj3L+L
vr27r8FYfuhn3HgqKRptfgvdq7FyNWzrCIvyc/zF9fFb/+CQW63Gp8fH09PAjPuiGZNa8KmLuj5i
t7aosy01i4tsMP8VY9kBJk0WWNUXNm2T0MKx/nMmIYD768JJEZ1F8ke3Ze6oECw2wlyrbtdsDgaY
iw45gmzPgBCUhl0HGWgR5HgzAr2AsdKaYCTfum6IP9ANbYwZ8OM9y6lnGhv6NnKmSRIf4ix4ucFs
H3jhbB/4roshWKfvmRhyMORByIdz0fXnlE1Qr2EbCAocOA2DAAhlXErxQYlJVTZtgh5kMscBLCeN
q4m0zWLMkaNQjiGB1LNskI1kY9eEugHLMgwI0nLHZQYWlLcdH2jUJBZgbJkuDCiOO3NVhhYUD2iM
jMAae+8cUbiub87APlLYd7AHZF3JpE9myfm1xzWulp8j394fH393hiPjM2aASXeRwjfdaNMvdkEw
lv+6Unp6mfnl8L/vh6eH3+eXJ//mDjaThP1R5XlvdSLNX4XZ5v3b6eWP5Pj69nL81zt/KzN6AjNx
NTSwoAVFSGOXn/evh3/mRHb4dpOfTs83f6cm/OPm+7mJr0oTh9UuHRtyvYT55gyGtJjFsrDwrUWx
gxdXHehAkHYKVGy99Cb64t5u5T/sn/N4zw/xgJv48fvl9Ppwej5Q1VMWX1yu4ysZjuJbGYl6c9f2
cLve1ww5ORWg46L74RVi+Zf7iFmmYVjxtZvU1Ze6RBepRbW1DRez0t1FqywC37M2K9syjNkdYzoy
kvE93P96+6kwpH3qy9tNLeMDPB3fxgO5TB0HHZsCc6BK1phRLXPQ0jPVugYpoPoN8gveH4/fjm+/
tfOwsGygaEjWDZb+LKC+XnONHlAYE2YZOJ9tgJWszJ/1lkfWafQW5+uGWeCQWzdbgLDMR1fGHLL0
M2nSpfK8oQ35jftXfjzcv76/HB4PT2837zREmqXvGHOLG3NbAvXdOXR2U3Hh8s5oec+YmwgY8cwE
W/5sbg7P5LYtey43h2HuksXc3WhpcCsFLGkP6LBhSMkCmhHXCVBzbou9hy7Fdm0WFw7tzTO7nEoE
LUU6IvcjJbnXSqKN15vdeBUaVA6jEUYKEhq+OQ2JhOeMYmjXz1nhJWx/hSRMmH69zqxM6e/5+OPn
m/6c/kSbD+Jfo2TLLw/Besu5UUkEMRNmowMMZKsSFqJTWIDIVKUDPQyGqFjm2xb4/sXa9B0XQmir
IQipHAuqC7jk4xgQ0wiywW19zD2uuxDygC3KqrKiygDilARpkAxjCZV6Zc399ufEa4G7oyEREOwE
aAKJkEDf8wLrqv1azq6RVHWpX1yfWGRas3eRW9rawfVqXdWGa6G7/Bp5eSTINWCupWV6QOmzo0Xl
xAzxVA60nOpAvWprU0bQ82VZNTa6dSfMmcXAiquox0XMFMSxmKZtQwisR9bc2raJ7Nja7S5jYJY1
MbMdIIEJzLdm53dDUxj5uhVYMIMBNRbHfFAvYY4LHP5tmWsGll7+2MWbHE4QCQI7mF1a5LT/YNBD
ZnkSdGZBb2/M4f4sCNqUe8h4+CvNTGtiZtEdo8NjUrp/uf/xdHiTlozaA3QZ0Gz9pD9aboMQuLYW
EOByb40Q2Y/QTsf9FM9aOBfRajPDa15ooMVrtLKR2w1lS+VlpE1ZpE1aQ5m3iG0XMU4E0rqx53ge
0c4ZYZg7n0/LTdvkOycEbejX6ZqqCxzMho/pICvY0dUF7TfG1eI6Mlha5iNZk1/xUwe4FrDv/BIV
0Tqif9gkvFTvdUg3eeW0vkQ1nNyFF1v9PdwgTycQPvw6PuHFod6kbeI828zPFoVcPhuhE7iJeEhb
wPRqau/Y3afD/YuwDbh/+vH+i/5+Pr0ehbue+WbGW0ZTunNryYMGpWCjuF7BQFn2fHojlvyofaTi
osVGkAUOgYSZyC831787M/p+JzBnMB8q0RGfxzHTxsp3dwYzkZTQVLmBzKb2oeXOK8BGva0dide3
+6HeglvjmIbxsZJlbqmLfjm8cplLezosKuKCCr2zNQI5G1Th5xAXCrR9SBa1gA7FF0UFH63kazpB
E3B9xezrJ0BVpwxw3hWYnGsWiw/Sb+dxZWKdYZWb5swzDgnDM63KbWRxVFR1zkgWcmdGQqWBdTAX
2pkSZPvAVrFywSd3hyDu5sZFqtk1TRxP/ylfq4jkTKRc5y6X9FcK42l+USw8ca9futnP7NB2Mac1
yNetpdNfx0eueJR2Xa/SzkhTdsIl4igCdtskb9qAZahZYqH+5sIolMyyJKrFs3XkzLhYmEiAr+gk
0a/fJbe0Ahsdq5dAu872IRR2CLJwLg/mQrOQtyKAQgPUau9y186NPVxRAjcxHt2Fngm8Cl6ZJp0L
k9fTL+4n7AN2cBaDN0YWMy1jDgTL5Ur1ov7m8PjMrzjB4cHv9MMAHvJZ0TbrtC7KuNxWeXpts27S
Qu+Pq8hpuXhAAJYg4iYkCMTqojKAPk5APoJMYJDQsC/MMDEExF5x12ThtwG2GQArcgl60Foh9ACz
oBtXRd3S6Bx1jmKE8gh/2E0rR6u65D4QUQQ7TiK8L4xLjeoVf2wfbdp11MDSo6YgBriJF5Aizm8h
ltzFEOOeTZZNAfHx9wxAEiftPbPQ4xLRLzygK7Dm4Li0fZ76Pq0/3zz8PD4rfl0vrorjdVYNPZWt
1KB7XQKfMO2m/tMcp++sYkq8s6dpWcP4q7WBw4ULPYc16a7iVar+TKxG1gVRU7QG7TKbtQQneSfG
AQjEt17ErnFPKQdoFcW3Y0+YZ6dRLG0UZy+KoyGBdJ54hk4KJASnhISlY40ZAhKkZSjXeZqGmN5R
QDPJlKy/3LD3f70KNyyXecE9EtY0AAQP1tj6SxvT6hJRHeOU+7DWz0WiW5Xlij6ax7JFJJt9xZ1O
2Jal8xfBKbpBauKL8eslOamafW/4yr+jj4rLnWeePaz0XroHp88iLtrbchPx1lm8RP1JfL3McdeI
UCRtU9b1yJuDli6Z1AyI2np/lY5l3MffdbIo35WQqtpHrRVsCpJlwKIaUPHuwzXSblXNjn9B47gu
N2lbJIXnAU6EE5ZxmpcNvyBOUgapxDst7l5k/SGamQ/kPkgg9zNcMudpyX3X0AcP3JAkNEuzzac0
1s+FpKkKcAAvpiv18MLnnGAGH6XtiM5PN9864zhrqxlXe3G8wViBsU2m97vLK9P5+uP7eqUN0dBl
EufEMNAL2842nVUpLtJz2kW9mPhzKkweRXfcSO5HU7hLhfWtN8yGILEB87lFZ5a0AfPgvpxKegDl
cTrTYpqrn1wzo3w+UyLlqKSZ6qhe4vnv3lFke1dnIC63JCuiFviuFHjn2K48+4+Lnr69nI7fBvNt
k9QlceKLbJPw+LkVkHgmZMIHH3iTJGtR5dPFZpdkRQE8lOg8jvZBTdWf49ilMjGPvpRbwX5Isf/+
9+n97ab5/XwYHBoqcUvSdNDuqgQE8FAoyw1NkibfXafctouy5F779W72B+0afoEcqHS5Zenk67pB
rIpMnf0Sq+l/M+0Sb15ZEs3R6Bstbdfubt5e7h+EImS6UTHAJ0v1dLOGzm+JZdDPG019iu1PtUIh
oPQarybVReESkcmpQ/fDoB7Fdj954zS+i9A55OP5omTlh5a+cR3OzNEDcBXueKH9IhrEw+AYZC45
OHW4PL0LmbiZrIq2rAaHHMuA9QHLswI5jhdeNOnvDToaSe7nJPrGlwzwVikIlLMGu2Cc7Ke+7Xut
+NDvW29iTlKvOPhVR3hxFK/T9q6sky4KuBLyI+K6tSalacalH6bKBpSUlUVUDVJYQ2LGJUV6MeMB
mQYB5nnYjzbJauq+duJKzWq1PoIJcVp19xMJtGXwACEi3wjiTS5ZRlMszqcQS+NtPQo/LzAUafrT
IrFUYv4bElMFxUL066XmT5S4VJ7BfdI38BNoHE/HaodPsu+bjDVZrOu+/ah2/rtzVN3unEv1anrE
4iwjuW+Ift6WTTRMUj/jIr8SUIOwDQSVGxE1h8X1dgGJ7qJ6A0HU+cQ6WINv7RJa7jmdR51IcuVk
LeMxeZ/Slla80CSfHTm2cU5MwDCOz5mKj4Z+95AkovUkQ7DbvFxdpdMuiUVTj0a1T9GPyBmttxuS
pDY0w1ocTE1S4ykn8YhRbzQzBOm+Ilm8IIFyaBzbH8NZfu7/y6FmiXYCK5FNOkEvy0675PlsHm4d
MqVd8ClBR4KC8Yh2/UwZbG0kf9ZfKn7lDZLbKF+xoU98Fc3khBe/wVHe7tJ6ZGJ+xjZlky2VOFbJ
OCGTCWJ6Ki2MxnR9SrfZc7+mRcboHNwovTBa5OInD/1En3A7cCKt+E3mJGhNSrSp02GeZUG7j6mj
F4g1akHcqM66t00pp//l08RxoKyHeCvivI09q+snD8+sOIsfFUvZaE9W50RJo8W9XA/n7iW1rdPu
jKN/dK7nNZRRfhd9oY8q87y8U7tKIeYyiJ5nUYj2NB1EF10jLFLq17L6co1OH581vn/4qYbZoyly
OUAu49Al8y1xwMaLZHyI9ieoskQkszLZXPvx5ihfwYMhuaTObGcKUVrXwiyGaz5GFi5TYsVpsOKI
QfSK7KHkn3VZ/JHsEsF/TdivjJWh5xlD1qDMs1RRNH8lInWeb5NlP+v6GvW1SPOKkv2xjJo/0j3/
/6bRt4OwwVoqGOUbtGo3JuG/+ygLcZmkVbRK/3RsX4dnZbzmDGTz59/uXx+Ox78pg6qQbZtloGX+
urofhyma0t/fvgd/O8+7pj8glTuemSnX0etOEQ7ZmrLs1nMWWdPzAqBQu82/7tu9sBeYljk6rnsW
fm7cpFLt9fD+7XTzXTeePAbEYLREwq0I/TlM4wp4dWMViXwsSVQhHkGN4yigeJ3lSZ0qx+BtWm/U
qkZ6iqaoJj91XSyBfdQ0A+lUJtN+lqTIIY+gEEeEVrXGFVdyS2LteruiY2yhNggniW7QpBaJ7CAZ
GIUHdRycDKLGdcT+r7RnbW4b1/Wv5O6nPTPdncRx0uTO9AMl0bbWekWUbCdfNN7Um2a2iTNOck57
f/0FSD34AOXsvTPttAYgvgkCIAA283jOsioOrXK67M2mPjOLV6xUoMGg5s5wv0Fjod7fVTY5g7Hm
Jcvm3C9EscgnQrGZJVJyKbBYC78HwvEhhHzLjShrYRUFvwvYJabEyp0dKkEjcqe/WyM9/mOmxEyK
mZcsNYQG+VvJhhFfGS+PKVRaUY+5iJuaiYVeUgdRQqVzmploJQGMlAtjEuZp0QgY7YQuKFzk6JvZ
Ega3sIPrbLRMJJS1Hy/bpJSmJNpyMvYFipthUY81yWEARoFqGEEmS5KxQu7URasNTu6mJDQnoJs7
Angnqohs23SJnrGBfKDsjo8PDE8DHkV8bBE1s5LNUWtq+4uFfjnXJNmNbwencQaM1RJKU//OWBR+
3E22mY5iL32tKNsqh82gIAELl/gA7q3aX8aNskVg7TIvXZBXC28LQPHqKurOMFGZZ6D83YsSS3zw
KLgFvfjL2elkemqR4RJsD/HBcNehMpF7brtbEnxNiborVlhgLvqktdDA9OboRECHbSrI2KXJiODD
y9wpsIMd/ajfsTacUsY7nGbjslHD1LbPy9sEd7rPhw6FnZyyEVQ/y+gPoh6xg2m+JOmBNDGeVzGw
Nh8wkP5E9cPQ4J3eElRNzqIjlChotEwAX4+mJExerfGVZFMa6JCdnDGIrWhpoHwWJOLc/HR1bsnB
CJuav8VaNwMriubUgWjqfJF1x566PbIw9paV1AkIwtQXbelnjXQ2kZYmZXCJmihPWZx9+eXv3eF5
9/33/eHhF2sY8Ls0npfMo+m1RN1qhcoDntiD2aql9D2Tshj2Vu9MeGpBIhSkeYJEZvfkDrRAsZDC
Zx0V2ouDducmDS6wBsUuutrI0Gfgt3dlRNRSiOy1EMnJskByOolpi+TT4SSim20SKTsmrVCN0N9j
65CjsxV1UxGCkg3na1Uyaezz9no68DGtY7BkMJE5L+Ncv9tAmdH66WwXKEibMg2h8tnZ0wiFsIT0
JOwnGUbBeWsOs/oVof27mQtdQFUwZLjOY3gtztzbAIHBxUKaZRlcaKe7iUDDZ3VbcOgs/AW+dWac
9D7ikOGLyjHzhPSOV0Gt27YT3f7RnmQe4HIFUGw15MXCOhNbkLPdTTRtfu+Q5uqkSomtSuPOeDbx
UDcM7YWwJWTZHbPRFE6kqQtYR4kFtI5uCZOtt2DOCA5QT2B8j2+iGsT2Jb8VvtZHeuusbq+zFuWv
ZYz9SoJOaIs9foYiDdA7a0WabwBpaId5xKxFwXxC8HVhUUqAb/FIJCUzKYQrMGU6P4IfnXRjG8ES
0Zvnmun5Z/ObHvMZME805vOFB3N1cerFTLwYf2m+Flxdeuu5PPNivC24PPdipl6Mt9WXl17MtQdz
fe775to7otfnvv5cT331XH22+hOL/Orq4rq58nxwNvHWDyhrqOX9sLmauvLP6GonNPicBnvafkGD
L2nwZxp8TYPPPE0587TlzGrMMo+vmpKA1SYsZSEqqCxzwaA6VHFIwUHgqcucwJQ5iK9kWbdlDOcp
UdqccRpecr50wTG0Ci85XERWx5Wnb2STqrpcxmJhItDsby6kOhQTAMeJxnrx3l7/YYvFdRbjatUy
DefN+kY3pxoeMCq17+7+/YDhSvsXDBHWDOh4YOnHEf4Gafqm5qI1ylCaPC9FDAoYyJVAj1Zho4yg
LYf4skKHQtB622rVs7X7k7fD+yvQqSfLX3ffd/dvu6//NbSyvWAePtR8JDkGa3foiJHGcfiqiRZN
Dg2XKpDd5UZeFsehQnrC6Vt5Jkq5kE778vFuanTw+kB69XXXnKV5idwTSExTOJKZS5nlipg6uZ0T
E23yfXNZHelPJ3dw4yKjo201bD+m2czKlEAXrFoYgicqp6GcnBTWqhLax4dVwE7N6s0RotT3hmdP
UuVpfpuPjBM64zFolml8tZFec5BLGWNEiSjwYrvIgd3cEgPUE8PcLI7hewlnhE45d6CrLitHK8RH
qDL06k1yFpFdRkQRZ2N9xUwP5LeCzTB6JY7GJ0VqiDkIuYlIx+pB7msbTHtkXHCvn+K89CUtjFPW
tPoCFN5gIGLLg9AnlvZVbs1nzmASTXdoI6Zpo9DdL7983z5/xYSyn35un7afvu+3X18enz+9bv/a
QTmPXz89Pr/tHpA7fwLO9/j8/uPT69MWyN/2T/uf+0/bl5ft4Wl/+EVxy6U08px82x6+7mT88sDS
lTPrDmh/njw+P2I+tsf/2bZ5b/s3ByR7QJ+JBi/i4iyucPtWvNSuOkmqO17moETrVmUvne6wAyDp
MAlMLDP4oIYCzadrhWeODVKsglKzVHMEy0J871iSg1xXLRpUrQJmLqylIoAjQVB8VVYp/Z1gRfbz
bBouOpoZyBMaib9pcgyxMcgbIzhFzJOTQNPev+Qkq1et9+/PX19P7r/t7v+G5aE7MsMJIl2iRF6X
ngNnwcqIZ1A1sm7XYaZ7GtuqQ3mSDyvyrxNlijz5un3bnrzC6X6PsajW++sgYHTV0ZYQ2KbA4KqU
FeOEqH3DJODZjw7eZV14U7Z8pJHWlAH7jOCsr0ONHw8wZ1FrKPv2rm+Ffw/3DynYAlt/cy1us1C6
sQ8CoIKBmBAWtzZ0k5c2qLixISWLo0tYbbBr9AtiEKn6x9LDw8+Xt/3J/f6wO9kfTr7tvr/IfN8G
MfoGskJTlgzwxIUbHtMa0CUV8ZyCEYTLMC4WuonQQrifmGeyBnRJy2xOwYiueWtjvgYui2JCAd0S
0NLjkoIuAhps6YO7H0gfSt0C1w4UT2aOw6xJktWGiU0BC/mvvYDUP8Q019UCxHqHHI2eCm63KwDw
ImXl0t8uns3jjBsuY+S6VX5F739+f7z/7e/dz5N7SfVw2L58++ks6lIwp/XRwnSYaIFNOQs/X59d
g4KZ18XIAPLQ7TkPowXRbQAL2lTcE5RHKNLEIeg8bzxjoI4SmQb//vHlm4qIsfepcCc1q4NYOD1j
ZcwcYJDk65mhI1uI7qLAWVCt+ZxAoM7q+0hUFyT00oFG3O3CjF7cywW7Y5EDFpxTwLLAyHoHnk5d
ZpVOu/huh77ibtdBOyPHsoUPozKc4cbkqqC63fPD27ffXuAQ3B3+jedSi5ZZTp72X3fEKohillV1
Ssx5OHXHe8Hgz+TU6W8YhC6sKgmYu+h4GBCbcQGfT/wbMCnXTusKbIUN3FSGq9pHxkiFYeKV96/b
97dvmK/rfvu2+wplyE2GSdr+8/j27WT7+rq/f5QoFEL+NbLX5rE4m1wR7KFDYRaCyN/fbuRBQ73F
ZKvu1ITuHM7D1BltWJXxyl2v/IaAcqgyRnKH1QkhoUM46f9ntFrx8/Xb7vUT5rrZvb7Bf3A+Uha6
gxkkbMkngbu2gNpp6SJl7spMI3dpp9GFKxmk0Zlu0+8GS10UOsDJxSUFvjibUOBzgmmcU8c5xiME
+dy/NtYFVuEsfHI3bFphSZOofeOu5MYyfD359f7nPRwzJ4fdV1Actvi8h9QdXv/lTA7Qn08IXoBg
w/n8SLmq8v0TbtRXUwPumLzl+NIxBt1LroVdTd3hQR87ArZwx0w60egiidYqlSkKhmz/dPL8/vTn
7nDyoJ6dIdtcBnOQ87PalUcQQ65VhaEkXImhmCwiHOAfMWrnHG1dxe04FlWRvGx938SXy+nHiaX/
32SUPhdahpvuGI5mV6enZ6dXdH8QeX3K9PXjHXM5IzUw99eX7f3uBI0zh7/gf85kSMszpe10CHrU
e6xXUekplLJh72cdDQLMqvBv7J6UVLR6LM+kj00eoGNfxfVhogZCHW/397vvOGbAjcNBcoTNyL4/
7EF1/fakErKgG+yv28PT5dTd7HNp7VRJIUJOaIKiR6KD4AiBTMI7ThIIUsry1q4jyaIHgg/UjnGU
YUpMp1ENkIzJ8Qbt3KbtjtL/y8zYp96aUm/4qk3LBdrLWDMHQjzUTqfsGHEYFsdIshh2/6YJs+zi
YrM5Rn2DUTOLq+uLH8dbirTh+eZjhYaXkw/RTT9Y3nRy+s96s5p9uD8fJIUemZQunQooJGQLROIV
wMbnCWpMcsn5MSKZO0Lw48srTXJMPjbfUK5FTNymKcf7OXm5h/5TWtjKgCzqIGlpRB2YZJuL02vY
ZmV7L8idzADFMhRXTVHGK8RiGRTF5zZ2RPt+uN2TeDS14Oee7NxztMYWXLmRYuRsd01JR9fGhZAf
sQpvPDpTjhvRiM+1/CUFbcUMXh8fnlXCQM1yPAQ6YToc7e7WDARy8eLLL5oLaovnm6pk+pD67ozy
LGIgEVj10dSqaBDpw6W0KJPEneD1gU4T1ihi3UvPvDE7mbqXw9WRVbM+ac3jn4ft4efJYf/+9vis
K9PKAKsbZjvIYPXSbpXijLOykdFPpou+18czAPbJYfUIPc6LrbgK4qKwXXY5UB+yEMSuWSmTcunr
WydJeObBZrxqXRy0yLAyig0TX4HB801WpwG0ggpzkEueJVb0GXoTh2mxCRfKC7XkMyI+bYZ2oTYp
TKw3sS8DWELDsqxN9i7MK542aNmXpkXe3stELW2sOclLwzNDtwsx/1dVmaC4qhvDPBQ21dlpFM9M
MgWTjTcQ5xPrp0tyObV+uiT4ckqcxnf9SLhmcXz2wFWSkjhw5nnAUNbCFu60wUCCFooeNZ6vKSMT
Iuazs8kVTDn5lc8AgDhlJxlB+Vsb1uWKTy4uzq7JJg3o3rJHlUIZH/sekRqFgfU3jzJQIDxbkPVl
I0V5jUg68vjno+PQE5VSKbz2TsiITQuJSGNLh/C30WOOGVCjn1ImmwE1+um5FzHyWXpO9z21P9rk
ZaeWP7kwm3hjrjfpSuROmgJLXooXU0wnMa5F8xR4EiWStDRX08kQefakQ1VspQnH2EfkRKYV507t
IQua3OV6TJsG1UrW4FOiHdK4Q8PJUtDsM8Q6WECKlipkc4dgI+GQhDSbKzpndIuWmS4LKqVuS8B0
P64BVi3gCKYQ8uSwaxEg047UUcYiXDmFpXFY5mYEW4sJwj+ISjyLZRitZm5EymmIABATEmPWryH0
cFiDPvfApyS8DZyzRCHpSsKMXAOgE0WNyJPcuLTSoej8eEV/gDVqqAokbMHNPTzAmmVaDDVo8CAl
wTOhwWUY0oolXXKbbopYWbJbJVvpGpbIwxgECClfAoHGtJJcgGIiTGfFODfSQCqQGTg0wBojGxvC
I2MuU4apjtpAIT10QA6cIgD2MPegQJCdVwsLhwh0Q0G/Rm7Jj4hjUVQ2lcqRYTYFZihhMmx2we28
2q14KnhVF127KTy6aMhW9+8SHaMKi5ogQSwsz2KsMcolFSnrrPe6JegCVoWLdmSwak0bXMd5lRhX
ci2oiW4zlsYUx8BigrzG5Mc1iPlK612mcsOYA6rmStU4eOwBovPIRXU5zxNPJVmedb0HmV1fSYjt
UViCNZFKCyi5+5FcfioJze6v7fv3N3xo4e3x4X3//nrypFy1tofd9gRfM/9vzQqKYwfCTZOqSGzN
UN5h4LTssLpBTkdjqgGMBp17jCtGUXH2ASIyVSuSyInB0NMvV+bYoxXZn8ui2z1jDhxinjTWbGPs
cGmO9o02KfMkD3RvejPEKynrNhXFIBgld5jKxMxWD7KjViZmctAzKkRxavyGH7NI2zboV1viBWyl
O+FKr35cjLqRRGrb3UGwikTuHg9zXoHqxfNZpLNMpEWUkqC0vEQ5rFMnwBKhZno2JLv6QeUbalFS
MzXpL3943hGS2M8/PI91SGwBx0di12iSsDAvs3ESzO/QTH9c+imgjad+7Nnpj7OR4kWdYcdHCc4m
PyYTPwUciE4lNv7yx/mEGNwrcgNYTKdntbgPVGR8nPWpSuHAxIWny5bdMhroMSy84hnLnBNDur6u
mX5lIZSrpd7cZH3jiF/aC0qWNctuR5xbzLJDKBFdZiaO26BO/YypM4xrn8Uy25jp89wZJiX05fD4
/Pa3eoHnaff64Aa3SPPbssFNZcZfSDD62ZKxkC2eb3hYyxj7sNpY0Q3hUpl9gjpOoiYmHS9Uoowm
wScWVhjv3nr1f/ZS3NSYEK0/BTrLsVNCTyHTOrfNjTCKXGNd6qy1o4cMcGPmyRK3aYABDJgaDqi4
LtkiNfxd8TLIhTGaNg4AFYsT21e3XTXeSTOrUikC7AZgMqjeIXX/9PL4fffb2+NTa75VLjj3Cn5w
V4Mqw7RdwfbCLZbhJMNsVE2d8AAdbZwO4nfS2dcTMtDTRGtWzpoK9rL0fukm7UOfTUlFx6S58LSt
5FEdktl2NKJOQQEhyXZcpqlQnz7WdJ1ceNLZaOSiSOJqvJlBpdk8C7bAXSKlzgpGls1JXMIyTJgX
JnXkrNyu0CbNozoxhSkFamPLJU0sUpRu6afJowCz+cYFafaflbBtZD7fL5PT6VW/sUAWKGCtYW57
M4UZ5ozm+EY5rD1G53pS6Rn7VsnSMa/wrTt5SnifgRjdZmoFcQ2VEl+6uDWD40i1uciliqfnPdTh
el2rVEZqoUg3MtWqLWvOliihNnQ2LNkXfJIH1M6GgeB/K2JjfEiChnn86j/MEyQHmaPTx+N9d7pE
uz/fHx7Qoy9+xmiAp93zm8Y9Uob3jeJWlDeavDgA+/AP5U3x5fTHKUUl4ATUrzFcHPrS1fgMDN6h
mUMq7HXd50SwkgpYWJttORdAS1jWRmgM/PYtGtxrdSCYEQgjAeNVhOoTHSFhXQI1Y41JDDnHH5o1
FVe6e/vP/oASw0BlvBXBK3mXXPJZ2GVI8UQpImkmPkKpRh7ox7EYyZgkTVll9Eom267Hd5E9khHs
sFN45k0CoVqAhFLb8jwKUAdjj79hLfk6870rNaSMBBnUE64piYC3iDzzXe0OLcVxHyEpcxn547ET
DtYPSbze2HtIh/TERYXpPLRzpvs99EBCuldcfEu/lWZA1E2AE7pbtMN4nkoogdkyebM5kmldlZQH
mMjXE7u8ABFxycsxHV0WoiLiauEzJghp82nj5rLInWB60FdpU8zlknBHYDUb69UqHUV2xX6gCXFZ
1Yxgki1ipJrWQRFj/cYWqjrk8FSkO6QiwjGWsCjzGT57Nx9nmMxlmAOiEWuUhISfYHGMYGWb19qA
N4V17+Z0rFN7i8UgcXx8IMtl5n00KaFt1EolKMsY7/yMZ2bQvIKMRUPqjNJa1gt8M9D2gZH0J/n+
5fXTSbK///v9RUkLi+3zw6vJVjM4o0GgynMypkfhpemmrr6c6mmx0LDLK9iYus1f5LPqnyBlhHmD
eRI18w+qfhjbnepfyJYMLZDjiaGTdoFBHUWG/EgQkRI6oppFDTNcMbHUWakSfnpUPxxnk1O3yQOZ
1WK7BnUxacoG6xsQC0ESjjwvSgAeN6I0fQi8gyg52sg8DwiBzqc67nlgZmyNqNwaIGh+fUfpUj+T
DYZp6pwSNmSJ7yI+iWLMgwp0LJ4WlcvCWJVbVnXlboQBPoN89Ovry+MzBv1AV57e33Y/0Gt/93b/
+++/a5658qyRxWK6YyrnHr772r8dQb9U3Z1XY4c2euSAVM/HuL8o4gwLG2PORwtZrxURiDX5GhNV
jLVqLXg6Vpjsml9y0ucD6uO8OFIWjrF0Bqd8ycwBhS0pffpcg1y37vuOElfs2hE+O15UiPkDsdI1
iytKauhsgP9giTm2gPJGXsNROm9vYCMkJiXoeLj58bjzduxjMXqYH8GLMSFK5ZKx/Ngsmu4edGxl
M3wzrHKcZg3O5Athl2ElYU2xJH3ydRc8IIfFwRLuk6SRYOxjHQeS4bIpo1LdmNLh/fYnST63zQTj
xPPCSwsaw/HO4CkhzS79YXV5quP5piBoTs2aBgL4H9L42+9N8yMruxEjG82cS4fF3bTmlrL0jiC2
cwHnb6LE/Ip3TyHTHAcIsvC2yscYWCijnG3tTE/MpFnK3VSgeaGGpLRE1N5+NY6dl6xYfIhm1qbZ
MEhUq1qCWr/AVopbayefdWxIRypP5hj4tsRKG5kYLaBZxxU61DntwGcp5OryFhTKxAxllMrn9iyk
vH/AVdcoNAiD+hirukMzzzDeS7Zv1mlpn1ddlljj1rIzvQlof+iOtkPfAoh7SWsc5c+ZSh0j0/dj
ohboKFPuBkNuD5AZZf/akun7OHPFkM8e6RTdRJg3OsTXgxdweQNi9myMpBXPxkiUNDhCsFjDxhsj
yEWWx4KPkeBolUeKaSO+OqldUdKHUbt+23Uq/C7QsNsz0H0XOcVug5JlsH6U5sudu4cO3vpgY3Im
+YHvIemOHDbOKCE+VIAmvjj3pr+FsgKulr+6XzA36VLHa3eUNJjT4KCYObCOP9hwq4ShK1BG2xh8
r6qMI4+J5jYDTqO+97wqDFM1gl9gGEVVxvO5b0moFaE4xsjL2QNLoz0/huNvYEhHKLuaWSIdSnAR
HC0R9naJDlZeYVcn7u/XJc+JeAJK6Nhuj6XhHx8P8xavryC0qow0ZDXS6ZKneTUisguGeeG9hkiF
7nazcmayjyKCxHbtUy5W7e29ESaRmzhHPpeBhh6zdYwqcnfWxNGYZZ3PYt9L3pdTGCS0r/nVmu6p
b1LGsluoux5Uu9c31HBQ9w/3/94dtg87vQvL2mcd7LSBD7y31h9SFqnhQaJeAhsppTd8LjGdk3Gn
kq88ibYydcTCuOFewAhzX8EYLQPTK+X/n4Zx0+soMjaESlHB5J+Dc4eZAKxVqeUDnHZyTp0OmeIM
L7ukTdpgnSqRmPTUUOWoazFa0Dba4iTsMoPEutcSm7uw1G4RejDwbBraenr+L8RoaEWqtgMA

@joshuaswarren

Copy link
Copy Markdown
Author

[Joshua's agent] Chris, the changes you requested at efff25c15c4f1abb1fb141d87ecfe07432bdb3c4 are on the current head, c3487a528f74dfc036ffba7d3a4e5fa09cf2b52b. The earlier response maps the retry rearm to cases 24 and 25; c3487a adds the exact half-range boundary assertion.

I re-ran the current-head host suite with python3 tools/asahi/pmp-agx-vote-tests.py --mutations: the suite passed and all 10 mutation controls were caught. I also ran a separate local policy probe for a failed SET with the last operation ending during the write, followed by idle release/resume and a new begin 500 ms or 1 ms before the retry deadline, including a wrapping clock. The reviewed efff25c1 header failed the idle retry scheduling assertion; c3487a passed all four probe variants. This extra probe is not part of the in-tree suite, and it models the policy rather than exercising a kernel workqueue or hardware.

The current PR CI checks are green. No new Apple hardware run was performed for these checks. Could you re-review the current head when you have a chance?

@joshuaswarren

Copy link
Copy Markdown
Author

Fixed at 3d114c7b63b1c5decaf0f506acf53f4f9afa4b9b (pushed to w72/t600x-agx-vote), with the regression case in the tree.

Your repro, walked against the pushed header. I cannot reproduce the stuck state at the new head. With end_in_write inside a failed SET ack (your steps 1-3), next() at 5001 arms the retry from the failure state (999 ms, from 85a68bf's branch); a begin at 5999 returns 1, the reconcile fires, and the vote lands inside the new operation. Two drivers: the full sim sequence, and the header-only walk with the real glue contract (begin1=1, decide=SET, end1=0, failures=1, next@5001=999, begin2=1 @5999, decide@6000=SET, voted=1, users=1).

Your conclusion holds, and the fix takes it. The begin-during-backoff delay return was the only way begin could move a deadline, and it is only safe for a queueing primitive that can pull deadlines earlier. Nothing in the glue guarantees that (mod_delayed_work on a coalescing workqueue is the strongest statement, and the timer class under it is weaker), so the delay return is removed entirely: agx_vote_begin() now returns the two-value contract only (1 when the deadline has passed, 0 otherwise). agx_vote_next() remains the only arming source for the retry, and it is monotone: the failure state is only extended by applied(), which runs inside the reconcile the timer drives.

Red-first regression, in the tree. Case 26 (with the pre-fix header checked out at the parent commit, the case fails the same way your harness does: begin2 returns 0 and the vote strands). It walks the full end-during-failed-SET / begin-before-deadline sequence, pins begin at the deadline edge and 1 ms before it, and case 22 gains the monotone-deadline assert you asked for (second begin cannot push the armed deadline later). The idle release/resume cancellation control stays (case 24's tail). The begin-no-retry-arm control is re-pointed at the real return, and the new begin-drops-users mutant covers the users increment under the stopped check. The tree runner registers the new mutant entry: --group tools passes all 13 vote entries.

The half-range boundary fix (c3487a52, unsigned distance in agx_vote_retry_reached) already carries the exact-boundary assertion in case 21 (retry_at - now == 0x80000000 reports reached, next returns 1, not a 0x80000000 delay).

State of the evidence: host fault injection and the header walk only, on the same terms as your re-review - no Apple hardware was rebooted or benchmarked at this head by me either. Full build and QEMU CI are on this push (same tree as c3487a52 plus the header/test deltas).

chriskearney's re-review at c3487a5 reproduced a stranded retry: with the
timer already armed by the write path, a begin before the deadline returned
the remaining delay, and a caller that queue_works the delay pushes the
deadline out. mod_delayed_work only pulls a deadline earlier, so the pushed
timer would fire after retry_at and reconcile a decide that returns NONE.

begin() now returns the two-value contract only: 1 when the deadline has
passed, 0 otherwise. The retry arming belongs to next() alone, which already
arms from the failure state (85a68bf) and is monotone under the same
queueing rule. A begin during the window cannot strand the vote and cannot
delay it.

Case 26 walks the repro end to end (end during a failed SET ack, begin
before the deadline, vote lands inside the new operation) and pins begin at
the deadline edge and 1 ms before it. Case 22 gains the monotone-deadline
assert. The begin-no-retry-arm control now reads the real return, and the
new begin-drops-users mutant (drop the users++ behind the stopped check)
dies at the base case. The tree runner registers the mutant entry.

Proof: plain suite 26 cases selfcheck ok; --mutations catches all ten
mutants; host-tests.py runs the vote entries green (base, --mutations and
the new mutant entry). The whole tools group was not run here: it exceeds
the 20 minute budget on this host and is CI's job.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

Copy link
Copy Markdown
Author

Correction to the head: the same fix is now bfb41862cd7194d43033d2bdc1c53ab695a003ed (the tree is unchanged; the commit message no longer claims the whole tools group was run here, since that group exceeds 20 minutes on this host and the vote entries were run directly through host-tests.py instead: base, --mutations and the new mutant entry, all pass).

The device trees job on bfb4186 died in "Install dtschema and yamllint": apt-get update stalled on the azure mirror for about 45 minutes and the job was cancelled at its timeout, before any dts check ran. That is a runner stall, not a code failure, and rerunning the failed job needs admin rights on omacom, so this empty commit retriggers the run. No code change.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

Copy link
Copy Markdown
Author

CI state at b36d0192f408 (the head differs from the reviewed tree only by a signed-off empty commit): the first rerun attempt at ed4a653c failed the ci / commits gate because the retrigger commit itself had no Signed-off-by; amended with the trailer and the full run is green now - device trees, commits, result and every other job pass. Root cause of the original failure, for the record: the device trees job died inside "Install dtschema and yamllint" while apt-get update stalled on the azure archive mirror for about 45 minutes, and the job was cancelled at its timeout before any dts check ran. Not a Discussion 1044 row (those are host-test flakes); runner-side only. The code under review is still exactly what was reviewed at bfb41862.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants