Compression Algorithm Improvements

docs/development/player.md

@@ -89,10 +89,23 @@ from that pitch. It saves a byte a tick directly, but the reason it matters is t
 cannot be transposed and an index can: the same figure played at several pitches is several
 copies in timer space and one entry plus a shift in index space.
 
+**A plane repeats from its own byte.** Four of the planes write a byte the APU only
+partly reads — two bits a pulse control byte wants set, the top nibble of a noise control
+byte, three bits above a noise period — and those spare bits carry the ticks the value
+repeats for. A rest costs one byte however long it lasts, and the planes that rest longest
+are exactly the ones with room to say so. The division follows from the register, so the
+block states none of it and the driver knows it by plane.
+
+**The triangle names its silence in the pitch it plays.** The channel sounds at one level,
+so the index its value plane carries is enough to say whether it sounds at all: an index
+above every pitch the table holds silences the linear counter. That spares a whole plane,
+in the block and on the tick path both.
+
 **Tokens.** A plane is written as holds, literals and phrase plays — the encoding is in
 [the format document](../formats/nsf.md#b4-the-token-streams). What matters here is that a
 token's count is a duration rather than a length, so one dictionary entry serves a figure
-however long it is held and whatever pitch it is played at.
+however long it is held and whatever pitch it is played at, and that a phrase may state
+the count its tokens play it at most often so those tokens carry none.
 
 **The cheapest reading, not a greedy one.** A plane is parsed as a shortest path: every way
 of covering a tick is an edge priced in the bytes its token takes, and the cheapest path
@@ -128,7 +141,8 @@ seeded into page zero, which no song occupies, and the advance passes it by, so
 the zero every plane starts from. A bend plane is stepped only on a tick its channel's new
 value flags, since it holds a value for those ticks alone. A plane's state carries where its next token
 lies, where in a phrase body it stands, how much of that body is left, how much of the
-current token is left, the value it last played, and the shift it is playing at. The three
+current token is left, the symbol it last played, the ticks that symbol still covers, the
+bits its own byte counts in, and the shift it is playing at. The three
 kinds of token fold into that one shape — a hold is a phrase of no bytes, a literal is a
 phrase whose bytes lie inline behind its opcode — so playing a tick is the same handful of
 instructions whichever token is standing.
@@ -138,11 +152,14 @@ is a base offset held in `X`, the way a channel's register base is, so every pla
 is one routine called again. The state lives in zero page, well inside what the driver
 leaves free, and the linker configuration keeps the two-segment memory model an NSF loads.
 
-**The one sum the driver performs is the bend.** A tone channel's value plane resolves to a
-divider through the timer table, and on a flagged tick its bend plane states the steps the
-tick stands away from it — sign-extended and added across both halves of the timer, with the
-high half reaching the register only where it changed. An unflagged tick adds nothing. Everything that keeps the sum in range is
-settled in Python, so what crosses into assembly stays a byte moved and a carry followed.
+**The driver's arithmetic is the bend and the repeat.** A tone channel's value plane
+resolves to a divider through the timer table, and on a flagged tick its bend plane states
+the steps the tick stands away from it — sign-extended and added across both halves of the
+timer, with the high half reaching the register only where it changed. An unflagged tick
+adds nothing. The other is a subtraction: a tick a symbol still covers steps the plane's
+count down and returns, which makes the common tick cheaper than reaching a new symbol.
+Everything that keeps either in range is settled in Python, so what crosses into assembly
+stays a byte moved and a carry followed.
 
 ## How it is verified
 

docs/formats/nsf.md

@@ -49,9 +49,9 @@ counted from the block's first byte, so the whole block plays from wherever the
 
 ```
 +0      header
-+55     timer table
++51     timer table
         phrase table:  count, then one offset per phrase
-        phrase bodies: each a length byte, then its values
+        phrase bodies: each a length byte, a count byte, then its values
         one token stream per plane, in plane order
 ```
 
@@ -66,7 +66,7 @@ counted from the block's first byte, so the whole block plays from wherever the
 | +7 | 2 | where the timer table begins |
 | +9 | 2 | where the phrase table begins |
 | +11 | `PLANE_COUNT`×2 | where each plane's stream begins, or `$FFFF` for an absent plane |
-| +33 | `PLANE_COUNT`×2 | where each plane's stream is re-entered once the song comes round, or `$FFFF` for an absent plane |
+| +31 | `PLANE_COUNT`×2 | where each plane's stream is re-entered once the song comes round, or `$FFFF` for an absent plane |
 
 All fields are little-endian, and the header runs to `SONG_HEADER_SIZE` bytes.
 
@@ -94,13 +94,17 @@ reconstruction's own generators render from.
 ```
 count               1 byte, how many phrases the table holds
 offsets             one uint16 per phrase, in id order
-bodies              each phrase: a length byte, then its values
+bodies              each phrase: a length byte, a count byte, then its values
 ```
 
 A **phrase** is a run of values a plane plays, stored at the pitch it was found at. Its position in the
 table is its **id**. The ids that fit inside a token's opcode are the cheap ones, so the phrases a song
 relies on most are listed first.
 
+**A phrase states the count its tokens play it at most often.** The count byte holds that count less one,
+the way a token states one. A token playing the phrase at that count names the phrase alone and spends a
+byte fewer.
+
 A song's phrases come from two places: the samples it plays, each offering the planes it writes, and a
 search over whatever those leave uncovered. Both are weighed the same way. A phrase keeps its entry when it
 saves the streams more bytes than the entry costs.
@@ -111,27 +115,36 @@ Each plane is a byte sequence of tokens. The opcode's top two bits name the kind
 its operand:
 
 ```
-00cccccc                 hold the value the plane reached, for c+1 ticks
-01nnnnnn b0..bn          the n+1 bytes that follow, one per tick
-10pppppp cccccccc        phrase p, for c+1 ticks
-11pppppp cccccccc tt     phrase p, for c+1 ticks, every value plus tt
+00cccccc                 hold the value the plane reached, for c+1 symbols
+01nnnnnn b0..bn          the n+1 symbols that follow, one each
+10dppppp cccccccc        phrase p, for c+1 symbols
+11dppppp cccccccc tt     phrase p, for c+1 symbols, every value plus tt
 ```
 
-`p == $3F` is an escape: the phrase's id is the byte that follows. That reaches every id in the table
+`p == $1F` is an escape: the phrase's id is the byte that follows. That reaches every id in the table
 while the low ids stay a byte cheaper. The shift `tt` is **added within the byte** and wraps. That is one
 addition on the 6502, and the encoder does the same addition.
 
+**The `d` bit says the count is the phrase's own.** A token carrying it names the phrase and no count, and
+plays the count the table states for that phrase. The count byte `cccccccc` is then absent. That bit is
+what leaves five bits for an id, so a phrase opcode names ids up to `$1F` outright where a hold or a
+literal counts to `$3F`.
+
 **A token's count is a duration, and it may run past the phrase.** Past its last value the plane holds
 that value, so a note whose envelope has finished keeps sounding. A count shorter than the body cuts the
 note off. One entry therefore serves every length a figure is played at and, with the shift, every pitch.
 
+**A token counts symbols.** On a plane whose byte carries a repeat count (§C) one symbol
+covers as many ticks as it states, so a token's reach in ticks is the ticks its symbols
+carry between them. On every other plane a symbol is a tick.
+
 ### B.5 Where a song comes round
 
 A song that repeats re-enters its streams partway through. The tick it returns to therefore begins a
-token on every plane, and that token names its values outright instead of relying on the value the plane
-had reached. A bend plane is re-entered at the value its channel's flags have reached by that tick
-(section C), which begins a token of its own. Coming round then means pointing each plane at the byte the
-header names and clearing what it was playing.
+symbol, and a token, on every plane, and that token names its values outright instead of relying on the
+value the plane had reached. A bend plane is re-entered at the value its channel's flags have reached by
+that tick (section C), which begins a token of its own. Coming round then means pointing each plane at the
+byte the header names and clearing what it was playing.
 
 The tick a song returns to is the export's choice: its first tick, the first tick of an order frame, or
 none. With none, the header has `$FFFF` and the song stops at its end.
@@ -142,17 +155,16 @@ The planes are written in this order, and each group belongs to one channel:
 
 | Plane | Carries | Reaches |
 |---|---|---|
-| pulse 1 control | duty cycle and volume | `$4000` |
+| pulse 1 control | duty cycle and volume, under a repeat count | `$4000` |
 | pulse 1 value | pitch index, and a flag for a bent tick | `$4002`, `$4003` |
 | pulse 1 bend | divider offset of each flagged tick | `$4002`, `$4003` |
-| pulse 2 control | duty cycle and volume | `$4004` |
+| pulse 2 control | duty cycle and volume, under a repeat count | `$4004` |
 | pulse 2 value | pitch index, and a flag for a bent tick | `$4006`, `$4007` |
 | pulse 2 bend | divider offset of each flagged tick | `$4006`, `$4007` |
-| triangle control | linear counter | `$4008` |
-| triangle value | pitch index, and a flag for a bent tick | `$400A`, `$400B` |
+| triangle value | pitch index, a flag for a bent tick, and the index that rests | `$4008`, `$400A`, `$400B` |
 | triangle bend | divider offset of each flagged tick | `$400A`, `$400B` |
-| noise control | volume | `$400C` |
-| noise value | period and mode | `$400E` |
+| noise control | volume, under a repeat count | `$400C` |
+| noise value | period and mode, around a repeat count | `$400E` |
 
 Splitting a channel's registers apart gives each plane something to repeat: a volume envelope and a pitch
 line are separate series that turn over at their own rates.
@@ -161,9 +173,25 @@ line are separate series that turn over at their own rates.
 the first tick to the last, which a hold covers in a few bytes. The driver therefore reads the same layout
 whichever channels a song sounds.
 
-**A plane playing zero throughout is absent.** Every plane starts at zero, so a plane that never leaves it
-needs no stream. Both its header entries are `ABSENT_STREAM` (`$FFFF`), and the driver leaves it at zero
-on every tick. A tone channel that never bends spends nothing on its bend plane.
+**A plane counts its repeats in the bits its register ignores.** A pulse control byte holds a duty cycle
+and a volume around two bits the hardware wants set. A noise control byte holds a volume under the same
+two. A noise period byte holds a mode bit above three the register reads nothing from. Those spare bits
+carry the ticks the value repeats for, so a run costs one byte however long it lasts. The register byte is
+the symbol masked and ored with the bits the hardware fixes, and counting one repeat off subtracts
+`COUNT_STEP`. Every other plane spends its whole byte and reads a symbol a tick. The division follows from
+the plane, so the block states none of it.
+
+**A plane standing at the value it is seeded to throughout is absent.** The driver seeds every plane
+before its first token: to the bits its register fixes where it has any, and to the index that rests on
+the triangle's value plane. A plane that never leaves that value takes no stream, and both its header
+entries are `ABSENT_STREAM` (`$FFFF`). A tone channel that never bends spends nothing on its bend plane, a
+pulse or noise channel that never sounds spends nothing on its control plane, and a song without a
+triangle spends nothing on its value plane.
+
+**The triangle names its silence in the pitch it plays.** The channel sounds at one level, so a tick
+states whether it sounds through the index its value plane names. The index standing above every pitch the
+table covers silences the linear counter, and the divider stays where the channel last sounded. That is a
+whole plane the block does without.
 
 **The noise channel has no bend plane.** It selects one of sixteen fixed periods, so there is no finer
 grid for a bend to reach, and its block leaves the plane out.
@@ -192,11 +220,12 @@ divider, and a divider halfway between two pitches goes to the higher one. Those
 the widest gap between neighboring pitches, 57 at the default tuning, so every divider the register holds
 reaches the planes.
 
-The driver sign-extends that byte and adds it across both halves of the timer. That is the only arithmetic
-it performs on a song's behalf. Python does everything that makes the sum land in range: `bent_timer`
-keeps the divider within `[MIN_TIMER, MAX_TIMER]`, the rule the generators render a bent frame by. The
-timer's high half therefore never exceeds three bits, never reaches the length-counter field beside them,
-and never collides with the `$FF` the driver marks an unwritten shadow by.
+The driver sign-extends that byte and adds it across both halves of the timer. That addition, and the
+repeat a symbol counts down, are the whole of the arithmetic it performs on a song's behalf. Python does
+everything that makes the sum land in range: `bent_timer` keeps the divider within
+`[MIN_TIMER, MAX_TIMER]`, the rule the generators render a bent frame by. The timer's high half therefore
+never exceeds three bits, never reaches the length-counter field beside them, and never collides with the
+`$FF` the driver marks an unwritten shadow by.
 
 ## D. Limits
 
@@ -205,11 +234,13 @@ and never collides with the `$FF` the driver marks an unwritten shadow by.
 | Program area | 32 KB from `$8000`, less the driver |
 | Song length | as many ticks as the streams fit in |
 | Offsets within the block | `uint16` |
-| Ticks one hold covers | 1–64 (`MAX_HOLD_TICKS`) |
-| Bytes one literal covers | 1–64 (`MAX_LITERAL_BYTES`) |
-| Ticks one phrase token covers | 1–256 (`MAX_PHRASE_TICKS`) |
+| Symbols one hold covers | 1–64 (`MAX_HOLD_TICKS`) |
+| Symbols one literal covers | 1–64 (`MAX_LITERAL_BYTES`) |
+| Symbols one phrase token covers | 1–256 (`MAX_PHRASE_TICKS`) |
+| Ticks one symbol covers | 1 to the plane's own count, 16 at the widest |
 | Values one phrase holds | up to 255 (`MAX_PHRASE_LENGTH`) |
 | Phrases one dictionary holds | up to 255 (`MAX_PHRASE_IDS`) |
+| Phrase ids a token's opcode names | up to 31 (`CHEAP_PHRASE_IDS`) |
 
 A song that reaches past the space behind the driver, or past what an offset field can hold, raises
 `SongTooLargeError` naming the part that overflowed. A project that offers more phrases than a dictionary

src/sampletones_core/constants/enums.py

@@ -91,14 +91,15 @@ class CQTWindow(StrEnum):
 
 ALL_CHANNELS: Final[FrozenSet[ChannelName]] = frozenset(ChannelName.items())
 
-TONE_CHANNELS: Final[FrozenSet[ChannelName]] = frozenset(
+PULSE_CHANNELS: Final[FrozenSet[ChannelName]] = frozenset(
     {
         ChannelName.PULSE1,
         ChannelName.PULSE2,
-        ChannelName.TRIANGLE,
     }
 )
 
+TONE_CHANNELS: Final[FrozenSet[ChannelName]] = PULSE_CHANNELS | {ChannelName.TRIANGLE}
+
 
 CHANNEL_ABBREVIATIONS: Final[Dict[ChannelName, Literal["P", "p", "T", "N"]]] = {
     ChannelName.PULSE1: "P",

src/sampletones_player/builder.py

@@ -129,7 +129,9 @@ def instructions_from_instruments(
     return instructions
 
 
-def loop_tick_from_instruments(instruments: Sequence[InstrumentExport]) -> Optional[int]:
+def loop_tick_from_instruments(
+    instruments: Sequence[InstrumentExport],
+) -> Optional[int]:
     """The tick a request's song returns to once it ends.
 
     A song repeats from its first tick where every slice it carries repeats, and ends at its

src/sampletones_player/clock/schedule.py

@@ -130,7 +130,10 @@ def fixed_point_step(self) -> FixedPointStep:
 
         Every answer the schedule gives counts in this step, so it is computed once and held.
         """
-        whole, fraction = divmod(round(self.ticks_per_play_call * FIXED_POINT_SCALE), FIXED_POINT_SCALE)
+        whole, fraction = divmod(
+            round(self.ticks_per_play_call * FIXED_POINT_SCALE),
+            FIXED_POINT_SCALE,
+        )
         return FixedPointStep(whole=whole, fraction=fraction)
 
     def maximum_drift(self, play_calls: int) -> int:

src/sampletones_player/compression/admit.py

@@ -1,13 +1,25 @@
 from typing import Final, List, NamedTuple, Sequence, Set, Tuple
 
-from sampletones_player.compression.dictionary.phrase import Phrase, phrase_entry_size
-from sampletones_player.compression.dictionary.table import PhraseTable, phrase_table
-from sampletones_player.compression.matches.cache import MIN_PHRASE_TICKS, MatchCache
+from sampletones_player.compression.dictionary.phrase import (
+    Phrase,
+    phrase_entry_size,
+)
+from sampletones_player.compression.dictionary.table import (
+    PhraseTable,
+    phrase_table,
+)
+from sampletones_player.compression.matches.cache import (
+    MIN_PHRASE_TICKS,
+    MatchCache,
+)
 from sampletones_player.compression.matches.shift import NO_SHIFT, asked_shift
 from sampletones_player.compression.options import CodecOptions
 from sampletones_player.compression.parse.result import Parse
 from sampletones_player.compression.tokens.sizes import phrase_size
-from sampletones_player.specification.compression import MAX_PHRASE_IDS, PHRASE_ID_ESCAPE
+from sampletones_player.specification.compression import (
+    MAX_PHRASE_IDS,
+    PHRASE_ID_ESCAPE,
+)
 from sampletones_shared.logger import logger
 
 CROWDED_PHRASE_ID: Final[int] = PHRASE_ID_ESCAPE
@@ -95,8 +107,8 @@ def _payment(
     if played == 0:
         return 0
 
-    stated = phrase_size(CROWDED_PHRASE_ID, UNSHIFTED_TOKEN_TRANSPOSE)
-    shifted = phrase_size(CROWDED_PHRASE_ID, SHIFTED_TOKEN_TRANSPOSE)
+    stated = phrase_size(CROWDED_PHRASE_ID, UNSHIFTED_TOKEN_TRANSPOSE, default=False)
+    shifted = phrase_size(CROWDED_PHRASE_ID, SHIFTED_TOKEN_TRANSPOSE, default=False)
     spent = stated + shifted * (played - 1) + phrase_entry_size(phrase.length)
     return paid - spent