Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,9 @@

- **Your own fonts: a stricter background check.** What the check found of a font is now kept on the
Kindle itself (`/var/local/grimoire`, a few hundred bytes a font) and tied to the font file by its
SHA-256. The check also times each glyph as Grimoire will draw it: a glyph that would take too long is
SHA-256. Grimoire keeps that folder to itself: anything else found at its name is moved aside untouched
(`grimoire.not-own-...` beside it), the folder is made afresh, and the fonts are checked once more. The
check also times each glyph as Grimoire will draw it: a glyph that would take too long is
drawn in Terminess, and a font whose letters would is "Took too long to read.". Each font is checked
once more after the update, and Grimoire reads in Terminess until the font in use has passed again.
Deleting `grimoire/fonts/.cache` still has every font checked again.
Expand Down
6 changes: 2 additions & 4 deletions docs/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -825,7 +825,8 @@ SHA-256, and the glyph limit; docs/TESTING.md has the device checks, none of whi
entry above: the font would then load before its family's Bold file has passed in every family, not only
in those whose Bold file sorts later.
- **Hashing a font while it is read** (one pass, not two), if the load time on the Scribe asks for it.
- **Removing `/var/local/grimoire` when the package is removed** (today: by hand, KINDLE.md).
- **Removing `/var/local/grimoire` when the package is removed**, with any `grimoire.not-own-*` entry moved aside
beside it (today: by hand, KINDLE.md).
- **A font with a slow glyph ahead of its broken ones** ends as "Took too long to read." (after its second
check) where the broken glyphs alone would have made it "Unreadable font.": the check ends at the first
count that is too many, and says what it knows then.
Expand All @@ -834,9 +835,6 @@ SHA-256, and the glyph limit; docs/TESTING.md has the device checks, none of whi
stand before it, and a second folder for one data folder that was named by two paths. An empty folder
with no `root` file cannot be told from one another data root's job is making at that moment, so it is
left. What holds in the end is the free-room test before every check.
- **The parent of Grimoire's own folder is not tested** (`/var/local` on the Kindle): the code holds the
folder itself and the two below it to "root's, and no one else can write it", and the parent is the
device check's (TESTING.md).

From the whole-release review of 2.1.0-beta.2 (the fix round, lane F3; the review's finding IDs). Fixed there: H8,
M17, M19, M20, the watchdog timeout that was a verdict for good (a second timeout is the verdict now), the
Expand Down
20 changes: 17 additions & 3 deletions docs/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -2478,9 +2478,23 @@ its bytes.** Four parts keep it (`fontcheck.rs`, `userfonts.rs`, `font.rs`):
`userfonts.rs`; so from the verdicts numbered 2 on): on the Kindle a folder of Grimoire's own outside the data root, `/var/local/grimoire`,
with a folder for each data root; in a build that names no such folder (`GRIMOIRE_STATE` unset), the
cache, as before. The own folder is walked name by name, by descriptor, from its parent: each name is
made by the job with mode 0700 when missing, opened without following a link, and must be a folder of the
app's user that no one else can write; verdicts are read and written from that descriptor. When it cannot
be used, its file system has too little room, or it takes no write (the job writes and removes a small
made by the job with mode 0700 when missing (the own folder itself by the first check of a run alone),
opened without following a link, and must be a folder of the
app's user that no one else can write; verdicts are read and written from that descriptor. Grimoire relies
on the parent (`/var/local` on the Kindle) only to hold the own folder's name: it tests the folder it opens
at every open, and never trusts the parent's owner or mode. When the entry at the own folder's name is not
Grimoire's own, the first check of a run moves it aside within the parent, to
`grimoire.not-own-<seconds>-<pid>`, and makes the folder afresh, at most once a run: the UI, which forks a
process for each check, gives the first check of a run alone leave to make the own folder or make it
afresh, from its own memory, never from a file. The entry is renamed by its name, from the parent's
descriptor, as it stands, by a rename that never replaces an entry, and is never followed, opened, read,
listed or deleted; its kind, owner and mode, as looked at just before the rename, are logged (an entry
another writer of the parent puts at the name in between is the one moved, and is left as it is too).
Every font is then checked once more, as after
`rm -r /var/local/grimoire`. The folders below the own folder are its own, which no one else can write, and
are never moved. When the own folder cannot be used (the move failed, say, the fresh folder is not
Grimoire's own either, or, after the first check of a run, it is gone or not Grimoire's own), its file
system has too little room, or it takes no write (the job writes and removes a small
file there before a font's check), the job ends with status 6 before any worker starts: nothing is
marked, and the session checks no more fonts (the ones that have their verdicts go on loading). The previews and the marks stay in the cache, and a verdict counts only with its companion
there (a pass with its preview, a refusal with `<key>.no`), so deleting `fonts/.cache` has every font
Expand Down
15 changes: 13 additions & 2 deletions docs/KINDLE.md
Original file line number Diff line number Diff line change
Expand Up @@ -645,7 +645,12 @@ What the check found of a font, its verdict, is kept on the Kindle itself and no
verdicts numbered 2 on, FORMAT.md; 2.1.0 kept its own in `grimoire/fonts/.cache/`):
`/var/local/grimoire/<16 hex digits>/fonts/<key>.font`, a few hundred bytes a font (at most 200 files of 8 KiB
for a data folder), in folders Grimoire makes for root alone (mode 0700; a folder that is already there is used
only while it is root's and no one else can write it). The 16 digits name the data folder (a second one, a
only while it is root's and no one else can write it). When the entry at `/var/local/grimoire` is not Grimoire's
own (a link, a file, or a folder of another user or that others can write), the first check after Grimoire starts
moves it aside as it is, never opened, to `/var/local/grimoire.not-own-<seconds>-<pid>`, and makes the folder
afresh: every font is then checked once more, as after `rm -r /var/local/grimoire`. That is done once a run: a
later check of the same run that finds such an entry there, or nothing, keeps nothing, and no font is checked
until Grimoire starts again. The 16 digits name the data folder (a second one, a
development copy, has its own, and the file `root` beside `fonts` says which folder it is of); what was kept for
a data folder that is gone is removed at the next check. The previews and marks stay in `grimoire/fonts/.cache/`,
and a verdict counts only with its preview (or, for a font the check refused, its `.no` file) there. So after the
Expand All @@ -654,7 +659,7 @@ after a factory reset, or with the storage moved to another Kindle, once more th
worker died (its `.bad` file in `grimoire/fonts/.cache/`) is not checked again by this: it stays out until
its file changes or the cache is deleted. `/var/local/grimoire` is
left when Grimoire is removed; `rm -r /var/local/grimoire` over SSH, with Grimoire closed, removes it, at the
price of one more check of every font. A `.font` file in `grimoire/fonts/.cache/` is 2.1.0's: it is not read,
price of one more check of every font. An entry moved aside is left too: Grimoire never opens or removes it. A `.font` file in `grimoire/fonts/.cache/` is 2.1.0's: it is not read,
and 2.1.0 finds it again if you go back.

What grimoire.log says (never a file's name or a family's: counts, sizes, timings, reasons):
Expand Down Expand Up @@ -685,6 +690,12 @@ What grimoire.log says (never a file's name or a family's: counts, sizes, timing
system that went read-only). No font is read, none is marked; the rows of unchecked fonts say "Unreadable font." for this session, and fonts that have their
verdicts go on working. `fonts: N data folder(s) are gone: what was kept for them is removed from Grimoire's
own folder`.
- `fonts: the entry at the name of Grimoire's own folder was not its own (<kind>, owner <uid>, mode <mode>): it
is moved aside as it was, and the folder is made afresh; every font is checked once more`: at most once a
run, by its first check, which then goes on in the fresh folder. When the move cannot be made, `... is not its own (...) and
cannot be moved aside (<why>)`, or when another entry stands at the name again at once, `fonts: the folder made
afresh is not Grimoire's own either (<why>); nothing more is moved aside`; either is followed by `what the
check finds cannot be kept (it is not Grimoire's own)` above.
- `fonts: the check ended badly; N file(s) marked unreadable`: the background job itself died on a file.
`fonts: what the check found could not be written. Is the storage, or Grimoire's own folder, full?`.
- `font: custom font loaded: S KB in T ms, bold by its weight axis|a Bold file|overprint, K glyph(s) from
Expand Down
62 changes: 43 additions & 19 deletions docs/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2913,36 +2913,46 @@ What the background check found of a font (its verdict) is kept on the Kindle it
will draw it. Automated: the tests of `rust/src/sha256.rs`, `rust/tests/test_sec_font_verdict.rs`, the
functions of `rust/tests/test_fontcheck.rs` about the glyph limit and the second try's mark, the unit tests
of `fontcheck.rs` and `userfonts.rs`, and (Linux, root, CI's `sandbox` job) the font test of
`rust/tests/test_sandbox_linux.rs`.
`rust/tests/test_sandbox_linux.rs` and its test of a folder of another user at the own folder's name.

**Not run on a Scribe, and it must be before a release.** This change can leave a Kindle with no custom font
(if Grimoire's own folder cannot be used there), can refuse a font that works today or hand some of its
glyphs to Terminess (the time limit), and changes what the sandboxed worker does for every glyph. Its
values that are placeholders, and wait for the answers below: the glyph limit (`s1::FONT_GLYPH_CPU_MS` in
`sandbox.rs`, 20 ms) and the size a draw is timed at (`FONT_TIME_PX` in `fontcheck.rs`, 160 px), which
steps 6 to 8 answer; the own folder's place (`STATE_BASE` in `userfonts.rs`, `/var/local/grimoire`), which
steps 1, 5 and 11 answer; and the two floors of free room beside it (`STATE_FREE_FLOOR`, 4 MB, and
`STATE_FILES_FLOOR`, 512 files), which step 1 answers.
**Step 1 is answered on a Scribe (below); the rest is not run on a Scribe yet, and it must be before a
release.** This change can leave a Kindle with no custom font (if Grimoire's own folder cannot be used there),
can refuse a font that works today or hand some of its glyphs to Terminess (the time limit), and changes what
the sandboxed worker does for every glyph. Its values that are placeholders, and wait for the answers below:
the glyph limit (`s1::FONT_GLYPH_CPU_MS` in `sandbox.rs`, 20 ms) and the size a draw is timed at
(`FONT_TIME_PX` in `fontcheck.rs`, 160 px), which steps 6 to 8 answer; and the own folder's place
(`STATE_BASE` in `userfonts.rs`, `/var/local/grimoire`), which steps 5 and 11 answer (step 1's part is
answered). The two floors of free room beside it are set from step 1's answers.

1. **Before installing, over SSH, with 2.1.0 still on.** `mount | grep /var/local`, `df -k /var/local`,
`df -i /var/local`, `ls -ld /var/local`. Write the answers here. /var/local must be a file system of its
own that keeps its files across a restart (not the tmpfs at /var) and that keeps owners and modes (ext3
or ext4, say; on one without them the test "root's, and no one else can write it" means nothing). It
must be root's, with no write bit for group or others (`drwxr-xr-x root`: Grimoire tests the folders it
makes there, never /var/local itself), with well over 4 MB and at least 512 inodes free (a file system
that counts no inodes, `df -i` showing 0, is fine). If it is not, stop: the place (`STATE_BASE`) and the
floors beside it are set from these numbers before a release. Also keep 2.1.0's grimoire.log lines
`df -i /var/local`, `ls -ld /var/local`. The answers on a Scribe (firmware 5.20.1.0.2, Grimoire 2.1.0):
/var/local is a file system of its own, `/dev/mmcblk0p9`, ext4, mounted read and write (`df` names it
`/var/base-local`); 996,780 KB in all with 868,764 KB free, and 65,536 inodes with 63,718 free. /var/local
itself is `drwxrwsr-x`, the system's `framework` user's and `javausers` group's (mode 2775: the group may
write it, the setgid bit is set). What Grimoire needs of it: a file system of its own that keeps its files
across a restart (not the tmpfs at /var) and that keeps owners and modes (ext3 or ext4, say; on one
without them the test "root's, and no one else can write it" means nothing), with well over 4 MB and at
least 512 inodes free (a file system that counts no inodes, `df -i` showing 0, is fine). Grimoire never
relies on /var/local itself: it tests its own folder at every open and moves aside an entry at its name
that is not its own (step 12), so the owner and mode of /var/local ask nothing of a release. The floors
stand on these numbers: `STATE_FREE_FLOOR`, 4 MB, and `STATE_FILES_FLOOR`, 512 files, against a check
that keeps a few hundred bytes and one file a font (at most 200 for a data folder). If a Kindle answers
otherwise (no file system of its own there, or one without owners and modes), stop: the place
(`STATE_BASE`) is set from its answers before a release. Also keep 2.1.0's grimoire.log lines
`fonts: file I of N passed: ... checked in T ms` and `font: custom font loaded: S KB in T ms`.
2. **The update.** Install, start Grimoire with a font of your own in use. The Library is in Terminess at
first; the font comes back by itself with one redraw once its check has passed (how long?). Settings >
Font shows "Checking..." rows meanwhile. grimoire.log has one `fonts: file I of N passed: ...` line a
font, and no `work: a font's worker made call N`, no `marked unreadable`, no `cannot be kept`. If the font
font, and no `work: a font's worker made call N`, no `marked unreadable`, no `cannot be kept`, no `moved
aside` (nothing stood at the folder's name before). If the font
in use has a Bold file whose name sorts after its own (`Name.ttf` with `NameBold.ttf`), look at text
Grimoire draws bold once the font is back: it is the regular overprinted until Grimoire is started
again, then the Bold file's (docs/BACKLOG.md, "Bold stays overprinted").
3. **Where it is kept.** `ls -ld /var/local/grimoire /var/local/grimoire/* /var/local/grimoire/*/fonts
/var/local/grimoire/*/root` (three folders, `drwx------`, and one file, `-rw-------`, all root's; `cat`
of the file gives the data folder's path); `ls -l /var/local/grimoire/*/fonts` (one small `.font` a
/var/local/grimoire/*/root` (three folders, `drwx------`, and one file, `-rw-------`, all root's; under a
/var/local with the setgid bit, as in step 1, the folders show `drwx--S---` and its group, which they take
from it and which has no rights in them; `cat` of the file gives the data folder's path);
`ls -l /var/local/grimoire/*/fonts` (one small `.font` a
font); `head -3` of one says `grimoire-font 2` and a 64-digit `hash=`, which is `sha256sum` of
the font file. `ls /mnt/us/grimoire/fonts/.cache` has a `.pgm` a font (a `.no` for one the check
refused), and 2.1.0's `.font` files untouched.
Expand Down Expand Up @@ -2974,6 +2984,20 @@ steps 1, 5 and 11 answer; and the two floors of free room beside it (`STATE_FREE
the worker's narrower system calls and this change's walk are in the same process.
11. When a firmware update next arrives: is /var/local/grimoire still there? (If not, fonts are checked once
more: note it in KINDLE.md.)
12. **What is not Grimoire's own is moved aside.** With Grimoire closed: `mv /var/local/grimoire
/var/local/grimoire.test && mkdir -m 775 /var/local/grimoire`. Start Grimoire with a font of your own in
use: grimoire.log has once `fonts: the entry at the name of Grimoire's own folder was not its own (a
folder, owner 0, mode 0775): it is moved aside as it was, and the folder is made afresh; every font is
checked once more` (`mode 2775` if mkdir gave the folder /var/local's setgid bit), then a pass line a font:
every font is checked once more, as after `rm -r /var/local/grimoire` (step 9), and the font comes back. `ls -ld
/var/local/grimoire*`: the folder made afresh as in step 3, the one mkdir made under
`grimoire.not-own-<seconds>-<pid>` with its mode as it was and nothing in it, and `grimoire.test`. A second
start moves nothing. Once a run: with Grimoire running, `mv /var/local/grimoire /var/local/grimoire.test2 &&
mkdir -m 775 /var/local/grimoire`, then copy a font that is not there yet into `grimoire/fonts`. Nothing more
is moved aside: grimoire.log has `cannot be kept (it is not Grimoire's own)` and `no font is checked until
Grimoire starts again`, once each, and that font's row says "Unreadable font.". The next start moves the folder
aside and checks every font once more. Then, with Grimoire closed, `rm -r /var/local/grimoire.not-own-*
/var/local/grimoire.test /var/local/grimoire.test2`.

## 2.2, From a file... and Use as template (issue #146): device checks

Expand Down
Loading
Loading