[EllianCarlos]

Stacked on #105. Base: fix/mojibake-gate. Merge #105 first.

Problem

A tokenizer can exit non-zero with a status that is not 33 (parser
crash) and not above 128 (signal). Without --abort-on-error, blobExec
then kept the original blob. The raw source went into the token tree
and the blob map, and no EXCLUDED line named the blob. Examples:

• The Rust tokenizer exits 1 on a .rs file that is not UTF-8.
• tokenBySha.pl and tokenize.pl use die. die exits with errno or 255.
• The worker answers 255 when the tokenizer cannot start.

The bug is on the per-blob path and on the --tokenizer-worker path.

Change

• Without --abort-on-error, a non-zero exit excludes the blob. The
  EXCLUDED failed blob line gives reason=tokenizer-error.
• BlobExec.run has a new onTokenizerError callback. The walker counts
  these blobs in blobsTokenizerFailed. The blobExec done: line and the
  summary of failed blobs show the count.
• With --abort-on-error, the run stops with exit 2, as before.
• The usage text, README.md and blobExec/README.org describe the new
  behaviour.

The change uses a new reason, not parser-crash. A test asserts that
exit 7 does not increment the parser-crash count. The parser-crash
warning says that srcML died, and that is wrong for a tokenizer error. No script reads the reason values, and
shard_merge.py ignores unknown keys on the done line. Thus no consumer
changes.

Not changed

• The exit status of blobExec. An excluded blob does not change it.
• Timeouts, parser crashes, signal deaths and a dead worker. They keep
  their reasons.
• The tokenizers and the tokenizer identity files.

Docs

• generate_dataset/DATASET.md lists reason=tokenizer-error and blobsTokenizerFailed.
• WORKER_PROTOCOL.md said that a worker that stops in a request gives a tokenizer error. The code reports a timeout, so the text now says that.

Tests

• BlobExecSpec: a tokenizer that exits 1 with output is excluded. Only
  onTokenizerError fires.
• TokenizerWorkerPoolSpec: a worker answer with exit 1 is excluded.
  With --abort-on-error, it aborts.
• FailedBlobSpec: a blob that exits 1 is absent from the tree and from
  the blob map, in serial, pipeline and pipeline-trees mode.

The 5 new tests fail on master. All CI suites pass: sbt 184 tests, prove
239, pytest 74, the shell suites, the worker test, the srcml2token and
Rust tests, and the legacy modules.