Respect the special flag on HF added tokens

[u7k4rs6]

Hey! Ran into this while comparing HFTokenizer output against the HF python tokenizers lib.

parse_special_tokens() puts every entry of added_tokens into special_token_map_ and ignores the "special" flag. So tokens marked "special": false get treated like real special tokens, which causes two problems:

1. decode with skip_special_tokens=true drops them. For Qwen2.5 / Qwen3 / DeepSeek-R1-Distill / SmolLM3 that means , and <tool_call> just vanish from the output, while HF keeps them.

2. their ids get removed from the BPE vocab, so merges can't produce them anymore. Gemma-3 is hit hard here, since most of its added tokens are non-special whitespace runs like "▁▁". Something like " Hello world " encodes to [26352, 236743, 1902, 236743] instead of HF's [26352, 138, 12392, 236743]. On a batch of real code/doc snippets, gemma-3-1b-it disagreed with HF on 161 out of 180.

The fix keeps a set of ids that are actually special (a missing "special" field still counts as special, so older files behave the same). Only those are removed from the vocab and skipped on decode. I added a small is_special_token_id_() hook on BPETokenizerBase that defaults to true, so tiktoken/tekken are untouched. vocab_size_ now counts distinct ids, since a non-special added token can also sit in the vocab.

I checked it against 24 tokenizer.json files from the Hub (llama, qwen, gemma, deepseek, phi, etc.) and got 0 encode/decode mismatches vs HF after the change. Added NonSpecialAddedTokens to test_hf_tokenizer.cpp, which fails on main and passes with the fix, and the full ctest suite passes (15/15).

[meta-cla]

Hi @u7k4rs6!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!