FacebookAI/xlm-roberta-large-finetuned-conll03-english

[DrvDispatch]

Hey
Does this work with FacebookAI/xlm-roberta-large-finetuned-conll03-english too? And where can I find base_vocabulary.json?

[yaireclipse]

Hi @AiTester950,
Sorry, I think it doesn't.
FacebookAI/xlm-roberta-large-finetuned-conll03-english is a finetuned XLM-RoBERTa, which means it uses XLMRobertaTokenizer, which uses SentencePiece. RobertaTokenizer uses byte-level Byte-Pair-Encoding, which is what's implemented in this repo.