fxmarty

/

tiny-llama-fast-tokenizer

like 9

I can neither load the tokenizer nor the model:

Python 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("fxmarty/tiny-llama-fast-tokenizer")
Downloading (…)okenizer_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 649/649 [00:00<00:00, 797kB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/davef/anaconda3/envs/pytorch310/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 674, in from_pretrained
    raise ValueError(
ValueError: Tokenizer class LlamaTokenizer does not exist or is not currently imported.
>>> model = AutoModelForCausalLM.from_pretrained("fxmarty/tiny-llama-fast-tokenizer")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/davef/anaconda3/envs/pytorch310/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 441, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "/home/davef/anaconda3/envs/pytorch310/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 904, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/home/davef/anaconda3/envs/pytorch310/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 610, in __getitem__
    raise KeyError(key)
KeyError: 'llama'
>>> import transformers
>>> transformers.__version__
'4.27.0.dev0'