bigcode

/

gpt_bigcode-santacoder

like 24

fixes the eos and bos token id in the config to match the tokenizer
The generation config picks eos_token from the config and not the tokenizer.
So, when specifying, min_new_tokens, the code crashes with the following error:

Traceback (most recent call last):
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/dccstor/mayankgpfs/scratch/GPTQ-for-SantaCoder/santacoder_inference.py", line 90, in <module>
    main()
  File "/dccstor/mayankgpfs/scratch/GPTQ-for-SantaCoder/santacoder_inference.py", line 86, in main
    simple_generation_test(tokenizer, model, args.prompt)
  File "/dccstor/mayankgpfs/scratch/GPTQ-for-SantaCoder/santacoder_inference.py", line 55, in simple_generation_test
    generated = model.generate(batch["input_ids"], do_sample=False, min_new_tokens=100, max_new_tokens=100)
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/site-packages/transformers/generation/utils.py", line 1438, in generate
    return self.greedy_search(
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/site-packages/transformers/generation/utils.py", line 2263, in greedy_search
    next_tokens_scores = logits_processor(input_ids, next_token_logits)
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/site-packages/transformers/generation/logits_process.py", line 92, in __call__
    scores = processor(input_ids, scores)
  File "/dccstor/mayankgpfs/conda/envs/bloom/lib/python3.8/site-packages/transformers/generation/logits_process.py", line 162, in __call__
    scores[:, i] = -float("inf")
IndexError: index 50256 is out of bounds for dimension 1 with size 49280