mistralai

/

Mixtral-8x22B-Instruct-v0.1

like 587

prompts_all=[
    "Who is Kobe?"
]
model = AutoModelForCausalLM.from_pretrained(
    model_path,    
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2"
)
tokenizer = AutoTokenizer.from_pretrained(model_path)   
tokenizer.pad_token = tokenizer.eos_token
inputs = tokenizer(prompts_all, return_tensors="pt", padding=True).to("cuda")
outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

the request is Who is Kobe? and the response is

So how can I remove Who is Kobe? in the response?