mistralai

/

Mixtral-8x7B-v0.1

like 1.56k

Hi @kxgong
I suggest to load the model in half-precision (torch_dtype=torch.float16) or in 4-bit precision load_in_4bit=True in order to load your model in the most memory efficient manner possible

Hi @kxgong
I suggest to load the model in half-precision (torch_dtype=torch.float16) or in 4-bit precision load_in_4bit=True in order to load your model in the most memory efficient manner possible

Thank you, I am using mixtral-8x7B for training. I wonder whether using 4bit will cause performance drop.

@kxgong if you use QLoRA you shouldn't expect performance drop with respect to full-finetuning. You can read more about QLoRA here: https://huggingface.co/blog/4bit-transformers-bitsandbytes and get started with resources on how to run QLoRA with this blogpost for example: https://pytorch.org/blog/finetune-llms/

Thanks for your help.