Is it possible to get GPTQ quants in 4 bpw?

by MrHillsss - opened Dec 11, 2023

Discussion

MrHillsss

Dec 11, 2023

Or any other quants to fit in a 3090/4090?

ArthurZ

Dec 11, 2023

I think the amazing @TheBloke is going to convert them and push them as always to his repos!
You can already use load_in_4bit = True for bitsandbytes quantization with transformers format

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment