Finetuning with PEFT - Some weights of MistralForSequenceClassification were not initialized from the model
Q1: Finetuning for a classification task (dataset imdb) using PEFT generates the following error:
Some weights of MistralForSequenceClassification were not initialized from the model checkpoint at mistralai/Mistral-7B-v0.1 and are newly initialized: ['score.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Q2: What is the max_length in mistral for Padding?
Thanks in advance
hi
@RobbieTheRobot
you need to properly set modules_to_save=["score"] in the peft config, can you provide more details on how you do fine-tuning with PEFT + sequence classification using mistral?
@ybelkada set module_to_save=["score"] does not eliminate the issue.
quantization_config = BitsAndBytesConfig(
load_in_4bit = True,
bnb_4bit_quant_type = 'nf4',
bnb_4bit_use_double_quant = True,
bnb_4bit_compute_dtype = torch.bfloat16
)
lora_config = LoraConfig(
r = 16,
lora_alpha = 8,
target_modules = ['q_proj', 'k_proj', 'v_proj', 'o_proj'],
lora_dropout = 0.05,
bias = 'none',
task_type = 'SEQ_CLS',
modules_to_save=["score"]
)
model = AutoModelForSequenceClassification.from_pretrained(
'mistralai/Mistral-7B-v0.1',
quantization_config=quantization_config,
num_labels=2
)
model = prepare_model_for_kbit_training(model)
model.config.use_cache = False
model.config.pad_token_id = tokenizer.pad_token_id
model = get_peft_model(model, lora_config)
Hi man, I met the same problem. Have u found the solution?
Any updates?