Qwen/Qwen-14B-Chat · int4量化Qwen/Qwen-14B-Chat运行出错

Sep 25, 2023

问题：int4量化后key.dtype为float16，但是query.dtype仍然为float32，在进行query和key点乘时报出类型错误
建议：在258行_attn函数中第一行加入判断query和key是否为同一类型的判断，并统一两者类型

Cornmonster

Sep 26, 2023

+1，load in 4bit 报错如下：

  File "/home/ubuntu/miniconda3/envs/wz/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/miniconda3/envs/wz/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/ubuntu/.cache/huggingface/modules/transformers_modules/qwen-chat-pytorch-14b/modeling_qwen.py", line 1109, in forward
    transformer_outputs = self.transformer(
  File "/home/ubuntu/miniconda3/envs/wz/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/miniconda3/envs/wz/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/ubuntu/.cache/huggingface/modules/transformers_modules/qwen-chat-pytorch-14b/modeling_qwen.py", line 930, in forward
    outputs = block(
  File "/home/ubuntu/miniconda3/envs/wz/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/miniconda3/envs/wz/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/ubuntu/.cache/huggingface/modules/transformers_modules/qwen-chat-pytorch-14b/modeling_qwen.py", line 631, in forward
    attn_outputs = self.attn(
  File "/home/ubuntu/miniconda3/envs/wz/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/miniconda3/envs/wz/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/ubuntu/.cache/huggingface/modules/transformers_modules/qwen-chat-pytorch-14b/modeling_qwen.py", line 556, in forward
    attn_output, attn_weight = self._attn(
  File "/home/ubuntu/.cache/huggingface/modules/transformers_modules/qwen-chat-pytorch-14b/modeling_qwen.py", line 314, in _attn
    attn_weights = torch.matmul(query, key.transpose(-1, -2))
RuntimeError: [address=127.0.0.1:36263, pid=21908] expected scalar type Half but found Float

JustinLin610

Qwen org Nov 13, 2023

How about using Qwen-14B-Int4? This one performs better than BNB. Check the section quantization in our github readme for more information.

jklj077 changed discussion status to closed Dec 21, 2023