zxdu20 commited on
Commit
fb85b4d
1 Parent(s): f8df870

Init commit

Browse files
LICENSE ADDED
@@ -0,0 +1,201 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright Zhengxiao Du
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
MODEL_LICENSE ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ The GLM-130B License
2
+
3
+ 1. Definitions
4
+
5
+ “Licensor” means the GLM-130B Model Team that distributes its Software.
6
+
7
+ “Software” means the GLM-130B model parameters made available under this license.
8
+
9
+ 2. License Grant
10
+
11
+ Subject to the terms and conditions of this License, the Licensor hereby grants to you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty-free copyright license to use the Software solely for your non-commercial research purposes.
12
+
13
+ The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
14
+
15
+ 3. Restriction
16
+
17
+ You will not use, copy, modify, merge, publish, distribute, reproduce, or create derivative works of the Software, in whole or in part, for any commercial, military, or illegal purposes.
18
+
19
+ You will not use the Software for any act that may undermine China's national security and national unity, harm the public interest of society, or infringe upon the rights and interests of human beings.
20
+
21
+ 4. Disclaimer
22
+
23
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
24
+
25
+ 5. Limitation of Liability
26
+
27
+ EXCEPT TO THE EXTENT PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER BASED IN TORT, NEGLIGENCE, CONTRACT, LIABILITY, OR OTHERWISE WILL ANY LICENSOR BE LIABLE TO YOU FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES, OR ANY OTHER COMMERCIAL LOSSES, EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
28
+
29
+ 6. Dispute Resolution
30
+
31
+ This license shall be governed and construed in accordance with the laws of People’s Republic of China. Any dispute arising from or in connection with this License shall be submitted to Haidian District People's Court in Beijing.
32
+
33
+ Note that the license is subject to update to a more comprehensive version. For any questions related to the license and copyright, please contact us at glm-130b@googlegroups.com.
README.md ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - zh
4
+ - en
5
+ tags:
6
+ - glm
7
+ - chatglm
8
+ - thudm
9
+ ---
10
+ # ChatGLM-6B-INT4
11
+ <p align="center">
12
+ 👋 Join our <a href="https://join.slack.com/t/chatglm/shared_invite/zt-1t4a8evfn-vduo2hhNcYqBUnZ71IXiqQ" target="_blank">Slack</a> and <a href="https://github.com/THUDM/ChatGLM-6B/blob/main/resources/WECHAT.md" target="_blank">WeChat</a>
13
+ </p>
14
+
15
+ ## 介绍
16
+ ChatGLM-6B 是一个开源的、支持中英双语问答的对话语言模型,基于 [General Language Model (GLM)](https://github.com/THUDM/GLM) 架构,具有 62 亿参数。结合模型量化技术,用户可以在消费级的显卡上进行本地部署(INT4 量化级别下最低只需 6GB 显存)。ChatGLM-6B 使用了和 [ChatGLM](https://chatglm.cn) 相同的技术,针对中文问答和对话进行了优化。经过约 1T 标识符的中英双语训练,辅以监督微调、反馈自助、人类反馈强化学习等技术的加持,62 亿参数的 ChatGLM-6B 已经能生成相当符合人类偏好的回答。
17
+
18
+ ChatGLM-6B-INT4 是 ChatGLM-6B 量化后的模型权重。具体的,ChatGLM-6B-INT4 对 ChatGLM-6B 中的 28 个 GLM Block 进行了 INT4 量化,没有对 Embedding 和 LM Head 进行量化。量化后的模型理论上 6G 显存(使用 CPU 即内存)即可推理,具有在嵌入式设备(如树莓派)上运行的可能。
19
+
20
+ 在 CPU 上运行时,会根据硬件自动编译 CPU Kernel ,请确保已安装 GCC 和 OpenMP (Linux一般已安装,对于Windows则需手动安装),以获得最佳并行计算能力。
21
+
22
+ ## 软件依赖
23
+
24
+ ```shell
25
+ pip install protobuf transformers==4.27.1 cpm_kernels
26
+ ```
27
+
28
+ ## 代码调用
29
+
30
+ 可以通过如下代码调用 ChatGLM-6B 模型来生成对话:
31
+
32
+ ```ipython
33
+ >>> from transformers import AutoTokenizer, AutoModel
34
+ >>> tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True)
35
+ >>> model = AutoModel.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).half().cuda()
36
+ >>> response, history = model.chat(tokenizer, "你好", history=[])
37
+ >>> print(response)
38
+ 你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。
39
+ >>> response, history = model.chat(tokenizer, "晚上睡不着应该怎么办", history=history)
40
+ >>> print(response)
41
+ 晚上睡不着可能会让你感到焦虑或不舒服,但以下是一些可以帮助你入睡的方法:
42
+
43
+ 1. 制定规律的睡眠时间表:保持规律的睡眠时间表可以帮助你建立健康的睡眠习惯,使你更容易入睡。尽量在每天的相同时间上床,并在同一时间起床。
44
+ 2. 创造一个舒适的睡眠环境:确保睡眠环境舒适,安静,黑暗且温度适宜。可以使用舒适的床上用品,并保持房间通风。
45
+ 3. 放松身心:在睡前做些放松的活动,例如泡个热水澡,听些轻柔的音乐,阅读一些有趣的书籍等,有助于缓解紧张和焦虑,使你更容易入睡。
46
+ 4. 避免饮用含有咖啡因的饮料:咖啡因是一种刺激性物质,会影响你的睡眠质量。尽量避免在睡前饮用含有咖啡因的饮料,例如咖啡,茶和可乐。
47
+ 5. 避免在床上做与睡眠无关的事情:在床上做些与睡眠无关的事情,例如看电影,玩游戏或工作等,可能会干扰你的睡眠。
48
+ 6. 尝试呼吸技巧:深呼吸是一种放松技巧,可以帮助你缓解紧张和焦虑,使你更容易入睡。试着慢慢吸气,保持几秒钟,然后缓慢呼气。
49
+
50
+ 如果这些方法无法帮助你入睡,你可以考虑咨询医生或睡眠专家,寻求进一步的建议。
51
+ ```
52
+
53
+ 关于更多的使用说明,包括如何运行命令行和网页版本的 DEMO,以及使用模型量化以节省显存,请参考我们的 [Github Repo](https://github.com/THUDM/ChatGLM-6B)。
54
+
55
+ ## 协议
56
+
57
+ 本仓库的代码依照 [Apache-2.0](LICENSE) 协议开源,ChatGLM-6B 模型的权重的使用则需要遵循 [Model License](MODEL_LICENSE)。
58
+
59
+ ## 引用
60
+
61
+ 如果你觉得我们的工作有帮助的话,请考虑引用下列论文:
62
+
63
+ ```
64
+ @inproceedings{
65
+ zeng2023glm-130b,
66
+ title={{GLM}-130B: An Open Bilingual Pre-trained Model},
67
+ author={Aohan Zeng and Xiao Liu and Zhengxiao Du and Zihan Wang and Hanyu Lai and Ming Ding and Zhuoyi Yang and Yifan Xu and Wendi Zheng and Xiao Xia and Weng Lam Tam and Zixuan Ma and Yufei Xue and Jidong Zhai and Wenguang Chen and Zhiyuan Liu and Peng Zhang and Yuxiao Dong and Jie Tang},
68
+ booktitle={The Eleventh International Conference on Learning Representations (ICLR)},
69
+ year={2023},
70
+ url={https://openreview.net/forum?id=-Aw0rrrPUF}
71
+ }
72
+ ```
73
+ ```
74
+ @inproceedings{du2022glm,
75
+ title={GLM: General Language Model Pretraining with Autoregressive Blank Infilling},
76
+ author={Du, Zhengxiao and Qian, Yujie and Liu, Xiao and Ding, Ming and Qiu, Jiezhong and Yang, Zhilin and Tang, Jie},
77
+ booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
78
+ pages={320--335},
79
+ year={2022}
80
+ }
81
+ ```
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "THUDM/chatglm-6b-int8",
3
+ "architectures": [
4
+ "ChatGLMModel"
5
+ ],
6
+ "auto_map": {
7
+ "AutoConfig": "configuration_chatglm.ChatGLMConfig",
8
+ "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration",
9
+ "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration"
10
+ },
11
+ "bos_token_id": 130004,
12
+ "eos_token_id": 130005,
13
+ "gmask_token_id": 130001,
14
+ "hidden_size": 4096,
15
+ "inner_hidden_size": 16384,
16
+ "layernorm_epsilon": 1e-05,
17
+ "mask_token_id": 130000,
18
+ "max_sequence_length": 2048,
19
+ "model_type": "chatglm",
20
+ "num_attention_heads": 32,
21
+ "num_layers": 28,
22
+ "pad_token_id": 3,
23
+ "position_encoding_2d": true,
24
+ "quantization_bit": 8,
25
+ "quantization_embeddings": false,
26
+ "torch_dtype": "float16",
27
+ "transformers_version": "4.27.1",
28
+ "use_cache": true,
29
+ "vocab_size": 130528
30
+ }
configuration_chatglm.py ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """ ChatGLM model configuration """
2
+
3
+ from transformers.configuration_utils import PretrainedConfig
4
+ from transformers.utils import logging
5
+
6
+ logger = logging.get_logger(__name__)
7
+
8
+
9
+ class ChatGLMConfig(PretrainedConfig):
10
+ r"""
11
+ This is the configuration class to store the configuration of a [`~ChatGLMModel`].
12
+ It is used to instantiate an ChatGLM model according to the specified arguments, defining the model
13
+ architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
14
+ the ChatGLM-6B [THUDM/ChatGLM-6B](https://huggingface.co/THUDM/chatglm-6b) architecture.
15
+
16
+ Configuration objects inherit from [`PretrainedConfig`] and can be used
17
+ to control the model outputs. Read the documentation from [`PretrainedConfig`]
18
+ for more information.
19
+
20
+
21
+ Args:
22
+ vocab_size (`int`, *optional*, defaults to 150528):
23
+ Vocabulary size of the ChatGLM-6B model. Defines the number of different tokens that can be represented by the
24
+ `inputs_ids` passed when calling [`~ChatGLMModel`] or
25
+ [`~TFChatGLMModel`].
26
+ hidden_size (`int`, *optional*, defaults to 4096):
27
+ Dimension of the encoder layers and the pooler layer.
28
+ num_hidden_layers (`int`, *optional*, defaults to 28):
29
+ Number of hidden layers in the Transformer encoder.
30
+ num_attention_heads (`int`, *optional*, defaults to 32):
31
+ Number of attention heads for each attention layer in the Transformer encoder.
32
+ inner_hidden_size (`int`, *optional*, defaults to 16384):
33
+ Dimension of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
34
+ max_sequence_length (`int`, *optional*, defaults to 512):
35
+ The maximum sequence length that this model might ever be used with.
36
+ Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
37
+ layernorm_epsilon (`float`, *optional*, defaults to 1e-5):
38
+ The epsilon used by the layer normalization layers.
39
+ use_cache (`bool`, *optional*, defaults to `True`):
40
+ Whether the model should return the last key/values attentions (not used by all models).
41
+ Example:
42
+
43
+ ```python
44
+ >>> from configuration_chatglm import ChatGLMConfig
45
+ >>> from modeling_chatglm import ChatGLMModel
46
+
47
+ >>> # Initializing a ChatGLM-6B THUDM/ChatGLM-6B style configuration
48
+ >>> configuration = ChatGLMConfig()
49
+
50
+ >>> # Initializing a model from the THUDM/ChatGLM-6B style configuration
51
+ >>> model = ChatGLMModel(configuration)
52
+
53
+ >>> # Accessing the model configuration
54
+ >>> configuration = model.config
55
+ ```
56
+ """
57
+ model_type = "chatglm"
58
+
59
+ def __init__(
60
+ self,
61
+ vocab_size=150528,
62
+ hidden_size=4096,
63
+ num_layers=28,
64
+ num_attention_heads=32,
65
+ layernorm_epsilon=1e-5,
66
+ use_cache=False,
67
+ bos_token_id=150004,
68
+ eos_token_id=150005,
69
+ mask_token_id=150000,
70
+ gmask_token_id=150001,
71
+ pad_token_id=0,
72
+ max_sequence_length=2048,
73
+ inner_hidden_size=16384,
74
+ position_encoding_2d=True,
75
+ quantization_bit=0,
76
+ quantization_embeddings=False,
77
+ pre_seq_len=None,
78
+ prefix_projection=False,
79
+ **kwargs
80
+ ):
81
+ self.num_layers = num_layers
82
+ self.vocab_size = vocab_size
83
+ self.hidden_size = hidden_size
84
+ self.num_attention_heads = num_attention_heads
85
+ self.max_sequence_length = max_sequence_length
86
+ self.layernorm_epsilon = layernorm_epsilon
87
+ self.inner_hidden_size = inner_hidden_size
88
+ self.use_cache = use_cache
89
+ self.bos_token_id = bos_token_id
90
+ self.eos_token_id = eos_token_id
91
+ self.pad_token_id = pad_token_id
92
+ self.mask_token_id = mask_token_id
93
+ self.gmask_token_id = gmask_token_id
94
+ self.position_encoding_2d = position_encoding_2d
95
+ self.quantization_bit = quantization_bit
96
+ self.quantization_embeddings = quantization_embeddings
97
+ self.pre_seq_len = pre_seq_len
98
+ self.prefix_projection = prefix_projection
99
+
100
+ super().__init__(
101
+ pad_token_id=pad_token_id,
102
+ bos_token_id=bos_token_id,
103
+ eos_token_id=eos_token_id,
104
+ **kwargs
105
+ )
modeling_chatglm.py ADDED
@@ -0,0 +1,1472 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """ PyTorch ChatGLM model. """
2
+
3
+ import math
4
+ import copy
5
+ import os
6
+ import warnings
7
+ import re
8
+ import sys
9
+
10
+ import torch
11
+ import torch.utils.checkpoint
12
+ import torch.nn.functional as F
13
+ from torch import nn
14
+ from torch.nn import CrossEntropyLoss, LayerNorm
15
+ from torch.nn.utils import skip_init
16
+ from typing import Optional, Tuple, Union, List, Callable, Dict, Any
17
+
18
+ from transformers.utils import (
19
+ add_code_sample_docstrings,
20
+ add_start_docstrings,
21
+ add_start_docstrings_to_model_forward,
22
+ )
23
+ from transformers.modeling_outputs import (
24
+ BaseModelOutputWithPast,
25
+ CausalLMOutputWithPast,
26
+ BaseModelOutputWithPastAndCrossAttentions,
27
+ )
28
+ from transformers.modeling_utils import PreTrainedModel
29
+ from transformers.utils import logging
30
+ from transformers.generation.logits_process import LogitsProcessor
31
+ from transformers.generation.utils import LogitsProcessorList, StoppingCriteriaList, GenerationConfig, ModelOutput
32
+
33
+ from .configuration_chatglm import ChatGLMConfig
34
+
35
+
36
+ # flags required to enable jit fusion kernels
37
+
38
+ if sys.platform != 'darwin':
39
+ torch._C._jit_set_profiling_mode(False)
40
+ torch._C._jit_set_profiling_executor(False)
41
+ torch._C._jit_override_can_fuse_on_cpu(True)
42
+ torch._C._jit_override_can_fuse_on_gpu(True)
43
+
44
+ logger = logging.get_logger(__name__)
45
+
46
+ _CHECKPOINT_FOR_DOC = "THUDM/ChatGLM-6B"
47
+ _CONFIG_FOR_DOC = "ChatGLM6BConfig"
48
+
49
+ CHATGLM_6B_PRETRAINED_MODEL_ARCHIVE_LIST = [
50
+ "THUDM/chatglm-6b",
51
+ # See all ChatGLM-6B models at https://huggingface.co/models?filter=chatglm
52
+ ]
53
+
54
+
55
+ class InvalidScoreLogitsProcessor(LogitsProcessor):
56
+ def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
57
+ if torch.isnan(scores).any() or torch.isinf(scores).any():
58
+ scores.zero_()
59
+ scores[..., 5] = 5e4
60
+ return scores
61
+
62
+
63
+ def load_tf_weights_in_chatglm_6b(model, config, tf_checkpoint_path):
64
+ """Load tf checkpoints in a pytorch model."""
65
+ try:
66
+ import re
67
+
68
+ import numpy as np
69
+ import tensorflow as tf
70
+ except ImportError:
71
+ logger.error(
72
+ "Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see "
73
+ "https://www.tensorflow.org/install/ for installation instructions."
74
+ )
75
+ raise
76
+ tf_path = os.path.abspath(tf_checkpoint_path)
77
+ logger.info(f"Converting TensorFlow checkpoint from {tf_path}")
78
+ # Load weights from TF model
79
+ init_vars = tf.train.list_variables(tf_path)
80
+ names = []
81
+ arrays = []
82
+ for name, shape in init_vars:
83
+ logger.info(f"Loading TF weight {name} with shape {shape}")
84
+ array = tf.train.load_variable(tf_path, name)
85
+ names.append(name)
86
+ arrays.append(array)
87
+
88
+ for name, array in zip(names, arrays):
89
+ name = name.split("/")
90
+ # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
91
+ # which are not required for using pretrained model
92
+ if any(
93
+ n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"]
94
+ for n in name
95
+ ):
96
+ logger.info(f"Skipping {'/'.join(name)}")
97
+ continue
98
+ pointer = model
99
+ for m_name in name:
100
+ if re.fullmatch(r"[A-Za-z]+_\d+", m_name):
101
+ scope_names = re.split(r"_(\d+)", m_name)
102
+ else:
103
+ scope_names = [m_name]
104
+ if scope_names[0] == "kernel" or scope_names[0] == "gamma":
105
+ pointer = getattr(pointer, "weight")
106
+ elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
107
+ pointer = getattr(pointer, "bias")
108
+ elif scope_names[0] == "output_weights":
109
+ pointer = getattr(pointer, "weight")
110
+ elif scope_names[0] == "squad":
111
+ pointer = getattr(pointer, "classifier")
112
+ else:
113
+ try:
114
+ pointer = getattr(pointer, scope_names[0])
115
+ except AttributeError:
116
+ logger.info(f"Skipping {'/'.join(name)}")
117
+ continue
118
+ if len(scope_names) >= 2:
119
+ num = int(scope_names[1])
120
+ pointer = pointer[num]
121
+ if m_name[-11:] == "_embeddings":
122
+ pointer = getattr(pointer, "weight")
123
+ elif m_name == "kernel":
124
+ array = np.transpose(array)
125
+ try:
126
+ assert (
127
+ pointer.shape == array.shape
128
+ ), f"Pointer shape {pointer.shape} and array shape {array.shape} mismatched"
129
+ except AssertionError as e:
130
+ e.args += (pointer.shape, array.shape)
131
+ raise
132
+ logger.info(f"Initialize PyTorch weight {name}")
133
+ pointer.data = torch.from_numpy(array)
134
+ return model
135
+
136
+
137
+ class PrefixEncoder(torch.nn.Module):
138
+ """
139
+ The torch.nn model to encode the prefix
140
+ Input shape: (batch-size, prefix-length)
141
+ Output shape: (batch-size, prefix-length, 2*layers*hidden)
142
+ """
143
+
144
+ def __init__(self, config):
145
+ super().__init__()
146
+ self.prefix_projection = config.prefix_projection
147
+ if self.prefix_projection:
148
+ # Use a two-layer MLP to encode the prefix
149
+ self.embedding = torch.nn.Embedding(config.pre_seq_len, config.hidden_size)
150
+ self.trans = torch.nn.Sequential(
151
+ torch.nn.Linear(config.hidden_size, config.hidden_size),
152
+ torch.nn.Tanh(),
153
+ torch.nn.Linear(config.hidden_size, config.num_layers * config.hidden_size * 2)
154
+ )
155
+ else:
156
+ self.embedding = torch.nn.Embedding(config.pre_seq_len, config.num_layers * config.hidden_size * 2)
157
+
158
+ def forward(self, prefix: torch.Tensor):
159
+ if self.prefix_projection:
160
+ prefix_tokens = self.embedding(prefix)
161
+ past_key_values = self.trans(prefix_tokens)
162
+ else:
163
+ past_key_values = self.embedding(prefix)
164
+ return past_key_values
165
+
166
+
167
+ @torch.jit.script
168
+ def gelu_impl(x):
169
+ """OpenAI's gelu implementation."""
170
+ return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * x *
171
+ (1.0 + 0.044715 * x * x)))
172
+
173
+
174
+ def gelu(x):
175
+ return gelu_impl(x)
176
+
177
+
178
+ class RotaryEmbedding(torch.nn.Module):
179
+ def __init__(self, dim, base=10000, precision=torch.half, learnable=False):
180
+ super().__init__()
181
+ inv_freq = 1. / (base ** (torch.arange(0, dim, 2).float() / dim))
182
+ inv_freq = inv_freq.half()
183
+ self.learnable = learnable
184
+ if learnable:
185
+ self.inv_freq = torch.nn.Parameter(inv_freq)
186
+ self.max_seq_len_cached = None
187
+ else:
188
+ self.register_buffer('inv_freq', inv_freq)
189
+ self.max_seq_len_cached = None
190
+ self.cos_cached = None
191
+ self.sin_cached = None
192
+ self.precision = precision
193
+
194
+ def _load_from_state_dict(self, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys,
195
+ error_msgs):
196
+ pass
197
+
198
+ def forward(self, x, seq_dim=1, seq_len=None):
199
+ if seq_len is None:
200
+ seq_len = x.shape[seq_dim]
201
+ if self.max_seq_len_cached is None or (seq_len > self.max_seq_len_cached):
202
+ self.max_seq_len_cached = None if self.learnable else seq_len
203
+ t = torch.arange(seq_len, device=x.device, dtype=self.inv_freq.dtype)
204
+ freqs = torch.einsum('i,j->ij', t, self.inv_freq)
205
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
206
+ emb = torch.cat((freqs, freqs), dim=-1).to(x.device)
207
+ if self.precision == torch.bfloat16:
208
+ emb = emb.float()
209
+
210
+ # [sx, 1 (b * np), hn]
211
+ cos_cached = emb.cos()[:, None, :]
212
+ sin_cached = emb.sin()[:, None, :]
213
+ if self.precision == torch.bfloat16:
214
+ cos_cached = cos_cached.bfloat16()
215
+ sin_cached = sin_cached.bfloat16()
216
+ if self.learnable:
217
+ return cos_cached, sin_cached
218
+ self.cos_cached, self.sin_cached = cos_cached, sin_cached
219
+ return self.cos_cached[:seq_len, ...], self.sin_cached[:seq_len, ...]
220
+
221
+ def _apply(self, fn):
222
+ if self.cos_cached is not None:
223
+ self.cos_cached = fn(self.cos_cached)
224
+ if self.sin_cached is not None:
225
+ self.sin_cached = fn(self.sin_cached)
226
+ return super()._apply(fn)
227
+
228
+ def rotate_half(x):
229
+ x1, x2 = x[..., :x.shape[-1] // 2], x[..., x.shape[-1] // 2:]
230
+ return torch.cat((-x2, x1), dim=x1.ndim - 1) # dim=-1 triggers a bug in earlier torch versions
231
+
232
+
233
+ @torch.jit.script
234
+ def apply_rotary_pos_emb_index(q, k, cos, sin, position_id):
235
+ # position_id: [sq, b], q, k: [sq, b, np, hn], cos: [sq, 1, hn] -> [sq, b, 1, hn]
236
+ cos, sin = F.embedding(position_id, cos.squeeze(1)).unsqueeze(2), \
237
+ F.embedding(position_id, sin.squeeze(1)).unsqueeze(2)
238
+ q, k = (q * cos) + (rotate_half(q) * sin), (k * cos) + (rotate_half(k) * sin)
239
+ return q, k
240
+
241
+
242
+ def attention_fn(
243
+ self,
244
+ query_layer,
245
+ key_layer,
246
+ value_layer,
247
+ attention_mask,
248
+ hidden_size_per_partition,
249
+ layer_id,
250
+ layer_past=None,
251
+ scaling_attention_score=True,
252
+ use_cache=False,
253
+ ):
254
+ if layer_past is not None:
255
+ past_key, past_value = layer_past[0], layer_past[1]
256
+ key_layer = torch.cat((past_key, key_layer), dim=0)
257
+ value_layer = torch.cat((past_value, value_layer), dim=0)
258
+
259
+ # seqlen, batch, num_attention_heads, hidden_size_per_attention_head
260
+ seq_len, b, nh, hidden_size = key_layer.shape
261
+
262
+ if use_cache:
263
+ present = (key_layer, value_layer)
264
+ else:
265
+ present = None
266
+
267
+ query_key_layer_scaling_coeff = float(layer_id + 1)
268
+ if scaling_attention_score:
269
+ query_layer = query_layer / (math.sqrt(hidden_size) * query_key_layer_scaling_coeff)
270
+
271
+ # ===================================
272
+ # Raw attention scores. [b, np, s, s]
273
+ # ===================================
274
+
275
+ # [b, np, sq, sk]
276
+ output_size = (query_layer.size(1), query_layer.size(2), query_layer.size(0), key_layer.size(0))
277
+
278
+ # [sq, b, np, hn] -> [sq, b * np, hn]
279
+ query_layer = query_layer.view(output_size[2], output_size[0] * output_size[1], -1)
280
+ # [sk, b, np, hn] -> [sk, b * np, hn]
281
+ key_layer = key_layer.view(output_size[3], output_size[0] * output_size[1], -1)
282
+
283
+ matmul_result = torch.zeros(
284
+ 1, 1, 1,
285
+ dtype=query_layer.dtype,
286
+ device=query_layer.device,
287
+ )
288
+
289
+ matmul_result = torch.baddbmm(
290
+ matmul_result,
291
+ query_layer.transpose(0, 1), # [b * np, sq, hn]
292
+ key_layer.transpose(0, 1).transpose(1, 2), # [b * np, hn, sk]
293
+ beta=0.0,
294
+ alpha=1.0,
295
+ )
296
+
297
+ # change view to [b, np, sq, sk]
298
+ attention_scores = matmul_result.view(*output_size)
299
+
300
+ if self.scale_mask_softmax:
301
+ self.scale_mask_softmax.scale = query_key_layer_scaling_coeff
302
+ attention_probs = self.scale_mask_softmax(attention_scores, attention_mask.contiguous())
303
+ else:
304
+ if not (attention_mask == 0).all():
305
+ # if auto-regressive, skip
306
+ attention_scores.masked_fill_(attention_mask, -10000.0)
307
+ dtype = attention_scores.dtype
308
+ attention_scores = attention_scores.float()
309
+ attention_scores = attention_scores * query_key_layer_scaling_coeff
310
+
311
+ attention_probs = F.softmax(attention_scores, dim=-1)
312
+
313
+ attention_probs = attention_probs.type(dtype)
314
+
315
+ # =========================
316
+ # Context layer. [sq, b, hp]
317
+ # =========================
318
+
319
+ # value_layer -> context layer.
320
+ # [sk, b, np, hn] --> [b, np, sq, hn]
321
+
322
+ # context layer shape: [b, np, sq, hn]
323
+ output_size = (value_layer.size(1), value_layer.size(2), query_layer.size(0), value_layer.size(3))
324
+
325
+ # change view [sk, b * np, hn]
326
+ value_layer = value_layer.view(value_layer.size(0), output_size[0] * output_size[1], -1)
327
+
328
+ # change view [b * np, sq, sk]
329
+ attention_probs = attention_probs.view(output_size[0] * output_size[1], output_size[2], -1)
330
+
331
+ # matmul: [b * np, sq, hn]
332
+ context_layer = torch.bmm(attention_probs, value_layer.transpose(0, 1))
333
+
334
+ # change view [b, np, sq, hn]
335
+ context_layer = context_layer.view(*output_size)
336
+
337
+ # [b, np, sq, hn] --> [sq, b, np, hn]
338
+ context_layer = context_layer.permute(2, 0, 1, 3).contiguous()
339
+
340
+ # [sq, b, np, hn] --> [sq, b, hp]
341
+ new_context_layer_shape = context_layer.size()[:-2] + (hidden_size_per_partition,)
342
+ context_layer = context_layer.view(*new_context_layer_shape)
343
+
344
+ outputs = (context_layer, present, attention_probs)
345
+
346
+ return outputs
347
+
348
+
349
+ def default_init(cls, *args, **kwargs):
350
+ return cls(*args, **kwargs)
351
+
352
+
353
+ class SelfAttention(torch.nn.Module):
354
+ def __init__(self, hidden_size, num_attention_heads,
355
+ layer_id, hidden_size_per_attention_head=None, bias=True,
356
+ params_dtype=torch.float, position_encoding_2d=True, empty_init=True):
357
+ if empty_init:
358
+ init_method = skip_init
359
+ else:
360
+ init_method = default_init
361
+ super(SelfAttention, self).__init__()
362
+
363
+ self.layer_id = layer_id
364
+ self.hidden_size = hidden_size
365
+ self.hidden_size_per_partition = hidden_size
366
+ self.num_attention_heads = num_attention_heads
367
+ self.num_attention_heads_per_partition = num_attention_heads
368
+ self.position_encoding_2d = position_encoding_2d
369
+ self.rotary_emb = RotaryEmbedding(
370
+ self.hidden_size // (self.num_attention_heads * 2)
371
+ if position_encoding_2d
372
+ else self.hidden_size // self.num_attention_heads,
373
+ base=10000,
374
+ precision=torch.half,
375
+ learnable=False,
376
+ )
377
+
378
+ self.scale_mask_softmax = None
379
+
380
+ if hidden_size_per_attention_head is None:
381
+ self.hidden_size_per_attention_head = hidden_size // num_attention_heads
382
+ else:
383
+ self.hidden_size_per_attention_head = hidden_size_per_attention_head
384
+
385
+ self.inner_hidden_size = num_attention_heads * self.hidden_size_per_attention_head
386
+
387
+ # Strided linear layer.
388
+ self.query_key_value = init_method(
389
+ torch.nn.Linear,
390
+ hidden_size,
391
+ 3 * self.inner_hidden_size,
392
+ bias=bias,
393
+ dtype=params_dtype,
394
+ )
395
+
396
+ self.dense = init_method(
397
+ torch.nn.Linear,
398
+ self.inner_hidden_size,
399
+ hidden_size,
400
+ bias=bias,
401
+ dtype=params_dtype,
402
+ )
403
+
404
+ @staticmethod
405
+ def attention_mask_func(attention_scores, attention_mask):
406
+ attention_scores.masked_fill_(attention_mask, -10000.0)
407
+ return attention_scores
408
+
409
+ def split_tensor_along_last_dim(self, tensor, num_partitions,
410
+ contiguous_split_chunks=False):
411
+ """Split a tensor along its last dimension.
412
+ Arguments:
413
+ tensor: input tensor.
414
+ num_partitions: number of partitions to split the tensor
415
+ contiguous_split_chunks: If True, make each chunk contiguous
416
+ in memory.
417
+ """
418
+ # Get the size and dimension.
419
+ last_dim = tensor.dim() - 1
420
+ last_dim_size = tensor.size()[last_dim] // num_partitions
421
+ # Split.
422
+ tensor_list = torch.split(tensor, last_dim_size, dim=last_dim)
423
+ # Note: torch.split does not create contiguous tensors by default.
424
+ if contiguous_split_chunks:
425
+ return tuple(chunk.contiguous() for chunk in tensor_list)
426
+
427
+ return tensor_list
428
+
429
+ def forward(
430
+ self,
431
+ hidden_states: torch.Tensor,
432
+ position_ids,
433
+ attention_mask: torch.Tensor,
434
+ layer_id,
435
+ layer_past: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
436
+ use_cache: bool = False,
437
+ output_attentions: bool = False,
438
+ ):
439
+ """
440
+ hidden_states: [seq_len, batch, hidden_size]
441
+ attention_mask: [(1, 1), seq_len, seq_len]
442
+ """
443
+
444
+ # [seq_len, batch, 3 * hidden_size]
445
+ mixed_raw_layer = self.query_key_value(hidden_states)
446
+
447
+ # [seq_len, batch, 3 * hidden_size] --> [seq_len, batch, num_attention_heads, 3 * hidden_size_per_attention_head]
448
+ new_tensor_shape = mixed_raw_layer.size()[:-1] + (
449
+ self.num_attention_heads_per_partition,
450
+ 3 * self.hidden_size_per_attention_head,
451
+ )
452
+ mixed_raw_layer = mixed_raw_layer.view(*new_tensor_shape)
453
+
454
+ # [seq_len, batch, num_attention_heads, hidden_size_per_attention_head]
455
+ (query_layer, key_layer, value_layer) = self.split_tensor_along_last_dim(mixed_raw_layer, 3)
456
+
457
+ if self.position_encoding_2d:
458
+ q1, q2 = query_layer.chunk(2, dim=(query_layer.ndim - 1))
459
+ k1, k2 = key_layer.chunk(2, dim=(key_layer.ndim - 1))
460
+ cos, sin = self.rotary_emb(q1, seq_len=position_ids.max() + 1)
461
+ position_ids, block_position_ids = position_ids[:, 0, :].transpose(0, 1).contiguous(), \
462
+ position_ids[:, 1, :].transpose(0, 1).contiguous()
463
+ q1, k1 = apply_rotary_pos_emb_index(q1, k1, cos, sin, position_ids)
464
+ q2, k2 = apply_rotary_pos_emb_index(q2, k2, cos, sin, block_position_ids)
465
+ query_layer = torch.concat([q1, q2], dim=(q1.ndim - 1))
466
+ key_layer = torch.concat([k1, k2], dim=(k1.ndim - 1))
467
+ else:
468
+ position_ids = position_ids.transpose(0, 1)
469
+ cos, sin = self.rotary_emb(value_layer, seq_len=position_ids.max() + 1)
470
+ # [seq_len, batch, num_attention_heads, hidden_size_per_attention_head]
471
+ query_layer, key_layer = apply_rotary_pos_emb_index(query_layer, key_layer, cos, sin, position_ids)
472
+
473
+ # [seq_len, batch, hidden_size]
474
+ context_layer, present, attention_probs = attention_fn(
475
+ self=self,
476
+ query_layer=query_layer,
477
+ key_layer=key_layer,
478
+ value_layer=value_layer,
479
+ attention_mask=attention_mask,
480
+ hidden_size_per_partition=self.hidden_size_per_partition,
481
+ layer_id=layer_id,
482
+ layer_past=layer_past,
483
+ use_cache=use_cache
484
+ )
485
+
486
+ output = self.dense(context_layer)
487
+
488
+ outputs = (output, present)
489
+
490
+ if output_attentions:
491
+ outputs += (attention_probs,)
492
+
493
+ return outputs # output, present, attention_probs
494
+
495
+
496
+ class GEGLU(torch.nn.Module):
497
+ def __init__(self):
498
+ super().__init__()
499
+ self.activation_fn = F.gelu
500
+
501
+ def forward(self, x):
502
+ # dim=-1 breaks in jit for pt<1.10
503
+ x1, x2 = x.chunk(2, dim=(x.ndim - 1))
504
+ return x1 * self.activation_fn(x2)
505
+
506
+
507
+ class GLU(torch.nn.Module):
508
+ def __init__(self, hidden_size, inner_hidden_size=None,
509
+ layer_id=None, bias=True, activation_func=gelu, params_dtype=torch.float, empty_init=True):
510
+ super(GLU, self).__init__()
511
+ if empty_init:
512
+ init_method = skip_init
513
+ else:
514
+ init_method = default_init
515
+ self.layer_id = layer_id
516
+ self.activation_func = activation_func
517
+
518
+ # Project to 4h.
519
+ self.hidden_size = hidden_size
520
+ if inner_hidden_size is None:
521
+ inner_hidden_size = 4 * hidden_size
522
+ self.inner_hidden_size = inner_hidden_size
523
+ self.dense_h_to_4h = init_method(
524
+ torch.nn.Linear,
525
+ self.hidden_size,
526
+ self.inner_hidden_size,
527
+ bias=bias,
528
+ dtype=params_dtype,
529
+ )
530
+ # Project back to h.
531
+ self.dense_4h_to_h = init_method(
532
+ torch.nn.Linear,
533
+ self.inner_hidden_size,
534
+ self.hidden_size,
535
+ bias=bias,
536
+ dtype=params_dtype,
537
+ )
538
+
539
+ def forward(self, hidden_states):
540
+ """
541
+ hidden_states: [seq_len, batch, hidden_size]
542
+ """
543
+
544
+ # [seq_len, batch, inner_hidden_size]
545
+ intermediate_parallel = self.dense_h_to_4h(hidden_states)
546
+
547
+ intermediate_parallel = self.activation_func(intermediate_parallel)
548
+
549
+ output = self.dense_4h_to_h(intermediate_parallel)
550
+
551
+ return output
552
+
553
+
554
+ class GLMBlock(torch.nn.Module):
555
+ def __init__(
556
+ self,
557
+ hidden_size,
558
+ num_attention_heads,
559
+ layernorm_epsilon,
560
+ layer_id,
561
+ inner_hidden_size=None,
562
+ hidden_size_per_attention_head=None,
563
+ layernorm=LayerNorm,
564
+ use_bias=True,
565
+ params_dtype=torch.float,
566
+ num_layers=28,
567
+ position_encoding_2d=True,
568
+ empty_init=True
569
+ ):
570
+ super(GLMBlock, self).__init__()
571
+ # Set output layer initialization if not provided.
572
+
573
+ self.layer_id = layer_id
574
+
575
+ # Layernorm on the input data.
576
+ self.input_layernorm = layernorm(hidden_size, eps=layernorm_epsilon)
577
+
578
+ self.position_encoding_2d = position_encoding_2d
579
+
580
+ # Self attention.
581
+ self.attention = SelfAttention(
582
+ hidden_size,
583
+ num_attention_heads,
584
+ layer_id,
585
+ hidden_size_per_attention_head=hidden_size_per_attention_head,
586
+ bias=use_bias,
587
+ params_dtype=params_dtype,
588
+ position_encoding_2d=self.position_encoding_2d,
589
+ empty_init=empty_init
590
+ )
591
+
592
+ # Layernorm on the input data.
593
+ self.post_attention_layernorm = layernorm(hidden_size, eps=layernorm_epsilon)
594
+
595
+ self.num_layers = num_layers
596
+
597
+ # GLU
598
+ self.mlp = GLU(
599
+ hidden_size,
600
+ inner_hidden_size=inner_hidden_size,
601
+ bias=use_bias,
602
+ layer_id=layer_id,
603
+ params_dtype=params_dtype,
604
+ empty_init=empty_init
605
+ )
606
+
607
+ def forward(
608
+ self,
609
+ hidden_states: torch.Tensor,
610
+ position_ids,
611
+ attention_mask: torch.Tensor,
612
+ layer_id,
613
+ layer_past: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
614
+ use_cache: bool = False,
615
+ output_attentions: bool = False,
616
+ ):
617
+ """
618
+ hidden_states: [seq_len, batch, hidden_size]
619
+ attention_mask: [(1, 1), seq_len, seq_len]
620
+ """
621
+
622
+ # Layer norm at the begining of the transformer layer.
623
+ # [seq_len, batch, hidden_size]
624
+ attention_input = self.input_layernorm(hidden_states)
625
+
626
+ # Self attention.
627
+ attention_outputs = self.attention(
628
+ attention_input,
629
+ position_ids,
630
+ attention_mask=attention_mask,
631
+ layer_id=layer_id,
632
+ layer_past=layer_past,
633
+ use_cache=use_cache,
634
+ output_attentions=output_attentions
635
+ )
636
+
637
+ attention_output = attention_outputs[0]
638
+
639
+ outputs = attention_outputs[1:]
640
+
641
+ # Residual connection.
642
+ alpha = (2 * self.num_layers) ** 0.5
643
+ hidden_states = attention_input * alpha + attention_output
644
+
645
+ mlp_input = self.post_attention_layernorm(hidden_states)
646
+
647
+ # MLP.
648
+ mlp_output = self.mlp(mlp_input)
649
+
650
+ # Second residual connection.
651
+ output = mlp_input * alpha + mlp_output
652
+
653
+ if use_cache:
654
+ outputs = (output,) + outputs
655
+ else:
656
+ outputs = (output,) + outputs[1:]
657
+
658
+ return outputs # hidden_states, present, attentions
659
+
660
+
661
+ class ChatGLMPreTrainedModel(PreTrainedModel):
662
+ """
663
+ An abstract class to handle weights initialization and
664
+ a simple interface for downloading and loading pretrained models.
665
+ """
666
+
667
+ is_parallelizable = False
668
+ supports_gradient_checkpointing = True
669
+ config_class = ChatGLMConfig
670
+ base_model_prefix = "transformer"
671
+ _no_split_modules = ["GLMBlock"]
672
+
673
+ def __init__(self, *inputs, **kwargs):
674
+ super().__init__(*inputs, **kwargs)
675
+
676
+ def _init_weights(self, module: nn.Module):
677
+ """Initialize the weights."""
678
+ return
679
+
680
+ def get_masks(self, input_ids, device):
681
+ batch_size, seq_length = input_ids.shape
682
+ context_lengths = [seq.tolist().index(self.config.bos_token_id) for seq in input_ids]
683
+ attention_mask = torch.ones((batch_size, seq_length, seq_length), device=device)
684
+ attention_mask.tril_()
685
+ for i, context_length in enumerate(context_lengths):
686
+ attention_mask[i, :, :context_length] = 1
687
+ attention_mask.unsqueeze_(1)
688
+ attention_mask = (attention_mask < 0.5).bool()
689
+
690
+ return attention_mask
691
+
692
+ def get_position_ids(self, input_ids, mask_positions, device, use_gmasks=None):
693
+ batch_size, seq_length = input_ids.shape
694
+ if use_gmasks is None:
695
+ use_gmasks = [False] * batch_size
696
+ context_lengths = [seq.tolist().index(self.config.bos_token_id) for seq in input_ids]
697
+ if self.position_encoding_2d:
698
+ position_ids = torch.arange(seq_length, dtype=torch.long, device=device).unsqueeze(0).repeat(batch_size, 1)
699
+ for i, context_length in enumerate(context_lengths):
700
+ position_ids[i, context_length:] = mask_positions[i]
701
+ block_position_ids = [torch.cat((
702
+ torch.zeros(context_length, dtype=torch.long, device=device),
703
+ torch.arange(seq_length - context_length, dtype=torch.long, device=device) + 1
704
+ )) for context_length in context_lengths]
705
+ block_position_ids = torch.stack(block_position_ids, dim=0)
706
+ position_ids = torch.stack((position_ids, block_position_ids), dim=1)
707
+ else:
708
+ position_ids = torch.arange(seq_length, dtype=torch.long, device=device).unsqueeze(0).repeat(batch_size, 1)
709
+ for i, context_length in enumerate(context_lengths):
710
+ if not use_gmasks[i]:
711
+ position_ids[context_length:] = mask_positions[i]
712
+
713
+ return position_ids
714
+
715
+ def _set_gradient_checkpointing(self, module, value=False):
716
+ if isinstance(module, ChatGLMModel):
717
+ module.gradient_checkpointing = value
718
+
719
+
720
+ CHATGLM_6B_START_DOCSTRING = r"""
721
+ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class.
722
+ Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general
723
+ usage and behavior.
724
+
725
+ Parameters:
726
+ config ([`~ChatGLM6BConfig`]): Model configuration class with all the parameters of the model.
727
+ Initializing with a config file does not load the weights associated with the model, only the configuration.
728
+ Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights.
729
+ """
730
+
731
+ CHATGLM_6B_INPUTS_DOCSTRING = r"""
732
+ Args:
733
+ input_ids (`torch.LongTensor` of shape `({0})`):
734
+ Indices of input sequence tokens in the vocabulary.
735
+
736
+ Indices can be obtained using [`ChatGLM6BTokenizer`].
737
+ See [`PreTrainedTokenizer.encode`] and
738
+ [`PreTrainedTokenizer.__call__`] for details.
739
+
740
+ [What are input IDs?](../glossary#input-ids)
741
+ attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*):
742
+ Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
743
+
744
+ - 1 for tokens that are **not masked**,
745
+ - 0 for tokens that are **masked**.
746
+
747
+ [What are attention masks?](../glossary#attention-mask)
748
+ token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*):
749
+ Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
750
+
751
+ - 0 corresponds to a *sentence A* token,
752
+ - 1 corresponds to a *sentence B* token.
753
+
754
+ [What are token type IDs?](../glossary#token-type-ids)
755
+ position_ids (`torch.LongTensor` of shape `({0})`, *optional*):
756
+ Indices of positions of each input sequence tokens in the position embeddings.
757
+ Selected in the range `[0, config.max_position_embeddings - 1]`.
758
+
759
+ [What are position IDs?](../glossary#position-ids)
760
+ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*):
761
+ Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
762
+
763
+ - 1 indicates the head is **not masked**,
764
+ - 0 indicates the head is **masked**.
765
+
766
+ inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*):
767
+ Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation.
768
+ This is useful if you want more control over how to convert *input_ids* indices into associated vectors
769
+ than the model's internal embedding lookup matrix.
770
+ output_attentions (`bool`, *optional*):
771
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
772
+ tensors for more detail.
773
+ output_hidden_states (`bool`, *optional*):
774
+ Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
775
+ more detail.
776
+ return_dict (`bool`, *optional*):
777
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
778
+ """
779
+
780
+
781
+ @add_start_docstrings(
782
+ "The bare ChatGLM-6B Model transformer outputting raw hidden-states without any specific head on top.",
783
+ CHATGLM_6B_START_DOCSTRING,
784
+ )
785
+ class ChatGLMModel(ChatGLMPreTrainedModel):
786
+ """
787
+
788
+ The model can behave as an encoder (with only self-attention) as well
789
+ as a decoder, in which case a layer of cross-attention is added between
790
+ the self-attention layers, following the architecture described in [Attention is
791
+ all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani,
792
+ Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.
793
+
794
+ To behave as an decoder the model needs to be initialized with the
795
+ `is_decoder` argument of the configuration set to `True`.
796
+ To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder`
797
+ argument and `add_cross_attention` set to `True`; an
798
+ `encoder_hidden_states` is then expected as an input to the forward pass.
799
+ """
800
+
801
+ def __init__(self, config: ChatGLMConfig, empty_init=True):
802
+ super().__init__(config)
803
+ if empty_init:
804
+ init_method = skip_init
805
+ else:
806
+ init_method = default_init
807
+ # recording parameters
808
+ self.max_sequence_length = config.max_sequence_length
809
+ self.hidden_size = config.hidden_size
810
+ self.params_dtype = torch.half
811
+ self.num_attention_heads = config.num_attention_heads
812
+ self.vocab_size = config.vocab_size
813
+ self.num_layers = config.num_layers
814
+ self.layernorm_epsilon = config.layernorm_epsilon
815
+ self.inner_hidden_size = config.inner_hidden_size
816
+ self.hidden_size_per_attention_head = self.hidden_size // self.num_attention_heads
817
+ self.position_encoding_2d = config.position_encoding_2d
818
+ self.pre_seq_len = config.pre_seq_len
819
+ self.prefix_projection = config.prefix_projection
820
+
821
+ self.word_embeddings = init_method(
822
+ torch.nn.Embedding,
823
+ num_embeddings=self.vocab_size, embedding_dim=self.hidden_size,
824
+ dtype=self.params_dtype
825
+ )
826
+ self.gradient_checkpointing = False
827
+
828
+ def get_layer(layer_id):
829
+ return GLMBlock(
830
+ self.hidden_size,
831
+ self.num_attention_heads,
832
+ self.layernorm_epsilon,
833
+ layer_id,
834
+ inner_hidden_size=self.inner_hidden_size,
835
+ hidden_size_per_attention_head=self.hidden_size_per_attention_head,
836
+ layernorm=LayerNorm,
837
+ use_bias=True,
838
+ params_dtype=self.params_dtype,
839
+ position_encoding_2d=self.position_encoding_2d,
840
+ empty_init=empty_init
841
+ )
842
+
843
+ self.layers = torch.nn.ModuleList(
844
+ [get_layer(layer_id) for layer_id in range(self.num_layers)]
845
+ )
846
+
847
+ # Final layer norm before output.
848
+ self.final_layernorm = LayerNorm(self.hidden_size, eps=self.layernorm_epsilon)
849
+
850
+ if self.pre_seq_len is not None:
851
+ for param in self.parameters():
852
+ param.requires_grad = False
853
+ self.prefix_tokens = torch.arange(self.pre_seq_len).long()
854
+ self.prefix_encoder = PrefixEncoder(config)
855
+ self.dropout = torch.nn.Dropout(0.1)
856
+
857
+ # total_params = sum(p.numel() for p in self.parameters())
858
+ # trainable_params = sum(p.numel() for p in self.parameters() if p.requires_grad)
859
+ # print("Using p-tuning v2: # trainable_params = {} / {}".format(trainable_params, total_params))
860
+
861
+ def get_input_embeddings(self):
862
+ return self.word_embeddings
863
+
864
+ def set_input_embeddings(self, new_embeddings: torch.Tensor):
865
+ self.word_embeddings = new_embeddings
866
+
867
+ def get_prompt(self, batch_size, device, dtype=torch.half):
868
+ prefix_tokens = self.prefix_tokens.unsqueeze(0).expand(batch_size, -1).to(device)
869
+ past_key_values = self.prefix_encoder(prefix_tokens).type(dtype)
870
+ past_key_values = past_key_values.view(
871
+ batch_size,
872
+ self.pre_seq_len,
873
+ self.num_layers * 2,
874
+ self.num_attention_heads,
875
+ self.hidden_size // self.num_attention_heads
876
+ )
877
+ # seq_len, b, nh, hidden_size
878
+ past_key_values = self.dropout(past_key_values)
879
+ past_key_values = past_key_values.permute([2, 1, 0, 3, 4]).split(2)
880
+ # past_key_values = [(v[0], v[1]) for v in past_key_values]
881
+ return past_key_values
882
+
883
+ @add_start_docstrings_to_model_forward(CHATGLM_6B_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
884
+ @add_code_sample_docstrings(
885
+ checkpoint=_CHECKPOINT_FOR_DOC,
886
+ output_type=BaseModelOutputWithPastAndCrossAttentions,
887
+ config_class=_CONFIG_FOR_DOC,
888
+ )
889
+ def forward(
890
+ self,
891
+ input_ids: Optional[torch.LongTensor] = None,
892
+ position_ids: Optional[torch.LongTensor] = None,
893
+ attention_mask: Optional[torch.Tensor] = None,
894
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
895
+ inputs_embeds: Optional[torch.LongTensor] = None,
896
+ use_cache: Optional[bool] = None,
897
+ output_attentions: Optional[bool] = None,
898
+ output_hidden_states: Optional[bool] = None,
899
+ return_dict: Optional[bool] = None,
900
+ ) -> Union[Tuple[torch.Tensor, ...], BaseModelOutputWithPast]:
901
+
902
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
903
+ output_hidden_states = (
904
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
905
+ )
906
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
907
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
908
+
909
+ if self.gradient_checkpointing and self.training:
910
+ if use_cache:
911
+ logger.warning_once(
912
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
913
+ )
914
+ use_cache = False
915
+
916
+ if input_ids is not None and inputs_embeds is not None:
917
+ raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
918
+ elif input_ids is not None:
919
+ batch_size, seq_length = input_ids.shape[:2]
920
+ elif inputs_embeds is not None:
921
+ batch_size, seq_length, _ = inputs_embeds.shape[:2]
922
+ else:
923
+ raise ValueError("You have to specify either input_ids or inputs_embeds")
924
+
925
+ if inputs_embeds is None:
926
+ inputs_embeds = self.word_embeddings(input_ids)
927
+
928
+ if past_key_values is None:
929
+ if self.pre_seq_len is not None:
930
+ past_key_values = self.get_prompt(batch_size=input_ids.shape[0], device=input_ids.device,
931
+ dtype=inputs_embeds.dtype)
932
+ else:
933
+ past_key_values = tuple([None] * len(self.layers))
934
+
935
+ if attention_mask is None:
936
+ attention_mask = self.get_masks(
937
+ input_ids,
938
+ device=input_ids.device
939
+ )
940
+
941
+
942
+ if position_ids is None:
943
+ MASK, gMASK = self.config.mask_token_id, self.config.gmask_token_id
944
+ seqs = input_ids.tolist()
945
+
946
+ mask_positions, use_gmasks = [], []
947
+ for seq in seqs:
948
+ mask_token = gMASK if gMASK in seq else MASK
949
+ use_gmask = mask_token == gMASK
950
+ mask_positions.append(seq.index(mask_token))
951
+ use_gmasks.append(use_gmask)
952
+
953
+ position_ids = self.get_position_ids(
954
+ input_ids,
955
+ mask_positions=mask_positions,
956
+ device=input_ids.device,
957
+ use_gmasks=use_gmasks
958
+ )
959
+
960
+ if self.pre_seq_len is not None and attention_mask is not None:
961
+ prefix_attention_mask = torch.ones(batch_size, 1, input_ids.size(-1), self.pre_seq_len).to(
962
+ attention_mask.device)
963
+ prefix_attention_mask = (prefix_attention_mask < 0.5).bool()
964
+ attention_mask = torch.cat((prefix_attention_mask, attention_mask), dim=3)
965
+
966
+ # [seq_len, batch, hidden_size]
967
+ hidden_states = inputs_embeds.transpose(0, 1)
968
+
969
+ presents = () if use_cache else None
970
+ all_self_attentions = () if output_attentions else None
971
+ all_hidden_states = () if output_hidden_states else None
972
+
973
+ if attention_mask is None:
974
+ attention_mask = torch.zeros(1, 1, device=input_ids.device).bool()
975
+
976
+ else:
977
+ attention_mask = attention_mask.to(input_ids.device)
978
+
979
+ for i, layer in enumerate(self.layers):
980
+
981
+ if output_hidden_states:
982
+ all_hidden_states = all_hidden_states + (hidden_states,)
983
+ layer_past = past_key_values[i]
984
+
985
+ if self.gradient_checkpointing and self.training:
986
+ layer_ret = torch.utils.checkpoint.checkpoint(
987
+ layer,
988
+ hidden_states,
989
+ position_ids,
990
+ attention_mask,
991
+ torch.tensor(i),
992
+ layer_past,
993
+ use_cache,
994
+ output_attentions
995
+ )
996
+ else:
997
+ layer_ret = layer(
998
+ hidden_states,
999
+ position_ids=position_ids,
1000
+ attention_mask=attention_mask,
1001
+ layer_id=torch.tensor(i),
1002
+ layer_past=layer_past,
1003
+ use_cache=use_cache,
1004
+ output_attentions=output_attentions
1005
+ )
1006
+
1007
+ hidden_states = layer_ret[0]
1008
+
1009
+ if use_cache:
1010
+ presents = presents + (layer_ret[1],)
1011
+
1012
+ if output_attentions:
1013
+ all_self_attentions = all_self_attentions + (layer_ret[2 if use_cache else 1],)
1014
+
1015
+ # Final layer norm.
1016
+ hidden_states = self.final_layernorm(hidden_states)
1017
+
1018
+ if output_hidden_states:
1019
+ all_hidden_states = all_hidden_states + (hidden_states,)
1020
+
1021
+ if not return_dict:
1022
+ return tuple(v for v in [hidden_states, presents, all_hidden_states, all_self_attentions] if v is not None)
1023
+
1024
+ return BaseModelOutputWithPast(
1025
+ last_hidden_state=hidden_states,
1026
+ past_key_values=presents,
1027
+ hidden_states=all_hidden_states,
1028
+ attentions=all_self_attentions,
1029
+ )
1030
+
1031
+
1032
+ class ChatGLMForConditionalGeneration(ChatGLMPreTrainedModel):
1033
+ def __init__(self, config: ChatGLMConfig, empty_init=True):
1034
+ super().__init__(config)
1035
+ if empty_init:
1036
+ init_method = skip_init
1037
+ else:
1038
+ init_method = default_init
1039
+
1040
+ # self.hidden_size = config.hidden_size
1041
+ # self.params_dtype = torch.half
1042
+ # self.vocab_size = config.vocab_size
1043
+ self.max_sequence_length = config.max_sequence_length
1044
+
1045
+ self.position_encoding_2d = config.position_encoding_2d
1046
+
1047
+ self.transformer = ChatGLMModel(config, empty_init=empty_init)
1048
+
1049
+ self.lm_head = init_method(
1050
+ nn.Linear,
1051
+ config.hidden_size,
1052
+ config.vocab_size,
1053
+ bias=False,
1054
+ dtype=torch.half
1055
+ )
1056
+
1057
+ self.config = config
1058
+
1059
+ self.quantized = False
1060
+
1061
+ if self.config.quantization_bit:
1062
+ self.quantize(self.config.quantization_bit, self.config.quantization_embeddings, use_quantization_cache=True, empty_init=True)
1063
+
1064
+ def get_output_embeddings(self):
1065
+ return self.lm_head
1066
+
1067
+ def set_output_embeddings(self, new_embeddings):
1068
+ self.lm_head = new_embeddings
1069
+
1070
+ def _update_model_kwargs_for_generation(
1071
+ self,
1072
+ outputs: ModelOutput,
1073
+ model_kwargs: Dict[str, Any],
1074
+ is_encoder_decoder: bool = False,
1075
+ standardize_cache_format: bool = False,
1076
+ ) -> Dict[str, Any]:
1077
+ # update past_key_values
1078
+ model_kwargs["past_key_values"] = self._extract_past_from_model_output(
1079
+ outputs, standardize_cache_format=standardize_cache_format
1080
+ )
1081
+
1082
+ # update attention mask
1083
+ if "attention_mask" in model_kwargs:
1084
+ attention_mask = model_kwargs["attention_mask"]
1085
+ if attention_mask is not None and attention_mask.dtype == torch.bool:
1086
+ attention_mask = torch.cat(
1087
+ [attention_mask, attention_mask.new_ones((*attention_mask.shape[:3], 1))], dim=3)
1088
+ new_attention_mask = attention_mask[:, :, -1:].clone()
1089
+ new_attention_mask[..., -1] = False
1090
+ model_kwargs["attention_mask"] = torch.cat(
1091
+ [attention_mask, new_attention_mask], dim=2
1092
+ )
1093
+
1094
+ # update position ids
1095
+ if "position_ids" in model_kwargs:
1096
+ position_ids = model_kwargs["position_ids"]
1097
+ new_position_id = position_ids[..., -1:].clone()
1098
+ new_position_id[:, 1, :] += 1
1099
+ model_kwargs["position_ids"] = torch.cat(
1100
+ [position_ids, new_position_id], dim=-1
1101
+ )
1102
+
1103
+ return model_kwargs
1104
+
1105
+ def prepare_inputs_for_generation(
1106
+ self,
1107
+ input_ids: torch.LongTensor,
1108
+ past: Optional[torch.Tensor] = None,
1109
+ past_key_values: Optional[torch.Tensor] = None,
1110
+ attention_mask: Optional[torch.Tensor] = None,
1111
+ position_ids: Optional[torch.Tensor] = None,
1112
+ **kwargs
1113
+ ) -> dict:
1114
+ batch_size, seq_length = input_ids.shape
1115
+ MASK, gMASK = self.config.mask_token_id, self.config.gmask_token_id
1116
+ seqs = input_ids.tolist()
1117
+ mask_positions, use_gmasks = [], []
1118
+ for seq in seqs:
1119
+ mask_token = gMASK if gMASK in seq else MASK
1120
+ use_gmask = mask_token == gMASK
1121
+ mask_positions.append(seq.index(mask_token))
1122
+ use_gmasks.append(use_gmask)
1123
+
1124
+ # only last token for input_ids if past is not None
1125
+ if past is not None or past_key_values is not None:
1126
+ last_token = input_ids[:, -1].unsqueeze(-1)
1127
+ if attention_mask is not None and attention_mask.dtype == torch.bool:
1128
+ attention_mask = attention_mask[:, :, -1:]
1129
+ else:
1130
+ attention_mask = None
1131
+ if position_ids is not None:
1132
+ position_ids = position_ids[..., -1:]
1133
+ else:
1134
+ context_lengths = [seq.index(self.config.bos_token_id) for seq in seqs]
1135
+ if self.position_encoding_2d:
1136
+ position_ids = torch.tensor(
1137
+ [[mask_position, seq_length - context_length] for mask_position, context_length in
1138
+ zip(mask_positions, context_lengths)], dtype=torch.long, device=input_ids.device).unsqueeze(-1)
1139
+ else:
1140
+ position_ids = torch.tensor([mask_position for mask_position in mask_positions], dtype=torch.long,
1141
+ device=input_ids.device).unsqueeze(-1)
1142
+
1143
+ if past is None:
1144
+ past = past_key_values
1145
+ return {
1146
+ "input_ids": last_token,
1147
+ "past_key_values": past,
1148
+ "position_ids": position_ids,
1149
+ "attention_mask": attention_mask
1150
+ }
1151
+ else:
1152
+ if attention_mask is not None and attention_mask.dtype != torch.bool:
1153
+ logger.warning_once(f"The dtype of attention mask ({attention_mask.dtype}) is not bool")
1154
+ attention_mask = None
1155
+ if attention_mask is None:
1156
+ attention_mask = self.get_masks(
1157
+ input_ids,
1158
+ device=input_ids.device
1159
+ )
1160
+ if position_ids is None:
1161
+ position_ids = self.get_position_ids(
1162
+ input_ids,
1163
+ device=input_ids.device,
1164
+ mask_positions=mask_positions,
1165
+ use_gmasks=use_gmasks
1166
+ )
1167
+
1168
+ return {
1169
+ "input_ids": input_ids,
1170
+ "past_key_values": past,
1171
+ "position_ids": position_ids,
1172
+ "attention_mask": attention_mask
1173
+ }
1174
+
1175
+ def forward(
1176
+ self,
1177
+ input_ids: Optional[torch.Tensor] = None,
1178
+ position_ids: Optional[torch.Tensor] = None,
1179
+ attention_mask: Optional[torch.Tensor] = None,
1180
+ past_key_values: Optional[Tuple[torch.FloatTensor]] = None,
1181
+ inputs_embeds: Optional[torch.Tensor] = None,
1182
+ labels: Optional[torch.Tensor] = None,
1183
+ use_cache: Optional[bool] = None,
1184
+ output_attentions: Optional[bool] = None,
1185
+ output_hidden_states: Optional[bool] = None,
1186
+ return_dict: Optional[bool] = None,
1187
+ ):
1188
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
1189
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1190
+
1191
+ transformer_outputs = self.transformer(
1192
+ input_ids=input_ids,
1193
+ position_ids=position_ids,
1194
+ attention_mask=attention_mask,
1195
+ past_key_values=past_key_values,
1196
+ inputs_embeds=inputs_embeds,
1197
+ use_cache=use_cache,
1198
+ output_attentions=output_attentions,
1199
+ output_hidden_states=output_hidden_states,
1200
+ return_dict=return_dict,
1201
+ )
1202
+
1203
+ hidden_states = transformer_outputs[0]
1204
+
1205
+ lm_logits = self.lm_head(hidden_states).permute(1, 0, 2).contiguous()
1206
+
1207
+ loss = None
1208
+ if labels is not None:
1209
+ lm_logits = lm_logits.to(torch.float32)
1210
+
1211
+ # Shift so that tokens < n predict n
1212
+ shift_logits = lm_logits[..., :-1, :].contiguous()
1213
+ shift_labels = labels[..., 1:].contiguous()
1214
+ # Flatten the tokens
1215
+ loss_fct = CrossEntropyLoss(ignore_index=-100)
1216
+ loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
1217
+
1218
+ lm_logits = lm_logits.to(hidden_states.dtype)
1219
+ loss = loss.to(hidden_states.dtype)
1220
+
1221
+ if not return_dict:
1222
+ output = (lm_logits,) + transformer_outputs[1:]
1223
+ return ((loss,) + output) if loss is not None else output
1224
+
1225
+ return CausalLMOutputWithPast(
1226
+ loss=loss,
1227
+ logits=lm_logits,
1228
+ past_key_values=transformer_outputs.past_key_values,
1229
+ hidden_states=transformer_outputs.hidden_states,
1230
+ attentions=transformer_outputs.attentions,
1231
+ )
1232
+
1233
+ @staticmethod
1234
+ def _reorder_cache(
1235
+ past: Tuple[Tuple[torch.Tensor, torch.Tensor], ...], beam_idx: torch.LongTensor
1236
+ ) -> Tuple[Tuple[torch.Tensor, torch.Tensor], ...]:
1237
+ """
1238
+ This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or
1239
+ [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct
1240
+ beam_idx at every generation step.
1241
+
1242
+ Output shares the same memory storage as `past`.
1243
+ """
1244
+ return tuple(
1245
+ (
1246
+ layer_past[0].index_select(1, beam_idx.to(layer_past[0].device)),
1247
+ layer_past[1].index_select(1, beam_idx.to(layer_past[1].device)),
1248
+ )
1249
+ for layer_past in past
1250
+ )
1251
+
1252
+ def process_response(self, response):
1253
+ response = response.strip()
1254
+ response = response.replace("[[训练时间]]", "2023年")
1255
+ punkts = [
1256
+ [",", ","],
1257
+ ["!", "!"],
1258
+ [":", ":"],
1259
+ [";", ";"],
1260
+ ["\?", "?"],
1261
+ ]
1262
+ for item in punkts:
1263
+ response = re.sub(r"([\u4e00-\u9fff])%s" % item[0], r"\1%s" % item[1], response)
1264
+ response = re.sub(r"%s([\u4e00-\u9fff])" % item[0], r"%s\1" % item[1], response)
1265
+ return response
1266
+
1267
+ @torch.no_grad()
1268
+ def chat(self, tokenizer, query: str, history: List[Tuple[str, str]] = None, max_length: int = 2048, num_beams=1,
1269
+ do_sample=True, top_p=0.7, temperature=0.95, logits_processor=None, **kwargs):
1270
+ if history is None:
1271
+ history = []
1272
+ if logits_processor is None:
1273
+ logits_processor = LogitsProcessorList()
1274
+ logits_processor.append(InvalidScoreLogitsProcessor())
1275
+ gen_kwargs = {"max_length": max_length, "num_beams": num_beams, "do_sample": do_sample, "top_p": top_p,
1276
+ "temperature": temperature, "logits_processor": logits_processor, **kwargs}
1277
+ if not history:
1278
+ prompt = query
1279
+ else:
1280
+ prompt = ""
1281
+ for i, (old_query, response) in enumerate(history):
1282
+ prompt += "[Round {}]\n问:{}\n答:{}\n".format(i, old_query, response)
1283
+ prompt += "[Round {}]\n问:{}\n答:".format(len(history), query)
1284
+ inputs = tokenizer([prompt], return_tensors="pt")
1285
+ inputs = inputs.to(self.device)
1286
+ outputs = self.generate(**inputs, **gen_kwargs)
1287
+ outputs = outputs.tolist()[0][len(inputs["input_ids"][0]):]
1288
+ response = tokenizer.decode(outputs)
1289
+ response = self.process_response(response)
1290
+ history = history + [(query, response)]
1291
+ return response, history
1292
+
1293
+ @torch.no_grad()
1294
+ def stream_chat(self, tokenizer, query: str, history: List[Tuple[str, str]] = None, max_length: int = 2048,
1295
+ do_sample=True, top_p=0.7, temperature=0.95, logits_processor=None, **kwargs):
1296
+ if history is None:
1297
+ history = []
1298
+ if logits_processor is None:
1299
+ logits_processor = LogitsProcessorList()
1300
+ logits_processor.append(InvalidScoreLogitsProcessor())
1301
+ gen_kwargs = {"max_length": max_length, "do_sample": do_sample, "top_p": top_p,
1302
+ "temperature": temperature, "logits_processor": logits_processor, **kwargs}
1303
+ if not history:
1304
+ prompt = query
1305
+ else:
1306
+ prompt = ""
1307
+ for i, (old_query, response) in enumerate(history):
1308
+ prompt += "[Round {}]\n问:{}\n答:{}\n".format(i, old_query, response)
1309
+ prompt += "[Round {}]\n问:{}\n答:".format(len(history), query)
1310
+ inputs = tokenizer([prompt], return_tensors="pt")
1311
+ inputs = inputs.to(self.device)
1312
+ for outputs in self.stream_generate(**inputs, **gen_kwargs):
1313
+ outputs = outputs.tolist()[0][len(inputs["input_ids"][0]):]
1314
+ response = tokenizer.decode(outputs)
1315
+ response = self.process_response(response)
1316
+ new_history = history + [(query, response)]
1317
+ yield response, new_history
1318
+
1319
+ @torch.no_grad()
1320
+ def stream_generate(
1321
+ self,
1322
+ input_ids,
1323
+ generation_config: Optional[GenerationConfig] = None,
1324
+ logits_processor: Optional[LogitsProcessorList] = None,
1325
+ stopping_criteria: Optional[StoppingCriteriaList] = None,
1326
+ prefix_allowed_tokens_fn: Optional[Callable[[int, torch.Tensor], List[int]]] = None,
1327
+ **kwargs,
1328
+ ):
1329
+ batch_size, input_ids_seq_length = input_ids.shape[0], input_ids.shape[-1]
1330
+
1331
+ if generation_config is None:
1332
+ generation_config = self.generation_config
1333
+ generation_config = copy.deepcopy(generation_config)
1334
+ model_kwargs = generation_config.update(**kwargs)
1335
+ bos_token_id, eos_token_id = generation_config.bos_token_id, generation_config.eos_token_id
1336
+
1337
+ if isinstance(eos_token_id, int):
1338
+ eos_token_id = [eos_token_id]
1339
+
1340
+ has_default_max_length = kwargs.get("max_length") is None and generation_config.max_length is not None
1341
+ if has_default_max_length and generation_config.max_new_tokens is None:
1342
+ warnings.warn(
1343
+ f"Using `max_length`'s default ({generation_config.max_length}) to control the generation length. "
1344
+ "This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we"
1345
+ " recommend using `max_new_tokens` to control the maximum length of the generation.",
1346
+ UserWarning,
1347
+ )
1348
+ elif generation_config.max_new_tokens is not None:
1349
+ generation_config.max_length = generation_config.max_new_tokens + input_ids_seq_length
1350
+ if not has_default_max_length:
1351
+ logger.warn(
1352
+ f"Both `max_new_tokens` (={generation_config.max_new_tokens}) and `max_length`(="
1353
+ f"{generation_config.max_length}) seem to have been set. `max_new_tokens` will take precedence. "
1354
+ "Please refer to the documentation for more information. "
1355
+ "(https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)",
1356
+ UserWarning,
1357
+ )
1358
+
1359
+ if input_ids_seq_length >= generation_config.max_length:
1360
+ input_ids_string = "decoder_input_ids" if self.config.is_encoder_decoder else "input_ids"
1361
+ logger.warning(
1362
+ f"Input length of {input_ids_string} is {input_ids_seq_length}, but `max_length` is set to"
1363
+ f" {generation_config.max_length}. This can lead to unexpected behavior. You should consider"
1364
+ " increasing `max_new_tokens`."
1365
+ )
1366
+
1367
+ # 2. Set generation parameters if not already defined
1368
+ logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()
1369
+ stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()
1370
+
1371
+ logits_processor = self._get_logits_processor(
1372
+ generation_config=generation_config,
1373
+ input_ids_seq_length=input_ids_seq_length,
1374
+ encoder_input_ids=input_ids,
1375
+ prefix_allowed_tokens_fn=prefix_allowed_tokens_fn,
1376
+ logits_processor=logits_processor,
1377
+ )
1378
+
1379
+ stopping_criteria = self._get_stopping_criteria(
1380
+ generation_config=generation_config, stopping_criteria=stopping_criteria
1381
+ )
1382
+ logits_warper = self._get_logits_warper(generation_config)
1383
+
1384
+ unfinished_sequences = input_ids.new(input_ids.shape[0]).fill_(1)
1385
+ scores = None
1386
+ while True:
1387
+ model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
1388
+ # forward pass to get next token
1389
+ outputs = self(
1390
+ **model_inputs,
1391
+ return_dict=True,
1392
+ output_attentions=False,
1393
+ output_hidden_states=False,
1394
+ )
1395
+
1396
+ next_token_logits = outputs.logits[:, -1, :]
1397
+
1398
+ # pre-process distribution
1399
+ next_token_scores = logits_processor(input_ids, next_token_logits)
1400
+ next_token_scores = logits_warper(input_ids, next_token_scores)
1401
+
1402
+ # sample
1403
+ probs = nn.functional.softmax(next_token_scores, dim=-1)
1404
+ if generation_config.do_sample:
1405
+ next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
1406
+ else:
1407
+ next_tokens = torch.argmax(probs, dim=-1)
1408
+
1409
+ # update generated ids, model inputs, and length for next step
1410
+ input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)
1411
+ model_kwargs = self._update_model_kwargs_for_generation(
1412
+ outputs, model_kwargs, is_encoder_decoder=self.config.is_encoder_decoder
1413
+ )
1414
+ unfinished_sequences = unfinished_sequences.mul((sum(next_tokens != i for i in eos_token_id)).long())
1415
+
1416
+ # stop when each sentence is finished, or if we exceed the maximum length
1417
+ if unfinished_sequences.max() == 0 or stopping_criteria(input_ids, scores):
1418
+ break
1419
+ yield input_ids
1420
+
1421
+ def quantize(self, bits: int, quantize_embeddings=False, use_quantization_cache=False, empty_init=False, **kwargs):
1422
+ if bits == 0:
1423
+ return
1424
+
1425
+ from .quantization import quantize, QuantizedEmbedding, QuantizedLinear, load_cpu_kernel
1426
+
1427
+ if self.quantized:
1428
+ if self.device == torch.device("cpu"):
1429
+ logger.info("Already quantized, reloading cpu kernel.")
1430
+ load_cpu_kernel(**kwargs)
1431
+ else:
1432
+ logger.info("Already quantized.")
1433
+ return self
1434
+
1435
+ self.quantized = True
1436
+
1437
+ self.config.quantization_bit = bits
1438
+ self.config.quantization_embeddings = quantize_embeddings
1439
+
1440
+ self.transformer = quantize(self.transformer, bits, use_quantization_cache=use_quantization_cache, empty_init=empty_init, **kwargs)
1441
+
1442
+ if self.device == torch.device("cpu"):
1443
+ dtype = torch.float32
1444
+ else:
1445
+ dtype = torch.half
1446
+
1447
+ if quantize_embeddings:
1448
+ logger.info("Applying quantization to embeddings")
1449
+ self.transformer.word_embeddings = QuantizedEmbedding(
1450
+ weight_bit_width=bits,
1451
+ weight_tensor=self.transformer.word_embeddings.weight.to(self.device),
1452
+ num_embeddings=self.transformer.word_embeddings.num_embeddings,
1453
+ embedding_dim=self.transformer.word_embeddings.embedding_dim,
1454
+ dtype=dtype,
1455
+ empty_init=empty_init,
1456
+ device=self.transformer.word_embeddings.weight.device,
1457
+ )
1458
+ self.lm_head = QuantizedLinear(
1459
+ weight_bit_width=bits,
1460
+ weight_tensor=self.lm_head.weight.to(self.device),
1461
+ bias_tensor=None,
1462
+ in_features=self.lm_head.in_features,
1463
+ out_features=self.lm_head.out_features,
1464
+ bias=False,
1465
+ quantized_weight=self.transformer.word_embeddings.weight,
1466
+ quantized_weight_scale=self.transformer.word_embeddings.weight_scale,
1467
+ dtype=dtype,
1468
+ empty_init=empty_init,
1469
+ device=self.lm_head.weight.device,
1470
+ )
1471
+
1472
+ return self
quantization.py ADDED
@@ -0,0 +1,515 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from torch.nn import Linear, Embedding
2
+ from torch.nn.parameter import Parameter
3
+ import torch.nn.functional as F
4
+
5
+ import os
6
+ import bz2
7
+ import torch
8
+ import base64
9
+ import ctypes
10
+ from transformers.utils import logging
11
+
12
+ from typing import List
13
+ from functools import partial
14
+
15
+ logger = logging.get_logger(__name__)
16
+
17
+ try:
18
+ from cpm_kernels.kernels.base import LazyKernelCModule, KernelFunction, round_up
19
+
20
+ class Kernel:
21
+ def __init__(self, code: bytes, function_names: List[str]):
22
+ self.code = code
23
+ self._function_names = function_names
24
+ self._cmodule = LazyKernelCModule(self.code)
25
+
26
+ for name in self._function_names:
27
+ setattr(self, name, KernelFunction(self._cmodule, name))
28
+
29
+ quantization_code = "$QlpoOTFBWSZTWU9yuJUAQHN//////////f/n/8/n///n//bt4dTidcVx8X3V9FV/92/v4B7/AD5FBQFAAAChSgKpFCFAFVSigUAAAEKhSgUUqgFBKigqVREQAABQBQIANDTTIGI00BkZBkNGE0A0BkBkGQGRkaNAaAGQNBoGgDIAAYIGTI0DQAQAaGmmQMRpoDIyDIaMJoBoDIDIMgMjI0aA0AMgaDQNAGQAAwQMmRoGgAgA0NNMgYjTQGRkGQ0YTQDQGQGQZAZGRo0BoAZA0GgaAMgABggZMjQNABABoaaZAxGmgMjIMhowmgGgMgMgyAyMjRoDQAyBoNA0AZAADBAyZGgaAAmqU1NEgJqnptU/Sn4jRR6J6epk2pqb1Q/SgAPUGgyNNGjQ2SBpoAZAAGg0NB6mgDIAAAAA2oaApSREBNAARhGiYEaEwU8pvImlP0k2aam1GaGqbFNM1MHpTwmkepmyU9R6nqPKekHqNNPUxNGhp6n6p6QaZ6o9TG1GMqcoV9ly6nRanHlq6zPNbnGZNi6HSug+2nPiZ13XcnFYZW+45W11CumhzYhchOJ2GLLV1OBjBjGf4TptOddTSOcVxhqYZMYwZXZZY00zI1paX5X9J+b+f4e+x43RXSxXPOdquiGpduatGyXneN696M9t4HU2eR5XX/kPhP261NTx3JO1Ow7LyuDmeo9a7d351T1ZxnvnrvYnrXv/hXxPCeuYx2XsNmO003eg9J3Z6U7b23meJ4ri01OdzTk9BNO96brz+qT5nuvvH3ds/G+m/JcG/F2XYuhXlvO+jP7U3XgrzPN/lr8Sf1n6j4j7jZs+s/T0tNaNNYzTs12rxjwztHlnire3Nzc3N1wuBwOBwXBvZfoHpD7rFmR99V5vj3aXza3xdBbXMalubTg/jIv5dfAi54Pdc75j4z412n3Npj3Ld/ENm7a3b/Cod6h/ret1/5vn/C+l+gdslMvgPSLJ8d8q+U66fevYn/tW1chleEtNTGlcHCbLRlq0tHzF5tsbbZZfHjjLgZu42XCuC3NrdjTasZGNzgxPIrGqp7r3p7L2p5XjnpPSmTd5XtzqnB6U87zzg1Ol0zd0zsLszxR6lkxp35u6/teL0L0W922cR7Lu1lpL9CsHirzuM2T+BgsyViT6LHcm0/Vr6U/7LGGyJeqTEjt0PHWhF5mCT7R9mtlDwriYv0Tyr/OxYt6qp5r0mPVT0608TqnqMZaarU2nFwrTzzlrs1ed7z1ux60wyr4ydCaTi3enW8x68x0zU7tXSlcmPSW1mGpWJMg4zmPC2lK96tp0OE80y4MfEvnZj8zGluR6b22ki1Ou9V2nCd9xovcPvcYMZYy0lvN60ScZ45vN6yeCeeXFb1lVjnnCar5fwXwE2bzJ4HI1XVPXfXZMm44GUsMpYsmLB65TuVdm0cl0b+i/wGNN66XjeV7zuPpHcnK/juhhjdfId5jMdE5nN0dGmmm2zZs2cexD5n9p/dY352XsvXHaZNWWsmmS1atjR452nYudzvqv2HMRyvNNnlMcDl3R2+yx2uVrBubTW9icHDVtbNXlZm7jma1rM4VurZZd2y6nUau7ZXZ7bVU+mnoOVxZGMrVmvX60605JwmzGZhhhjTWtaaaMaaGTGmNMZasY0iX8VMUl8eepaIrzGSpemWOQyZORk2bNpjUybMmxqYmknCGCFynutfksaZpjTNMaaatM0xsxcGR0sociNqxNSmhhR1ZJPbsn8qyF0t2qH6iYBclclalbtTTcHTDsPaX6rlnElph2Jyumumtynv2Kk8GI7rsvXbIcJgHJOSaSXnnGaI3m87RtVXJOZ/YtgdTE6Wpha6ZlE8ayXkef1fh602r2WwvfMXtMdLlkfnLFdYYwYso+bWqm7yJqHXZGw2nrS5ZanSYnWlxBxMF1V940K2wdrI7R6OYf7DGGamMmTSbRhlS45xmVOumF1EyPCmHrrN8wwZOOrdNtLeMtzFzDlWnfTBxMk2NaXIZHBYxYLD4w8yju0ao65Vz1OIXoS9dLanwCe1PWrYuWMqf1if1z2k2yYfKJ741PDgno1ZQ8DRqvUny3mNoWTzGO6m1DkrJI8JiR5cSd+vZdGOO8nrMoc5+NDUFsMSXaZJeNlMmGLtJsovOsUp7I9S5VojKxF6bTVEelXqlfJobQr3LozSh2Jk7VcrVMfhXqszGWMzNqGhqZY0OadxkyyMssKugZR0KNFXBHlqwmJgTE/BNVMk6ItJXZMR0H47GpXv/DMOvNkmVuaV1PRfEdxuqc7Hcd+ZV/zTLaRxWk0nl9CdCeM6mn5rstHIBcpiuwmUZXeq81DacHI2rmrZ5SuE5mOZd6LQrZg9mx32TprA8BMo5jKN6yLTCi3WzQaZSuhzTtM1fUTGVpG8Tw+KXI0tjEpiWxtLYynOlktSbVlaI5kxP8TDH8kx50xoxi5KcA4pcja8KWLRlO/Ks6q06ergnvm1ca3Tq8Uw7LTUsmWyctXPWmpitl/uvGcWTGXGuAXDfhqazGmjkxcJW5hMMMMpYsXl2TZYtVOddG3XCarUt6Ptq9CZXSNzyuRzqRZOjsxdBbFVz6OA5HI43r1jityVlVpVkxmOsyaYWE1NTGq1sOVh36mHMcxtSvcy70edG0ZGR3I1Go1GRlV7mWWo1G0ZGRqlvH40l7o4m5xMWLLLYyNjnqc8556mdPqLJ31n/1nWOncxzG1tizrHs/Z+d2vP/B/l8wdJ6rHUn2nbbDq4p6htFtYzMMMTaZis1K5GKzGNmxhmUx2DDlZ/qNnIx41xnaMfCZWYaZWtNLTNW8ND4Fw1MyZOCdM428suKG1ehW8TesOydg7J+YYcD4cYR+8dFK6M4E3HM9ZfRNNL+Sn6rsl4DsrDl2HpPCnfxjGXtbZtYys1ttlyJ4T+BvexjGWRjMszK4Jpc77D3GyuVD7q0+G8m9G+2+rGm7cOR2y7FdtY2XUYx/oNlfRYxhMYyYZkyyg55enna9Kt/FFi6GMMwYwdwxWgxGMLKYmUyGExTKMZkMFhkymKuh0NOBNnBu+23LdwDoZYYzGGMxtORaTU1pjTGWTTGGtMrNWUsyyTTLLG1qy2ZjbK2DBllWqxMtBMaYZQmcE7zvvRcTkclUwdkxTaSdyySt/7fpL+T1v516Ji97fwr5JbLu305zMn5+GMTTZ9F+y7ExwmGVfG44yxn3dLv6l5i+Wth1jCrDq21nW9LqvvDzz3Vf3LLH/O/32TJ/erx3bXftO4eF+G956D952K/An4NfvOpjFjExjevP/UmE0fIoZXx6/w6lX/no3D0bLt+ixjieBM6ksRd0yB4Lt2SwYNE+gd1detlZWUnpiZfGfFaK+4PyCa/v18V8X75pe9fLXzp7l3VjF76vWZmHwGz1IZNWT7b8yddJ4q5kyrVdfru6atWc7bVYztL9Jf4GXvT+Y8m9/YsXP6H018a8D4XVOqvfzqeR+6yZOD8dPv0+U7/q5Pl+2dNb0MjzGVH5p6MNQ7cOWvw62U9aHE8DprDek+McLyvDz+te+9Zhq5+YTruufMcWMabqysTmZVWjKPfnK0wyVcrsuhjZRdLkHNvD72b9abriOSGIxiLixMOoalNPXzy+wT/tf+U6HHONfsz+xe8ufHBdQWWGWLA9if0rsnmrxK5LvRZQeWsTCsrmOYy8VteVfuRfcVTtDLItLIsMYxZLdU/DbtSemxF6Z6Zo5WBXE4tFdCyVMMXMTEMZXVlS6Xec2T4e0tHsRcEuWshcJ2YsNF5rUx1E8ifCq6Z+ZP7qdCeu/aTwFd53l16/o0NOw6O3dLavP4Hbi4RdmuDk6DoYaninC0+o4uZjbJ7Rxeu0/FbuFg+q7DVS6fQe0rZ6NDGUNNU6DEqOaLTicKnYZMnBWruljQxoaS3dZhocDge0bSTyOvdAbG5hxe2xji7E/L55xX13wWNDi6HCekcFxfCPGxY0MXC+s7afWaMdDyjyr+o8Rudm/NabOZvdl274zH4f5XK9z6On1Pe/K5TdPAslg77BjuO6Y3eO7GqvOPG/stknp1leyvLL0Z7bl9I4noMvLkzytLhWYzrOZzLXCORe028rORzOg4N/L0HlMOQ3Pgmnbb6KczlabORpu980q37TBqRu0/p3PO6234Bl03Ynuz+9W7gnsEcmvYaYY3aMYY0wx3pYd+ujsXauWdaY5Xkbtl23fPzFHiDB/QMo0yFjBllYxTQYYyxkrwn7JufwJ/PfgJ+C83X69ni6zvXcnyXabv0ncbLwsceS+RNlyN2mnneJtX0ngYO0+e+0+UnA+Wch3ji8hj5an4h+i6XBySU4n+R0roVcbw5yvHrmr4Yw8Y7x6c+9POPYHI5HI5HI5HI5HGXGww4nE4nrVyOR8XeqPEO7PLOiukYa3Novk5hV4cdtYZLI93e+uxff2jRo0aNGjRo0aNG1bVtW1dy3m83m8+tQ5ZzHw3nObwOu8La9Rc1dtkdS8A3eTk823tnktXWlxN6Oixe06zrN70Isd9jiOgZFq9yfkPqP/SLhN2Myl8jDM43bl1nbcb4cO57jlh8Jow6pzXZdL4dyODTuuhu77FyO27DdwdRxmvO+O+3N2+BdqyTwLHVczDVY4UPE4O66/ZO2cx1LFzVdSXtF7G4HMbrauOHRw6c8FdZ5m9fHZHYZXfTlZquyynSyTTKke6vcffSD9pzPA/G7n7jxPmuhc1DHMynPMrGL6AdewYmwu5ko+UUyTwrMv27rPH1v1nGqd87+p6N6LU8k3NEng53xXyHS97+44OSg/sy/hn+Se6yfYNjW0/uTgP+PvWYzLMmjhcLB/gGpri6H83/84eUXWT6T9Hsv7785z/7z4icpW+zfXypuR7rx/gMdZb1/wC678pcs8/2a3mDitGHxl9mfPlll5MafWWqxk/eYuTDgcNMzDGWLWvsuglNxs53GtN6uWpktlW1tZZYcuinMMWmnNnJydze3b2Y1McBxrBkXw799izLMZZYyy0TkbsGM4p03S2uVu5s/XXUdSdec6smVxZYYGpVmT8A+8ajuEyV5FatkvVru2x6uxGXXbH4A+jvgP4GMYy3iPLXzq/6z65+E005ey+cwMZD3fZcqc6xpjTFjQ0P3U+e++cPYmTIwj0nrK5NPTfl3WvpfLtXDcb2HQMudYOxFXQBor4L4T6vrOauFctYXJQ++NUWmJe5bmx1jDiZS1dTqWxo4GR8jm3fttpmPHppk9PEyv4/y8/sO07XacOmcqc0x2Vi9BvNJvN5oW8x4mOsydpidRxMYJPx06m1bqPzq9KtK8sxXNXFodD/+MYYaJTLwOhc9brCsV18oOR1i4tXChyTkq4lf4y1Ke+9axjDHqs1mfBbMXuP4Hzi+X7t8vzv7bHerrUPgPCxhjre4fXdfLNtNM+Jd+Zdh8xd8wP87uNPoPgv4W7/5P2BuxfsMabNnMnza+54Pdi5U671GPZY8CehX8Voeoo7FHpkeEc6715FwHZrIrUrHaviPUbPZHND+IhczrP6FcYvhOZ0Di/ETt0OI+YwNWR9r7tpf6WDeZKZDB1+z2IthOl1mPyb5FluvEx9h9d0NnM0Y1XPFkWIsk1WotJ0PBMmkvjvQTd0e71tfeV+8r8lQ/tpzpsmxJ+InrI/dj2UajUajVTUajatRqNRtGo1Go1Go4wjeMpZFMVV9CHbofPraLsJ3JpWV2XOoanCuFky4y3PPNxucK2uKC1Lbdb1eo+m5XomN6HfeZsabHLHRX/K+offtNGGmHWctcVcG44MdSqsOLY9VzX+Zxfxn2HPdWTpzWvkrtJ8M5zorrKcquRytJ5N5DZmcaW02l76nWO+BqPXm1A2Ry/0q71dH/mqrqeFjkYxjEXtsX8qubTk67rGycyqsdm4tZx5D6D5hhi0waaWmiaMP81Yjii5qxPlPuU/GfTL1Y5E6Jyfiq63qTa39A4J0sOGDgO9WF9bOXl0XfPRbsY2bPNKPy1YrFYrFYmRhhlTIyMjJWJYZHXuCXI8OoXsvfljGLFicNifpp2XunoPiG1wtx3p1Tah+/DD66OnVtVXP9rKbVxOnL0tR/rHtqB5UDErUVcl11D4qqvjpOcxX7armUNJB3LpW6bxVvD08e8h3odKKvyCFZBdSh2FVcST9xV3n3T8t1j7Kr9qgrqXg+13Pt5U7JCvFXVIV1YG5lRhkVYZJYYDDD4KOIMoHCp26WS8GB7uBh2zIdgq/PKyInjV2STShuoapUdCpX1yTwqq/z1VvET7Kh5nVPkO8YyxjLt2MaaMmWTLQvx3qnzltnXW0p2jxgbEtSny/Osv8Y9pLMXYoHVPAhkVdWVeODhR6q9/Sxe2liwwZWMVvFXfRkeIDxAePUPIrdJ4ey6yquzH+PD/bUOWAu05qVHtFd8rrKHSoeNIOUqrYr3FXyToqfYJgwmJdKpXXOwYYegNNGMzfZPp/t3t/DVs4zjNTN61rRqaWaa4NYbRjTa0tWwy2Y2tGN8ZO8ofNKq4j9SL7I+cSm4/6ovLV5HNXLI0jJidwrtk6ynCaP6Z++GjRlWS3tLeW129Mi9evxU9mtz6s5J3Z7M2ngTgnKvmpomxpaLCzPfmx0JWE+m3NLDDGOX47RctdYYNK5jakdqLkRlI39n590T5zctGSwwZZDJj6kW8XSi6ot2MmWWJ0DUT3nuvebBudScjZ79g8cWJ8av0k+/bE5WKd5MdbFpbDVMxu1DVMmtNZGJvq1mtRbn6M+g/kP0FwDwr7quZs7xosNGpbscyxhhd9TyJyFwbLcxlTasg75vW7TsV5K7ji44XPMMrdoj+Y3rT0Hie62nlYV/pwczzOmdLqLhYkzGMzCZWGMQzGMSsZYY6Di1t4nlJ+Em63mJxrVLxPbYxNEdgc1dU2iOKyoYYWjNrEeHTYybVk0atSa7ehuwsWMWTqn1TrnS6hYsi71d1+s+k+ic70e20fzE/VaTdxT9ZtU4GIXdeNx3X77guYYfpHeTQjaMX6brOu4OY4K7Y2d9mbHarI5ox3p4GpJ2Vd/Tst60f7j999pppjR+Q/Qf8J/VaORs3cji7FfFuN61+ui9s8hix1OCh5KGVV23BPXvZfz3CLyHpix+exi8z/KnCnosY2eunor+cxyPO/xJ0vKey9OvE9VjqaYu0x3Z3jd6o2b1T12D+F8l232lwaaacD5LE8LBxu7WTlbWraWpew8Xexjel3E+wWD4APITdNqR8F3R3T0lunCQ4GaE9R37DxeCYfcHi4xci5ovKfxVs55y2hf+65E/Xdp6jR5nrebTmi5incpkyOjs50JvrZwstbbW6kfuuQw+2mykf/EXNFzxfKTrxew929TR6bWnGL//F3JFOFCQT3K4lQ"
30
+
31
+ kernels = Kernel(
32
+ bz2.decompress(base64.b64decode(quantization_code)),
33
+ [
34
+ "int4WeightCompression",
35
+ "int4WeightExtractionFloat",
36
+ "int4WeightExtractionHalf",
37
+ "int8WeightExtractionFloat",
38
+ "int8WeightExtractionHalf",
39
+ ],
40
+ )
41
+ except Exception as exception:
42
+ kernels = None
43
+ logger.warning("Failed to load cpm_kernels:", exception)
44
+
45
+
46
+ class W8A16Linear(torch.autograd.Function):
47
+ @staticmethod
48
+ def forward(ctx, inp: torch.Tensor, quant_w: torch.Tensor, scale_w: torch.Tensor, weight_bit_width):
49
+ ctx.inp_shape = inp.size()
50
+ ctx.weight_bit_width = weight_bit_width
51
+ out_features = quant_w.size(0)
52
+ inp = inp.contiguous().view(-1, inp.size(-1))
53
+ weight = extract_weight_to_half(quant_w, scale_w, weight_bit_width)
54
+ ctx.weight_shape = weight.size()
55
+ output = inp.mm(weight.t())
56
+ ctx.save_for_backward(inp, quant_w, scale_w)
57
+ return output.view(*(ctx.inp_shape[:-1] + (out_features,)))
58
+
59
+ @staticmethod
60
+ def backward(ctx, grad_output: torch.Tensor):
61
+ inp, quant_w, scale_w = ctx.saved_tensors
62
+ weight = extract_weight_to_half(quant_w, scale_w, ctx.weight_bit_width)
63
+ grad_output = grad_output.contiguous().view(-1, weight.size(0))
64
+ grad_input = grad_output.mm(weight)
65
+ grad_weight = grad_output.t().mm(inp)
66
+ return grad_input.view(ctx.inp_shape), grad_weight.view(ctx.weight_shape), None, None
67
+
68
+
69
+ class W8A16LinearCPU(torch.autograd.Function):
70
+ @staticmethod
71
+ def forward(ctx, inp: torch.Tensor, quant_w: torch.Tensor, scale_w: torch.Tensor, weight_bit_width, quantization_cache=None):
72
+ ctx.inp_shape = inp.size()
73
+ ctx.weight_bit_width = weight_bit_width
74
+ out_features = quant_w.size(0)
75
+ inp = inp.contiguous().view(-1, inp.size(-1))
76
+ weight = extract_weight_to_float(quant_w, scale_w, weight_bit_width, quantization_cache=quantization_cache)
77
+ ctx.weight_shape = weight.size()
78
+ output = inp.mm(weight.t())
79
+ ctx.save_for_backward(inp, quant_w, scale_w)
80
+ return output.view(*(ctx.inp_shape[:-1] + (out_features,)))
81
+
82
+ @staticmethod
83
+ def backward(ctx, grad_output: torch.Tensor):
84
+ inp, quant_w, scale_w = ctx.saved_tensors
85
+ weight = extract_weight_to_float(quant_w, scale_w, ctx.weight_bit_width)
86
+ grad_output = grad_output.contiguous().view(-1, weight.size(0))
87
+ grad_input = grad_output.mm(weight)
88
+ grad_weight = grad_output.t().mm(inp)
89
+ return grad_input.view(ctx.inp_shape), grad_weight.view(ctx.weight_shape), None, None
90
+
91
+
92
+ default_cpu_kernel_code_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "quantization_kernels.c")
93
+ default_cpu_kernel_code = "QlpoOTFBWSZTWXLbSoQAAgzbgERwQXxmTwAAr/ff3kABt0Q2oRVT0hpo9RtEAAAAyBEiSQ9EGjQGQAAAwANGhowjJoNGmgMEUplMTNSMJ5TQaDJpsoMyRMj8P4mZzFSVVwqSXG8GG7MlVwiToYEQwVD7noBxMhNfkeZYtYFtbgOBUSIGtIQjhNHCEnPJsadhb3yBmRIOD3TeAtNLSaU5GgvKUBWSNuuOIHmVt0YhW6rsmDMDUjeUJGJ64R1Jm5lrh0Aa0tKjhFwPdWcGogxLDSXPWQUWTM8Sd3Qz1HMYNxx3HMeiNqNo4jeRDEfZ3gUSHIcU/heomq0vEzL1Msz5KKGxH8FrNOYw3KaxdqaEmNHYMxJFgQbR0DyRknL2L4kwUSxKRdhjRpEtUqilVfggFL1klaMS3PPRDfNqbBOPWO7m4JTVGhS9QTBDDJaEbLbrUQNB+IpJSKQbG5SZZ5gkwJEhJ3aYKJipZ/i7kinChIOW2lQg"
94
+ default_cpu_parallel_kernel_code_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "quantization_kernels_parallel.c")
95
+ default_cpu_parallel_kernel_code = "QlpoOTFBWSZTWUzax5EAALXbgERwSX1mTwAAr/ff3kACNyXUbZYwBpoaNGIyAaADQwRSaVP9QoMg0A2oAPU0AEUkU9GaaKMaQB6gA09T1ARRKnpk0niaJkaaNDJ6g0DTIKVKfZ/g6v1Kem5LJLa0WmkukkuCIHUqWbtJGJMsCSQFiPEIYHgBIZDzR8R6REbYxIqD2Cu7lMkFoPu6LmHeOAy0GF83Tc40jgmTs4HnCe60QfJa2bDBZ0Y1lhgbiZjW8SNsAKCk42UOEdjWN3KoiCIYeQUCCKWIyHewhtSoInLKSG22l4jKM2ZDCVKtBm3OTYBl3jsVqMImtj7PQw7xKxLXQzwgJaPPgW1fRhrvPJICl4YFDYfNbkbBh5JDgrazFml50xEQQwQUjxNwE0IDSofLzSg7UNVKn+Rr1KErzBHUxBqdHRlXzqYsIa5K9Y0UuE2ugw3g5KYofm7AaGNTzJSMhcchhxdaU4JZ0F1UNgQ8XcGDguypqYza8yFaEoGgNRcLej+g2t0feGKFE5OY2PFluQ3q4HgycxlfvzHqo0KcM0JI8OKXtzayJFgsqC1NdUQVu8rChnA6FO3MFyGOoC9KO8ITPpYM5pRqTlczFkLES/4u5IpwoSCZtY8i"
96
+
97
+ cpu_kernels = None
98
+
99
+
100
+ class CPUKernel:
101
+ def __init__(self, kernel_file="", source_code=default_cpu_kernel_code_path, compile_parallel_kernel=None, parallel_num=None):
102
+ self.load =False
103
+ self.int8WeightExtractionFloat = None
104
+ self.int4WeightExtractionFloat = None
105
+ self.int4WeightCompression = None
106
+ self.SetNumThreads = lambda x: x
107
+
108
+ try:
109
+ if not os.path.exists(default_cpu_kernel_code_path):
110
+ with open(default_cpu_kernel_code_path, "w", encoding="utf-8") as file:
111
+ code = default_cpu_kernel_code
112
+ cpu_quantization_code = bz2.decompress(base64.b64decode(code)).decode()
113
+ file.write(cpu_quantization_code)
114
+
115
+ if not os.path.exists(default_cpu_parallel_kernel_code_path):
116
+ with open(default_cpu_parallel_kernel_code_path, "w", encoding="utf-8") as file:
117
+ code = default_cpu_parallel_kernel_code
118
+ cpu_quantization_code = bz2.decompress(base64.b64decode(code)).decode()
119
+ file.write(cpu_quantization_code)
120
+
121
+ except Exception as ex:
122
+ print("Error when generating default cpu kernel code(can be ignored when using custom kernels).")
123
+
124
+ if compile_parallel_kernel is None:
125
+ compile_parallel_kernel = bool(int(os.cpu_count()) >= 4)
126
+
127
+ if compile_parallel_kernel and source_code == default_cpu_kernel_code_path:
128
+ source_code = default_cpu_parallel_kernel_code_path
129
+
130
+ kernels = None
131
+
132
+ if (not kernel_file) or (not os.path.exists(kernel_file)):
133
+ print("No compiled kernel found.")
134
+ try:
135
+ if os.path.exists(source_code):
136
+ print("Compiling kernels :", source_code)
137
+ kernel_file = source_code[:-2] + ".so"
138
+
139
+ if compile_parallel_kernel:
140
+ compile_command = "gcc -O3 -fPIC -pthread -fopenmp -std=c99 {} -shared -o {}".format(source_code, kernel_file)
141
+ print("Compiling", compile_command)
142
+ exit_state = os.system(compile_command)
143
+ if not exit_state:
144
+ try:
145
+ kernels = ctypes.cdll.LoadLibrary(kernel_file)
146
+ print("Load kernel :", kernel_file)
147
+ except:
148
+ kernels = None
149
+ print("Load parallel cpu kernel failed, using default cpu kernel code:")
150
+ import traceback
151
+ exception = traceback.format_exc()
152
+ print(exception)
153
+ else:
154
+ print("Compile default cpu kernel failed, using default cpu kernel code.")
155
+
156
+ if kernels is None: # adjust config, use default cpu kernel
157
+ compile_parallel_kernel = False
158
+ source_code = default_cpu_kernel_code_path
159
+ kernel_file = source_code[:-2] + ".so"
160
+
161
+ if kernels is None:
162
+ compile_command = "gcc -O3 -fPIC -std=c99 {} -shared -o {}".format(source_code, kernel_file)
163
+ print("Compiling", compile_command)
164
+ exit_state = os.system(compile_command)
165
+ if not exit_state:
166
+ try:
167
+ kernels = ctypes.cdll.LoadLibrary(kernel_file)
168
+ print("Load kernel :", kernel_file)
169
+ except:
170
+ kernels = None
171
+ print("Load default cpu kernel failed:")
172
+ import traceback
173
+ exception = traceback.format_exc()
174
+ print(exception)
175
+ else:
176
+ print("Compile default cpu kernel failed.")
177
+ else:
178
+ print("Kernel source code not found.")
179
+ return
180
+ except:
181
+ print("Failed to build cpu kernel:")
182
+ import traceback
183
+ exception = traceback.format_exc()
184
+ print(exception)
185
+ return
186
+ else:
187
+ try:
188
+ kernels = ctypes.cdll.LoadLibrary(kernel_file)
189
+ print("Load kernel :", kernel_file)
190
+ except:
191
+ kernels = None
192
+ print("Load custom cpu kernel failed:")
193
+ import traceback
194
+ exception = traceback.format_exc()
195
+ print(exception)
196
+
197
+ if kernels is not None:
198
+ self.int8WeightExtractionFloat = kernels.extract_int8_weight_to_float
199
+ self.int4WeightExtractionFloat = kernels.extract_int4_weight_to_float
200
+ self.int4WeightCompression = kernels.compress_int4_weight
201
+ if compile_parallel_kernel:
202
+ try:
203
+ self.SetNumThreads = kernels.set_num_threads
204
+ except:
205
+ print("No set_num_threads() found in kernel.")
206
+ self.load = True
207
+ else:
208
+ print("Failed to load kernel.")
209
+ return
210
+
211
+ if compile_parallel_kernel:
212
+ if parallel_num is None:
213
+ parallel_num = max(os.cpu_count() // 2, 1)
214
+ print("Setting CPU quantization kernel threads to", parallel_num)
215
+ if parallel_num < 4:
216
+ print("Parallel kernel is not recommended when parallel num < 4.")
217
+ self.SetNumThreads(parallel_num)
218
+
219
+ self.parallel_num = parallel_num
220
+
221
+
222
+ def compress_int4_weight(weight: torch.Tensor): # (n, m)
223
+ """compress weight on cpu or cuda to int4"""
224
+ if weight.device == torch.device("cpu"):
225
+ assert isinstance(cpu_kernels, CPUKernel)
226
+ n, m = weight.size(0), weight.size(1)
227
+ assert m % 2 == 0
228
+ m = m // 2
229
+ out = torch.empty(n, m, dtype=torch.int8, device="cpu")
230
+ cpu_kernels.int4WeightCompression(
231
+ ctypes.c_void_p(weight.data_ptr()),
232
+ ctypes.c_void_p(out.data_ptr()),
233
+ ctypes.c_int32(n),
234
+ ctypes.c_int32(m)
235
+ )
236
+ return out
237
+ else:
238
+ with torch.cuda.device(weight.device):
239
+ n, m = weight.size(0), weight.size(1)
240
+ assert m % 2 == 0
241
+ m = m // 2
242
+ out = torch.empty(n, m, dtype=torch.int8, device="cuda")
243
+ stream = torch.cuda.current_stream()
244
+
245
+ gridDim = (n, 1, 1)
246
+ blockDim = (min(round_up(m, 32), 1024), 1, 1)
247
+
248
+ kernels.int4WeightCompression(
249
+ gridDim,
250
+ blockDim,
251
+ 0,
252
+ stream,
253
+ [ctypes.c_void_p(weight.data_ptr()), ctypes.c_void_p(out.data_ptr()), ctypes.c_int32(n), ctypes.c_int32(m)],
254
+ )
255
+ return out
256
+
257
+
258
+ def extract_weight_to_half(weight: torch.Tensor, scale_list: torch.Tensor, source_bit_width: int):
259
+ if source_bit_width == 8:
260
+ func = kernels.int8WeightExtractionHalf
261
+ elif source_bit_width == 4:
262
+ func = kernels.int4WeightExtractionHalf
263
+ else:
264
+ assert False, "Unsupported bit-width"
265
+
266
+ with torch.cuda.device(weight.device):
267
+ n, m = weight.size(0), weight.size(1)
268
+ out = torch.empty(n, m * (8 // source_bit_width), dtype=torch.half, device="cuda")
269
+ stream = torch.cuda.current_stream()
270
+
271
+ gridDim = (n, 1, 1)
272
+ blockDim = (min(round_up(m, 32), 1024), 1, 1)
273
+
274
+ func(
275
+ gridDim,
276
+ blockDim,
277
+ 0,
278
+ stream,
279
+ [
280
+ ctypes.c_void_p(weight.data_ptr()),
281
+ ctypes.c_void_p(scale_list.data_ptr()),
282
+ ctypes.c_void_p(out.data_ptr()),
283
+ ctypes.c_int32(n),
284
+ ctypes.c_int32(m),
285
+ ],
286
+ )
287
+ return out
288
+
289
+
290
+ def extract_weight_to_float(weight: torch.Tensor, scale_list: torch.Tensor, source_bit_width: int, quantization_cache=None):
291
+ """extract weight on cpu to float32"""
292
+ if source_bit_width == 8:
293
+ func = cpu_kernels.int8WeightExtractionFloat
294
+ elif source_bit_width == 4:
295
+ func = cpu_kernels.int4WeightExtractionFloat
296
+ else:
297
+ assert False, "Unsupported bit-width"
298
+
299
+ n, m = weight.size(0), weight.size(1)
300
+
301
+ if quantization_cache is not None:
302
+ out = quantization_cache
303
+ func(
304
+ ctypes.c_void_p(weight.data_ptr()),
305
+ ctypes.c_void_p(scale_list.data_ptr()),
306
+ ctypes.c_void_p(out.data_ptr()),
307
+ ctypes.c_int32(n),
308
+ ctypes.c_int32(m)
309
+ )
310
+ return out.tensor
311
+ else:
312
+ out = torch.empty(n, m * (8 // source_bit_width), dtype=torch.float, device="cpu")
313
+ func(
314
+ ctypes.c_void_p(weight.data_ptr()),
315
+ ctypes.c_void_p(scale_list.data_ptr()),
316
+ ctypes.c_void_p(out.data_ptr()),
317
+ ctypes.c_int32(n),
318
+ ctypes.c_int32(m)
319
+ )
320
+ return out
321
+
322
+
323
+ class CacheTensor():
324
+ def __init__(self, *args, **kwargs):
325
+ self.tensor = torch.empty(*args, **kwargs)
326
+
327
+ def to(self, *args, **kwargs):
328
+ self.tensor = self.tensor.to(*args, **kwargs)
329
+
330
+ def data_ptr(self):
331
+ return self.tensor.data_ptr()
332
+
333
+
334
+ class QuantizedLinear(Linear):
335
+ def __init__(self, weight_bit_width: int, weight_tensor=None, bias_tensor=None, quantized_weight=None, quantized_weight_scale=None, quantization_cache=None, empty_init=False, *args, **kwargs):
336
+ super(QuantizedLinear, self).__init__(*args, **kwargs)
337
+ self.weight_bit_width = weight_bit_width
338
+ self.quantization_cache = quantization_cache
339
+
340
+ if (quantized_weight is not None) and (quantized_weight_scale is not None):
341
+ del self.weight
342
+ self.weight = Parameter(quantized_weight.to(kwargs["device"]), requires_grad=False)
343
+ self.weight_scale = Parameter(quantized_weight_scale.to(kwargs["device"]), requires_grad=False)
344
+ else:
345
+ shape = self.weight.shape
346
+ del self.weight
347
+
348
+ if weight_tensor is None or empty_init:
349
+ self.weight = torch.empty(
350
+ shape[0], shape[1] * weight_bit_width // 8, dtype=torch.int8, device=kwargs["device"]
351
+ )
352
+ self.weight_scale = torch.empty(shape[0], dtype=kwargs["dtype"], device=kwargs["device"])
353
+ else:
354
+ self.weight_scale = (weight_tensor.abs().max(dim=-1).values / ((2 ** (weight_bit_width - 1)) - 1)).to(kwargs["dtype"])
355
+ self.weight = torch.round(weight_tensor / self.weight_scale[:, None]).to(torch.int8)
356
+ if weight_bit_width == 4:
357
+ self.weight = compress_int4_weight(self.weight)
358
+
359
+ self.weight = Parameter(self.weight.to(kwargs["device"]), requires_grad=False)
360
+ self.weight_scale = Parameter(self.weight_scale.to(kwargs["device"]), requires_grad=False)
361
+
362
+ if bias_tensor is not None:
363
+ self.bias = Parameter(bias_tensor.to(kwargs["device"]), requires_grad=False)
364
+ else:
365
+ self.bias = None
366
+
367
+ def reset_parameters(self):
368
+ """To accelerate initialization"""
369
+ pass
370
+
371
+ def forward(self, input):
372
+ if self.weight.device == torch.device("cpu"):
373
+ output = W8A16LinearCPU.apply(input, self.weight, self.weight_scale, self.weight_bit_width, self.quantization_cache)
374
+ else:
375
+ output = W8A16Linear.apply(input, self.weight, self.weight_scale, self.weight_bit_width)
376
+ if self.bias is not None:
377
+ output = output + self.bias
378
+ return output
379
+
380
+ def _apply(self, fn):
381
+ self_obj = super()._apply(fn)
382
+ if self.quantization_cache is not None:
383
+ self.quantization_cache.to(self_obj.weight.device)
384
+ self.quantization_cache.to(self_obj.weight_scale.dtype)
385
+ return self_obj
386
+
387
+
388
+ class QuantizedEmbedding(Embedding): # TODO: backward, check empty_init
389
+ def __init__(self, weight_bit_width: int, weight_tensor=None, quantized_weight=None, quantized_weight_scale=None, empty_init=False, *args, **kwargs):
390
+ super(QuantizedEmbedding, self).__init__(*args, **kwargs)
391
+ self.weight_bit_width = weight_bit_width
392
+
393
+ if (quantized_weight is not None) and (quantized_weight_scale is not None):
394
+ del self.weight
395
+ self.weight = Parameter(quantized_weight.to(kwargs["device"]), requires_grad=False)
396
+ self.weight_scale = Parameter(quantized_weight_scale.to(kwargs["device"]), requires_grad=False)
397
+ else:
398
+ shape = self.weight.shape
399
+ del self.weight
400
+
401
+ if weight_tensor is None or empty_init:
402
+ self.weight = torch.empty(
403
+ shape[0], shape[1] * weight_bit_width // 8, dtype=torch.int8, device=kwargs["device"]
404
+ )
405
+ self.weight_scale = torch.empty(shape[0], dtype=kwargs["dtype"], device=kwargs["device"])
406
+ else:
407
+ self.weight_scale = (weight_tensor.abs().max(dim=-1).values / ((2 ** (weight_bit_width - 1)) - 1)).half()
408
+ self.weight = torch.round(weight_tensor / self.weight_scale[:, None]).to(torch.int8)
409
+ if weight_bit_width == 4:
410
+ self.weight = compress_int4_weight(self.weight)
411
+
412
+ self.weight = Parameter(self.weight.to(kwargs["device"]), requires_grad=False)
413
+ self.weight_scale = Parameter(self.weight_scale.to(kwargs["device"]), requires_grad=False)
414
+
415
+ def forward(self, input):
416
+ if self.weight.device == torch.device("cpu"):
417
+ original_weight = extract_weight_to_float(weight=self.weight, scale_list=self.weight_scale, source_bit_width=self.weight_bit_width)
418
+ else:
419
+ original_weight = extract_weight_to_half(weight=self.weight, scale_list=self.weight_scale, source_bit_width=self.weight_bit_width)
420
+ output = F.embedding(
421
+ input, original_weight, self.padding_idx, self.max_norm,
422
+ self.norm_type, self.scale_grad_by_freq, self.sparse
423
+ )
424
+ return output
425
+
426
+
427
+ def load_cpu_kernel(**kwargs):
428
+ global cpu_kernels
429
+ cpu_kernels = CPUKernel(**kwargs)
430
+ assert cpu_kernels.load
431
+
432
+
433
+ def quantize(model, weight_bit_width, use_quantization_cache=False, empty_init=False, **kwargs):
434
+ """Replace fp16 linear with quantized linear"""
435
+
436
+ query_key_value_quantization_cache = None
437
+ dense_quantization_cache = None
438
+ dense_h_to_4h_quantization_cache = None
439
+ dense_4h_to_h_quantization_cache = None
440
+
441
+ try:
442
+ load_cpu_kernel(**kwargs)
443
+ except:
444
+ if kernels is None: # CUDA kernels failed
445
+ print("Cannot load cpu or cuda kernel, quantization failed:")
446
+ assert kernels is not None
447
+ print("Cannot load cpu kernel, don't use quantized model on cpu.")
448
+
449
+ current_device = model.device
450
+
451
+ if model.device == torch.device("cpu"):
452
+ dtype=torch.float32
453
+ else:
454
+ dtype = torch.half
455
+
456
+ QuantizedLinearWithPara = partial(
457
+ QuantizedLinear,
458
+ weight_bit_width=weight_bit_width,
459
+ bias=True,
460
+ dtype=dtype,
461
+ empty_init=empty_init
462
+ )
463
+
464
+ if use_quantization_cache:
465
+ print("Using quantization cache")
466
+ layer = model.layers[0]
467
+ weight = layer.attention.query_key_value.weight
468
+ n, m = weight.size(0), weight.size(1)
469
+ query_key_value_quantization_cache = CacheTensor(n, m, dtype=dtype, device=current_device, requires_grad=False)
470
+ weight = layer.attention.dense.weight
471
+ n, m = weight.size(0), weight.size(1)
472
+ dense_quantization_cache = CacheTensor(n, m, dtype=dtype, device=current_device, requires_grad=False)
473
+ weight = layer.mlp.dense_h_to_4h.weight
474
+ n, m = weight.size(0), weight.size(1)
475
+ dense_h_to_4h_quantization_cache = CacheTensor(n, m, dtype=dtype, device=current_device, requires_grad=False)
476
+ weight = layer.mlp.dense_4h_to_h.weight
477
+ n, m = weight.size(0), weight.size(1)
478
+ dense_4h_to_h_quantization_cache = CacheTensor(n, m, dtype=dtype, device=current_device, requires_grad=False)
479
+
480
+ print("Applying quantization to glm layers")
481
+
482
+ for layer in model.layers:
483
+ layer.attention.query_key_value = QuantizedLinearWithPara(
484
+ weight_tensor=layer.attention.query_key_value.weight.to(current_device),
485
+ bias_tensor=layer.attention.query_key_value.bias,
486
+ in_features=layer.attention.query_key_value.in_features,
487
+ out_features=layer.attention.query_key_value.out_features,
488
+ device=layer.attention.query_key_value.weight.device,
489
+ quantization_cache=query_key_value_quantization_cache
490
+ )
491
+ layer.attention.dense = QuantizedLinearWithPara(
492
+ weight_tensor=layer.attention.dense.weight.to(current_device),
493
+ bias_tensor=layer.attention.dense.bias,
494
+ in_features=layer.attention.dense.in_features,
495
+ out_features=layer.attention.dense.out_features,
496
+ device=layer.attention.dense.weight.device,
497
+ quantization_cache=dense_quantization_cache
498
+ )
499
+ layer.mlp.dense_h_to_4h = QuantizedLinearWithPara(
500
+ weight_tensor=layer.mlp.dense_h_to_4h.weight.to(current_device),
501
+ bias_tensor=layer.mlp.dense_h_to_4h.bias,
502
+ in_features=layer.mlp.dense_h_to_4h.in_features,
503
+ out_features=layer.mlp.dense_h_to_4h.out_features,
504
+ device=layer.mlp.dense_h_to_4h.weight.device,
505
+ quantization_cache=dense_h_to_4h_quantization_cache
506
+ )
507
+ layer.mlp.dense_4h_to_h = QuantizedLinearWithPara(
508
+ weight_tensor=layer.mlp.dense_4h_to_h.weight.to(current_device),
509
+ bias_tensor=layer.mlp.dense_4h_to_h.bias,
510
+ in_features=layer.mlp.dense_4h_to_h.in_features,
511
+ out_features=layer.mlp.dense_4h_to_h.out_features,
512
+ device=layer.mlp.dense_4h_to_h.weight.device,
513
+ quantization_cache=dense_4h_to_h_quantization_cache
514
+ )
515
+ return model
quantization_kernels.c ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ void compress_int4_weight(void *weight, void *out, int n, int m)
2
+ {
3
+ for(int i=0;i<n*m;i++)
4
+ {
5
+ (*(unsigned char*)(out)) = ((*(unsigned char*)(weight)) << 4);
6
+ weight += sizeof(char);
7
+ (*(unsigned char*)(out)) |= ((*(unsigned char*)(weight)) & 15);
8
+ weight += sizeof(char);
9
+ out += sizeof(char);
10
+ }
11
+ }
12
+
13
+ void extract_int8_weight_to_float(void *weight, void *scale_list, void *out, int n, int m)
14
+ {
15
+ for(int i=0;i<n;i++)
16
+ for(int j=0;j<m;j++)
17
+ (*(float*)(out + sizeof(float) * (i * m + j))) = (*(float*)(scale_list + sizeof(float) * i)) * (*(char*)(weight + sizeof(char) * (i * m + j)));
18
+ }
19
+
20
+ void extract_int4_weight_to_float(void *weight, void *scale_list, void *out, int n, int m)
21
+ {
22
+ for(int i=0;i<n;i++)
23
+ {
24
+ for(int j=0;j<m;j++)
25
+ {
26
+ (*(float*)(out)) = (*(float*)(scale_list)) * ((*(char*)(weight)) >> 4);
27
+ out += sizeof(float);
28
+ (*(float*)(out)) = (*(float*)(scale_list)) * (((char)((*(unsigned char*)(weight)) << 4))>> 4);
29
+ out += sizeof(float);
30
+ weight += sizeof(char);
31
+ }
32
+ scale_list += sizeof(float);
33
+ }
34
+ }
quantization_kernels_parallel.c ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #include <omp.h>
2
+
3
+ void set_num_threads(int n_threads)
4
+ {
5
+ omp_set_num_threads(n_threads);
6
+ }
7
+
8
+ int get_num_threads()
9
+ {
10
+ return omp_get_num_threads();
11
+ }
12
+
13
+ void compress_int4_weight(void *weight, void *out, int n, int m)
14
+ {
15
+ #pragma omp parallel for
16
+ for(int i=0;i<n;i++)
17
+ {
18
+ for(int j=0;j<m;j++)
19
+ {
20
+ (*(unsigned char*)(out + sizeof(unsigned char) * (i * m + j))) = ((*(unsigned char*)(weight + sizeof(unsigned char) * (i * (m << 1) + (j << 1)))) << 4);
21
+ (*(unsigned char*)(out + sizeof(unsigned char) * (i * m + j))) |= (((*(unsigned char*)(weight + sizeof(unsigned char) * (i * (m << 1) + ((j << 1) | 1)))) & 15));
22
+ }
23
+ }
24
+ }
25
+
26
+ void extract_int8_weight_to_float(void *weight, void *scale_list, void *out, int n, int m)
27
+ {
28
+ #pragma omp parallel for
29
+ for(int i=0;i<n;i++)
30
+ {
31
+ for(int j=0;j<m;j++)
32
+ (*(float*)(out + sizeof(float) * (i * m + j))) = (*(float*)(scale_list + sizeof(float) * i)) * (*(char*)(weight + sizeof(char) * (i * m + j)));
33
+ }
34
+ }
35
+
36
+ void extract_int4_weight_to_float(void *weight, void *scale_list, void *out, int n, int m)
37
+ {
38
+ #pragma omp parallel for
39
+ for(int i=0;i<n;i++)
40
+ {
41
+ for(int j=0;j<m;j++)
42
+ {
43
+ (*(float*)(out + sizeof(float) * (i * (m << 1) + (j << 1)))) = (*(float*)(scale_list + sizeof(float) * i)) * ((*(char*)(weight + sizeof(char) * (i * m + j))) >> 4);
44
+ (*(float*)(out + sizeof(float) * (i * (m << 1) + ((j << 1) | 1)))) = (*(float*)(scale_list + sizeof(float) * i)) * (((char)((*(unsigned char*)(weight + sizeof(char) * (i * m + j))) << 4))>> 4);
45
+ }
46
+ }
47
+ }
tokenization_chatglm.py ADDED
@@ -0,0 +1,430 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Tokenization classes for ChatGLM."""
2
+ from typing import List, Optional, Union
3
+ import os
4
+
5
+ from transformers.tokenization_utils import PreTrainedTokenizer
6
+ from transformers.utils import logging, PaddingStrategy
7
+ from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
8
+ from typing import Dict
9
+ import sentencepiece as spm
10
+ import numpy as np
11
+
12
+ logger = logging.get_logger(__name__)
13
+
14
+ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
15
+ "THUDM/chatglm-6b": 2048,
16
+ }
17
+
18
+
19
+ class TextTokenizer:
20
+ def __init__(self, model_path):
21
+ self.sp = spm.SentencePieceProcessor()
22
+ self.sp.Load(model_path)
23
+ self.num_tokens = self.sp.vocab_size()
24
+
25
+ def encode(self, text):
26
+ return self.sp.EncodeAsIds(text)
27
+
28
+ def decode(self, ids: List[int]):
29
+ return self.sp.DecodeIds(ids)
30
+
31
+ def tokenize(self, text):
32
+ return self.sp.EncodeAsPieces(text)
33
+
34
+ def convert_tokens_to_ids(self, tokens):
35
+ return [self.sp.PieceToId(token) for token in tokens]
36
+
37
+ def convert_token_to_id(self, token):
38
+ return self.sp.PieceToId(token)
39
+
40
+ def convert_id_to_token(self, idx):
41
+ return self.sp.IdToPiece(idx)
42
+
43
+ def __len__(self):
44
+ return self.num_tokens
45
+
46
+
47
+ class SPTokenizer:
48
+ def __init__(
49
+ self,
50
+ vocab_file,
51
+ num_image_tokens=20000,
52
+ max_blank_length=80,
53
+ byte_fallback=True,
54
+ ):
55
+ assert vocab_file is not None
56
+ self.vocab_file = vocab_file
57
+ self.num_image_tokens = num_image_tokens
58
+ self.special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "<unused_0>", "<sop>", "<eop>", "<ENC>", "<dBLOCK>"]
59
+ self.max_blank_length = max_blank_length
60
+ self.byte_fallback = byte_fallback
61
+ self.text_tokenizer = TextTokenizer(vocab_file)
62
+
63
+ def _get_text_tokenizer(self):
64
+ return self.text_tokenizer
65
+
66
+ @staticmethod
67
+ def get_blank_token(length: int):
68
+ assert length >= 2
69
+ return f"<|blank_{length}|>"
70
+
71
+ @staticmethod
72
+ def get_tab_token():
73
+ return f"<|tab|>"
74
+
75
+ @property
76
+ def num_text_tokens(self):
77
+ return self.text_tokenizer.num_tokens
78
+
79
+ @property
80
+ def num_tokens(self):
81
+ return self.num_image_tokens + self.num_text_tokens
82
+
83
+ @staticmethod
84
+ def _encode_whitespaces(text: str, max_len: int = 80):
85
+ text = text.replace("\t", SPTokenizer.get_tab_token())
86
+ for i in range(max_len, 1, -1):
87
+ text = text.replace(" " * i, SPTokenizer.get_blank_token(i))
88
+ return text
89
+
90
+ def _preprocess(self, text: str, linebreak=True, whitespaces=True):
91
+ if linebreak:
92
+ text = text.replace("\n", "<n>")
93
+ if whitespaces:
94
+ text = self._encode_whitespaces(text, max_len=self.max_blank_length)
95
+ return text
96
+
97
+ def encode(
98
+ self, text: str, linebreak=True, whitespaces=True, add_dummy_prefix=True
99
+ ) -> List[int]:
100
+ """
101
+ @param text: Text to encode.
102
+ @param linebreak: Whether to encode newline (\n) in text.
103
+ @param whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding.
104
+ @param special_tokens: Whether to encode special token ([MASK], [gMASK], etc.) in text.
105
+ @param add_dummy_prefix: Whether to add dummy blank space in the beginning.
106
+ """
107
+ text = self._preprocess(text, linebreak, whitespaces)
108
+ if not add_dummy_prefix:
109
+ text = "<n>" + text
110
+ tmp = self._get_text_tokenizer().encode(text)
111
+ tokens = [x + self.num_image_tokens for x in tmp]
112
+ return tokens if add_dummy_prefix else tokens[2:]
113
+
114
+ def decode(self, text_ids: List[int]) -> str:
115
+ ids = [int(_id) - self.num_image_tokens for _id in text_ids]
116
+ ids = [_id for _id in ids if _id >= 0]
117
+ text = self._get_text_tokenizer().decode(ids)
118
+ text = text.replace("<n>", "\n")
119
+ text = text.replace(SPTokenizer.get_tab_token(), "\t")
120
+ for i in range(2, self.max_blank_length + 1):
121
+ text = text.replace(self.get_blank_token(i), " " * i)
122
+ return text
123
+
124
+ def tokenize(
125
+ self, text: str, linebreak=True, whitespaces=True, add_dummy_prefix=True
126
+ ) -> List[str]:
127
+ """
128
+ @param text: Text to encode.
129
+ @param linebreak: Whether to encode newline (\n) in text.
130
+ @param whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding.
131
+ @param special_tokens: Whether to encode special token ([MASK], [gMASK], etc.) in text.
132
+ @param add_dummy_prefix: Whether to add dummy blank space in the beginning.
133
+ """
134
+ text = self._preprocess(text, linebreak, whitespaces)
135
+ if not add_dummy_prefix:
136
+ text = "<n>" + text
137
+ tokens = self._get_text_tokenizer().tokenize(text)
138
+ return tokens if add_dummy_prefix else tokens[2:]
139
+
140
+ def __getitem__(self, x: Union[int, str]):
141
+ if isinstance(x, int):
142
+ if x < self.num_image_tokens:
143
+ return "<image_{}>".format(x)
144
+ else:
145
+ return self.text_tokenizer.convert_id_to_token(x - self.num_image_tokens)
146
+ elif isinstance(x, str):
147
+ if x.startswith("<image_") and x.endswith(">") and x[7:-1].isdigit():
148
+ return int(x[7:-1])
149
+ else:
150
+ return self.text_tokenizer.convert_token_to_id(x) + self.num_image_tokens
151
+ else:
152
+ raise ValueError("The key should be str or int.")
153
+
154
+
155
+ class ChatGLMTokenizer(PreTrainedTokenizer):
156
+ """
157
+ Construct a ChatGLM tokenizer. Based on byte-level Byte-Pair-Encoding.
158
+
159
+ Args:
160
+ vocab_file (`str`):
161
+ Path to the vocabulary file.
162
+ """
163
+
164
+ vocab_files_names = {"vocab_file": "ice_text.model"}
165
+ max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
166
+ model_input_names = ["input_ids", "attention_mask", "position_ids"]
167
+
168
+ def __init__(
169
+ self,
170
+ vocab_file,
171
+ do_lower_case=False,
172
+ remove_space=False,
173
+ bos_token='<sop>',
174
+ eos_token='<eop>',
175
+ end_token='</s>',
176
+ mask_token='[MASK]',
177
+ gmask_token='[gMASK]',
178
+ padding_side="left",
179
+ pad_token="<pad>",
180
+ unk_token="<unk>",
181
+ num_image_tokens=20000,
182
+ **kwargs
183
+ ) -> None:
184
+ super().__init__(
185
+ do_lower_case=do_lower_case,
186
+ remove_space=remove_space,
187
+ padding_side=padding_side,
188
+ bos_token=bos_token,
189
+ eos_token=eos_token,
190
+ end_token=end_token,
191
+ mask_token=mask_token,
192
+ gmask_token=gmask_token,
193
+ pad_token=pad_token,
194
+ unk_token=unk_token,
195
+ num_image_tokens=num_image_tokens,
196
+ **kwargs
197
+ )
198
+
199
+ self.do_lower_case = do_lower_case
200
+ self.remove_space = remove_space
201
+ self.vocab_file = vocab_file
202
+
203
+ self.bos_token = bos_token
204
+ self.eos_token = eos_token
205
+ self.end_token = end_token
206
+ self.mask_token = mask_token
207
+ self.gmask_token = gmask_token
208
+
209
+ self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens)
210
+
211
+ """ Initialisation """
212
+
213
+ @property
214
+ def gmask_token_id(self) -> Optional[int]:
215
+ if self.gmask_token is None:
216
+ return None
217
+ return self.convert_tokens_to_ids(self.gmask_token)
218
+
219
+ @property
220
+ def end_token_id(self) -> Optional[int]:
221
+ """
222
+ `Optional[int]`: Id of the end of context token in the vocabulary. Returns `None` if the token has not been
223
+ set.
224
+ """
225
+ if self.end_token is None:
226
+ return None
227
+ return self.convert_tokens_to_ids(self.end_token)
228
+
229
+ @property
230
+ def vocab_size(self):
231
+ """ Returns vocab size """
232
+ return self.sp_tokenizer.num_tokens
233
+
234
+ def get_vocab(self):
235
+ """ Returns vocab as a dict """
236
+ vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
237
+ vocab.update(self.added_tokens_encoder)
238
+ return vocab
239
+
240
+ def preprocess_text(self, inputs):
241
+ if self.remove_space:
242
+ outputs = " ".join(inputs.strip().split())
243
+ else:
244
+ outputs = inputs
245
+
246
+ if self.do_lower_case:
247
+ outputs = outputs.lower()
248
+
249
+ return outputs
250
+
251
+ def _tokenize(self, text, **kwargs):
252
+ """ Returns a tokenized string. """
253
+ text = self.preprocess_text(text)
254
+
255
+ seq = self.sp_tokenizer.tokenize(text)
256
+
257
+ return seq
258
+
259
+ def _decode(
260
+ self,
261
+ token_ids: Union[int, List[int]],
262
+ skip_special_tokens: bool = False,
263
+ clean_up_tokenization_spaces: bool = True,
264
+ **kwargs
265
+ ) -> str:
266
+ if isinstance(token_ids, int):
267
+ token_ids = [token_ids]
268
+ if len(token_ids) == 0:
269
+ return ""
270
+ if self.pad_token_id in token_ids: # remove pad
271
+ token_ids = list(filter((self.pad_token_id).__ne__, token_ids))
272
+ return self.sp_tokenizer.decode(token_ids)
273
+
274
+ def _convert_token_to_id(self, token):
275
+ """ Converts a token (str) in an id using the vocab. """
276
+ return self.sp_tokenizer[token]
277
+
278
+ def _convert_id_to_token(self, index):
279
+ """Converts an index (integer) in a token (str) using the vocab."""
280
+ return self.sp_tokenizer[index]
281
+
282
+ def save_vocabulary(self, save_directory, filename_prefix=None):
283
+ """
284
+ Save the vocabulary and special tokens file to a directory.
285
+
286
+ Args:
287
+ save_directory (`str`):
288
+ The directory in which to save the vocabulary.
289
+ filename_prefix (`str`, *optional*):
290
+ An optional prefix to add to the named of the saved files.
291
+
292
+ Returns:
293
+ `Tuple(str)`: Paths to the files saved.
294
+ """
295
+ if os.path.isdir(save_directory):
296
+ vocab_file = os.path.join(
297
+ save_directory, self.vocab_files_names["vocab_file"]
298
+ )
299
+ else:
300
+ vocab_file = save_directory
301
+
302
+ with open(self.vocab_file, 'rb') as fin:
303
+ proto_str = fin.read()
304
+
305
+ with open(vocab_file, "wb") as writer:
306
+ writer.write(proto_str)
307
+
308
+ return (vocab_file,)
309
+
310
+ def build_inputs_with_special_tokens(
311
+ self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
312
+ ) -> List[int]:
313
+ """
314
+ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
315
+ adding special tokens. A BERT sequence has the following format:
316
+
317
+ - single sequence: `[CLS] X [SEP]`
318
+ - pair of sequences: `[CLS] A [SEP] B [SEP]`
319
+
320
+ Args:
321
+ token_ids_0 (`List[int]`):
322
+ List of IDs to which the special tokens will be added.
323
+ token_ids_1 (`List[int]`, *optional*):
324
+ Optional second list of IDs for sequence pairs.
325
+
326
+ Returns:
327
+ `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
328
+ """
329
+ gmask_id = self.sp_tokenizer[self.gmask_token]
330
+ eos_id = self.sp_tokenizer[self.eos_token]
331
+ token_ids_0 = token_ids_0 + [gmask_id, self.sp_tokenizer[self.bos_token]]
332
+ if token_ids_1 is not None:
333
+ token_ids_0 = token_ids_0 + token_ids_1 + [eos_id]
334
+ return token_ids_0
335
+
336
+ def _pad(
337
+ self,
338
+ encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
339
+ max_length: Optional[int] = None,
340
+ padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
341
+ pad_to_multiple_of: Optional[int] = None,
342
+ return_attention_mask: Optional[bool] = None,
343
+ ) -> dict:
344
+ """
345
+ Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
346
+
347
+ Args:
348
+ encoded_inputs:
349
+ Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`).
350
+ max_length: maximum length of the returned list and optionally padding length (see below).
351
+ Will truncate by taking into account the special tokens.
352
+ padding_strategy: PaddingStrategy to use for padding.
353
+
354
+ - PaddingStrategy.LONGEST Pad to the longest sequence in the batch
355
+ - PaddingStrategy.MAX_LENGTH: Pad to the max length (default)
356
+ - PaddingStrategy.DO_NOT_PAD: Do not pad
357
+ The tokenizer padding sides are defined in self.padding_side:
358
+
359
+ - 'left': pads on the left of the sequences
360
+ - 'right': pads on the right of the sequences
361
+ pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
362
+ This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
363
+ `>= 7.5` (Volta).
364
+ return_attention_mask:
365
+ (optional) Set to False to avoid returning attention mask (default: set to model specifics)
366
+ """
367
+ # Load from model defaults
368
+ bos_token_id = self.sp_tokenizer[self.bos_token]
369
+ mask_token_id = self.sp_tokenizer[self.mask_token]
370
+ gmask_token_id = self.sp_tokenizer[self.gmask_token]
371
+ assert self.padding_side == "left"
372
+
373
+ required_input = encoded_inputs[self.model_input_names[0]]
374
+ seq_length = len(required_input)
375
+
376
+ if padding_strategy == PaddingStrategy.LONGEST:
377
+ max_length = len(required_input)
378
+
379
+ if max_length is not None and pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
380
+ max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
381
+
382
+ needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len(required_input) != max_length
383
+
384
+ # Initialize attention mask if not present.
385
+ if max_length is not None:
386
+ if "attention_mask" not in encoded_inputs:
387
+ if bos_token_id in required_input:
388
+ context_length = required_input.index(bos_token_id)
389
+ else:
390
+ context_length = seq_length
391
+ attention_mask = np.ones((1, seq_length, seq_length))
392
+ attention_mask = np.tril(attention_mask)
393
+ attention_mask[:, :, :context_length] = 1
394
+ attention_mask = np.bool_(attention_mask < 0.5)
395
+ encoded_inputs["attention_mask"] = attention_mask
396
+
397
+ if "position_ids" not in encoded_inputs:
398
+ if bos_token_id in required_input:
399
+ context_length = required_input.index(bos_token_id)
400
+ else:
401
+ context_length = seq_length
402
+ position_ids = np.arange(seq_length, dtype=np.int64)
403
+ mask_token = mask_token_id if mask_token_id in required_input else gmask_token_id
404
+ if mask_token in required_input:
405
+ mask_position = required_input.index(mask_token)
406
+ position_ids[context_length:] = mask_position
407
+ block_position_ids = np.concatenate(
408
+ [np.zeros(context_length, dtype=np.int64),
409
+ np.arange(1, seq_length - context_length + 1, dtype=np.int64)])
410
+ encoded_inputs["position_ids"] = np.stack([position_ids, block_position_ids], axis=0)
411
+
412
+ if needs_to_be_padded:
413
+ difference = max_length - len(required_input)
414
+
415
+ if "attention_mask" in encoded_inputs:
416
+ encoded_inputs["attention_mask"] = np.pad(encoded_inputs["attention_mask"],
417
+ pad_width=[(0, 0), (difference, 0), (difference, 0)],
418
+ mode='constant', constant_values=True)
419
+ if "token_type_ids" in encoded_inputs:
420
+ encoded_inputs["token_type_ids"] = [self.pad_token_type_id] * difference + encoded_inputs[
421
+ "token_type_ids"
422
+ ]
423
+ if "special_tokens_mask" in encoded_inputs:
424
+ encoded_inputs["special_tokens_mask"] = [1] * difference + encoded_inputs["special_tokens_mask"]
425
+ if "position_ids" in encoded_inputs:
426
+ encoded_inputs["position_ids"] = np.pad(encoded_inputs["position_ids"],
427
+ pad_width=[(0, 0), (difference, 0)])
428
+ encoded_inputs[self.model_input_names[0]] = [self.pad_token_id] * difference + required_input
429
+
430
+ return encoded_inputs
tokenizer_config.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name_or_path": "THUDM/chatglm-6b-int4",
3
+ "bos_token": "<sop>",
4
+ "eos_token": "<eop>",
5
+ "end_token": "</s>",
6
+ "gmask_token": "[gMASK]",
7
+ "mask_token": "[MASK]",
8
+ "pad_token": "<pad>",
9
+ "unk_token": "<unk>",
10
+ "remove_space": false,
11
+ "do_lower_case": false,
12
+ "tokenizer_class": "ChatGLMTokenizer",
13
+ "num_image_tokens": 0,
14
+ "auto_map": {
15
+ "AutoTokenizer": [
16
+ "tokenization_chatglm.ChatGLMTokenizer",
17
+ null
18
+ ]
19
+ }
20
+ }