ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
Paper
โข
2406.12793
โข
Published
โข
33
ChatGLM2-6B
๐ป Github Repo โข ๐ฆ Twitter โข ๐ [GLM@ACL 22] [GitHub] โข ๐ [GLM-130B@ICLR 23] [GitHub]
๐ Join our Slack and WeChat
๐Experience the larger-scale ChatGLM model at chatglm.cn
ไป็ป
ChatGLM2-6B ๆฏๅผๆบไธญ่ฑๅ่ฏญๅฏน่ฏๆจกๅ ChatGLM-6B ็็ฌฌไบไปฃ็ๆฌ๏ผๅจไฟ็ไบๅไปฃๆจกๅๅฏน่ฏๆต็ ใ้จ็ฝฒ้จๆง่พไฝ็ญไผๅคไผ็ง็นๆง็ๅบ็กไนไธ๏ผChatGLM2-6B ๅผๅ ฅไบๅฆไธๆฐ็นๆง๏ผ
- ๆดๅผบๅคง็ๆง่ฝ๏ผๅบไบ ChatGLM ๅไปฃๆจกๅ็ๅผๅ็ป้ช๏ผๆไปฌๅ จ้ขๅ็บงไบ ChatGLM2-6B ็ๅบๅบงๆจกๅใChatGLM2-6B ไฝฟ็จไบ GLM ็ๆททๅ็ฎๆ ๅฝๆฐ๏ผ็ป่ฟไบ 1.4T ไธญ่ฑๆ ่ฏ็ฌฆ็้ข่ฎญ็ปไธไบบ็ฑปๅๅฅฝๅฏน้ฝ่ฎญ็ป๏ผ่ฏๆต็ปๆๆพ็คบ๏ผ็ธๆฏไบๅไปฃๆจกๅ๏ผChatGLM2-6B ๅจ MMLU๏ผ+23%๏ผใCEval๏ผ+33%๏ผใGSM8K๏ผ+571%๏ผ ใBBH๏ผ+60%๏ผ็ญๆฐๆฎ้ไธ็ๆง่ฝๅๅพไบๅคงๅน ๅบฆ็ๆๅ๏ผๅจๅๅฐบๅฏธๅผๆบๆจกๅไธญๅ ทๆ่พๅผบ็็ซไบๅใ
- ๆด้ฟ็ไธไธๆ๏ผๅบไบ FlashAttention ๆๆฏ๏ผๆไปฌๅฐๅบๅบงๆจกๅ็ไธไธๆ้ฟๅบฆ๏ผContext Length๏ผ็ฑ ChatGLM-6B ็ 2K ๆฉๅฑๅฐไบ 32K๏ผๅนถๅจๅฏน่ฏ้ถๆฎตไฝฟ็จ 8K ็ไธไธๆ้ฟๅบฆ่ฎญ็ป๏ผๅ ่ฎธๆดๅค่ฝฎๆฌก็ๅฏน่ฏใไฝๅฝๅ็ๆฌ็ ChatGLM2-6B ๅฏนๅ่ฝฎ่ถ ้ฟๆๆกฃ็็่งฃ่ฝๅๆ้๏ผๆไปฌไผๅจๅ็ปญ่ฟญไปฃๅ็บงไธญ็้่ฟ่กไผๅใ
- ๆด้ซๆ็ๆจ็๏ผๅบไบ Multi-Query Attention ๆๆฏ๏ผChatGLM2-6B ๆๆด้ซๆ็ๆจ็้ๅบฆๅๆดไฝ็ๆพๅญๅ ็จ๏ผๅจๅฎๆน็ๆจกๅๅฎ็ฐไธ๏ผๆจ็้ๅบฆ็ธๆฏๅไปฃๆๅไบ 42%๏ผINT4 ้ๅไธ๏ผ6G ๆพๅญๆฏๆ็ๅฏน่ฏ้ฟๅบฆ็ฑ 1K ๆๅๅฐไบ 8Kใ
- ๆดๅผๆพ็ๅ่ฎฎ๏ผChatGLM2-6B ๆ้ๅฏนๅญฆๆฏ็ ็ฉถๅฎๅ จๅผๆพ๏ผๅจๅกซๅ้ฎๅท่ฟ่ก็ป่ฎฐๅไบฆๅ ่ฎธๅ ่ดนๅไธไฝฟ็จใ
ChatGLM2-6B is the second-generation version of the open-source bilingual (Chinese-English) chat model ChatGLM-6B. It retains the smooth conversation flow and low deployment threshold of the first-generation model, while introducing the following new features:
- Stronger Performance: Based on the development experience of the first-generation ChatGLM model, we have fully upgraded the base model of ChatGLM2-6B. ChatGLM2-6B uses the hybrid objective function of GLM, and has undergone pre-training with 1.4T bilingual tokens and human preference alignment training. The evaluation results show that, compared to the first-generation model, ChatGLM2-6B has achieved substantial improvements in performance on datasets like MMLU (+23%), CEval (+33%), GSM8K (+571%), BBH (+60%), showing strong competitiveness among models of the same size.
- Longer Context: Based on FlashAttention technique, we have extended the context length of the base model from 2K in ChatGLM-6B to 32K, and trained with a context length of 8K during the dialogue alignment, allowing for more rounds of dialogue. However, the current version of ChatGLM2-6B has limited understanding of single-round ultra-long documents, which we will focus on optimizing in future iterations.
- More Efficient Inference: Based on Multi-Query Attention technique, ChatGLM2-6B has more efficient inference speed and lower GPU memory usage: under the official implementation, the inference speed has increased by 42% compared to the first generation; under INT4 quantization, the dialogue length supported by 6G GPU memory has increased from 1K to 8K.
- More Open License: ChatGLM2-6B weights are completely open for academic research, and free commercial use is also allowed after completing the questionnaire.
่ฝฏไปถไพ่ต
pip install protobuf transformers==4.30.2 cpm_kernels torch>=2.0 gradio mdtex2html sentencepiece accelerate
ไปฃ็ ่ฐ็จ
ๅฏไปฅ้่ฟๅฆไธไปฃ็ ่ฐ็จ ChatGLM-6B ๆจกๅๆฅ็ๆๅฏน่ฏ๏ผ
>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True)
>>> model = AutoModel.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True).half().cuda()
>>> model = model.eval()
>>> response, history = model.chat(tokenizer, "ไฝ ๅฅฝ", history=[])
>>> print(response)
ไฝ ๅฅฝ๐!ๆๆฏไบบๅทฅๆบ่ฝๅฉๆ ChatGLM-6B,ๅพ้ซๅ
ด่งๅฐไฝ ,ๆฌข่ฟ้ฎๆไปปไฝ้ฎ้ขใ
>>> response, history = model.chat(tokenizer, "ๆไธ็กไธ็ๅบ่ฏฅๆไนๅ", history=history)
>>> print(response)
ๆไธ็กไธ็ๅฏ่ฝไผ่ฎฉไฝ ๆๅฐ็ฆ่ๆไธ่ๆ,ไฝไปฅไธๆฏไธไบๅฏไปฅๅธฎๅฉไฝ ๅ
ฅ็ก็ๆนๆณ:
1. ๅถๅฎ่งๅพ็็ก็ ๆถ้ด่กจ:ไฟๆ่งๅพ็็ก็ ๆถ้ด่กจๅฏไปฅๅธฎๅฉไฝ ๅปบ็ซๅฅๅบท็็ก็ ไน ๆฏ,ไฝฟไฝ ๆดๅฎนๆๅ
ฅ็กใๅฐฝ้ๅจๆฏๅคฉ็็ธๅๆถ้ดไธๅบ,ๅนถๅจๅไธๆถ้ด่ตทๅบใ
2. ๅ้ ไธไธช่้็็ก็ ็ฏๅข:็กฎไฟ็ก็ ็ฏๅข่้,ๅฎ้,้ปๆไธๆธฉๅบฆ้ๅฎใๅฏไปฅไฝฟ็จ่้็ๅบไธ็จๅ,ๅนถไฟๆๆฟ้ด้้ฃใ
3. ๆพๆพ่บซๅฟ:ๅจ็กๅๅไบๆพๆพ็ๆดปๅจ,ไพๅฆๆณกไธช็ญๆฐดๆพก,ๅฌไบ่ฝปๆ็้ณไน,้
่ฏปไธไบๆ่ถฃ็ไนฆ็ฑ็ญ,ๆๅฉไบ็ผ่งฃ็ดงๅผ ๅ็ฆ่,ไฝฟไฝ ๆดๅฎนๆๅ
ฅ็กใ
4. ้ฟๅ
้ฅฎ็จๅซๆๅๅกๅ ็้ฅฎๆ:ๅๅกๅ ๆฏไธ็งๅบๆฟๆง็ฉ่ดจ,ไผๅฝฑๅไฝ ็็ก็ ่ดจ้ใๅฐฝ้้ฟๅ
ๅจ็กๅ้ฅฎ็จๅซๆๅๅกๅ ็้ฅฎๆ,ไพๅฆๅๅก,่ถๅๅฏไนใ
5. ้ฟๅ
ๅจๅบไธๅไธ็ก็ ๆ ๅ
ณ็ไบๆ
:ๅจๅบไธๅไบไธ็ก็ ๆ ๅ
ณ็ไบๆ
,ไพๅฆ็็ตๅฝฑ,็ฉๆธธๆๆๅทฅไฝ็ญ,ๅฏ่ฝไผๅนฒๆฐไฝ ็็ก็ ใ
6. ๅฐ่ฏๅผๅธๆๅทง:ๆทฑๅผๅธๆฏไธ็งๆพๆพๆๅทง,ๅฏไปฅๅธฎๅฉไฝ ็ผ่งฃ็ดงๅผ ๅ็ฆ่,ไฝฟไฝ ๆดๅฎนๆๅ
ฅ็กใ่ฏ็ๆ
ขๆ
ขๅธๆฐ,ไฟๆๅ ็ง้,็ถๅ็ผๆ
ขๅผๆฐใ
ๅฆๆ่ฟไบๆนๆณๆ ๆณๅธฎๅฉไฝ ๅ
ฅ็ก,ไฝ ๅฏไปฅ่่ๅจ่ฏขๅป็ๆ็ก็ ไธๅฎถ,ๅฏปๆฑ่ฟไธๆญฅ็ๅปบ่ฎฎใ
ๅ ณไบๆดๅค็ไฝฟ็จ่ฏดๆ๏ผๅ ๆฌๅฆไฝ่ฟ่กๅฝไปค่กๅ็ฝ้กต็ๆฌ็ DEMO๏ผไปฅๅไฝฟ็จๆจกๅ้ๅไปฅ่็ๆพๅญ๏ผ่ฏทๅ่ๆไปฌ็ Github Repoใ
For more instructions, including how to run CLI and web demos, and model quantization, please refer to our Github Repo.
Change Log
- v1.0
ๅ่ฎฎ
ๆฌไปๅบ็ไปฃ็ ไพ็ ง Apache-2.0 ๅ่ฎฎๅผๆบ๏ผChatGLM2-6B ๆจกๅ็ๆ้็ไฝฟ็จๅ้่ฆ้ตๅพช Model Licenseใ
ๅผ็จ
ๅฆๆไฝ ่งๅพๆไปฌ็ๅทฅไฝๆๅธฎๅฉ็่ฏ๏ผ่ฏท่่ๅผ็จไธๅ่ฎบๆใ
If you find our work helpful, please consider citing the following paper.
@misc{glm2024chatglm,
title={ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools},
author={Team GLM and Aohan Zeng and Bin Xu and Bowen Wang and Chenhui Zhang and Da Yin and Diego Rojas and Guanyu Feng and Hanlin Zhao and Hanyu Lai and Hao Yu and Hongning Wang and Jiadai Sun and Jiajie Zhang and Jiale Cheng and Jiayi Gui and Jie Tang and Jing Zhang and Juanzi Li and Lei Zhao and Lindong Wu and Lucen Zhong and Mingdao Liu and Minlie Huang and Peng Zhang and Qinkai Zheng and Rui Lu and Shuaiqi Duan and Shudan Zhang and Shulin Cao and Shuxun Yang and Weng Lam Tam and Wenyi Zhao and Xiao Liu and Xiao Xia and Xiaohan Zhang and Xiaotao Gu and Xin Lv and Xinghan Liu and Xinyi Liu and Xinyue Yang and Xixuan Song and Xunkai Zhang and Yifan An and Yifan Xu and Yilin Niu and Yuantao Yang and Yueyan Li and Yushi Bai and Yuxiao Dong and Zehan Qi and Zhaoyu Wang and Zhen Yang and Zhengxiao Du and Zhenyu Hou and Zihan Wang},
year={2024},
eprint={2406.12793},
archivePrefix={arXiv},
primaryClass={id='cs.CL' full_name='Computation and Language' is_active=True alt_name='cmp-lg' in_archive='cs' is_general=False description='Covers natural language processing. Roughly includes material in ACM Subject Class I.2.7. Note that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'}
}
- Downloads last month
- 561,959
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
๐
Ask for provider support
Model tree for zai-org/chatglm2-6b
Spaces using zai-org/chatglm2-6b 100
Papers for zai-org/chatglm2-6b
GLM-130B: An Open Bilingual Pre-trained Model
Paper
โข
2210.02414
โข
Published
โข
3
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Paper
โข
2103.10360
โข
Published
โข
2
Fast Transformer Decoding: One Write-Head is All You Need
Paper
โข
1911.02150
โข
Published
โข
9