Tokenizer

#41

by startlightquyet - opened Apr 13, 2023

Apr 13, 2023

How to train tokenizer custum dataset language not en and chinese . Continue 13000 vocab english and chinese model chatglm6b concat tokenizer custum after traning

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment