philschmid

/

gemma-tokenizer-chatml

like 27

Hi @philschmid thanks for sharing this.

Can you share how you updated the vocab with the replacement of the chatml tokens, amongst other values? I'm trying to use some of the <unusedXX> tokens, but my changes are vanishing after saving the tokenizer and then re-loading those changes.