A PHP implementation of OpenAI's BPE tokenizer tiktoken.
-
Updated
Jun 16, 2024 - PHP
A PHP implementation of OpenAI's BPE tokenizer tiktoken.
Add a description, image, and links to the special-tokens topic page so that developers can more easily learn about it.
To associate your repository with the special-tokens topic, visit your repo's landing page and select "manage topics."