⏳ js-tiktoken on Cloudflare Pages

An elegant LLM chat UI forked from chatgpt-demo of @anse-app. Index site at https://free-chat.asia

OpenShield is a new generation security layer for AI models

This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.

Count tokens in a text file.

The client connects to a frontend server using a middleware load balancer that manages Docker instances with a "least connected" algorithm. New instances register dynamically, and a monitoring endpoint shows logs and request counts. The service counts tokens in text.

Byte Pair Encoding (BPE) in the Zig programming language (0.13.0)

LLM trained on reddit short stories!

A fast RWKV Tokenizer written in Rust

Count tokens to determine cost differential of ChatGPT Plus subscription and ChatGPT API

Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion

Tiktoken-counter as standalone API

C API for tiktoken-rs

PHP port for openai/tiktoken (most)

This is a port of the tiktoken

The projects which made by me while self learning.

Kotlin multiplatform BPE tokenizer library for OpenAI models

Tiktoken for Crystalists

Online playground for OpenAPI tokenizers

This project uses the CrewAI framework to automate stock analysis, enabling AI agents to collaborate and execute complex tasks efficiently. Example stock: Nvidia. Technologies include Python, CrewAI, Unstructured, PyOWM, Tools, Wikipedia, yFinance, SEC-API, tiktoken, faiss-cpu, python-dotenv, langchain-community, langchain-core, and OpenAI.