Skip to content

Pull requests: openai/tiktoken

New pull request New
71 Open 137 Closed
71 Open 137 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Handle Data Gym vocab files without trailing newline
#598 opened Aug 18, 2026 by tandede Loading…
Validate base64 tokens when loading BPE files
#592 opened Aug 12, 2026 by weivwang Loading…
Add get_token_length methods to more cheaply calculate token lengths
#591 opened Aug 11, 2026 by thomasaarholt Loading…
Add DisallowedSpecialTokenError (ValueError subclass) for disallowed special tokens (#290)
#590 opened Aug 7, 2026 by JayYarlagadda Loading…
1
Fix remaining panics from #400 in encode_ordinary/_encode_bytes/encode_with_unstable
#587 opened Jul 30, 2026 by yanmo42 Loading…
fix: Unable to route GET requests through proxy
#584 opened Jul 23, 2026 by thiagobarbosa Loading…
2 tasks done
1
Avoid panic when byte_pair_encode is given an empty piece
#583 opened Jul 20, 2026 by devYRPauli Loading…
Handle surrogate pairs in encode_with_unstable
#581 opened Jul 16, 2026 by devYRPauli Loading…
1
3
Add CI support for Windows ARM64
#580 opened Jul 14, 2026 by chinazhangchao Loading…
3
Return ValueError instead of panicking on duplicate mergeable_ranks
#579 opened Jul 13, 2026 by devYRPauli Loading…
fix: apply surrogate-pair fixup in encode_with_unstable
#575 opened Jul 8, 2026 by steps-re Loading…
1
fix: reject encoders missing single-byte tokens to prevent reachable panic (#568)
#570 opened Jun 14, 2026 by yen0304 Loading…
4 tasks done
1
fix(model): add exact mapping for gpt-4.5
#569 opened Jun 12, 2026 by yen0304 Loading…
fix(py): make encode_with_unstable handle surrogates like encode (#541)
#567 opened Jun 7, 2026 by MasterOfLogic1 Loading…
1
perf: speed up batch tokenization and decoding hot paths
#561 opened May 27, 2026 by eonr Loading…
Fix CoreBPE::new type inference failure
#558 opened May 24, 2026 by fallintoplace Loading…
1
fix(py): make encode_with_unstable handle surrogates like encode (#541)
#556 opened May 24, 2026 by LeSingh1 Loading…
1
Repair surrogates in encode_with_unstable (Fixes #541)
#553 opened May 23, 2026 by jbbqqf Loading…
1
Add lexer fast path for o200k_base, cl100k_base, and gpt-2 pretokenization patterns; custom regexes fall back to fancy_regex
#552 opened May 22, 2026 by augustasio Loading…
Add 2-byte pair lookup table to skip hashmap for initial BPE merge scan
#551 opened May 22, 2026 by augustasio Loading…
bench: add tail-latency benchmark for encode_ordinary_batch
#548 opened May 20, 2026 by alobroke Loading…
1
build: enable line tables in release builds so backtraces show file:line (#103)
#546 opened May 18, 2026 by LeSingh1 Loading…
1
fix(py): report element count in buffer.shape, not byte length (#405)
#545 opened May 18, 2026 by LeSingh1 Loading…
1
1
Use user-specific cache directory by default
#544 opened May 15, 2026 by MukundaKatta Contributor Loading…
1
Add HTTP timeout to read_file network fetch
#539 opened May 9, 2026 by Spectual Loading…
1 of 3 tasks
ProTip! no:milestone will show everything without a milestone.