Search results

Expose spans for mapping tokens to input characters

Status: Open.

#39 In alasdairforsythe/tokenmonster;

· iiLaurens opened on Jan 28, 2025

Preferred citation <code>bibtex</code>

Status: Open.

#35 In alasdairforsythe/tokenmonster;

· konstantinjdobler opened on May 8, 2024

Hangs with PyTorch data loaders when <code>num_workers &gt; 0</code>

Status: Open.

#34 In alasdairforsythe/tokenmonster;

· ntoxeg opened on Mar 22, 2024

Update on multilingual

Status: Open.

#32 In alasdairforsythe/tokenmonster;

· kerighan opened on Feb 27, 2024

Inquiry on Extending Algorithm to Other Languages

Status: Open.

#30 In alasdairforsythe/tokenmonster;

· dsdanielpark opened on Dec 12, 2023

Special tokens not showing up correctly when tokenized.

Status: Open.

#29 In alasdairforsythe/tokenmonster;

· amazingvince opened on Nov 5, 2023

What is the difference between <code>50256-consistent-oneword</code> and <code>50256-consistent</code>?

Status: Open.

#24 In alasdairforsythe/tokenmonster;

· Calvinnncy97 opened on Sep 6, 2023

C implementation

Status: Open.

#17 In alasdairforsythe/tokenmonster;

· abb128 opened on Aug 3, 2023

Meaning of C and D

Status: Open.

#15 In alasdairforsythe/tokenmonster;

· Maxscha opened on Aug 3, 2023

HUggingface tokenizer coming soon?

Status: Open.

#4 In alasdairforsythe/tokenmonster;

· kyegomez opened on Jun 4, 2023