BPE implementation tokenize beyond word boundaries?

[asafam]

Thanks for a great review and clear code. I have a question: Shouldn't the training in BPE be applied at the word level? Your implementation can generate merged tokens that include multiple words.

[rasbt]

Thanks for the comment. I thought the words = line.split() would prevent that. Do you have an example where you had this issue occurring?

[asafam]

In the initial tokenization step int the train() method, it looks like the code processes processed_text as a flat sequence of characters, rather than splitting it into words first (i.e., no whitespace-level chunking before tokenization):

token_ids = [self.inverse_vocab[char] for char in processed_text]

To my best knowledge, standard BPE tokenization expects the input to be at the word level before applying merges. Without word boundaries, the merging ends up crossing word boundaries.

Would love to hear your thoughts — maybe there’s a reason behind this choice, I'm getting to the depths of it now and may be missing a key point.

Anyhow, thanks for putting it out. That's a great blog post and it was very helpful to me.