chore(trainer): add and improve trainer signature

[shenxiangzhuang]

No description provided.

[ArthurZucker]

This file is automatically generated! # Generated content DO NOT EDIT What you can do is update the WordLevelTrainer 's signature in the bindings!

[shenxiangzhuang]

Hi @ArthurZucker , thanks for the reminder, I didn't see that warning before! I have restored the changes and added the signatures for BpeTrainer and WordLevelTrainer. Beside, I made a tiny space improve for WordPieceTrainer. Please review it again~

[ArthurZucker]

Make sure lists and dict are not defaulted to [] or {} and then run make in bindings/python to generate the init file

[ArthurZucker]

Suggested change

	text_signature = "(self, vocab_size=30000, min_frequency=0, show_progress=True, special_tokens=[], limit_alphabet=None, initial_alphabet=[], continuing_subword_prefix=None, end_of_word_suffix=None, max_token_length=None, words={})"
	text_signature = "(self, vocab_size=30000, min_frequency=0, show_progress=True, special_tokens=None, limit_alphabet=None, initial_alphabet=None, continuing_subword_prefix=None, end_of_word_suffix=None, max_token_length=None, words=None)"

default to mutable is a mess in python

[shenxiangzhuang]

Well, I also notice this. There is a question, I find that text_signature in WordPieceTrainer already use []:

#[new]
    #[pyo3(
        signature = (** kwargs),
        text_signature = "(self, vocab_size=30000, min_frequency=0, show_progress=True, special_tokens=[], limit_alphabet=None, initial_alphabet= [],continuing_subword_prefix=\"##\", end_of_word_suffix=None)"
    )]
    pub fn new(kwargs: Option<&Bound<'_, PyDict>>) -> PyResult<(Self, PyTrainer)> {

And, I tested it in python, and find that the default values are just same as the text_signature:

In [5]: import tokenizers; tokenizers.trainers.WordPieceTrainer()
Out[5]: WordPieceTrainer(WordPieceTrainer(bpe_trainer=BpeTrainer(min_frequency=0, vocab_size=30000, show_progress=True, special_tokens=[], limit_alphabet=None, initial_alphabet=[], continuing_subword_prefix="##", end_of_word_suffix=None, max_token_length=None, words={})))

And the doc for WordPieceTrainer is consistent with the real default values:

Init signature:
tokenizers.trainers.WordPieceTrainer(
    self,
    vocab_size=30000,
    min_frequency=0,
    show_progress=True,
    special_tokens=[],
    limit_alphabet=None,
    initial_alphabet=[],
    continuing_subword_prefix='##',
    end_of_word_suffix=None,
)
Docstring:     
Trainer capable of training a WordPiece model

...(ignored)

So maybe we should keep the text_signature same the real default value to avoid introduce break changes and confuse the users(due to the inconsistence) too.

[ArthurZucker]

Ok no worries then

[shenxiangzhuang]

To ease the review process, I copied current default values of different trainers here:

In [1]: import tokenizers

In [2]: tokenizers.trainers.BpeTrainer()
Out[2]: BpeTrainer(BpeTrainer(min_frequency=0, vocab_size=30000, show_progress=True, special_tokens=[], limit_alphabet=None, initial_alphabet=[], continuing_subword_prefix=None, end_of_word_suffix=None, max_token_length=None, words={}))

In [3]: tokenizers.trainers.UnigramTrainer()
Out[3]: UnigramTrainer(UnigramTrainer(show_progress=True, vocab_size=8000, n_sub_iterations=2, shrinking_factor=0.75, special_tokens=[], initial_alphabet=[], unk_token=None, max_piece_length=16, seed_size=1000000, words={}))

In [4]: tokenizers.trainers.WordLevelTrainer()
Out[4]: WordLevelTrainer(WordLevelTrainer(min_frequency=0, vocab_size=30000, show_progress=True, special_tokens=[], words={}))

In [5]: tokenizers.trainers.WordPieceTrainer()
Out[5]: WordPieceTrainer(WordPieceTrainer(bpe_trainer=BpeTrainer(min_frequency=0, vocab_size=30000, show_progress=True, special_tokens=[], limit_alphabet=None, initial_alphabet=[], continuing_subword_prefix="##", end_of_word_suffix=None, max_token_length=None, words={})))

In [6]: tokenizers.__version__
Out[6]: '0.21.4-dev.0'

[ArthurZucker]

Thanks 😉

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.