More information about the training of these models can be found in the article Learning Word Vectors for 157 Languages. We used the Stanford word segmenter for Chinese, Mecab for Japanese and UETsegmenter for Vietnamese.įor languages using the Latin, Cyrillic, Hebrew or Greek scripts, we used the tokenizer from the Europarl preprocessing tools.įor the remaining languages, we used the ICU tokenizer. These text models can easily be loaded in Python using the following code: import ioįin = io.open(fname, 'r', encoding= 'utf-8', newline= '\n', errors= 'ignore')ĭata] = map(float, tokens) In the text format, each line contain a word followed by its vector.Įach value is space separated, and words are sorted by frequency in descending order. Where the file oov_words.txt contains out-of-vocabulary words. Using the binary models, vectors for out-of-vocabulary words can be obtained with $.
The word vectors are available in both binary and text formats. Or save it for later use: > ft.save_model( 'cc.en.100.bin') Then you can use ft model object as usual: > ft.get_word_vector( 'hello').shape ft = fasttext.load_model( 'cc.en.300.bin')
