Difference Between Max Length Of Word Ngrams And Size Of Context Window

In the description of the fasttext library for python https://github.com/facebookresearch/fastText/tree/master/python for training a supervised model there are different arguments,

Solution 1:

First, we use the train_unsupervised API to create a Word-Representation Model. There are two techniques that we can use, skipgram and cbow. On the other hand, we use the train_supervised API to create Text Classification Model. You are asking about the train_supervised API, so I will stick to it.

The way that text classification work in fasttext, is to first represent the word using skipgram by default. Then, use these word-vectors learned from the skipgram model to classify your input text. The two parameters that you asked about (ws and wordNgrams) are related to the skipgram/cbow model.

The following image contains a simplified illustration of how we are using our input text to train skipgram model. Here, we defined the ws parameter as 2 and wordNgrams as 1.

As we can see, we have only one text in our training data which is The quick brown fox jumps over the lazy dog. We defined the context window to be two, which means that we will create a window whose center is center word and the next/previous two words within the window are target words. Then, we move this window a word at a time. The bigger the window size is, the more training samples you have for your model, the more overfitted the model becomes given a small sample of data.

That's for our first argument ws. According to the second argument wordNgrams, if we set wordNgrams to 2, it will consider two-word pairs like the following image. (The ws in the following image is one for simplicity)

Ref

• Check this link which contains the source code for the train_supervised method.

• There is a major difference between skipgram and cbow that can be summarized in the following image: