Features

Introduction

This document describes all the features provided by the wordpredictor package. It first describes how to generate n-gram models. Next it describes how to evaluate the performance of the n-gram models. Finally it describes how to make word predictions using the n-gram model.

Model Generation

The wordpredictor package provides several classes that can be used to generate n-gram models. These classes may be used to generate n-gram models step by step. An alternative is to use the ModelGenerator class which combines all the steps and provides a single method for generating n-gram models.

The following steps are involved in generating n-gram models:

Data Exploration

The first step in generating a n-gram model is data exploration. This involves determining the type of textual content and various text related statistics. The type of text may be news content, blog posts, Twitter feeds, product reviews, customer chat history etc. Example of text related statistics are line count, word count, average line length and input file size.

It is also important to determine the unwanted words and symbols in the data such as vulgar words, punctuation symbols, non-alphabetical symbols etc. The wordpredictor package provides the DataAnalyzer class which can be used to find out statistics about the input data. The following example shows how to get statistics on all text files within a folder:

The word count of a text file can be fetched using the command: cat file-name | wc -w. This command should work on all Unix based systems.

Data Sampling

The next step is to generate training, testing and validation samples from the input text file. If there are many input text files, then they can be combined to a single file using the command: cat file-1 file-2 file3 > output-file. The contents of the combined text file may need to be randomized.

The wordpredictor package provides the DataSampler class which can be used to generate a random sample containing given number of lines. The following example shows how to generate a random sample of size 10 Mb from an input text file:

Usually we need a train data set for generating the n-gram model. A test data set for testing the model and a validation data set for evaluating the performance of the model. The following example shows how to generate the train, test and validation files. The train file contains the first 80% of the lines, the test set contains the next 10% of the lines. The remaining lines are in the validation set.

The data in the validation file must be different from the data in the train file. Otherwise it can result in over-fitting of the model. When a model is over-fitted, the model evaluation results will be exaggerated, overly optimistic and unreliable. So care should be taken to ensure that the data in the validation and train files is different.

In the above example, dir parameter is the directory containing the input.txt file and the generated test, validation and train data files.

Data Cleaning

The next step is to remove unwanted symbols and words from the input text file. This reduces the memory requirement of the n-gram model and makes it more efficient. Example of unwanted words are vulgar words, words that are not part of the vocabulary, punctuation, numbers, non-printable characters and extra spaces.

The wordpredictor package provides the DataCleaner class which can be used to remove unwanted words and symbols from text files. The following example shows how to clean a given text file:

The clean_file method reads a certain number of lines at a time, cleans the lines of text and saves them to an output text file. It can be used for cleaning large text files.

Tokenization

The next step is to generate n-gram tokens from the cleaned text file. The TokenGenerator class allows generating n-gram tokens of given size from a given input text file. The following example shows how to generate n-grams tokens of size 1,2,3 and 4:

The above code generates the files n1.RDS, n2.RDS, n3.RDS and n4.RDS in the data directory. These files contains n-gram tokens along with their frequencies. N-grams of larger size provide more context. Usually n-grams of size 4 are generated.

Two important customization options supported by the TokenGenerator class are min_freq and stem_words. min_freq sets minimum frequency for n-gram tokens. All n-gram tokens with frequency less than min_freq are excluded.

The stem_words option is used to transform n-gram prefix components to their stems. The next word is not transformed.

The n-gram token frequencies may be analyzed using the DataAnalyzer class. The following example displays the top most occurring 2-gram tokens:

The following example shows the distribution of word frequencies:

The following example returns top 10 2-gram tokens that start with and_:

Prefix Frequency
and_the 2
and_cart 1
and_fired 1
and_forget 1
and_leave 1
and_open 1
and_out 1
and_phrase 1
and_say 1
and_tea 1

Transition Probabilities

The next step in generating the n-gram model is to generate transition probabilities (tp) from the n-gram files. The TPGenerator class is used to generate the tps. For each n-gram token file a corresponding tp file is generated.

The tp files are then combined into a single file containing tp data for n-grams of size 1, 2, 3, 4 etc.

The following example shows how to generate combined tps for n-grams of size 1, 2, 3 and 4:

The above code produces the file model-4.RDS.

The model file

The final step is to generate a n-gram model file from the files generated in the previous steps. The Model class contains the method load_model, which reads the combined tps files and other files that are used by the model. An instance of the Model class represents the n-gram model.

Generating the model in one step

All the previous steps may be combined into a single step. The ModelGenerator class allows generating the final n-gram model using a single method call. The following example generates a n-gram model using default data cleaning and tokenization options:

Evaluating the model performance

The wordpredictor package provides the ModelEvaluator class for evaluating the performance of the generated n-gram model. Intrinsic and Extrinsic evaluation are supported. Also the performance of several n-gram models may be compared.

The following example performs Intrinsic evaluation. It measures the Perplexity score for each sentence in the validation.txt file, that was generated in the data sampling step. It returns the minimum, mean and maximum Perplexity score for each line.

# The required files
rf <- c("def-model.RDS", "validate-clean.txt")
# The test environment is setup
ed <- setup_env(rf, ve)
#>  [1] "CITATION"    "DESCRIPTION" "INDEX"       "LICENSE"     "Meta"       
#>  [6] "NAMESPACE"   "NEWS.md"     "R"           "demo"        "examples"   
#> [11] "extdata"     "help"        "html"

# The model file name
mfn <- paste0(ed, "/def-model.RDS")
# The path to the cleaned validation file
vfn <- paste0(ed, "/validate-clean.txt")
# ModelEvaluator class object is created
me <- ModelEvaluator$new(mf = mfn, ve = ve)
# The intrinsic evaluation is performed on first 20 lines
stats <- me$intrinsic_evaluation(lc = 20, fn = vfn)

# The test environment is cleaned up
clean_up(ve)

The following example performs Extrinsic evaluation. It measures the accuracy score for each sentence in validation.txt file. For each sentence the model is used to predict the last word in the sentence given the previous words. If the last word was correctly predicted, then the prediction is considered to be accurate.

# The required files
rf <- c("def-model.RDS", "validate-clean.txt")
# The test environment is setup
ed <- setup_env(rf, ve)
#>  [1] "CITATION"    "DESCRIPTION" "INDEX"       "LICENSE"     "Meta"       
#>  [6] "NAMESPACE"   "NEWS.md"     "R"           "demo"        "examples"   
#> [11] "extdata"     "help"        "html"

# The model file name
mfn <- paste0(ed, "/def-model.RDS")
# The path to the cleaned validation file
vfn <- paste0(ed, "/validate-clean.txt")
# ModelEvaluator class object is created
me <- ModelEvaluator$new(mf = mfn, ve = ve)
# The intrinsic evaluation is performed on first 100 lines
stats <- me$extrinsic_evaluation(lc = 100, fn = vfn)

# The test environment is cleaned up
clean_up(ve)

Making word predictions

The n-gram model generated in the previous step can be used to predict the next word given a set of words. The following example shows how to predict the next word. It returns the 3 possible next words along with their probabilities.

# The required files
rf <- c("def-model.RDS")
# The test environment is setup
ed <- setup_env(rf, ve)
#>  [1] "CITATION"    "DESCRIPTION" "INDEX"       "LICENSE"     "Meta"       
#>  [6] "NAMESPACE"   "NEWS.md"     "R"           "demo"        "examples"   
#> [11] "extdata"     "help"        "html"

# The model file name
mfn <- paste0(ed, "/def-model.RDS")
# An object of class ModelPredictor is created. The mf parameter is the name of
# the model file that was generated in the previous example.
mp <- ModelPredictor$new(mf = mfn, ve = ve)
# Given the words: "how are", the next word is predicted. The top 3 most likely
# next words are returned along with their respective probabilities.
res <- mp$predict_word(words = "how are", 3)

# The test environment is cleaned up
clean_up(ve)