What is word segmentation?

Word segmentation (also called tokenization) is the process of splitting text into a list of words. Humans can do this pretty easily, but computers need help sometimes. At a higher level, you can think of segmentation as a way of boosting character-level models that also makes them more human-interpretable.

Setup

First, download the IMDB Movie Reviews Dataset from here. Consolidate the reviews into a reviews.txt file, with one review per line.

Basic segmentation methods

The Python standard library comes with many useful methods for strings. The split method is one that can be used for very basic segmentation tasks.

Let’s pick a random movie review.

import re, random

reviews = open('reviews.txt').readlines()
text = random.choice(reviews)
words = re.findall('\w+',text)
print(words)

Split method

Now let’s try it out.

words = text.split()
print(words)

Here is the result:

[‘This’, ‘film’, ‘is’, ‘undoubtedly’, ‘one’, ‘of’, ‘the’, ‘greatest’, ‘landmarks’, ‘in’, ‘history’, ‘of’, ‘cinema.’, ‘By’, ‘seeing’, ‘this’, ‘film,we’, ‘can’, ‘only’, ‘retrospectively’, ‘notice’, ‘that’, ‘world’, ‘cinema’, ‘in’, ‘1950s’, ‘had’, ‘such’, ‘a’, ‘purely’, ‘humanistic’, ‘dramaturgy,such’, ‘a’, ‘strong’, ‘and’, ‘adequate’, ‘use’, ‘of’, ‘sound-image’, ‘montage,and’, ‘almost’, ‘religious’, ‘admiration’, ‘of’, ‘ethical’, ‘choices’, ‘in’, ‘human’, ‘life.’, ‘Cinema’, ‘was’, ‘then’, ‘not’, ‘only’, ‘one’, ‘form’, ‘of’, ‘arts.’, ‘It’, ‘was’, ‘much’, ‘higher’, ‘than’, ‘ordinary’, ‘life’, ‘and’, ‘it’, ‘gave’, ‘many’, ‘people’, ‘hope’, ‘to’, ‘live’, ‘after’, ‘the’, ‘tragic’, ‘war.’, ‘It’, ‘is’, ‘said,’, ‘that’, ‘even’, ‘Picasso’, ‘was’, ‘moved’, ‘and’, ‘cried’, ‘that’, ‘such’, ‘a’, ‘work’, ‘of’, ‘art’, ‘can’, ‘appear’, ‘only’, ‘once’, ‘in’, ‘100’, ‘years!’, ‘Audience’, ‘that’, ‘time’, ‘was’, ‘also’, ‘different.’, ‘I’, ‘read’, ‘that’, ‘after’, ‘seeing’, “Kurosawa’s”, ‘”Ikiru(Live)”‘, ‘in’, ‘its’, ‘first’, ‘release,’, ‘young’, ‘couple’, ‘quietly’, ‘told’, ‘each’, ‘other,”It’, ‘is’, ‘a’, ‘good’, ‘film,’, “isn’t”, ‘it?”.’, ‘I’, ‘think,contemporary’, ‘cinema,’, ‘though’, ‘technically’, ‘developed’, ‘and’, ‘opened’, ‘some’, ‘new’, ‘narrative’, ‘perspective,’, ‘has’, ‘lost’, ‘the’, ‘most’, ‘important—reliance’, ‘of’, ‘audience.Cienma’, ‘was’, ‘once’, ‘really’, ‘the’, ‘most’, ‘popular’, ‘art’, ‘from’, ‘and,’, ‘unlike’, ‘modern’, ‘fine’, ‘arts’, ‘and’, ‘contemporary’, ‘music,gave’, ‘millions’, ‘of’, ‘people’, ‘hope’, ‘and’, ‘ideals.’, ‘In’, ‘this’, ‘point’, ‘of’, ‘view,”Letyat’, ‘zhuravli”‘, ‘must’, ‘be’, ‘in’, ‘the’, ‘pantheon’, ‘of’, ‘classics’, ‘of’, ‘all’, ‘the’, ‘time,’, ‘as’, ‘”City’, ‘light”,”Ikiru”‘, ‘and’, ‘”La’, ‘Strada”.’]

This result isn’t very good, because punctuation is mixed-in with regular words.

Regular expressions

Let’s use re.

words = re.findall('\w+',text)
print(words)

Here is the result:

[‘This’, ‘film’, ‘is’, ‘undoubtedly’, ‘one’, ‘of’, ‘the’, ‘greatest’, ‘landmarks’, ‘in’, ‘history’, ‘of’, ‘cinema’, ‘By’, ‘seeing’, ‘this’, ‘film’, ‘we’, ‘can’, ‘only’, ‘retrospectively’, ‘notice’, ‘that’, ‘world’, ‘cinema’, ‘in’, ‘1950s’, ‘had’, ‘such’, ‘a’, ‘purely’, ‘humanistic’, ‘dramaturgy’, ‘such’, ‘a’, ‘strong’, ‘and’, ‘adequate’, ‘use’, ‘of’, ‘sound’, ‘image’, ‘montage’, ‘and’, ‘almost’, ‘religious’, ‘admiration’, ‘of’, ‘ethical’, ‘choices’, ‘in’, ‘human’, ‘life’, ‘Cinema’, ‘was’, ‘then’, ‘not’, ‘only’, ‘one’, ‘form’, ‘of’, ‘arts’, ‘It’, ‘was’, ‘much’, ‘higher’, ‘than’, ‘ordinary’, ‘life’, ‘and’, ‘it’, ‘gave’, ‘many’, ‘people’, ‘hope’, ‘to’, ‘live’, ‘after’, ‘the’, ‘tragic’, ‘war’, ‘It’, ‘is’, ‘said’, ‘that’, ‘even’, ‘Picasso’, ‘was’, ‘moved’, ‘and’, ‘cried’, ‘that’, ‘such’, ‘a’, ‘work’, ‘of’, ‘art’, ‘can’, ‘appear’, ‘only’, ‘once’, ‘in’, ‘100’, ‘years’, ‘Audience’, ‘that’, ‘time’, ‘was’, ‘also’, ‘different’, ‘I’, ‘read’, ‘that’, ‘after’, ‘seeing’, ‘Kurosawa’, ‘s’, ‘Ikiru’, ‘Live’, ‘in’, ‘its’, ‘first’, ‘release’, ‘young’, ‘couple’, ‘quietly’, ‘told’, ‘each’, ‘other’, ‘It’, ‘is’, ‘a’, ‘good’, ‘film’, ‘isn’, ‘t’, ‘it’, ‘I’, ‘think’, ‘contemporary’, ‘cinema’, ‘though’, ‘technically’, ‘developed’, ‘and’, ‘opened’, ‘some’, ‘new’, ‘narrative’, ‘perspective’, ‘has’, ‘lost’, ‘the’, ‘most’, ‘important’, ‘reliance’, ‘of’, ‘audience’, ‘Cienma’, ‘was’, ‘once’, ‘really’, ‘the’, ‘most’, ‘popular’, ‘art’, ‘from’, ‘and’, ‘unlike’, ‘modern’, ‘fine’, ‘arts’, ‘and’, ‘contemporary’, ‘music’, ‘gave’, ‘millions’, ‘of’, ‘people’, ‘hope’, ‘and’, ‘ideals’, ‘In’, ‘this’, ‘point’, ‘of’, ‘view’, ‘Letyat’, ‘zhuravli’, ‘must’, ‘be’, ‘in’, ‘the’, ‘pantheon’, ‘of’, ‘classics’, ‘of’, ‘all’, ‘the’, ‘time’, ‘as’, ‘City’, ‘light’, ‘Ikiru’, ‘and’, ‘La’, ‘Strada’]

This result isn’t too bad, but there are still some problems. First, there is information loss associated with the absence of punctuation and other non-alphanumeric characters, especially in UTF-8. Second, this method does not consider words that contain spaces. “New York City” could really be considered one word, even though it contains spaces. Let’s see if third party libraries can do any better.

Popular word segmentation libraries

NLTK

NLTK stands for Natural Language Toolkit, initially released in 2001. It was designed for research, and is not typically considered “production-ready”, but has stood the test of time. It is probably the most well-known natural language processing library in Python, and is very good for some tasks, such as parse trees.

Required packages:

nltk.download('punkt')

Tokenization method:

from nltk.tokenize import word_tokenize

tokens = word_tokenize(text)

SpaCy

SpaCy is an industrial strength NLP library with a beautiful API. It is extensible, and includes built-in methods for performing common tasks, such as entity recognition.

Required packages:

python -m spacy download en

Tokenization method:

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp(text)
tokens = [token.text for token in doc]

stanfordnlp

Stanford has been a trailblazer in NLP since the beginning. stanfordnlp is a library written on their Java-based CoreNLP library. Their library currently supports a Bi-Directional Long-Short Term Memory Network with GPU acceleration.

Required packages:

stanfordnlp.download('en')

Tokenization method:

import stanfordnlp

nlp = stanfordnlp.Pipeline()
doc = nlp(text)
print([word.text for sentence in doc.sentences for word in sentence.tokens])

Other libraries

There are other popular ones, such as TextBlob, flair, Intel’s NLP Architect, and AllenNLP, but they don’t use their own segmentation methods. Gensim provides a tokenization function in their utils module as a convenience, but the results are alphanumeric only and lowercase, similar to the regular expression previously mentioned. Most algorithm specific libraries use other methods, for example BERT uses an import of a previously made vocab file.

Unsupervised word segmentation using SentencePiece

Segmentation works in the reverse direction. Given a dictionary of all known words and a token ID sequence, we can reconstruct the original text. This implies there is no information loss in the tokenized version.

Instead of using a large, language-specific package, we can use a grammar-agnostic model that learns a segmentation model from raw bytes. This means the same architecture can be used for many different types of text, including logographic languages like Chinese. We can also skip downloading large language models, and updating new terms is a breeze.

Now let’s do the necessary imports. If you do not have sentencepiece installed, use pip install sentencepiece.

import os
import sentencepiece as spm

Time to train our model.

if not os.path.isfile('reviews.model'):
    spm.SentencePieceTrainer.Train('--input=reviews.txt --model_prefix=reviews --vocab_size=20000 --split_by_whitespace=False')

sp = spm.SentencePieceProcessor()
sp.load('reviews.model')

Let’s inspect our vocabulary, in the newly created pos-train.vocab file.

Co	-10.4889
▁Old	-10.4895
▁bigger	-10.4899
▁or▁two	-10.4903
▁give▁it▁a	-10.4903
▁grace	-10.4904
▁carried	-10.4904
▁love▁and	-10.4912
▁on▁film	-10.4915
oe	-10.4918
▁gags	-10.492
uring	-10.4922
▁legendary	-10.4924
▁department	-10.4936
▁creates	-10.4939
▁as▁a▁child	-10.4943
ress	-10.4948
▁VHS	-10.4951
▁vi	-10.4955
▁profound	-10.4955
▁broke	-10.4964
▁up▁for	-10.4964
▁years▁of	-10.4967
▁but▁I▁think	-10.4968
t▁do	-10.4969
▁heavily	-10.497
▁tender	-10.497
▁contest	-10.4972
▁news	-10.4974
▁fly	-10.4975
▁Britain	-10.4975

Now let’s use the SentencePiece tokenizer model to tokenize an unseen sentence.

sentence = "The quick brown fox jumps over the lazy dog"

sp.EncodeAsPieces(sentence)
>> ['▁The', '▁quick', '▁brown', '▁fox', '▁jumps', '▁over▁the', '▁lazy', '▁dog']

sp.EncodeAsIds(sentence)
>> [25, 3411, 11826, 5786, 8022, 2190, 11302, 2048]

Benchmark comparison