AI

3 NLTK Tricks for Advanced Text Preprocessing and Linguistic Analysis

Learn three practical NLTK techniques—MWE tokenization, POS-aware lemmatization, and collocation extraction—to improve text preprocessing pipelines.

By Editorial Team ·

3 NLTK Tricks for Advanced Text Preprocessing and Linguistic Analysis

3 NLTK Tricks for Advanced Text Preprocessing & Linguistic Analysis

Natural language processing (NLP) has undergone an obvious paradigm shift in recent years, with large language models (LLMs) and transformers handling complex end-to-end understanding tasks. However, in any practical NLP workflow, raw text must still be tokenized, normalized, and analyzed before it ever reaches a model. While modern NLP libraries and ecosystems like SpaCy or Hugging Face are fantastic for building general-purpose deep learning pipelines or integrating with LLMs, the Natural Language Toolkit (NLTK) remains a viable, transparent option for fine-grained structural linguistics, custom text normalization, and statistical corpus analysis.

Unfortunately, many developers incorrectly believe that LLMs render traditional text preprocessing obsolete, or they write text preprocessing code using naive methods that discard critical linguistic structure. They split multi-word expressions like “machine learning” into separate, meaningless words; they perform context-blind lemmatization that yields inaccurate base forms; or they rely on simple raw frequency counts that miss meaningful word associations.

To build robust, semantically accurate NLP models, you need to preserve structural and linguistic context at the preprocessing stage. In this article, we will walk through three essential NLTK tricks to elevate your text preprocessing:

  1. Preserving phrase integrity with the MWETokenizer
  2. Context-aware lemmatization with Part-of-Speech (POS) mapping
  3. Statistical collocation extraction using association measures

1. Preserving Domain Terminology with the Multi-Word Expression Tokenizer

Tokenization is the foundation of any NLP pipeline. However, standard tokenizers split sentences strictly by whitespace and punctuation. This becomes problematic when dealing with domain-specific multi-word expressions — such as "neural network", "decision tree", or "San Francisco" — where the individual words combine to form a single semantic concept.

If a tokenizer splits "neural network" into "neural" and "network", a downstream vectorizer (like Bag-of-Words or TF-IDF) will treat them as unrelated features, diluting the signal and introducing noise. Developers often try to fix this by writing search-and-replace regular expressions on the raw text before tokenizing.

Using character-level replacements (e.g. text.replace("neural network", "neural_network")) is brittle. It fails to respect word boundaries, handles punctuation poorly, and is incredibly slow to execute across large datasets. The optimized approach is to tokenize the text first and then run NLTK’s native MWETokenizer to merge these tokens cleanly.

The naive approach of regex replacement relies on character-level string manipulation, which does not scale well and can inadvertently modify substrings inside unrelated words:

import re
import time

# Sample corpus
raw_texts = [
    "We are studying neural networks and deep learning.",
    "The decision tree is a popular model in machine learning.",
    "A neural network can have many layers."
] * 5000

cleaned_texts = []
for text in raw_texts:
    # Manual string replacements for domain terms
    text = re.sub(r"\bneural networks?\b", "neural_network", text, flags=re.IGNORECASE)
    text = re.sub(r"\bdecision trees?\b", "decision_tree", text, flags=re.IGNORECASE)
    text = re.sub(r"\bmachine learnings?\b", "machine_learning", text, flags=re.IGNORECASE)
    
    # Tokenize the processed string
    tokens = text.lower().split()
    cleaned_texts.append(tokens)

print("Sample tokens:", cleaned_texts[0])

Output:

Sample tokens: ['we', 'are', 'studying', 'neural_network', 'and', 'deep', 'learning.']

Now let’s try using NLTK’s tokenizers. We first tokenize using the standard word_tokenize method and then pass the token streams through an initialized MWETokenizer that handles merging on token boundaries efficiently:

import nltk
from nltk.tokenize import word_tokenize, MWETokenizer
import time

# Ensure NLTK resources are downloaded
nltk.download('punkt', quiet=True)

raw_texts = [
    "We are studying neural networks and deep learning.",
    "The decision tree is a popular model in machine learning.",
    "A neural network can have many layers."
] * 5000

# Initialize tokenizer and register MWE tuples
mwe_tokenizer = MWETokenizer([
    ('neural', 'network'),
    ('neural', 'networks'),
    ('decision', 'tree'),
    ('decision', 'trees'),
    ('machine', 'learning')
], separator='_')

cleaned_texts_mwe = []
for text in raw_texts:
    # Tokenize words using NLTK's standard tokenizer
    tokens = word_tokenize(text.lower())
    # Merge specified multi-word expressions
    merged_tokens = mwe_tokenizer.tokenize(tokens)
    cleaned_texts_mwe.append(merged_tokens)

print("Sample tokens:", cleaned_texts_mwe[0])

We get the same output, but in a more elegant, linguistically accurate, and scalable approach:

Sample tokens: ['we', 'are', 'studying', 'neural_network', 'and', 'deep', 'learning.']

Using the MWETokenizer shifts the operation from slow character-level string matches to token-level comparison.

  • We define the multi-word expressions as tuples of independent tokens: ('neural', 'network').
  • By setting separator='_', the tokenizer merges the matching sequence into a single string token: "neural_network".
  • Because it acts directly on token arrays, it is immune to boundary matching bugs and handles trailing punctuation (like "neural networks." splitting into "neural", "networks", "." first, then safely merging to "neural_networks", ".") correctly. It executes faster and scales cleanly to hundreds of domain terms.

2. Context-Aware Lemmatization with POS-Tag Mapping

Lemmatization is the process of reducing a word to its base dictionary form (its lemma) — “running” -> “run”, “better” -> “good”. This is an essential normalization step, as it groups different grammatical inflections of the same word together.

However, NLTK’s WordNetLemmatizer defaults to treating every word as a noun. If you pass verbs or adjectives without specifying their POS category, the lemmatizer will return the word unchanged. For example:

  • lemmatizer.lemmatize("running") yields "running" (instead of “run”)
  • lemmatizer.lemmatize("better") yields "better" (instead of “good”)

To solve this, we must dynamically identify the grammatical role of each word in the sentence using NLTK’s POS tagger, map those tags to WordNet’s simplified categories (noun, verb, adjective, adverb), and pass them to the lemmatizer.

This naive approach feeds words directly to the lemmatizer. It misses verb and adjective conversions, resulting in suboptimal vocabulary normalization:

import nltk
from nltk.stem import WordNetLemmatizer
from nltk.tokenize import word_tokenize

nltk.download('punkt', quiet=True)
nltk.download('wordnet', quiet=True)

sentence = "The feet of the running runners are getting better and faster."
tokens = word_tokenize(sentence.lower())

lemmatizer = WordNetLemmatizer()

# Naive lemmatization: assumed to be all nouns
naive_lemmas = [lemmatizer.lemmatize(token) for token in tokens]
print("Tokens:      ", tokens)
print("Naive Lemmas:", naive_lemmas)

Output:

Tokens:       ['the', 'feet', 'of', 'the', 'running', 'runners', 'are', 'getting', 'better', 'and', 'faster', '.']
Naive Lemmas: ['the', 'foot', 'of', 'the', 'running', 'runner', 'are', 'getting', 'better', 'and', 'faster', '.']

Let’s look at an optimized version: we write a clean helper dictionary mapping Penn Treebank tags (returned by NLTK’s pos_tag) to WordNet POS constants, ensuring every word type is lemmatized accurately:

import nltk
from nltk.stem import WordNetLemmatizer
from nltk.tokenize import word_tokenize
from nltk.corpus import wordnet

# Download POS tagger resources
nltk.download('punkt', quiet=True)
nltk.download('wordnet', quiet=True)
nltk.download('averaged_perceptron_tagger', quiet=True)

sentence = "The feet of the running runners are getting better and faster."
tokens = word_tokenize(sentence.lower())

# Generate POS tags for each token
pos_tags = nltk.pos_tag(tokens)

# Map Penn Treebank tags to WordNet tags
def get_wordnet_pos(treebank_tag):
    if treebank_tag.startswith('J'):
        return wordnet.ADJ
    elif treebank_tag.startswith('V'):
        return wordnet.VERB
    elif treebank_tag.startswith('N'):
        return wordnet.NOUN
    elif treebank_tag.startswith('R'):
        return wordnet.ADV
    else:
        # Default to WordNet's default noun handling
        return None

lemmatizer = WordNetLemmatizer()

# Lemmatize utilizing mapped POS tags
context_lemmas = []
for token, tag in pos_tags:
    wn_tag = get_wordnet_pos(tag)
    if wn_tag:
        lemma = lemmatizer.lemmatize(token, pos=wn_tag)
    else:
        lemma = lemmatizer.lemmatize(token)
    context_lemmas.append(lemma)

print("POS Tagged:    ", pos_tags)
print("Context Lemmas:", context_lemmas)

Output:

POS Tagged:     [('the', 'DT'), ('feet', 'NNS'), ('of', 'IN'), ('the', 'DT'), ('running', 'NN'), ('runners', 'NNS'), ('are', 'VBP'), ('getting', 'VBG'), ('better', 'RBR'), ('and', 'CC'), ('faster', 'RBR'), ('.', '.')]
Context Lemmas: ['the', 'foot', 'of', 'the', 'running', 'runner', 'be', 'get', 'well', 'and', 'faster', '.']

NLTK’s pos_tag labels words using the Penn Treebank tagset (e.g. 'VBG' for a gerund verb, 'JJR' for a comparative adjective).

  • Our helper function get_wordnet_pos() inspects the first character of the tag. In line with WordNet’s POS standards, if it starts with ‘J’, we map it to WordNet’s Adjective tag (wordnet.ADJ); if it starts with ‘V’, to Verb (wordnet.VERB), and so on.
  • By feeding the correct POS tag into lemmatizer.lemmatize(token, pos=wn_tag), the lemmatizer successfully resolves “running” to “run”, “are” to “be”, “getting” to “get”, “better” to “good”, and “faster” to “fast”. This preserves the semantic core of the sentence, drastically reducing vocabulary sparsity for downstream ML models.

3. Statistical Phrase Extraction Using Collocation Finders

Extracting key phrases or multi-word concepts from text is valuable for topic modeling, search indexing, and sentiment analysis. These phrases are known as collocations — sequences of words that co-occur more often than would be expected by chance.

Rather than relying on simple raw frequency counts, NLTK’s collocation finders use statistical association measures such as Pointwise Mutual Information (PMI) and chi-square scoring to identify word pairs and trigrams that form meaningful, non-random combinations. This approach surfaces genuinely informative phrases while filtering out high-frequency but semantically weak pairings, giving downstream models a richer and more accurate representation of the text’s conceptual structure.

Together, these three techniques — MWE tokenization, POS-aware lemmatization, and statistical collocation extraction — form a powerful preprocessing toolkit that preserves linguistic structure at every stage of the NLP pipeline. Applying them systematically leads to cleaner vocabularies, more accurate feature representations, and stronger model performance across a wide range of text analysis tasks.