AI with Python – NLTK Package

AI with Python – Getting Started with NLTK

The Natural Language Toolkit (NLTK) is a comprehensive Python library for various Natural Language Processing (NLP) tasks. This chapter guides you through setting up NLTK and exploring its core functionalities.

Why NLTK?

NLP applications face challenges due to the inherent ambiguity and contextual nature of human language. NLTK provides tools and resources to build applications that can process and understand text in a more human-like way. It’s particularly valuable for learning NLP concepts due to its extensive collection of datasets (corpora) and modular components for tasks like tokenization, stemming, tagging, parsing, and classification.

Installing NLTK

You can install NLTK using pip, Python’s package installer:

pip install nltk

If you are using Anaconda, you can install it via conda:

conda install -c anaconda nltk

After installation, import NLTK in your Python script or interactive session:

import nltk

Downloading NLTK Data

NLTK comes with many corpora, pre-trained models, and other data resources. You need to download these separately. You can do this interactively in Python:

nltk.download()

This command will open the NLTK Downloader window. You can choose to download individual packages (like ‘punkt’ for tokenization, ‘wordnet’ for lemmatization, ‘stopwords’) or common collections like ‘book’ (all data needed for examples in the NLTK book). For general use, downloading ‘popular’ or ‘all’ is also an option, though ‘all’ can be quite large.

Other Useful Packages for NLP

While NLTK is powerful, you might also use other libraries in conjunction with it or for more advanced tasks:

gensim

gensim is a robust library for topic modeling, document similarity analysis, and working with word embeddings (like Word2Vec). Install it with:

pip install gensim

spaCy

spaCy is an industrial-strength NLP library known for its speed and efficiency, offering pre-trained models for various languages and tasks like Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and dependency parsing. Install with:

pip install spacy
python -m spacy download en_core_web_sm # Download a small English model

scikit-learn

scikit-learn provides tools for text feature extraction (e.g., CountVectorizer, TfidfVectorizer) and various machine learning algorithms suitable for text classification tasks. Install with:

pip install scikit-learn

Fundamental NLP Tasks: Tokenization, Stemming, and Lemmatization

These are foundational preprocessing steps in many NLP pipelines.

Tokenization

Tokenization is the process of breaking down a text (a sequence of characters) into smaller units called tokens. These tokens are typically words, numbers, or punctuation marks. It’s also known as word segmentation.

For example, the input string: “NLTK is great for learning NLP! It costs $0.”

Could be tokenized into: ['NLTK', 'is', 'great', 'for', 'learning', 'NLP', '!', 'It', 'costs', '$', '0', '.']

NLTK provides several tokenizers:

• Sentence Tokenizer (sent_tokenize): Splits text into sentences.
• Word Tokenizer (word_tokenize): Splits text (or sentences) into words and punctuation.
• WordPunctTokenizer: Splits text based on whitespace and also separates punctuation marks from adjacent words.

Example using word_tokenize and sent_tokenize (requires ‘punkt’ data package):

from nltk.tokenize import sent_tokenize, word_tokenize

# Ensure 'punkt' is downloaded: nltk.download('punkt')

text_example = "NLTK is great for learning NLP! It costs $0. What do you think?"

sentences = sent_tokenize(text_example)
print(f"Sentences: {sentences}")

first_sentence_words = word_tokenize(sentences[0])
print(f"Words in first sentence: {first_sentence_words}")

Stemming

Stemming is a heuristic process of reducing words to their ‘stem’ or base form by chopping off derivational or inflectional affixes (prefixes/suffixes). The goal is to group different forms of a word under a common representation. For example, ‘running’, ‘runs’, ‘ran’ might all be stemmed to ‘run’.

NLTK offers several stemmers:

• PorterStemmer: One of the oldest and most widely used stemmers. It’s relatively gentle.
• LancasterStemmer: A more aggressive stemmer, often resulting in shorter, sometimes less intuitive stems.
• SnowballStemmer: Supports multiple languages and is generally an improvement over Porter. It’s a good balance of aggressiveness and accuracy.

Example with PorterStemmer:

from nltk.stem.porter import PorterStemmer

porter = PorterStemmer()
words_to_stem = ["running", "studies", "leaves", "activity", "connection"]
stemmed_words = [porter.stem(word) for word in words_to_stem]
print(f"Stemmed words (Porter): {stemmed_words}")
# Output: Stemmed words (Porter): ['run', 'studi', 'leav', 'activ', 'connect']

Lemmatization

Lemmatization aims to reduce words to their dictionary form, known as the ‘lemma’. Unlike stemming, lemmatization considers the word’s meaning and part of speech. It uses a vocabulary (like WordNet) and morphological analysis.

For example, ‘better’ would be lemmatized to ‘good’ (if its POS is adjective), and ‘is’, ‘are’, ‘was’ to ‘be’. ‘leaves’ could be lemmatized to ‘leaf’ (noun) or ‘leave’ (verb).

NLTK’s WordNetLemmatizer is commonly used (requires ‘wordnet’ data package):

from nltk.stem import WordNetLemmatizer
from nltk.corpus import wordnet # To specify Part-of-Speech for lemmatizer

# Ensure 'wordnet' and 'omw-1.4' (Open Multilingual Wordnet) are downloaded:
# nltk.download('wordnet')
# nltk.download('omw-1.4')

lemmatizer = WordNetLemmatizer()

print(f"Lemmatizing 'studies': {lemmatizer.lemmatize('studies', pos=wordnet.NOUN)}") # 'study'
print(f"Lemmatizing 'studies' (verb): {lemmatizer.lemmatize('studies', pos=wordnet.VERB)}") # 'study'
print(f"Lemmatizing 'better': {lemmatizer.lemmatize('better', pos=wordnet.ADJ)}") # 'good'
print(f"Lemmatizing 'is': {lemmatizer.lemmatize('is', pos=wordnet.VERB)}") # 'be'

Lemmatization generally produces more linguistically correct base forms than stemming but is computationally more expensive.

Chunking: Identifying Simple Phrases

Chunking (or shallow parsing) is the process of identifying and segmenting text into syntactically correlated groups of words, like noun phrases (NP), verb phrases (VP), etc. It doesn’t build a full parse tree but provides a simpler, often more robust, structural analysis.

For example, in “The clever fox was jumping over the lazy dog,” chunking might identify “The clever fox” and “the lazy dog” as Noun Phrases.

Noun-Phrase (NP) Chunking Example

To perform chunking, you typically first need Part-of-Speech (POS) tags for your tokens. NLTK provides POS taggers (e.g., nltk.pos_tag, requires ‘averaged_perceptron_tagger’).

Steps:

1. Tokenize the sentence.
2. POS tag the tokens.
3. Define a chunk grammar (often using regular expressions over POS tags).
4. Create a chunk parser with this grammar.
5. Apply the parser to the tagged tokens.

import nltk

# Ensure required NLTK data is downloaded:
# nltk.download('averaged_perceptron_tagger')
# nltk.download('punkt')

sentence_text = "The clever fox was jumping over the lazy dog."
tokens = nltk.word_tokenize(sentence_text)
tagged_tokens = nltk.pos_tag(tokens)
print(f"Tagged Tokens: {tagged_tokens}")

# Define a grammar for Noun Phrases (NP)
# NP: an optional Determiner (DT), any number of Adjectives (JJ*), then a Noun (NN)
# Or a Proper Noun (NNP), or Pronoun (PRP)
grammar = r"""
    NP: {<DT>?<JJ>*<NN>}
        {<DT>?<JJ>*<NNS>}
        {<DT>?<JJ>*<NNP>}
        {<DT>?<JJ>*<NNPS>}
        {<PRP>}
"""

chunk_parser = nltk.RegexpParser(grammar)
chunked_sentence_tree = chunk_parser.parse(tagged_tokens)

print("\nChunked Sentence Tree:")
# To print the tree structure textually (instead of drawing):
chunked_sentence_tree.pprint() # pretty print

# To show just the chunks:
# for subtree in chunked_sentence_tree.subtrees():
#     if subtree.label() == 'NP':
#         print(subtree)

The chunked_sentence_tree.pprint() output will look something like this, showing the tree structure with identified Noun Phrases:

(S
  (NP The/DT clever/JJ fox/NN)
  was/VBD
  jumping/VBG
  over/IN
  (NP the/DT lazy/JJ dog/NN)
  ./.)

This textual representation shows that ‘The clever fox’ and ‘the lazy dog’ have been correctly identified as Noun Phrases (NP).

Bag-of-Words (BoW) Model

The Bag-of-Words (BoW) model is a fundamental technique for feature extraction from text. It represents text (like a sentence or document) as an unordered collection (a ‘bag’) of its words, disregarding grammar and word order but keeping track of word frequency.

How it Works

1. Vocabulary Creation: A vocabulary of all unique words in the entire corpus (collection of documents) is created.
2. Vectorization: Each document is represented as a numerical vector. The length of this vector is the size of the vocabulary. Each dimension corresponds to a word in the vocabulary, and the value in that dimension is typically the count of that word in the document.

Example:

• Sentence 1: “The cat sat on the mat.”
• Sentence 2: “The dog chased the cat.”

Vocabulary: ['The', 'cat', 'sat', 'on', 'mat', 'dog', 'chased'] (ignoring case and punctuation for simplicity)

Vectors:

• Sentence 1: [2, 1, 1, 1, 1, 0, 0] (for ‘The’, ‘cat’, ‘sat’, ‘on’, ‘mat’, ‘dog’, ‘chased’ respectively)
• Sentence 2: [1, 1, 0, 0, 0, 1, 1]

Term Frequency-Inverse Document Frequency (TF-IDF)

BoW uses raw counts. TF-IDF is a refinement that reflects how important a word is to a document in a collection or corpus.

Term Frequency (TF)

Measures how frequently a term appears in a document. TF(t, d) = (Number of times term t appears in document d) / (Total number of terms in document d).

Inverse Document Frequency (IDF)

Measures how important a term is. It downscales words that appear frequently across many documents (like ‘the’, ‘is’) and upscales rare words. IDF(t, D) = log(Total number of documents D / Number of documents containing term t).

TF-IDF score for a term t in document d is TF(t,d) * IDF(t,D).

Building BoW and TF-IDF Models with scikit-learn

scikit-learn provides convenient tools for BoW and TF-IDF:

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

sentences = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?'
]

# Bag-of-Words (CountVectorizer)
count_vectorizer = CountVectorizer()
bow_matrix = count_vectorizer.fit_transform(sentences)
print("\nBoW Vocabulary:", count_vectorizer.vocabulary_)
print("BoW Matrix (sparse):")
print(bow_matrix.toarray())

# TF-IDF (TfidfVectorizer)
tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix = tfidf_vectorizer.fit_transform(sentences)
print("\nTF-IDF Vocabulary:", tfidf_vectorizer.vocabulary_)
print("TF-IDF Matrix (sparse):")
print(tfidf_matrix.toarray())

These matrices (BoW or TF-IDF) serve as feature vectors for training machine learning models.

Applications & Problem Solving with NLTK & scikit-learn

1. Text Category Prediction (Newsgroup Classification)

Predicting the category of a document (e.g., sports, politics, tech). We’ll use the 20 Newsgroups dataset from scikit-learn, TF-IDF, and a Naive Bayes classifier.

from sklearn.datasets import fetch_20newsgroups
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline # For chaining vectorizer and classifier

# Define categories to load for simplicity
categories = ['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.med']

# Fetch training data
train_data = fetch_20newsgroups(subset='train', categories=categories, shuffle=True, random_state=42)

# Create a pipeline: TF-IDF Vectorizer -> Multinomial Naive Bayes Classifier
model = make_pipeline(TfidfVectorizer(), MultinomialNB())

# Train the model
model.fit(train_data.data, train_data.target)

# Example new texts to predict
new_texts = [
    "God is love",
    "OpenGL on the GPU is fast",
    "Christianity and science can coexist",
    "Medical imaging with advanced algorithms"
]

predicted_categories_indices = model.predict(new_texts)

print("\nCategory Predictions:")
for text, index in zip(new_texts, predicted_categories_indices):
    print(f'Input: "{text}" => Predicted Category: {train_data.target_names[index]}')

This example demonstrates a basic text classification pipeline.

2. Gender Identification from Names (Illustrative Example)

A simple classifier to predict gender based on names, using features like the last letter(s) of a name. NLTK’s names corpus provides labeled data.

import random
from nltk.corpus import names # nltk.download('names')
from nltk import NaiveBayesClassifier
from nltk.classify import accuracy as nltk_accuracy

# Feature extractor: uses last N letters of a name
def gender_features(word, n=2):
    return {'last_letters': word[-n:].lower()}

# Prepare data
# nltk.download('names') # Ensure names corpus is downloaded
male_names = [(name, 'male') for name in names.words('male.txt')]
female_names = [(name, 'female') for name in names.words('female.txt')]
all_names = male_names + female_names
random.shuffle(all_names)

# Create feature sets
N_letters = 2 # Use last 2 letters as features
featuresets = [(gender_features(name, N_letters), gender) for (name, gender) in all_names]

# Split into training and test sets
train_size = int(0.8 * len(featuresets))
train_set, test_set = featuresets[:train_size], featuresets[train_size:]

# Train Naive Bayes classifier
classifier = NaiveBayesClassifier.train(train_set)

# Test accuracy
print(f"\nGender Finder (features: last {N_letters} letters):")
print(f"Accuracy: {nltk_accuracy(classifier, test_set) * 100:.2f}%")

# Predict for new names
names_to_test = ['Neo', 'Trinity', 'Alex', 'Jordan', 'Taylor']
for name in names_to_test:
    guess = classifier.classify(gender_features(name, N_letters))
    print(f"Name: {name}, Predicted Gender: {guess}")

This is a simplistic model and its accuracy will be limited. More sophisticated features and models are needed for reliable gender identification, which itself is a complex and nuanced task.

Topic Modeling: Discovering Themes in Text

Topic modeling is an unsupervised technique to discover abstract ‘topics’ or hidden thematic structures within a collection of documents. Each topic is a distribution over words, and each document is a mixture of topics.

Common Algorithms

• Latent Dirichlet Allocation (LDA): A probabilistic generative model. Most popular. gensim provides a good implementation.
• Latent Semantic Analysis (LSA/LSI): Uses matrix factorization (SVD) on the document-term matrix.
• Non-Negative Matrix Factorization (NMF): Another matrix factorization technique.

Applications

• Document Organization and Browsing: Grouping documents by themes.
• Improving Text Classification: Using topics as features.
• Recommender Systems: Recommending content based on shared topics.

Example with gensim for LDA (conceptual outline - requires data preprocessing):

from gensim.corpora import Dictionary
from gensim.models import LdaModel
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords # nltk.download('stopwords')
import string

# Sample documents (replace with your actual text data)
documents_sample = [
    "Sugar is bad to consume. My sister likes to have sugar, but not my father.",
    "My father spends a lot of time driving my sister around to dance practice.",
    "Doctors suggest that driving may cause increased stress and blood pressure.",
    "Sometimes I feel pressure to perform well at school, but my father never seems to drive my sister to do better.",
    "Health experts say that Sugar is not good for your lifestyle."
]

# Preprocessing (basic example)
# nltk.download('stopwords')
# nltk.download('punkt')
stop_words = set(stopwords.words('english'))
punctuation = set(string.punctuation)

processed_docs = []
for doc in documents_sample:
    tokens = word_tokenize(doc.lower())
    cleaned_tokens = [token for token in tokens if token.isalpha() and token not in stop_words and token not in punctuation]
    processed_docs.append(cleaned_tokens)

# Create a dictionary and a corpus for LDA
id2word = Dictionary(processed_docs)
corpus = [id2word.doc2bow(doc) for doc in processed_docs]

# Build LDA model (e.g., find 2 topics)
if corpus and id2word: # Ensure corpus and dictionary are not empty
    lda_model = LdaModel(corpus=corpus, id2word=id2word, num_topics=2, random_state=100,
                         update_every=1, chunksize=100, passes=10, alpha='auto', per_word_topics=True)

    print("\nLDA Topics (example):")
    for idx, topic in lda_model.print_topics(-1):
        print(f"Topic: {idx} \nWords: {topic}")
else:
    print("\nLDA Topics: Corpus or dictionary is empty, cannot train LDA model.")

This example shows the basic steps: preprocess text, create a dictionary (mapping words to IDs) and a BoW corpus, then train the LDA model. The output will show the top words associated with each discovered topic. Real-world topic modeling involves more careful preprocessing and hyperparameter tuning.