Ch4. Machine Learning — Natural Language Processing (NLP) Fundamentals

What Is Natural Language Processing?

Natural Language Processing (NLP) is the field of AI that enables computers to understand and generate human language. It combines machine learning with linguistics to work on text and speech.

Core NLP tasks:
  Text classification   → spam filtering, sentiment analysis, topic labeling
  Information extraction→ named entity recognition (NER), relation extraction
  Machine translation   → English ↔ French, English ↔ Japanese
  Question answering    → search engines, chatbots
  Text summarization    → news digests, document compression
  Text generation       → GPT-family language models

NLP is challenging because language is inherently ambiguous, context-dependent, and culturally nuanced. “I saw the man with the telescope” — did I use the telescope, or did the man have it?

Text Preprocessing Pipeline

Raw text cannot be fed directly into most models. A preprocessing pipeline converts it into a machine-readable form.

Step 1: Tokenization

Split text into units called tokens.

Sentence: "Machine learning is incredibly useful!"

Word tokenization:
["Machine", "learning", "is", "incredibly", "useful", "!"]

Subword tokenization (BPE — used by modern LLMs):
["Machine", " learn", "ing", " is", " incred", "ibly", " use", "ful", "!"]

Step 2: Normalization

import re

text = "Hello!!! This is a TEST... visit http://example.com"

text = text.lower()
# "hello!!! this is a test... visit http://example.com"

text = re.sub(r'http\S+', '', text)
# "hello!!! this is a test... visit "

text = re.sub(r'[^a-z\s]', '', text)
# "hello  this is a test  visit "

text = re.sub(r'\s+', ' ', text).strip()
# "hello this is a test visit"

Step 3: Stop Word Removal

Stop words: "the", "a", "an", "is", "are", "in", "at", "to", "and", ...
→ Frequently occurring words that carry little semantic information

Original: "The cat sat on the mat"
After stop word removal: ["cat", "sat", "mat"]

Caution: don't always remove stop words blindly
→ In sentiment analysis: "not good" → "good" (meaning reversed if "not" is removed)

Step 4: Stemming vs. Lemmatization

Stemming (rule-based, approximate root):
  running   → run
  studies   → studi  (may be incorrect)

Lemmatization (dictionary-based, exact base form):
  running   → run
  studies   → study
  better    → good   (comparative → base)

→ Lemmatization is more accurate but slower
→ Use stemming when speed is critical

Bag of Words (BoW)

The simplest text representation — counts word occurrence frequency while ignoring word order.

Document 1: "The cat sat on the mat"
Document 2: "The dog sat on the mat"
Document 3: "The cat chased the dog"

Vocabulary: {cat, dog, sat, mat, chased}

BoW vectors:
         cat  dog  sat  mat  chased
Doc 1:    1    0    1    1     0
Doc 2:    0    1    1    1     0
Doc 3:    1    1    0    0     1

Limitations:

Order lost: “I like you” vs. “You like me” → identical representation
No semantic similarity: “car” and “automobile” treated as unrelated
Sparse vectors: most values are 0 for large vocabularies

TF-IDF

An improvement over raw counts — assigns higher weight to words that are frequent in a document but rare across documents.

TF (Term Frequency):
TF(t, d) = count of term t in document d / total words in document d

IDF (Inverse Document Frequency):
IDF(t) = log(total documents / documents containing term t)

TF-IDF(t, d) = TF(t, d) × IDF(t)

Intuition:
"today", "and", "is" → appear in every document
→ IDF low → TF-IDF low (common words, low importance)

"neural", "backpropagation", "transformer" → appear in few documents
→ IDF high → TF-IDF high (key domain words, high importance)

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "machine learning algorithm training",
    "deep learning neural network training",
    "natural language processing text analysis",
]

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(corpus)

print(vectorizer.get_feature_names_out())
# tfidf_matrix.shape = (3, n_features)

Word Embeddings

BoW and TF-IDF produce sparse high-dimensional vectors. Word embeddings represent words as dense low-dimensional real-valued vectors where semantically similar words are close in vector space.

Word2Vec — Intuition

Released by Google in 2013, Word2Vec is based on the distributional hypothesis: “words appearing in similar contexts have similar meanings.”

Training method 1 — CBOW (Continuous Bag of Words):
  Predict the center word from surrounding context words
  [The, __, is, fast] → [cat]

Training method 2 — Skip-gram:
  Predict surrounding context words from the center word
  [cat] → [The, is, fast]

Fascinating properties learned by Word2Vec:

Vector arithmetic:
  vec("king") − vec("man") + vec("woman") ≈ vec("queen")
  vec("Paris") − vec("France") + vec("Japan") ≈ vec("Tokyo")

Semantic distance:
  cos_similarity(vec("cat"), vec("dog")) > cos_similarity(vec("cat"), vec("car"))

Using Pre-trained Embeddings

import gensim.downloader as api

# Load pre-trained Word2Vec (trained on billions of words)
wv = api.load("word2vec-google-news-300")

# Find similar words
wv.most_similar("king")
# [('queen', 0.71), ('monarch', 0.64), ('prince', 0.62), ...]

# Word vector (300 dimensions)
vec = wv["machine"]  # shape: (300,)

Sentiment Analysis in Practice

Sentiment analysis automatically classifies text as positive, negative, or neutral — one of the most widely deployed NLP applications.

Rule-based vs. Machine Learning Approaches

Rule-based:
→ Uses a sentiment lexicon: "good"(+1), "bad"(−1), "very"(×2)
→ Sum scores to determine sentiment
→ Simple to implement; hard to transfer to new domains

ML-based:
→ Train a classifier on labeled review data
→ TF-IDF + Logistic Regression, Random Forest
→ Deep learning: LSTM, BERT fine-tuning

TF-IDF + Logistic Regression Pipeline

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

texts = [
    "Absolutely delicious — best meal I've had in years!",
    "Food was too salty and overpriced.",
    "Great atmosphere and friendly staff.",
    "Waited an hour; totally unacceptable.",
]
labels = [1, 0, 1, 0]  # 1=positive, 0=negative

X_train, X_test, y_train, y_test = train_test_split(
    texts, labels, test_size=0.2, random_state=42
)

pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(ngram_range=(1, 2))),  # unigrams + bigrams
    ('clf',   LogisticRegression(max_iter=1000)),
])

pipeline.fit(X_train, y_train)
print(classification_report(y_test, pipeline.predict(X_test)))

Text Classification Pipeline

1. Data collection and labeling
   → Crawling, crowdsourcing, internal annotation

2. Preprocessing
   → Normalize → Tokenize → Remove stop words → Stem / Lemmatize

3. Feature extraction
   → BoW / TF-IDF / Word2Vec / BERT embeddings

4. Model training
   → Naïve Bayes, Logistic Regression, SVM, LSTM, BERT

5. Evaluation
   → Accuracy, F1, Precision, Recall
   → (Use F1 when classes are imbalanced)

6. Error analysis
   → Inspect misclassified samples
   → Improve features, preprocessing, or data balance

N-grams

N-gram: sequence of N consecutive tokens
→ Captures some word-order information that BoW misses

Sentence: "the food was not good"
1-gram:   ["the", "food", "was", "not", "good"]
2-gram:   ["the food", "food was", "was not", "not good"]
3-gram:   ["the food was", "food was not", "was not good"]

→ Larger N captures more context but increases sparsity and memory
→ In practice, mixing 1-gram and 2-gram is the most common approach

Key Concept Cards

Tokenization ★★★★★
The first step in NLP: split text into tokens. Modern LLMs use subword tokenization (BPE). Important: tokenization differs significantly between languages.

TF-IDF intuition ★★★★★
High TF-IDF = frequent in this document AND rare across all documents. Words like “the” score near zero because they appear everywhere.

Word2Vec distributional hypothesis ★★★★★
“Words in similar contexts have similar meanings.” vec(“king”) − vec(“man”) + vec(“woman”) ≈ vec(“queen”).

BoW limitations ★★★☆☆
Ignores word order, ignores semantic similarity, produces sparse vectors. N-grams partially address order; embeddings address semantics.

Stop word removal trade-off ★★★☆☆
Removes noise but can destroy meaning in certain tasks. Never remove negations (“not”, “never”) in sentiment analysis.

Practice Quiz

Q1. Why do words like “the” and “and” get very low TF-IDF scores?

These words appear in almost every document, so IDF = log(total docs / docs containing word) ≈ log(1) ≈ 0. Since TF-IDF = TF × IDF, the near-zero IDF makes the final score negligible, correctly indicating that these words carry little discriminating information.

Q2. What is the core idea behind Word2Vec?

The distributional hypothesis: words that appear in similar contexts share similar meaning. Word2Vec trains a neural network to predict either the center word from context (CBOW) or context words from the center word (Skip-gram). The resulting weight vectors encode semantic relationships, so similar words end up close in vector space.

Q3. Why can removing stop words hurt sentiment analysis?

Negations like “not”, “never”, and “hardly” are often classified as stop words, but removing them reverses the sentiment: “not good” → “good.” For sentiment tasks, always customize your stop word list to preserve negations and intensifiers that fundamentally alter meaning.

Q4. What is the difference between bigrams and unigrams, and when are bigrams more useful?

A unigram is a single token; a bigram is a two-token sequence. Bigrams capture local word order — for example, “not good” as a bigram preserves the negation that unigrams would split. Bigrams are most useful in sentiment analysis and any task where short phrases carry meaning that individual words do not.