Natural Language Processing (NLP)

What is NLP?

Natural Language Processing is the branch of AI concerned with the interaction between computers and human language. It encompasses everything from basic tasks like identifying the parts of speech in a sentence to complex ones like understanding sarcasm, translating between languages, or generating coherent text [1].

Language is one of the hardest problems in AI because it is inherently ambiguous, context-dependent, and culturally shaped. The same word can mean different things in different contexts (‘bank’ as a financial institution vs. a riverbank), and the meaning of a sentence often depends on what came before it. NLP develops the computational methods to handle this complexity.

The field has evolved dramatically since the 1950s, from hand-crafted rules to statistical models to the deep learning approaches that power today’s large language models.

Three eras of NLP

NLP has passed through three major paradigms [1]:

Rule-based NLP (1950s–1980s): Early systems relied on hand-crafted grammatical rules and expert knowledge. Linguists manually encoded syntax, vocabulary, and transformation rules. These systems were brittle—they worked in narrow domains but broke when language deviated from expected patterns. Notable examples include Chomsky’s formal grammar (1957) and the ELIZA chatbot (1966).

Statistical NLP (late 1980s–2000s): The availability of large text corpora and increased computing power enabled data-driven approaches. Hidden Markov Models, n-gram language models, and statistical machine translation replaced hand-written rules with probabilistic models learned from data. Word2vec (2013) and GloVe introduced distributed word representations that captured semantic relationships.

Neural NLP (2010s–present): Deep learning transformed the field. Recurrent neural networks (RNNs), Long Short-Term Memory (LSTM) networks, and attention mechanisms enabled models to process sequences of text with contextual understanding. The Transformer architecture (Vaswani et al., 2017) replaced RNNs entirely and became the foundation for modern large language models [2].

Key NLP tasks

NLP encompasses a wide range of tasks [1]:

Understanding tasks:

• Text classification — assigning categories to text (spam detection, topic labeling)
• Sentiment analysis — determining the emotional tone of text (positive, negative, neutral)
• Named entity recognition (NER) — identifying and classifying entities like people, organizations, locations, and dates
• Question answering — extracting answers from text given a question
• Semantic similarity — measuring how close in meaning two pieces of text are

Generation tasks:

• Machine translation — converting text between languages while preserving meaning
• Text summarization — condensing longer text while retaining key information
• Text generation — producing coherent, contextually appropriate text
• Dialogue systems — generating conversational responses

Structural tasks:

• Part-of-speech tagging — labeling each word with its grammatical role
• Parsing — analyzing the grammatical structure of sentences
• Coreference resolution — determining which words refer to the same entity

The Transformer revolution

The Transformer architecture, introduced in the 2017 paper ‘Attention Is All You Need’ by Vaswani et al., fundamentally changed NLP [2].

Before Transformers, NLP models processed text sequentially—one word at a time—which limited their ability to capture long-range dependencies and parallelize computation. Transformers use self-attention mechanisms to process all words in a sequence simultaneously, weighing each word’s relevance to every other word.

Two families of Transformer models emerged:

• BERT (Bidirectional Encoder Representations from Transformers) — released by Google in 2018, BERT uses the encoder half of the Transformer. It reads text bidirectionally and excels at understanding tasks like classification, question answering, and NER [3].
• GPT (Generative Pre-trained Transformer) — released by OpenAI starting in 2018, GPT uses the decoder half. It generates text left-to-right and, when scaled to billions of parameters (GPT-3, GPT-4), became the basis for general-purpose large language models [4].

The relationship is hierarchical: Transformer architecture → BERT-style encoders and GPT-style decoders → both are Large Language Models → LLMs are the dominant modern approach within NLP.

NLP and LLMs

Large Language Models (LLMs) are the current state of the art in NLP. They represent a specific approach within the broader field: training massive neural networks on enormous text corpora to learn general-purpose language capabilities [4].

The key insight of LLMs is that a single model, pre-trained on a broad corpus, can perform many NLP tasks without task-specific training. A model trained to predict the next word in a sequence learns grammar, facts, reasoning patterns, and even some degree of common sense along the way.

But NLP is broader than LLMs. The field also includes:

• Information retrieval — finding relevant documents in large collections
• Knowledge graphs — representing structured relationships between entities
• Speech recognition — converting spoken language to text
• Multimodal NLP — combining text with images, audio, or video

LLMs are the dominant paradigm, but they are not the entirety of NLP.

Why NLP matters for agents

NLP is the foundation of how AI agents communicate and understand instructions:

• Natural language interfaces — agents accept instructions in human language and generate responses that humans can understand
• Instruction following — parsing complex, multi-step instructions into actionable plans requires NLP capabilities
• Context understanding — agents must understand context, ambiguity, and intent in user requests
• Tool interaction — many agent tools process or generate text (search results, API responses, documentation)

Every conversational AI agent—from customer service bots to coding assistants—depends on NLP to bridge the gap between human intent and machine execution.

Limitations and open challenges

Despite remarkable progress, NLP faces ongoing challenges:

• Ambiguity — human language is inherently ambiguous. Resolving ambiguity often requires world knowledge and common sense that models approximate but do not truly possess.
• Bias — NLP models trained on human-generated text inherit societal biases present in the training data. Gender, racial, and cultural biases can manifest in translations, sentiment analysis, and text generation.
• Hallucination — large language models can generate fluent, confident text that is factually incorrect. This is a structural feature of how they generate language, not a bug to be easily fixed.
• Low-resource languages — most NLP research and training data focuses on English and a handful of high-resource languages. Performance degrades significantly for languages with limited digital text.
• Evaluation — measuring NLP quality remains difficult. Automated metrics (BLEU, ROUGE, perplexity) correlate imperfectly with human judgment, especially for open-ended generation tasks.

Frequently asked questions

Is NLP the same as AI?

No. NLP is a subfield of AI focused specifically on language. AI encompasses many other areas—computer vision, robotics, planning, game playing—that don’t involve language processing. But NLP is one of the most visible and impactful areas of AI, especially since the rise of large language models.

What is the difference between NLP and NLU?

NLU (Natural Language Understanding) is a subset of NLP focused specifically on comprehension—understanding the meaning, intent, and structure of text. NLP is the broader field that includes both understanding (NLU) and generation (NLG). When you ask a chatbot a question, NLU parses your intent; NLG generates the response.

How do large language models relate to NLP?

LLMs are the current state of the art within NLP. They are massive neural networks trained on enormous text corpora that can perform many NLP tasks (translation, summarization, question answering) without task-specific training. But NLP is broader than LLMs—it also includes information retrieval, speech recognition, and knowledge representation.

What was the biggest breakthrough in NLP?

The Transformer architecture (2017) is widely considered the biggest breakthrough. It replaced sequential processing with self-attention mechanisms, enabling models to process all words simultaneously and capture long-range dependencies. This made it possible to train the large language models (BERT, GPT) that dominate modern NLP.

Sources

[1] Daniel Jurafsky, James H. Martin. Speech and Language Processing (3rd Edition Draft). January 6, 2026.

[2] Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. Attention Is All You Need. June 12, 2017.

[3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutova (Google AI Language). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. October 11, 2018.

[4] Tom B. Brown, Benjamin Mann, Nick Ryder, et al. (OpenAI). Language Models are Few-Shot Learners (GPT-3). May 28, 2020.