Neural networks

## What is a neural network?

A **neural network** (or **artificial neural network**, ANN) is a computational system loosely inspired by the structure of biological brains. It is made up of layers of interconnected nodes—called **artificial neurons** or **units**—that process information by passing signals through adjustable connections.

Each connection has a **weight**, a number that determines how much influence one neuron has on the next. During training, the network adjusts these weights based on examples, gradually learning to map inputs (like images or text) to outputs (like labels or predictions). This ability to learn from data, rather than following hand-coded rules, is what makes neural networks the foundation of modern AI [1].

## How neural networks work

A neural network processes data in layers:

1. **Input layer** – receives raw data (pixels, words, numbers)
2. **Hidden layers** – transform the data through weighted connections and activation functions
3. **Output layer** – produces the final prediction or classification

Each artificial neuron in a hidden layer:
– Takes inputs from the previous layer
– Multiplies each input by a learned weight
– Sums the results and adds a bias term
– Passes the sum through an **activation function** (like ReLU or sigmoid) that decides whether the neuron “fires”

The network learns by comparing its output to the correct answer, calculating the error, and adjusting weights backward through the network—a process called **backpropagation** [2].

## Types of neural network architectures

Different problems call for different network structures:

– **Feedforward networks** – the simplest type; data flows in one direction, from input to output. Good for tabular data and basic classification [3].
– **Convolutional neural networks (CNNs)** – designed for spatial data like images. Use filters that slide across the input to detect patterns (edges, textures, objects). Dominated computer vision after AlexNet’s 2012 ImageNet breakthrough [4].
– **Recurrent neural networks (RNNs)** – designed for sequential data like text or time series. Each step feeds back into the next, giving the network a form of memory. LSTMs and GRUs improved on early RNNs by solving the vanishing gradient problem [3].
– **Transformers** – the architecture behind modern large language models. Use **self-attention** mechanisms to process all parts of a sequence in parallel, rather than step by step. This parallelism makes it possible to train on massive datasets. Introduced in the 2017 paper “Attention Is All You Need” [5].

## Shallow vs. deep networks

A network with one or two hidden layers is considered **shallow**. A network with many hidden layers (sometimes hundreds) is considered **deep**—hence the term **deep learning**.

For decades, training deep networks was impractical. The 1986 backpropagation paper by Rumelhart, Hinton, and Williams showed that multi-layer networks could learn useful internal representations [2], but computational limits kept networks small.

The breakthrough came in 2012, when **AlexNet**—a deep convolutional neural network—won the ImageNet competition by a wide margin, demonstrating that deep networks could dramatically outperform traditional approaches when given enough data and computing power [4]. This launched the modern deep learning era.

## Historical milestones

– **1943** – Warren McCulloch and Walter Pitts proposed the first mathematical model of an artificial neuron [6]
– **1957** – Frank Rosenblatt invented the **perceptron**, the first neural network that could learn from data [7]
– **1969** – Minsky and Papert published *Perceptrons*, highlighting limitations of single-layer networks and dampening research funding for over a decade [8]
– **1986** – Rumelhart, Hinton, and Williams published the backpropagation algorithm, making multi-layer networks trainable [2]
– **1998** – LeCun et al. created **LeNet-5**, an early CNN for handwritten digit recognition [9]
– **2012** – AlexNet won ImageNet, triggering the deep learning revolution [4]
– **2017** – Vaswani et al. introduced the transformer architecture, enabling modern LLMs [5]
– **2022** – ChatGPT brought transformer-based neural networks into mainstream awareness [10]

## Neural networks and the agentic economy

Neural networks are the computational engine behind every major AI capability in the agentic economy:

– **Large language models (LLMs)** are transformer-based neural networks trained on massive text corpora to generate and understand language [5]
– **Computer vision** agents use CNNs to interpret visual information
– **Speech recognition** and **text-to-speech** systems use specialized neural architectures
– **Reinforcement learning** agents (like those controlling robots or playing games) use neural networks to estimate value functions and policies

When an AI agent reasons about a problem, selects a tool, or generates a response, it is running inference on a neural network. The shift from rule-based software to neural-network-powered agents is what makes the agentic economy possible [10].

## Limitations

Neural networks are powerful but have important limitations:

– **Black-box problem** – their internal reasoning is difficult to interpret, which creates challenges for high-stakes applications like healthcare and law [11]
– **Data hunger** – they typically need large amounts of labeled training data
– **Computational cost** – training large models requires significant hardware (GPUs, TPUs) and energy
– **Hallucination** – generative models can produce confident-sounding but false outputs [12]
– **Adversarial vulnerability** – carefully crafted inputs can fool neural networks in ways humans would not be fooled [13]

These limitations are active areas of research and are central to ongoing debates about AI safety and governance.