Beginner · Neural Networks
Neural Network
Visual diagram · Math · Worked example · 3 difficulty levels.
TL;DR. A neural network is a computer system that learns patterns from data, loosely inspired by the human brain. It's the foundation of modern AI.
Intuitive Explanation
Think of a neural network like a team of people playing a game of telephone. The first person hears a message (input data), whispers it to the next, who interprets and passes it on. Each person adds their own twist (weights). After many rounds, the team gets better at passing accurate messages — that's learning.
Technical Definition
A neural network is a parameterized function f: ℝⁿ → ℝᵐ composed of alternating linear transformations (matrix multiplications with learned weights W and biases b) and element-wise non-linear activation functions. Training minimizes a loss function L(f(x; θ), y) over a dataset using gradient-based optimization, where θ represents all learnable parameters.
How it works
Neural networks operate through two phases: forward propagation and backward propagation. **Forward Pass:** Input data flows through layers. Each neuron computes z = Σ(wᵢxᵢ) + b, then applies an activation function a = f(z). The output of one layer becomes the input to the next. **Backward Pass (Training):** The network computes the loss (error) between predicted and actual output. Using the chain rule of calculus, gradients flow backward through every layer, telling each weight how to adjust to reduce the error. **Key Components:** - **Input Layer:** Receives raw data (pixels, text tokens, numbers) - **Hidden Layers:** Transform data through learned representations - **Output Layer:** Produces predictions (classifications, values, probabilities) - **Weights & Biases:** Learnable parameters adjusted during training - **Activation Functions:** Introduce non-linearity (ReLU, sigmoid, tanh) **Why Depth Matters:** Deeper networks can learn hierarchical features. In image recognition, early layers detect edges, middle layers detect shapes, and deep layers detect objects. This compositionality is what makes deep learning so powerful. **Training Dynamics:** Networks are typically trained with mini-batch stochastic gradient descent. Batch size, learning rate, and architecture choices significantly affect convergence and final performance.
Mathematical Notation
y = f(Σ wᵢxᵢ + b)Each neuron computes a weighted sum of its inputs (wᵢxᵢ), adds a bias term (b), then passes the result through an activation function f. The weights wᵢ determine how much each input contributes to the output.
Visual Explanation (layers)
Input Layer (x₁, x₂, x₃) → Hidden Layer 1 (4 neurons) → Hidden Layer 2 (4 neurons) → Output Layer (ŷ)
Real-World Use Cases
- Tesla: Vision neural networks for autonomous driving perception
- Google: Neural networks power Search ranking, Gmail spam filtering, and Google Photos
- Netflix: Recommendation engine using deep neural networks for personalized content
- DeepMind: AlphaFold uses neural networks to predict 3D protein structures
Related Concepts
- Backpropagation — An algorithm that efficiently computes gradients by propagating errors backward through the network using the chain rule.
- Activation Function — A non-linear function applied to a neuron's output, enabling the network to learn complex, non-linear relationships.
- Deep Learning — A subset of machine learning using neural networks with many layers to learn hierarchical representations from large datasets.