AI GlossaryTransformerENTRY — Deep Learning
[Deep Learning]
Transformer.
A neural network architecture based on self-attention mechanisms, powering modern language models.
In-depth explanation
01Introduced in "Attention is All You Need" (2017), Transformers replaced RNNs for many sequence tasks. They use self-attention to weigh the importance of different input elements regardless of distance. This enables parallel processing and better handling of long-range dependencies. Transformers power GPT, BERT, and modern LLMs.
Examples
02EX. 01
GPT-4
EX. 02
BERT
EX. 03
Vision Transformer (ViT)
More in Deep Learning
0301Attention MechanismA technique that allows models to focus on relevant parts of the input when producing output.02Convolutional Neural Network (CNN)A neural network architecture designed for processing grid-like data such as images.03DropoutA regularization technique that randomly drops neurons during training to prevent overfitting.04Fine-TuningAdapting a pre-trained model to a new task by training on task-specific data.05LSTM (Long Short-Term Memory)An RNN variant with gates that control information flow, enabling learning of long-term dependencies.06Recurrent Neural Network (RNN)A neural network architecture designed for sequential data with connections between nodes forming cycles.
[NEXT] — APPLY THE CONCEPT
Master Transformer.
Learn how to apply this concept with hands-on projects in our comprehensive AI programs.