AI GlossaryAttention MechanismENTRY — Deep Learning
[Deep Learning]
Attention Mechanism.
A technique that allows models to focus on relevant parts of the input when producing output.
In-depth explanation
01Attention computes weighted combinations of input elements based on their relevance to the current task. Self-attention relates different positions within a single sequence. Cross-attention relates positions across different sequences (like source and target in translation). Attention enables dynamic, context-dependent processing of information.
Examples
02EX. 01
Translation focusing on relevant words
EX. 02
Image captioning attending to regions
More in Deep Learning
0301Convolutional Neural Network (CNN)A neural network architecture designed for processing grid-like data such as images.02DropoutA regularization technique that randomly drops neurons during training to prevent overfitting.03Fine-TuningAdapting a pre-trained model to a new task by training on task-specific data.04LSTM (Long Short-Term Memory)An RNN variant with gates that control information flow, enabling learning of long-term dependencies.05Recurrent Neural Network (RNN)A neural network architecture designed for sequential data with connections between nodes forming cycles.06Transfer LearningUsing knowledge learned from one task to improve performance on a different but related task.
[NEXT] — APPLY THE CONCEPT
Master Attention Mechanism.
Learn how to apply this concept with hands-on projects in our comprehensive AI programs.