Self-Attention Mechanism In Transformers | Transformers

Self-Attention Mechanism is a key building block of the Transformer architecture. It enables the model to weigh the importance of different words in a sentence when processing each word. This mechanism is crucial for capturing relationships between words in a sequence, especially in natural language processing tasks.

Don't take img serious if you don't understand

Here's how it works:

Input Representation: Assume you have a sentence or sequence of words. Each word is initially represented as an embedding vector, which encodes its meaning. These embeddings are the input to the self-attention mechanism.

Query, Key, and Value: The self-attention mechanism computes three sets of vectors for each word:

Query: This is the vector for the current word that we want to calculate attention scores for.

Key: These are the vectors for all words in the input sequence. Keys represent the words that we compare against to decide the importance of the current word.

Value: These are also vectors for all words. Values are the ones we'll eventually use to compute the weighted sum.

Calculating Attention Scores: To calculate the attention score for a word, we take the dot product between the query vector of the current word and the key vectors of all words. This measures how well the current word "matches" with all other words in the sequence. These dot products are then scaled and passed through a softmax function to get a probability distribution over all words. This distribution represents the importance or attention that the current word assigns to other words in the sequence.

Weighted Sum: The final step is to calculate a weighted sum of the value vectors using the attention scores obtained in the previous step. This weighted sum becomes the representation of the current word after self-attention.

Multi-Head Attention: In practice, the self-attention mechanism is used with multiple sets of query, key, and value vectors in parallel. These are called "attention heads." This allows the model to learn different types of relationships between words.

The self-attention mechanism is powerful because it can capture both local and global dependencies between words, making it suitable for various sequence-to-sequence tasks, including machine translation and text summarization.

Here's a simplified Python code snippet to help you understand the concept:

import numpy as np

# Define query, key, and value vectors for three words

query = np.array([0.2, 0.1, 0.3])

keys = np.array([[0.1, 0.2, 0.3], [0.2, 0.3, 0.1], [0.3, 0.1, 0.2]])

values = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]])

# Calculate attention scores

attention_scores = np.dot(query, keys.T)

attention_scores = attention_scores / np.sqrt(3)  # Scaling

# Compute softmax to get attention weights

attention_weights = np.exp(attention_scores) / np.sum(np.exp(attention_scores), axis=0)

# Calculate the weighted sum using attention weights

output = np.dot(attention_weights, values)

print("Attention Scores:", attention_scores)

print("Attention Weights:", attention_weights)

print("Output after Attention:", output)

This is a simplified example, and in practice, Transformer models use multi-head self-attention and learn these parameters during training. 

TAGS:

#transformers

#attentionIsAllYouNeed