I blog about stuff I learn about AI! This post is about AliBi, which stands for Attention with Linear Biases. I found out about this when looking into Jina Embeddings V2, so I decided to do a deepdive into this. TLDR, ALiBi, as a transformer model, achieves higher efficiency by not adding positional embeddings. Instead, it incorporates a bias towards elements that are proximate to the target element within the sequence, as opposed to distant elements.

What is AliBi

The method AliBi, in the paper "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" addresses a fundamental question in the realm of transformer models: how to achieve extrapolation at inference time for sequences that are longer than those encountered during training. The method introduces a novel position representation technique termed Attention with Linear Biases (ALiBi), which contrasts with traditional methods by not adding positional embeddings to word embeddings. Instead, ALiBi biases the attention scores between query and key with a penalty that varies linearly with their distance. This new approach has demonstrated its efficacy by enabling the training of a 1.3 billion parameter model on input sequences of length 1024, and successfully extrapolating to sequences of length 2048 without loss in performance. This performance and perplexity matched that of a model utilizing sinusoidal position embedding but showcased an 11% improvement in training speed and an 11% reduction in memory usage. Furthermore, the inductive bias of ALiBi towards recency led to its superior performance over several robust position methods on the WikiText-103 benchmark.

This approach simplifies the position representation method, thereby enabling extrapolation more efficiently compared to existing methods. The core idea behind ALiBi is that by merely altering the position representation method, the model's extrapolation capabilities can be significantly enhanced. Although existing methods were found to be inefficient for extrapolation, ALiBi was introduced as a more straightforward and more efficient alternative. This innovation provides a simpler position method, which not only enables efficient extrapolation but also outperforms other position methods, particularly in handling longer input sequences.

What is a Transformer Model

What is a Transformer model? A transformer model is a type of artificial neural network architecture designed primarily for handling sequence data efficiently. It was introduced in the paper "Attention is All You Need" by Vaswani et al. in 2017 and has since become a foundational architecture for many state-of-the-art models in natural language processing (NLP), and it's also being used in other domains like computer vision.

Here are the key components and concepts associated with transformer models:

1. Self-Attention Mechanism:

   • At the heart of the transformer is the self-attention mechanism, which allows the model to weigh the importance of different parts of the input sequence relative to each other. This mechanism helps in capturing long-range dependencies in the data which is crucial for understanding context in sequences.

2. Positional Encodings:

   • Unlike recurrent neural networks (RNNs) or convolutional neural networks (CNNs), transformers do not have an inherent sense of order or position. To address this, positional encodings are added to the embeddings of the tokens to provide the model with information about the position of each token within the sequence.

3. Encoder and Decoder Stacks:

   • The original transformer architecture consists of an encoder and a decoder. Each of these is composed of a stack of identical layers, with each layer having two sub-layers: a multi-head self-attention mechanism and a position-wise fully connected feed-forward network.

4. Multi-Head Attention:

   • Multi-head attention is an extension of the self-attention mechanism that allows the model to focus on different parts of the input sequence in parallel, enabling it to learn multiple types of relationships between tokens.

5. Layer Normalization and Residual Connections:

   • Each sub-layer (both in the encoder and decoder) has a residual connection around it, and the output is passed through a layer normalization step. These design choices help in training deep networks.

6. Modular and Scalable Design:

   • The transformer's design is modular and can be scaled up by increasing the number of layers, attention heads, or the model dimensions, making it adaptable to different tasks and data sizes.

7. Parallel Processing:

   • One of the significant advantages of transformers is their ability to process all tokens in the input sequence in parallel, as opposed to RNNs which process tokens sequentially. This characteristic makes transformers highly efficient for processing large sequences of data.

Transformers have led to the development of several highly effective models like BERT, GPT (and its iterations like GPT-3), T5, and many others, which have set new performance benchmarks across a wide range of tasks in NLP and beyond.

Inductive Bias of AliBi Towards Recency

"inductive bias of ALiBi towards recency" refers to a particular predisposition or assumption in the Attention with Linear Biases (ALiBi) method, which is oriented towards favoring recent or nearby information when processing sequences of data.

1. Inductive Bias:

   • In machine learning, inductive bias refers to the set of assumptions that a model makes to predict outputs for unseen data based on the training data it has encountered. It's the bias that helps the model to generalize beyond the specific examples it has seen.

2. Towards Recency:

   • The bias towards recency in the context of ALiBi means that the method has a preference for considering recent or nearby tokens (units of data) in a sequence when calculating attention scores. The term "recency" in this context refers to how close or recent certain pieces of information are in relation to others within the input data.

3. In Context of ALiBi:

   • ALiBi modifies the way attention scores are computed in transformer models by introducing a penalty for attention scores between query-key pairs that are far apart in the sequence. This penalty increases as the distance between a key and a query increases, which essentially biases the model towards paying more attention to nearby or recent tokens when processing sequences. This bias helps ALiBi to handle longer sequences efficiently during inference, even if the training was performed on shorter sequences.

In summary, the inductive bias of ALiBi towards recency is a design choice that helps the method to efficiently process and extrapolate on longer sequences by favoring more recent or nearby information when determining attention relationships within the data.

Sources and Reading Material

https://ar5iv.labs.arxiv.org/html/2108.12409
https://arxiv.org/abs/1706.03762