Introducing Infini-attention: Enhancing large language models, efficiently processing lengthy sequences despite memory constraints by combining long-term memory with local attention.
Memory is significant for intelligence as it helps recall and apply past experiences to current situations. However, because of how their attention mechanism works, both conventional Transformer models and Transformer-based Large Language Models (LLMs) have limitations regarding context-dependent memory. This attention mechanism’s memory consumption and computation time are both quadratic in complexity.
Compressive memory systems present a viable substitute for managing very lengthy sequences. They aim to be more efficient and scalable and keep storage and computation costs in check by maintaining a constant number of parameters for storing and retrieving information, in contrast to classical attention mechanisms that need memory to expand with the duration of the input sequence.
This system’s parameter adjustment process aims to assimilate new information into memory while maintaining its retrievability. However, existing LLMs have not yet adopted an efficient compressive memory method that compromises simplicity and quality.
To overcome these limitations, a team of researchers from Google has proposed a unique solution that allows Transformer LLMs to handle arbitrarily lengthy inputs with a constrained memory footprint and computing power. A key component of their approach is an attention mechanism known as Infini-attention, which combines long-term linear attention and masked local attention into a single Transformer block and includes compressive memory in the conventional attention process.
The primary breakthrough of Infini-attention is its capacity to manage memory while processing lengthy sequences effectively. The model can store and recall data with a fixed set of parameters by using compressive memory, which eliminates the requirement for memory to expand with the length of the input sequence. This keeps computing costs within reasonable bounds and helps control memory consumption.
The team has shared that this method has been effective in several tasks, such as book summarising tasks with input sequences of 500,000 tokens, passkey context block retrieval for sequences up to 1 million tokens in length, and long-context language modeling benchmarks. LLMs of sizes ranging from 1 billion to 8 billion parameters have been used to solve these tasks.
Also Read: Google Cloud Next 2024: AI Takes Center Stage with New Tools and Partnerships
One of this approach’s main advantages is the ability to include minimal bounded memory parameters, that is, to limit and anticipate the model’s memory requirements. Also, the suggested approach has made fast streaming inference for LLMs possible, allowing for efficient sequential input analysis in real-time or almost real-time circumstances.
The team has summarized their primary contributions as follows,
- The team has presented Infini-attention, a unique mechanism that blends local causal attention with long-term compressive memory. This method is useful and effective since it effectively represents contextual dependencies over short and long distances.
- The standard scaled dot-product attention mechanism must only be slightly altered to accommodate infinite attention. This enables plug-and-play continuous pre-training and long-context adaptation, making incorporation into current Transformer structures simple.
- The method keeps constrained memory and computational resources while allowing Transformer-based LLMs to accommodate endlessly long contexts. The approach guarantees optimal resource utilization by processing very long inputs in a streaming mode, which enables LLMs to function well in large-scale data real-world applications.
In conclusion, this study is a major step forward for LLMs, allowing for the efficient handling of very long inputs in computation and memory utilization.
