At their annual Meta Connect event in California, the social media giant introduced "Llama 2 Long," an advanced AI model that promises to generate coherent and relevant responses to exceptionally long user queries.
This innovation has quietly taken the spotlight, surpassing some of its formidable competitors in the field.
Introducing Llama 2 Long
The original Llama 2 was already a formidable open-source AI model, capable of learning from a wide range of data sources and performing multiple tasks, including coding, mathematics, language understanding, common sense reasoning, and even engaging in natural conversations.
However, what makes Llama 2 Long truly remarkable is its ability to handle more extensive sequences of information, Meta researchers tell us in a recent paper.
It has been trained on a dataset containing longer texts and modified to overcome the limitations faced by its competitors, such as OpenAI's GPT-3.5 Turbo and Claude 2, which struggle with extensive contextual understanding.
Impressive AI Power
Interesting Engineering reports that Meta's researchers used various versions of Llama 2, ranging from 7 billion to a whopping 70 billion parameters-these parameters define how the AI model adjusts as it learns from data.
Additionally, they added a staggering 400 billion tokens of data that contained longer texts than the original Llama 2 dataset.
A key technique employed was the introduction of Rotary Positional Embedding (RoPE), a method that maps each token to a point on a 3D graph.
This innovative approach allows Llama 2 Long to comprehend the relationships between tokens, even when they are scattered across extensive passages of text.
The reduced rotation angle of RoPE encoding in Llama 2 Long enabled the inclusion of more distant or less frequent tokens, enhancing the model's knowledge base.
But the brilliance does not stop there. The researchers used reinforcement learning from human feedback (RLHF), a method where the AI model is rewarded for correct answers and corrected by human evaluators.
They also utilized synthetic data generated by Llama 2 chat itself to fine-tune its performance across various tasks.
Unleashing Llama 2 Long
The most astonishing revelation is that Llama 2 Long can craft high-quality responses to user prompts that stretch up to a staggering 200,000 characters.
To put this into perspective, that is roughly equivalent to composing 40 pages of text! Whether it is history, science, literature, or sports, Llama 2 Long seems to excel in understanding and responding to a wide array of topics.
The study that led to Llama 2 Long's creation delves into the development of a series of long-context language models (LLMs). These models are designed to understand and generate text with an impressively long context, extending to a staggering 32,768 words.
Meta's researchers are well aware of the ethical and social implications of such a powerful AI model. They emphasize the need for responsible use and further research on how to harness these capabilities for the benefit of society.
Stay posted here at Tech Times.
Related Article : Google Launches Tool for Web Publishers to Opt out of Data Training for Bard, Other AI Models
