MBZUAI New AI Model K2-65B Sets Global Benchmark In Sustainability And Transparency

The UAE is making significant strides in the field of artificial intelligence (AI) with the announcement of a new, cost-effective, open-source Large Language Model (LLM) named K2-65B. This development comes as a result of a collaborative effort between the Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Petuum, and LLM360. The K2-65B model stands out for setting new benchmarks in transparency and performance within the open-source AI community. It serves as a cornerstone for advancing knowledge sharing, fundamental research, and technological transfer in generative AI.

K2-65B boasts a 65-billion parameter framework. It is designed to provide a detailed blueprint for the documentation and study of LLMs' full lifecycle, incorporating all details necessary for reproduction. This initiative is part of a broader vision to promote peer-reviewed, transparent, reproducible, and collaborative research and development in the realm of artificial general intelligence (AGI). Notably, K2-65B is freely available globally under the Apache 2.0 license, marking it as a significant contribution to the open-source community.

Petuum's Head of Engineering, Hector Liu, highlighted the importance of K2-65B's reproducibility for advancing global knowledge and development in LLMs. The model's transparency is facilitated through the LLM360 Pretraining and Developer Suites, which provide detailed guides, checkpoints, and evaluation results, ensuring reproducibility and auditability.

In terms of technical achievements, K2-65B was trained on 1.4 trillion tokens using NVIDIA's DGX Cloud, employing 480 A100 GPUs. Remarkably, it utilized 35% fewer resources than its counterpart, Llama 2 70B, positioning it as one of the most sustainable models in its category. K2-65B also demonstrates competitive performance in areas of strategic interest, such as mathematical and logical reasoning, even when compared to larger models like GPT-4.

The model's development underscores the UAE's commitment to establishing itself as a leading destination for AI innovation. This ambition is further exemplified by the creation of the world's most advanced Arabic LLM, Jais, highlighting the nation's advancements in natural language processing (NLP). MBZUAI President and University Professor, Eric Xing, emphasized the UAE’s ambition in LLM development and the significance of an open and collaborative approach to creating efficient and transformative LLMs.

K2-65B's training involved two stages and 22 multidisciplinary assessments to ensure a comprehensive evaluation of its performance across various domains, including mathematics, coding, and medicine. It has demonstrated superior performance across these areas compared to Llama 2 70B. Additionally, its chat model, K2-Chat, has shown exceptional capabilities in generating human-like responses across diverse scenarios, outperforming Llama 2 70B Chat in every aspect of evaluation.

One of the notable aspects of K2-65B is its efficiency and reduced energy consumption, supporting sustainable computing practices worldwide. The LLM360 Research Suite offers comprehensive resources for studying training dynamics, presenting researchers and developers with invaluable tools for further exploration.

UAE companies have made important progress in this field in recent years, notably the world’s most advanced Arabic LLM Jais – developed in partnership with Core42, MBZUAI, and Cerebras Systems.

Xing, who supported the development of K2, as well as other large-scale AI systems under the university’s Institute of Foundation Models (IFM), said, “The launch of K2-65B demonstrates the UAE’s growing prowess in superior LLM development. The model epitomises the importance of embracing an open and collaborative approach to create LLMs with unmatched performance and efficiency and the power to bring transformation to all sectors and organisations.”

Trained in two stages, K2-65B underwent rigorous evaluation through 22 multidisciplinary assessments, ensuring a comprehensive performance assessment across various domains including math, coding, and medicine, among others, with the new model surpassing Llama 2 70B across each of these areas.