- August 10, 2024
- 4 min read
Hugging Face Acquires XetHub to Build and Scale Millions of Large LLMs
Hugging Face Acquires XetHub to Unleash the Power of Large Language Models at Scale
In a groundbreaking move, Hugging Face, the renowned AI and machine learning platform, has acquired XetHub, a Seattle-based company specializing in scaling Git for large datasets and AI models. This strategic acquisition aims to propel Hugging Face’s capabilities in managing and versioning massive datasets and models, a crucial need as the AI community embraces the era of large language models (LLMs) and ever-growing data volumes.
“Big models are here to stay,” said Clement Delangue, Chief of Hugging Face. “What we want is to make the development of AI closer to what software engineering is — make it drastically faster.”
In a world where AI models are rapidly expanding in size and complexity, the ability to efficiently handle and collaborate on terabyte-scale repositories is paramount. XetHub has developed groundbreaking technology that enables Git to scale to these immense file sizes, empowering teams to work seamlessly with evolving datasets and models.
Revolutionizing AI Collaboration and Development
Hugging Face’s acquisition of XetHub aligns with the company’s long-term vision of optimizing storage and versioning for AI development, transcending the limitations of Git LFS, which was not designed to handle the colossal file sizes typical in the AI realm. As Julien Chaumond, CTO of Hugging Face, explained, “The XetHub team will help us unlock the next 5 years of growth of HF datasets and models by switching to our own, better version of LFS as storage backend for the Hub’s repos.”
XetHub’s innovative technology promises to unlock significant growth for the Hugging Face platform by enabling more efficient data management. Instead of re-uploading entire files, users will only need to upload modified chunks, streamlining updates and reducing storage needs. This improvement is crucial as AI models continue to grow in size, with trillion-parameter models like the BigLlama-3.1-1T already on the horizon.
Scaling AI Infrastructure for the Future
Founded in 2021 by Yucheng Low, Ajit Banerjee, and Rajat Arya, XetHub was built by a team with extensive experience in scaling AI infrastructure, including work on Apple’s internal machine learning infrastructure. The team will now integrate XetHub’s cutting-edge technology into the Hugging Face platform, aiming to make AI collaboration and development easier for its vast community of users.
“Data is at the heart of AI’s evolution, and we’re thrilled to join Hugging Face to continue our mission of enhancing AI collaboration at scale,” said Yucheng Low, co-founder of XetHub.
Hugging Face is currently handling a significant volume of data, with over 1.3 million model repositories, 450,000 datasets, and 680,000 spaces, totaling 12 petabytes of data stored in LFS. The acquisition of XetHub is expected to help manage this growing demand more efficiently, paving the way for a future where AI models can be developed, shared, and deployed at an unprecedented scale.
Expanding the Hugging Face Ecosystem
This acquisition is the latest in a series of strategic moves by Hugging Face to expand its ecosystem and capabilities. Previously, the company acquired Spanish startup Agrilla for $10 million, which specializes in collaborative software for AI professionals, focusing on data annotation and enhancing NLP with human-machine collaboration.
Hugging Face’s infrastructure team is also actively expanding and hiring to support the ongoing development of its platform. With OpenAI’s ChatGPT and other large language models gaining widespread adoption, the demand for efficient and scalable AI development tools is at an all-time high.
Embracing the Future of AI
As the AI landscape continues to evolve at a rapid pace, Hugging Face’s acquisition of XetHub positions the company as a trailblazer in enabling the development and deployment of large language models at scale. By addressing the challenges of managing and versioning massive datasets and models, Hugging Face is paving the way for a future where AI can be harnessed to its full potential, driving innovation and transforming industries across the globe.
“This is just the beginning,” said Delangue. “With XetHub’s technology, we’re unlocking the next frontier of AI development, empowering our community to build and scale millions of large language models and push the boundaries of what’s possible.”
The AI industry is on the cusp of a revolution, and Hugging Face’s acquisition of XetHub is a testament to the company’s commitment to leading the charge, enabling developers, researchers, and businesses to harness the power of large language models and shape the future of AI.
Explore More on UBOS:
- ChatGPT and Telegram integration
- Revolutionizing marketing with generative AI
- February product update on UBOS
- Generative AI agents for businesses
- Scaling AI in organizations