- August 9, 2024
- 4 min read
Hugging Face Acquires XetHub to Build and Scale Millions of Large LLMs
In a move that could revolutionize the development and scaling of large language models (LLMs), Hugging Face, a leading AI and machine learning platform, has acquired XetHub, a Seattle-based company specializing in scaling Git for massive datasets and AI models. This acquisition aims to enhance Hugging Face’s capabilities in managing and versioning colossal datasets and models, a critical need as the AI community continues to push the boundaries with ever-larger models and datasets.
Hugging Face: Paving the Way for AI Collaboration
Founded in 2016, Hugging Face has emerged as a pioneering force in the AI and machine learning landscape, fostering an ecosystem of collaboration and open-source innovation. With over 1.3 million model repositories, 450,000 datasets, and 680,000 spaces, the platform has become a hub for AI enthusiasts, researchers, and industry professionals alike.
Clement Delangue, Chief of Hugging Face, expressed his excitement about the acquisition, stating, “Big models are here to stay. What we want is to make the development of AI closer to what software engineering is — make it drastically faster.” This sentiment underscores the company’s commitment to streamlining and optimizing the AI development process, a mission that aligns perfectly with XetHub’s expertise.
XetHub: Scaling Git for Terabyte-sized Repositories
Founded in 2021 by Yucheng Low, Ajit Banerjee, and Rajat Arya, XetHub has developed groundbreaking technology that enables Git to handle terabyte-scale repositories, allowing teams to work efficiently with evolving datasets and models. This capability is crucial as AI models continue to grow in size, with trillion-parameter models like the BigLlama-3.1-1T already on the horizon.
Yucheng Low, co-founder of XetHub, highlighted the importance of data in AI’s evolution, expressing excitement about joining Hugging Face to continue their mission of enhancing AI collaboration at scale. “Data is the fuel that powers AI’s evolution,” Low said. “By joining forces with Hugging Face, we can accelerate the development and scaling of AI models, enabling more teams to unlock the full potential of this transformative technology.”
Revolutionizing AI Development and Collaboration
The acquisition of XetHub aligns with Hugging Face’s long-term goal of optimizing storage and versioning for AI development, moving away from the limitations of Git LFS, which was not designed to handle the immense file sizes typical in AI. Julien Chaumond, CTO of Hugging Face, explained, “The XetHub team will help us unlock the next 5 years of growth of HF datasets and models by switching to our own, better version of LFS as storage backend for the Hub’s repos.”
By integrating XetHub’s technology, Hugging Face aims to streamline data management and updates for its vast community of users. Instead of re-uploading entire files, users will only need to upload modified chunks, reducing storage needs and improving efficiency. This improvement is crucial as AI models continue to grow in size, with trillion-parameter models like the GPT-4 already on the horizon.
“This is the real 🍓—welcome to @xetdata. We’re just getting started!” – Clement Delangue, Chief of Hugging Face
Empowering the AI Community
Hugging Face’s acquisition of XetHub is a significant step forward in empowering the AI community to develop and scale large language models more efficiently. By addressing the limitations of Git LFS and enabling seamless collaboration on massive datasets and models, this partnership has the potential to accelerate innovation and unlock new frontiers in AI development.
As the demand for AI continues to surge across industries, from retail and marketing to enterprise solutions, Hugging Face’s commitment to fostering an open and collaborative ecosystem will be instrumental in driving the next wave of AI innovation.
Conclusion
The acquisition of XetHub by Hugging Face marks a significant milestone in the AI industry, paving the way for more efficient development and scaling of large language models. By combining Hugging Face’s collaborative platform with XetHub’s cutting-edge technology for handling terabyte-scale repositories, this partnership promises to empower the AI community with unprecedented capabilities.
As the industry continues to push the boundaries of what’s possible with AI, the integration of XetHub’s technology into the Hugging Face platform is poised to unlock new horizons, accelerating innovation and driving the next generation of AI-powered solutions. The future of AI development is bright, and Hugging Face, together with XetHub, is leading the charge.
