Industry-leading tool to merge pre-trained models to create smaller, more efficient models without additional training. Original capabilities of the models are preserved while AI performance and versatility are enhanced.
Get started with the open source library.
Merges can be run entirely on CPUs with as little as 8 GB of VRAM. The resulting model outranks the input models. You boost the performance and accuracy of your models in a cost-efficient and time-efficient way.
Here at Arcee AI, we use model merging as one of the key techniques in the world-class proprietary pipeline we developed to train the small language models (SLMs) that power Arcee Orchestra.
When you merge models that have been trained on the same tasks
Better performance and improved out-of-domain generalization
When you merge models that have been trained on different tasks
Synergistic effects can actually boost task performance above input level
With more than 5,000 stars on GitHub, MergeKit is the premier tool for merging pre-trained models, and offers unparalleled flexibility and efficiency in model integration.
While the library itself is a powerful resource, achieving optimal merging results requires deep expertise and precision–so we also offer a MergeKit Enterprise License (BSL) that includes direct access to our team, including the creator of MergeKit, Charles Goddard. Get in touch for licensing details.
MergeKit is a great tool for combining the strengths of multiple language models. With MergeKit, you can train a model on just the data that matters to you, and add the capabilities of cutting edge open source models without further training. This brings benefits of efficiency and cost savings, and also reduces complexity in your LLM pipeline.
MergeKit supports most decoder-only language models implemented in the transformers library. MergeKit features first-class support for Llama, Mistral, Phi-2, StableLM, GPT-NeoX, Mixtral, and many more.
Yes, MergeKit will remain open source and we will be extending and expanding it to be the most powerful toolkit in the world for model merging.
MergeKit is utilized under the hood of the Arcee product in our Adaptation and Alignment layers.
Absolutely, you can go to the examples folder in the repo for a variety of examples.
The easiest way to get started with model merging and MergeKit is to follow the open source repo guide to installation and getting started.
Contact us for a custom demo.
Subscribe to our newsletter
.svg)
