Tools and libraries for LLM Fine-Tuning and Inference

Due to LLM popularity at the moment, there are a lot of different tools for what seem to be very similar tasks — generating responses from the LLM and fine-tuning LLMs for your own needs. To understand this market better myself, I decided to write this post, summarizing knowledge that I already had and recently learned on the topic. All the information presented in the post is accurate as of April 2025 and may change, as the ecosystem around LLMs changes quickly.

LLAMA.CPP

First, I want to mention llama.cpp, which is both used as a backend for some other popular tools and could be used on its own for user-friendly and high-throughput inference.

Disclaimer: this is not true for ALL LLM inference tools, as some of them use custom backends or other backends such as TensorRT-LLM, but it is hard to highlight enough the importance of llama.cpp for the LLM inference ecosystem.

Links:
https://github.com/ggml-org/llama.cpp

User-Friendly LLM Interfaces

These tools focus on user experience. Most come with GUIs, are relatively simple to install, and allow you to quickly start chatting with an LLM. Some also support fine-tuning and provide API access to local models. Many of these tools use llama.cpp or similar backends and are compatible with GGUF quantized models. Most are open source and support local inference, though multi-GPU support is usually limited or dependent on the backend used.

Links:
https://lmstudio.ai/work
https://ollama.com
https://docs.gpt4all.io/gpt4all_desktop/models.html
https://github.com/Mintplex-Labs/anything-llm
https://github.com/menloresearch/jan
https://cortex.so/models/

Tools for High-Performance Inference

These tools are mostly used in production for high-performance inference and support quantization, batching, and other optimization techniques. Most of them provide Docker images as well.

Links:
https://github.com/turboderp-org/exllamav2
https://blog.vllm.ai/2023/06/20/vllm.html
https://github.com/huggingface/text-generation-inference
https://huggingface.co/docs/text-generation-inference/multi_backend_support
https://nvidia.github.io/TensorRT-LLM/torch.html

Tools for High-Performance Fine-Tuning

These tools are used for fine-tuning, where you train models for your specific needs. They support quantization, multi-GPU setups, and different model families. They use different approaches for fine-tuning, such as a code-first approach with Unsloth or fine-tuning through configuration and command line with Torchtune.

Links:
https://github.com/pytorch/torchtune
https://unsloth.ai/introducing
https://github.com/axolotl-ai-cloud/axolotl