Exploring Open-Source AI: Running Local LLMs on Ubuntu
To Nha Notes | Feb. 20, 2025, 9:40 p.m.
🔍 1. Discover & Browse Local LLMs
These platforms allow you to explore and download open-source AI models:
- Hugging Face Model Hub – The biggest repository for LLMs like LLaMA, Mistral, Falcon, and more.
- TheBloke on Hugging Face – Optimized quantized models for running efficiently on local hardware.
- ModelScope – A model hub focused on Chinese and multilingual models.
- CivitAI – AI models and LLMs focused on image generation but expanding into text models.
📥 2. Download & Run Local LLMs
These tools allow you to run local models efficiently on Ubuntu:
Easy-to-Use LLM Runtimes
- Ollama – Simplest way to run local LLMs with ollama run mistral.
- LM Studio – GUI application for discovering and running local models.
- GPT4All – Cross-platform LLM runner with chat UI.
Command-Line & Web UIs
- Text Generation Web UI – Web UI for running models like LLaMA, Mistral, and GPT-J.
- KoboldCpp – Optimized for running large models using CPU/GPU.
- LLM CLI – Simple terminal-based local LLM runner.
For Advanced GPU Users
- LMDeploy – NVIDIA’s optimized LLM deployment toolkit.
- vLLM – High-performance LLM inference for GPUs.
- ExLlama – Optimized for running quantized LLaMA models.
🏃 3. Running a Local LLM on Ubuntu
Example: Using Ollama to run Mistral locally
# Install Ollama curl -fsSL https://ollama.com/install.sh | sh # Download and run Mistral-7B ollama run mistral
Example: Using LM Studio
- Download LM Studio.
- Open LM Studio and browse models.
- Download and run any model with a GUI.
Would you like recommendations based on your hardware specs (CPU/GPU)? 😊