Local LLM with Ollama

Interesting links with tutorials

  1. https://itsfoss.com/ollama-setup-linux/
  2. https://itsfoss.com/install-docker-ubuntu/
  3. https://www.arsturn.com/blog/running-ollama-on-nvidia-gpus-a-comprehensive-guide
  4. https://dev.to/korak997/how-to-set-up-a-local-ubuntu-server-to-host-ollama-models-with-a-webui-ok7
  5. https://ibarcena.net/installing-and-running-ollama-openwebui-in-ubuntu-with-cuda-support/
  6. https://www.howto-do.it/install-ollama-on-ubuntu-24/

Troubleshooting links

  1. https://forums.developer.nvidia.com/t/ubuntu-and-nvidia-provided-packages-conflict-breaking-installation/259150

But the basic steps are:

  1. Prepare Ubuntu (see [4])
  2. Download and Install Docker (see [2])
  3. Download NVIDIA drivers (see [4])
  4. Download and install Ollama
  5. Download a Ollama model
  6. Setting up OpenUI

Steps I actually took

Prepare Ubuntu

sudo apt update
sudo apt upgrade
sudo apt install build-essential dkms linux-headers-$(uname -r) software-properties-common

Docker

Download and install

sudo apt install docker.io docker-compose

Test which version is installed with:

docker -v

NVIDIA drivers

Add the NVIDIA PPA

sudo add-apt-repository ppa:graphics-drivers/ppa -y
sudo apt update

NB: not quite sure why I don’t have to do sudo apt upgrade at this point??? I had an error when trying to install the nvidia-driver-560 (“unmet dependencies”), so I did a upgrade.

Detect the recommended driver:

ubuntu-drivers devices

Which gave me:

udevadm hwdb is deprecated. Use systemd-hwdb instead.
[...]
== /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0 ==
modalias : pci:v000010DEd0000134Esv0000103Csd0000822Fbc03sc02i00
vendor : NVIDIA Corporation
model : GM108M [GeForce 930MX]
driver : nvidia-driver-550 - third-party non-free
driver : nvidia-driver-565 - third-party non-free
driver : nvidia-driver-545 - third-party non-free
driver : nvidia-driver-470 - third-party non-free
driver : nvidia-driver-535-server - distro non-free
driver : nvidia-driver-470-server - distro non-free
driver : nvidia-driver-535 - third-party non-free
driver : nvidia-driver-390 - third-party non-free
driver : nvidia-driver-560 - third-party non-free recommended
driver : xserver-xorg-video-nouveau - distro free builtin

Install the recommended driver:

sudo apt install nvidia-driver-560

This time it worked. Now reboot and verify:

nvidia-smi

Ollama

curl -fsSL https://ollama.com/install.sh | sh

Pull a model and run it.