Running AI Locally

I have been using AI for a long time now. I used chatbots like ChatGPT, Claude, Gemini, and many more. I also used Opencode with Gemini’s API key and the free tier which Opencode provides.

Getting Started with Local AI

A few days back I decided to try and run a local AI model on my laptop. I quickly downloaded LMStudio and downloaded a small model from its store and tried to run it. And IT WORKED! It ran!

It was an amazing thing to witness. An AI model running on my laptop, which is a very old laptop with just 4GB of VRAM and 16GB of RAM.

I then decided to download some agents like Hermes and Bionic to try to run them as well. But my laptop couldn’t handle the large context size, which is required by these agents. My laptop is good but not that good.

So, I decided to install Linux again on my laptop, since Linux is basically the best OS for this kind of thing. I downloaded Arch Linux as I was already experienced in it, as I used to daily drive it. I configured it with Hyprland and installed opencode. Used opencode to make it look good.

I then installed Ollama and using Ollama I installed a few models like gemma4:4b, qwen3.5:4b, qwen3.5:9b and a small STT model (more on this in another blog). Everything ran amazingly. I think it ran better than Windows, of course.

llama.cpp: Building from Source

Then I came across llama.cpp. I researched it and found that it is the base on which Ollama, LM Studio and other tools are built. I thought I will use this to try and run models on my laptop. So that was the main goal for me now.

I saw a guide and read it from start to finish. I then checked if llama.cpp is there in my AUR. It was there, but I thought that it would be better to build it from source, as it will be better optimized like that.

So, I tried to build it from source. When I was building it, my CPU went to 100°C. As it was taking time and I was hesitant to keep my CPU running for so long. I cancelled the build and searched if there was anything I could do to speed it up.

I didn’t find anything.

So, I finally gave in and just let it build. It took some time and after some errors, it finally built. I quickly downloaded some models from the HuggingFace website. And tried to run them with llama.cpp.

First Run

The first thing I came across was the OOM error. As I was trying to offload all the layers to the GPU. It was throwing this error. Once I reduced the offloaded layers and used some flags which I found online, I was able to run the model using llama.cpp.

Now, llama.cpp has its own web interface similar to ChatGPT. The first thing I found once I ran my first prompt, was that it was underwhelming. I thought I would get almost double the speed for some reason, but it was not that.

I quickly realized that there is a bigger and deeper hole with llama.cpp. There are many flags and environment variables which can optimize the build and optimize the model’s performance as per my hardware.

What’s Next

I decided not to go into that hole right now. Maybe in the future. My main goal was to run local LLM models on my laptop and the final goal was to use llama.cpp to run models locally. Both of these goals were achieved and I am quite happy with that, considering my old laptop.

Once I get better hardware, I will run bigger models and will dive into the llama.cpp hole, but until then I’m just going to use Opencode with some subscription and try to incorporate AI into my life more.

Conclusion

Running LLM locally really helped me get a better understanding on this subject. I learnt terms like inference, MOE, quantization, which would have easily gone over my head previously. I’ll still keep learning more as this field is evolving so fast and there is always something waiting to be discovered.