This article explains a set-up I have found useful for LLM applications using a locally-hosted small-language model. My wife runs a popular events calendar for Baroque music in Bavaria, and we use AI to find and post events. A local LLM is fine for this purpose, since we define the workflow in terms of narrow tasks.

I run small LLMs on a separate Linux server equipped with 64 GB RAM, a 20-core Intel i7, and an Nvidia RTX 4060 Ti with 16 GB VRAM and 4352 CUDA cores. The set-up described below allows me to use the models hosted there from a laptop or separate development workstation, and makes for a flexible arrangement.

Why run models locally, rather than calling a public LLM from OpenAI, Anthropic, or Google? While those models are more powerful for complex inference, small models provide adequate (and rapidly improving) performance for narrower tasks, and preserve data privacy to enable digital sovereignty. They are also cheaper to run on a token-by-token basis, but you need to factor in the costs of the hardware and maintaining it, so this is not always a clear advantage.

Setting up the model server

I now use llama.cpp for running small language models, because it uses fewer resources and is more flexible than Ollama, which I wrote about earlier. Although some distributions provide llama.cpp through their package managers, I recommend installing the latest full-featured version directly from their web site, as documented on the GitHub page:

curl -LsSf https://llama.app/install.sh | sh

The default installation includes CUDA support, and gets installed as ~/.local/bin/llama.

Downloading and running a model

LLama.cpp can directly download and run any of the models on Hugging Face, by using the -hf flag followed by the Hugging Face identifier of the model, which you can find on the Hugging Face models page:

llama serve -hf nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB

Be sure to pick a model that fits on your graphics card, although there are tricks to load a larger mixture-of-experts model by moving some of the layers into RAM. It is hard to tell the VRAM usage from the parameter count or quantization in the model name, so run nvidia-smi after loading a model to be sure. The above Nvidia model runs in a remarkably slim 4430 MB of VRAM.

Note that Hugging Face models downloaded by llama.cpp are saved in your ~/.cache/huggingface directory. They are not downloaded again if they already exist in your cache.

The above command uses serve to create an OpenAI-compatible API endpoint, on the default port 9931 (use --port 9999 to change the port), for use by LangChain and other libraries (see example below). You can also use cli to create a text-mode chatbot, useful for exploring the model’s capabilities, e.g.,

llama cli -hf nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB

If you have previously downloaded a model in GGUF format, you can use it directly with the -m flag, like this:

llama serve -m mymodels/Qwen3.5-9B-Q4_K_M.gguf

Finally, if you want to be able to access the model from anywhere on your local network, you need to specify a host IP of 0.0.0.0, like this:

llama serve --host 0.0.0.0 -m mymodels/Qwen3.5-9B-Q4_K_M.gguf

Accessing the model from an application

If you serve using the --host 0.0.0.0 flag above, you can access the model from LangChain or another library that can call OpenAI via the API.

To do so with a simple LangChain agent (which can iteratively call tools), define the client separately, so you can override the base URL to point to your server (mine has a local host name gpu, although I could have used the IP address on the local network):

from langchain_openai import ChatOpenAI
from langchain.agents import create_agent

client = ChatOpenAI(base_url = "http://gpu:9931/v1", api_key = 'no-key')

# Create a Langchain agent using locally hosted Qwen, with OpenAI API, 
# with tools
agent = create_agent(
    model = client,
    tools = [search_tool, read_file, get_webpage],
    system_prompt = "You are an expert research assistant. You must answer the user's question by searching the internet, downloading relevant webpage content, reading it, and synthesizing a comprehensive answer. You have access to tools for searching and downloading web pages."
)

Note that we import LangChain’s OpenAI library, even though we are calling llama.cpp, since that is the API that is emulated. There needs to be a phony API key to match the library, but this is ignored.

Also note that the above example binds some tools, which you can define using LangChain’s elegant tool syntax. For example, here is the definition for a simple tool that reads a file on the local file system:

from langchain_core.tools import tool

@tool
def read_file(filename: str) -> str:
    """
    Reads a text file and returns the contents.
    """
    print('*** read_file:', filename)
    try:
        return(open(filename).read())
    except Exception as e:
        return f"Failed to read file: {str(e)}"

The agent can then be run with a prompt, so that it calls your local LLM for reasoning, acts on any tool requests suggested by the LLM, and iterates until it can formulate an answer. Here is the code we use to get Qwen to find the upcoming events for our Baroque music events calendar in JSON format.

query = f"""
Read the list of URLs in the file sites.txt, 
then read each page and tell me the dates, venues, and descriptions of all events found. 
Descend into any links found on the pages listed. 
Output in JSON format, with fields date, venue, description for each event.
Put dates in yyyy-mm-dd format.
Only include dates on or after {today}.
"""

# Call the LLM, invokes tools as required
response = agent.invoke({
    "messages": [ { "role": "user", "content": query } ]
})

# Show all results
for m in response["messages"]:
    print(m.content)
    print('---')