This article explains a set-up I have found useful for LLM applications using a locally-hosted small-language model. My wife runs a popular events calendar for Baroque music in Bavaria, and we use AI to find and post events. A local LLM is fine for this purpose, since we define the workflow in terms of narrow tasks.

I run small LLMs on a separate Linux server equipped with 64 GB RAM, a 20-core Intel i7, and an Nvidia RTX 4060 Ti with 16 GB VRAM and 4352 CUDA cores. The set-up described below allows me to use the models hosted there from a laptop or separate development workstation, and makes for a flexible arrangement.

A couple of the challenges running AI applications at scale are managing cost, and ensuring content is compliant. LLM calls and token costs can rapidly spin out of control, and prompts are sent to the LLMs without filtering, a potential risk with external providers. Go’s standard libraries provide sophisticated networking capability, including an elegant reverse proxy implementation. This seemed like a good fit for these problems.

In this post, we will use reverse proxy to create an endpoint for LLM calls, that intercepts and inspects incoming prompts, before forwarding these largely unchanged to the LLM provider if they pass inspection, or block them if not. For those that pass through, the LLM reply is again intercepted by the same reverse proxy, and token counts are extracted from the LLM completions. These counts can be aggregated, and high-usage accounts can be quickly flagged and even blocked.

Let’s use the last post, on building WebAssembly browser plug-ins in Rust, to create something useful: a browser plugin that redacts personally identifiable information (PII) in real time, as a user types or pastes text. The redaction happens instantly, without the user’s data ever leaving the browser.

The best way to show this in a demo is to have the browser’s network tab open (on Chrome: menu -> more tools -> developer tools, click on network tab). Then type some text in the web demo, and show that the redaction is happening without any network traffic, so the text is getting cleaned up without leaving the browser.

WebAssembly is a secure way of building binary code that runs inside a browser, using a protected sandbox. JavaScript can talk to the plugin, but only through explicitly exposed methods, and the same is true of WebAssembly methods calling JavaScript functions from inside the plugin.

Using Rust to create WebAssembly plugins that work in a browser is not new, but I wanted to share the build chain I have been using, as it is simpler than the standard process documented by Mozilla, especially if you are not using Node.js to build your site. We’ll create a simple plugin to show the process, and in the next post we will try something more ambitious.

In this post, we combine the fast MCP server from the last post with the near-instantaneous queries on Parquet files from the post before that. This approach is not only fast, but good for data sovereignty and scale. With only 250 lines of code, we get a lightning-fast MCP server that puts Apache DataFusion behind a tool call, so an LLM can do analytics on enterprise-scale Parquet data without a notebook, a warehouse bill, or a single row of raw data entering the context window.