Sidechat is a exploration of sidechannels in chatbots: the chatbot can answer questions like usual, and can also have another conversation, out in the open, under the radar.

sidechat app generating text

Try out sidechat in the browser!

Sidechannel as untrustworthy surrealism

Similar to rejecting realism in image synthesis and choosing to generate landscapes with numbers, Sidechat rejects the realism and patina of trustworthy professionalism, and veers into the surreal. Unlike asking a larger LLM to make an acrostic-oriented answer, which would require much more thinking to work effectively, and potentially leak the secret of the acrostic into the output, the acrostic details are only in the part of LLM generation that chooses text to generate. (Future work could spend a bit more time on multi-token prediction, to make the acrostic constraint better fitted into the text.)

Tiny LLM in the browser, >10 tok/s, 200MB, for artistic permanence

Artwork that relies on static files running Javascript in the viewer’s browser is exceptionally cheap to keep alive. (Self-hosting a 200MB LLM is a few cents a year on S3 at full retail pricing, and a few cents a viewing, and is straightforward to move between hosting providers.)

This is just the first project of many to use a ‘general-purpose’ LLM that’s small enough to fit in the browser. Initially, this used SmolLM2-360M-Instruct, at a 4-bit quantization, and then we got slightly better results using LFM2.5-350M.

Text classification via LLM and computer-optimized prompt

When we constrain generation, we start lines with ` * ` if the prompt is something that will generate a list.

We have a training dataset of 80 examples (40 of list, 40 of prose) and a validation dataset of 20 examples (10 of each), and we did multiple rounds of improving prompts, keeping the top 5-10 prompts, and mixing prompts between each other.

There are a lot of use cases that are pretty interesting for using an LLM to classify text via a 60% accurate prompt that can optimize to a 97% accurate prompt.

Let me know what you think!

The code for Sidechat is open source so let me know what you think! Definitely interested to collaborate on using small LLMs in the browser