DistillJS

Here's a new thing: DistillJS.

DistillJS allows you to recreate a specific LLM behavior in Javascript.

I built this because in various projects, I kept having to call LLMs via API for simple tasks, such as summarizing or classifying text. It would be much better if that functionality were part of the application itself: lower latency, no usage charges, and no API keys to manage.

I heard o3 was really smart, so I figured I'd assign it the challenge of writing a minified JS file that delivers 80-90% of the functionality of an LLM on a specific task. DistillJS is the result.

It works in three steps:

1. Define the task: Write a prompt that describes the task you want the LLM to perform (e.g. "summarize this text in 2-3 sentences").
2. Provide examples: Give a few good examples of input/output pairs (DistillJS can generate the outputs for you using gpt-4o-mini).
3. Distill: Ask o3 (or o4-mini) to write a minified JS function that reproduces similar outputs for a wide variety of inputs, mutatis mutandis.

I've mainly been testing this on summarization tasks, and the files o3 comes up with work surprisingly well (but aren't perfect). You can try the summarization demo yourself on distilljs.com by clicking "autofill" on the prompt and "auto-generate" on the examples.

There's a possibility DistillJS is a fool's errand, because it's essentially asking a deterministic Javascript file to play in the same league as a machine learning model.

• The reason machine learning has become ubiquitous is that it's proven dramatically more effective than deterministic programming at producing better results across a wide range of tasks and inputs.

But I think this project is worth a shot for a few reasons:

• It's a pain to call LLM APIs for simple tasks – it's computationally and financially more expensive. People want a better solution.
• o3 is really smart. o4 will be even smarter. They're full of surprises.
• Even if o3 or o4 can't achieve >90% performance in Javascript, even 80% or 70% performance would be worth the trade-off in certain cases.

I heard a quote recently – was it from Sam Altman? – that if you find yourself building a product that's right on the cusp of current model capabilities, that's a good place to be.

DistillJS feels like it's on that cusp. Try it out and email me what you think!