---
title: Running AI client-side
teaser: Running AI client-side is a great way to provide privacy and save costs. Learn
how to do it and when it is useful.
tags: javascript,artificial intelligence,machine learning,web
author: Matheus Richard
published_on: 2024-09-17
---
Everyone is doing AI nowadays. While [using LLMs] or [diffusion models] via API
is a [great way] to use AI, it is also possible to run AI client-side. This
article will show you how to do it and some of the trade-offs involved.
## What can be done?
Actually, quite a lot. We're going to use the [Transformers.js] library to run
AI client-side. It supports a variety of tasks, in different categories. Some
examples are:
- **Natural Language Processing**: [summarization], [translation], and [question answering].
- **Computer Vision**: [object detection], [upscaling images], and [removing backgrounds].
- **Audio**: [automatic speech recognition] and [text to speech].
You can see the [full list of tasks and models here]. I'll use a Rails app to
demonstrate some of the features. The full code is available [in this repo].
For the sake of brevity, I'll use the simplest models in the examples. They're not the best available, but they're good enough for a demo.
Feel free to explore the full list of tasks and models to find the best one for your use case.
## Describing images
Writing alt text for images is [important for accessibility], and AI can help
giving us a head start. Install the `@xenova/transformers` package (I used `yarn`, but you might have a different setup in your project), and you're
good to go.
The idea is to use an image to text model to automatically add a description to
an image when the user uploads it. Unfortunately, Action text doesn't have
events for when an image finishes uploading -- [should be out in Rails 8],
though --, so we'll put the description in an `` tag and let the user
copy from there.
Assuming we have a basic Article model with a title and content, on the
`app/views/articles/_form.html.erb` view we can add the following:
```erb
<%= form_with(
model: article,
data: {
controller: "autocaption",
action: "trix-attachment-add->autocaption#saveAttachment"
}) do |form| %>
<%= button_tag "Describe image", type: :button, class: "secondary", data: {action: "autocaption#describeImage", autocaption_target: "trigger"} %>
Description:
<% end %>
```
After setting up the controller and the targets, we save the attachment when
the user uploads one. The `autocaption_controller.js` file will look like this:
```javascript
import { Controller } from "@hotwired/stimulus"
import { pipeline, RawImage } from '@xenova/transformers';
export default class extends Controller {
static targets = ["output", "trigger"]
async initialize() {
this.triggerTarget.disabled = true;
this.captioner = await pipeline('image-to-text');
}
saveAttachment(event) {
this.attachment = event.attachment
this.triggerTarget.disabled = false;
}
async describeImage() {
const previousLabel = this.triggerTarget.textContent;
this.triggerTarget.textContent = 'Analyzing...';
this.triggerTarget.disabled = true;
const img = await RawImage.fromBlob(this.attachment.file);
const caption = (await this.captioner(img))[0].generated_text;
this.outputTarget.textContent = caption;
this.triggerTarget.textContent = previousLabel;
}
}
```
After loading the model, we use to describe the image and update the output text
with the description. Here's that in action:
<%= @article.title %>
<%= button_tag "Read aloud", type: :button, class: "secondary", data: { action: "text-to-speech#play", text_to_speech_target: "trigger" } %>
<%= @article.content %>
```
Again, we setup the controller and the targets. We also have a button to start
the reading. The `text_to_speech_controller.js` file will look like this:
```javascript
import { Controller } from "@hotwired/stimulus"
import { pipeline } from '@xenova/transformers';
const speaker_embeddings = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/speaker_embeddings.bin';
export default class extends Controller {
static targets = ["text", "trigger"]
async initialize() {
this.triggerTarget.disabled = true;
this.synthesizer = await pipeline('text-to-speech');
this.triggerTarget.disabled = false;
}
async play() {
const previousLabel = this.triggerTarget.textContent;
this.triggerTarget.textContent = 'Preparing...';
this.triggerTarget.disabled = true;
const audioData = await this.synthesizer(this.textTarget.textContent, { speaker_embeddings });
this.triggerTarget.textContent = 'Reading...';
this.#playAudio(audioData);
this.triggerTarget.textContent = previousLabel;
this.triggerTarget.disabled = false;
}
#playAudio(audioData) {
const audioContext = new (window.AudioContext || window.webkitAudioContext)();
const audioBuffer = audioContext.createBuffer(1, audioData.audio.length, audioData.sampling_rate);
audioBuffer.copyToChannel(audioData.audio, 0, 0);
const source = audioContext.createBufferSource();
source.buffer = audioBuffer;
source.connect(audioContext.destination);
source.start();
}
}
```
Pretty similar to the previous one, but with some extra work to play the audio
-- in a production scenario you could save the audio as a wav file and play it --.
It will look like this: