Hey Laayo: voice ordering over WebRTC.

Hey Laayo is the voice assistant in the Laayo customer app. You tap the mic, say what you feel like eating, and it searches, suggests, opens the restaurant, adds dishes with their options and takes you to checkout. It speaks Nepali or English. Underneath it is OpenAI's Realtime API over WebRTC.

Who does what

The split that worked:

• The server mints a short-lived client secret for each session and holds the persona prompt and the tool definitions. Changing what the assistant says, or which tools it has, does not need an app release.
• The phone does the WebRTC offer and answer directly with the model, so the audio never passes through my servers.
• The tools run in the app. Search and menus call the same APIs as the screens. Adding to the cart goes through the same code as the Add button, so voice and touch can't disagree about prices or options.

Starting a session is rate limited per customer, and every session has a hard length cap and an idle timeout. A live audio session costs money for every second it is open.

The assistant never places an order on its own judgement. The place order tool needs two arguments: a flag saying the customer confirmed out loud, and the total the assistant read back to them. The app compares that total with the real payable amount and refuses if they differ. If the model misheard a quantity, the totals don't match and nothing is placed.

Every tool returns a short status string, like added, options_required, different_restaurant or store_closed. Those strings are a contract between the prompt and the tools file, so I change both in the same commit.

// What the app hands back to the model after adding a dish.
type AddResult = {
  status: "added" | "options_required" | "different_restaurant" | "store_closed";
  item: string;
  price: number;
  price_spoken: { en: string; ne: string }; // "four hundred fifty rupees" / "चार सय पचास रुपैयाँ"
  total_spoken?: { en: string; ne: string };
};

// What the app hands back to the model after adding a dish.
type AddResult = {
  status: "added" | "options_required" | "different_restaurant" | "store_closed";
  item: string;
  price: number;
  price_spoken: { en: string; ne: string }; // "four hundred fifty rupees" / "चार सय पचास रुपैयाँ"
  total_spoken?: { en: string; ne: string };
};

The price_spoken fields came later. In Nepali the model kept reading digits badly, sometimes mixing words and numerals inside one number. Now the app writes every amount out in words, in both languages, next to the number, and the prompt says to say those words and never convert digits itself.

Turn taking is most of the work

A voice assistant feels broken when it talks over you or waits too long.

The voice activity detector has an eagerness setting. High cut people off when they paused mid-sentence to think. Low felt slow. It sits at medium.

The greeting had its own bug. The mic was open while the assistant said hello, so it sometimes heard itself and answered its own greeting. The mic now stays muted from connect until the greeting has finished playing, and the input buffer is cleared before it opens.

Interrupting the assistant mid-sentence needed the phone's echo canceller. The app reads the microphone level after echo cancellation. If the customer's voice stays clearly louder than the leftover echo for a short moment, the assistant stops. If the phone has no voice processing, or the echo is too loud twice, it falls back to tap to stop.

Start-up speed came from prewarming. When your finger touches the mic, before you lift it, the app starts creating the session. If you don't complete the tap, the warm session is kept for twenty seconds in case you try again, then thrown away.

Holding a language

The most common complaint: the customer starts in Nepali, a tool result comes back in English, and the assistant switches to English.

My first fix was a language section in the middle of the prompt. It did nothing. What worked was moving the rule to the very top, right after the first line, as the most important rule: if the customer has spoken Nepali in this call, answer in Nepali, and treat tool results as data, not as a language cue. I added four short example exchanges. I also had to write the checkout question in Nepali, because the model was copying "Shall I place it?" word for word from the English instructions. Nepali typed in English letters counts as Nepali.

Two more rules fixed the most visible failures:

• Pace. At most two short sentences. One suggestion at a time. One question first when the request is vague.
• Know before you offer. When the customer names a dish, search first, and only offer kinds that are in the results. Before that rule, "momo" got the reply "veg, chicken or mutton?" from a menu with no mutton.

Each tool call also shows a small action card in the voice panel, with a tick or a cross and the real reason. Before that, the assistant could switch the payment method and nothing on screen showed it. The cards also make screenshots of a bad session much easier to read.

Testing a voice prompt without talking to it

Prompt changes kept breaking things I wasn't looking at, so I built a bench. It runs the real model in text mode against a set of scenarios, with real tool calls against a test environment, and scores each one: right language, one option at a time, correct price words, short lines while a tool runs. It runs before every prompt change, and I compare scores before and after.

Text mode is not audio, but it catches prompt regressions without anyone talking to a phone.

One lesson from the bench: if it uses the same API account as production, it shares the realtime rate limit with real customers. Replays in a loop can push the live assistant into rate limiting. Benches now use a separate key and run spaced out.