Voice & chat

One conversation

• The orb talks. Aa opens the chat panel. Typed messages go to the same live session as speech, and the reply streams back as chat bubbles.
• Opening chat pauses the microphone but keeps the session. Nothing is lost, and there’s no second model or second endpoint.
• The panel header has two controls. – (Minimize chat) hides the panel, pauses voice, and keeps the conversation. End (End conversation) finishes it, the same as × on the orb.
• Voice off means silent. Whatever pauses voice (the orb, the panel’s mic button, minimize, the idle pause, or End) stops the microphone and the assistant’s audio at the same moment. Replies that arrive while voice is paused show as text only.
• The panel’s mic button (“Start voice” / “Pause voice”) switches between typing and talking in one tap, in the same conversation: no second greeting, and the history stays.
• A fresh session greets the user at most once, and never once the user has typed: a message sent while the session is still connecting gets a reply, not a greeting first.
• If a session ends (idle timeout, maximum length, or a lost connection) and the user keeps going, the new session starts with a recap of the recent chat instead of a greeting, so it picks up where it left off. End, sign-out, and switching organization clear everything, and the next conversation starts fresh.
• The session survives route changes and React remounts. When the user moves to a new page, Usher tells the model where they are now, so it won’t offer the page they’re already on.
• Voice and chat turn on with voqalKey alone: the browser microphone and speaker are wired for you. Without a key, the panel runs the local text-only mode, and tapping the orb opens and closes the panel.

Hands-free or push-to-talk Optional

Recommended in noisy or public settings, so bystanders and speakers can't trigger the assistant.

• "auto" (default): the server detects speech and starts a turn when the user talks. Talking over the assistant interrupts it. It’s the simplest mode, but in a noisy room, or through laptop speakers without headphones, the assistant can pick up other voices.
• "push-to-talk": the user presses and holds the orb to talk and lets go to send. Only held speech reaches the model, so nothing else can interrupt or trigger it. The orb reads “Hold to talk” at rest. Releasing the pointer, or dragging it off the orb, ends the turn.

What every control does

The panel’s status line

A message typed while the socket is down isn’t sent. It goes to voice.onError as LiveSessionNotConnectedError so you can tell the user to try again.

Orb states

How Usher decides

In a live session, the chat panel shows the assistant’s own words, streamed as it speaks. When it offers a page, a suggestion card appears under those words. Tapping it navigates through the same validation chain, and the panel then adds the fixed line “Took you to {page}.” The other fixed lines in the last column appear only in local text mode.

The suggestion card

• A tinted card with a page icon, the destination’s title, one line from its semanticDescription, and a chevron. The whole card is one button, named “Go to {title}” for screen readers. A muted “No thanks” sits below it.
• The description shows only when it adds something: a blank one, or one that just repeats the title, leaves the card with the title alone. Well-written semanticDescriptions in your map now show up in the UI, so write them for users as well as for the model. See Destinations.
• Once the user answers, the card collapses to one muted line: “Opened {title}” when the page changes to it (a tap, or a spoken yes), “Stayed on this page” after No thanks, and “Suggested {title}” when the user moves on by typing or speaking something else. A stale card never stays clickable.
• A refusal’s fallback (the parent page, when a record couldn’t be found) uses the same card without “No thanks”.
• No card is ever offered for the page the user is on at the time.

In a live session

The model decides between navigate, offer, clarify, declining a request it has no page for, and a plain reply. For a question your knowledge base answers, it gives the fact first and may offer the page after, never instead. It is told to navigate only on a clear request, to offer on “how do I” or “where is” questions, to ask when unsure, never to move the user to the page they’re on, and to always say something out loud. Whatever it picks, the validation chain runs before your router does.

• Greetings, emoji, and acknowledgements (“hi”, “👍”, “ok”) never navigate. If the model tries, the move isn’t run and onAction gets refused with not_requested.
• A clear yes (“ok”, “yeah”, “sure”) to an offer, or to a question that named a page, accepts it and navigates.
• It never navigates to, or offers, the page the user is on. If the model tries, nothing moves, no card appears, the assistant is told the user is already there and helps them on that page, and onAction gets refused with already_here. On a dynamic page, only the same record counts: from one case, another case is still a real move. Tapping an older card for the page the user is now on moves nothing either, and the panel says “You're already here.”
• Asked to do something (“book it for me”), it uses one of your tools if one fits. Otherwise it says plainly that it can’t, then explains how to do it: right there if it’s on the current page, otherwise on the page where it’s done.
• Asked to open something your app opens with a navigation tool, it uses that tool alone, not its built-in navigation as well. If the user says “stop” while that tool is still running, Usher aborts it and the assistant is told nothing was done.
• Where spoken announcements are on for your account (available on request), the assistant says a short line before a tool that opens something runs, or before it moves the user to another page: “Opening Birch House.” Typed messages are never announced. See A line before it runs.
• A vague request (“take me somewhere”) gets a question back: where to, with a few pages suggested.
• The assistant never calls itself Usher or Voqal. It uses your title, or “the assistant” without one.

Every message gets an answer

• Before anything is on screen, the panel’s placeholder says “Connecting…”. Once the user sends a message, “Thinking…” stays until the reply starts.
• If the model ends a turn without saying anything, Usher nudges it. If it still says nothing, or no reply arrives within 20 seconds of a typed message, the panel shows “Sorry — I didn’t catch that. Could you say a bit more?” The user is never left with nothing.
• If the key’s rate limit refuses a new session, the panel opens with “Too many requests right now — please try again in a moment.” and the status line reads “Too many requests — try again in a moment”.

In local text mode

A deterministic policy weighs each suggestion using the phrasing and the match score:

• Answer when the user is already there, asks what the assistant can do, or is just saying hello or thanks.
• Clarify when the top two pages both score 0.3 or more and are within 0.12 of each other.
• Answer when the best score is under 0.3.
• Navigate on act-now phrasing (“take me to”, “open”, “I need to change…”) with a score of 0.5 or more.
• Otherwise offer. The policy can make a decision gentler, never bolder.

Voice options

The voice prop is optional. Pass one to set the interaction mode, the greeting, or error handling. It needs a voqalKey: a voice prop without a key throws.

Microphone and audio

• The browser asks for microphone permission on the first tap. The page must be served over HTTPS (localhost is fine for development).
• Audio is captured at 16 kHz mono in 100 ms blocks, with echo cancellation, noise suppression, and auto gain on. Replies play at 24 kHz.
• Capture uses an AudioWorklet loaded from a blob: URL. A strict Content Security Policy must allow it; see Security.
• The session language defaults to en-US. The model may answer in another language when the user speaks one, but the beta doesn’t configure or guarantee that.
• isLiveAudioSupported() tells you whether the browser can capture audio at all. It returns false during server rendering.

Idle timeout & session limits

So an open microphone never listens forever, a session winds down on its own. The defaults suit most apps.

• Activity means recognised speech from the user, a typed message, a tap to resume, or the assistant speaking or calling a tool. The microphone level and background noise never count.
• A timer never cuts the assistant off mid-turn.
• After an idle or maximum-length end, the transcript stays on screen and the status reads “Paused — send a message to continue”. The next tap or message starts a new session that picks up from a recap of the recent chat, without a second greeting.
• In noisy or public places, where nearby conversation keeps the microphone open, use push-to-talk.

Change the limits

<Usher  voqalKey={import.meta.env.VITE_VOQAL_KEY}  router={adapter}  destinations={destinations}  idleTimeout={{ maxSessionMs: 20 * 60_000 }} // allow 20-minute sessions; other defaults stay/>

idleTimeout?:  | {      pauseMicAfterMs?: number | false; // default 30_000 (DEFAULT_PAUSE_MIC_AFTER_MS)      endSessionAfterMs?: number | false; // default 300_000 (DEFAULT_END_SESSION_AFTER_MS)      maxSessionMs?: number | false; // default 600_000 (DEFAULT_MAX_SESSION_MS)    }  | false; // false turns every limit off; 0 or false turns one off

0 or false turns one limit off, and idleTimeout={false} turns them all off. The defaults are exported as DEFAULT_PAUSE_MIC_AFTER_MS, DEFAULT_END_SESSION_AFTER_MS, and DEFAULT_MAX_SESSION_MS. The timers report idle:pause-mic, idle:end-session, max-session:closing-line, and max-session:end through voice.onDiagnostic.

Reconnects and the maximum length

They do different jobs. Reconnecting handles unplanned drops inside a session, including the voice provider’s periodic connection limit; the new connection gets a recap of the recent chat, and messages sent meanwhile are held and re-sent. The maximum length is a deliberate limit that ends the session. With the default, a session normally ends at the maximum before a provider limit comes into play. If you raise the maximum or turn it off, reconnecting keeps long sessions going.