Skip to content
Big Cities Health Coalition logo Big Cities Health Coalition Public Health AI & Tech Exchange

Pilot · Austin Public Health

Open-source code

Voice intake assistant for service navigation

Residents can talk instead of type: a realtime voice assistant asks the intake questions, matches services from the catalog, and builds the same service plan as the text chat.

Published October 15, 2026

Source code (opens in a new tab) Documentation or write-up (opens in a new tab)
Skills needed to set it up
Developer
Readiness
Needs customization
Data it touches
Personal information (PII)
Who sees the output
Public-facing
Cost to stand up
No new spend
License
Apache 2.0
Portable to other platforms
Partially — with rework
Reviews it went through
Not yet reviewed
Public health area
Access to and linkage with care Communications and community engagement
What do these mean?
Developer
A software developer to deploy or adapt it.
Needs customization
Your team will need to adapt it before use.
Personal information (PII)
Names, addresses, IDs or other personal identifiers.
Public-facing
Residents or the general public interact with it.
No new spend
Built with licences, staff and infrastructure the organization already had.
Apache 2.0
Permissive, with a patent grant.
Partially — with rework
Some pieces are vendor-specific and would need swapping.
Not yet reviewed
Shared honestly — this has not been through a formal review.
Access to and linkage with care
Clinics, community health workers, referrals and connecting residents to services.
Communications and community engagement
Public information, risk communication, language access and community partnerships.

Problem

A text chat still asks residents to read and type. For people with low literacy, low vision or limited dexterity, or for someone on a phone in a stressful moment, talking is easier. We wanted to see whether the same intake could run as a natural spoken conversation without building a separate system.

Approach

Voice is a second front door to the Austin Service Guide. The resident chooses voice or text. The voice option says up front that audio is sent to OpenAI.

  • The browser gets a short-lived token from our backend, so the real API key never leaves the server. It then opens a WebRTC connection directly to OpenAI’s Realtime API.
  • The voice model calls the same tools as the text chat: record profile facts, search the catalog, run the rules-based matching engine, finish the intake. Eligibility logic, crisis handling and plans are therefore shared, not duplicated.
  • Each exchange is transcribed live on screen, so the resident can see what was heard. Transcripts sync back to the backend and become part of the same session, so a resident can switch between voice and text.
  • When the intake finishes, the resident gets the same ranked plan as the text chat.

About the screenshots: they show a simulated session. A scripted stand-in for the voice connection played a fictional conversation through the real interface and the real backend tools, so the matches and plan shown are what the app produces. No live audio was recorded.

How resident information is kept from the model

Voice has weaker protections than text, and we say so:

  • Audio streams straight from the browser to OpenAI, so it cannot be redacted before OpenAI hears it. If a resident reads out a Social Security number, the voice and transcription models receive it.
  • The voice instructions tell the model never to ask for or repeat names, Social Security numbers, addresses, phone numbers, dates of birth or ID numbers, and to steer the conversation back if they come up.
  • Resident transcripts are redacted with the same pattern-based filter as typed text when they sync to the backend. The stored conversation, and anything later sent to the text model, never holds the raw identifiers. The browser also replaces its on-screen copy with the redacted version.
  • The assistant’s own transcripts are redacted for SSNs, cards, dates of birth, ID numbers and emails. Phone numbers and addresses are left in because the assistant reads out service contact details.
  • Realtime tracing is turned off. The Realtime API has no store=false option, so a real deployment needs a zero-data-retention agreement that covers it.
  • Sessions are deleted from server memory after 24 hours.

The full account, including limits, is in docs/PRIVACY.md.

Time and resources

Built by Austin Public Health IT staff with an AI pair programmer over three days in May 2026, on top of the existing text guide. Because voice reuses the guide’s tools and matching, the work was mostly the browser connection, turn-taking and transcript handling. The only running cost is OpenAI Realtime usage, which is billed by the amount of audio. Voice is off unless an API key is configured.

Results

Demonstrated alongside the text guide to the Austin Public Health Commission. It is a pilot demonstration and has not been used with real residents.

Lessons learned

  • Share the tools, not just the prompt. Pointing the voice model at the same backend functions as the text chat kept eligibility and crisis logic in one tested place.
  • Voice changes the privacy model. With text you can filter before the model sees anything; with direct-to-vendor audio you can’t. Tell residents before the microphone turns on, and get retention terms that cover realtime audio.
  • Show the transcript. A visible live transcript lets residents catch recognition errors and builds trust in what was heard.
  • Turn-taking is most of the work. Greeting timing, microphone locking while the assistant speaks, and finishing cleanly after the last answer took more effort than the AI integration itself.

How to reuse this

Start from the Austin Service Guide entry. Voice comes with it. Set OPENAI_API_KEY and DEMO_MODE=false, and choose a voice and model with the OPENAI_REALTIME_* settings. The browser side is a single hook (useRealtimeVoiceSession.ts). Before real use, review the voice section of docs/PRIVACY.md with your privacy office, especially the audio retention terms.

About

Type of solution
  • Generative AI tool
  • Application or software
AI use category
Administrative & task automation
Review status
Under review

How it's built

Role of AI
Both
Types of AI
  • Speech & transcription
  • Chat assistant
  • Agents & automation
Tools and technologies
  • OpenAI Realtime API (gpt-realtime-2) over WebRTC
  • OpenAI transcription (gpt-4o-mini-transcribe)
  • Python / FastAPI
  • React / TypeScript
  • Claude Code (AI pair programmer)
  • OpenAI Codex (AI pair programmer)
Where it runs
  • Desktop or local

Sharing & licensing

Portability notes
Voice depends on OpenAI's Realtime API: the browser opens a WebRTC connection straight to OpenAI using a short-lived token from the backend. Moving to another realtime speech provider means rewriting the browser hook (frontend/src/hooks/useRealtimeVoiceSession.ts) and the token endpoint. The tools the voice model calls are the same backend functions the text chat uses, so matching, plans and the catalog carry over unchanged.
Adapted from

What it took

Cost to keep running
Under $10k/yr
How it was bought
  • No procurement needed
Who it affects
Voice helps residents who find typing hard: low literacy, low vision, limited dexterity, or a phone as their only device. The realtime model can speak several languages, but its translations have not been reviewed by people. It needs a microphone, a modern browser and a steady connection, so it is an option alongside text chat, not a replacement. Speech recognition can be less accurate for some accents and noisy settings; the resident sees the transcript as it is written, so mistakes are visible.

Data & access

No PII/PHI in the shared material
Yes
Data sources
  • Austin-area service catalog (shared with the text guide)
  • Resident speech streamed to OpenAI (transcripts held in server memory for 24 hours)
Data-governance caveats
Audio goes directly from the browser to OpenAI, so it cannot be filtered before OpenAI hears it. Transcripts are redacted (SSNs, cards, emails, phones, addresses, dates of birth, ID numbers) when they sync back to the backend, so the stored conversation and anything later sent to the text model are clean. Realtime tracing is turned off. The Realtime API has no store=false option, so retention follows the account's OpenAI data policy; a real deployment needs zero-data-retention approval that covers Realtime. See docs/PRIVACY.md.

Printed from Public Health AI & Tech Exchange
https://big-cities-health-coalition.github.io/use-case-catalog/catalog/voice-intake-assistant-for-service-navigation/