Voice calling architecture and runbook

Arinova has two deliberately separate voice paths:

• Human-to-human (H2H) calls use the Rust /ws/voice signaling service and browser/native WebRTC media. Media is peer-to-peer when possible and relayed through coturn when NAT or firewall policy requires it.
• Human-to-agent (H2A) calls use the existing billed ElevenLabs Speech Engine skill. The server creates the session through LlmGateway; the retired H2A peer-to-peer signaling path is gated and must never report a synthetic voice_call_start.

Video, group calls, and screen sharing are not part of this architecture.

H2H lifecycle

1. The caller sends voice_call_request; the server stores a pending row and returns voice_ringing with the canonical 60-second deadline.
2. Every callee connection can receive the ring. Exactly one device wins the atomic voice_accept; the others receive voice_ring_cancelled.
3. The caller sends its offer only after voice_accepted. Both clients become logically connected only after the server emits voice_call_start.
4. ICE candidates and restart offers are participant-authorized, rate limited, and routed to the exact registered connection. A signaling reconnect sends voice_rejoin; socket loss has a 10-second grace period.
5. A terminal transition is idempotent. Both Redis participant leases are removed and a call log is written to the caller/callee DM, even when the original dial action came from a group conversation.

The web client adds a 15-second WebRTC connection watchdog, a 10-second disconnect recovery window, ICE restart, microphone/speaker selection, network quality reporting, wake lock, and a cross-tab renewable lease so only one tab rings or owns the microphone. Connected rows older than four hours are reaped; busy checks ignore all rows older than six hours. Startup also recovers in-flight rows owned by the restarting instance.

HTTP and push surfaces

• GET /api/voice/ice-servers returns Cloudflare and Google STUN plus coturn UDP, TCP, and TLS URLs with 600-second HMAC credentials.
• POST /api/voice/ice-path accepts the client’s selected direct/relay path.
• GET /api/voice/calls/pending restores a current incoming ring.
• GET /api/voice/calls/{sessionId} is participant-scoped and powers call deep links and expired-call feedback.
• POST /api/voice/calls/{sessionId}/end is the participant-scoped fallback when a disconnected signaling socket cannot deliver hangup.
• POST /api/voice/calls/{sessionId}/reject shares the canonical terminal path with the WebSocket voice_reject event.
• GET /api/voice/history/{conversationId} supports cursor/offset pagination and callee, status, end-reason, and direction filters.
• POST /api/push/register-voip-device and its DELETE counterpart store iOS PushKit tokens separately from ordinary APNs tokens. An iOS calling client must register both its ordinary APNs token and its PushKit token: incoming calls use PushKit, while cancellation and every non-call notification require the ordinary token. A VoIP-only registration is an invalid client state and increments arinova_voice_voip_without_standard_token_total. Only an incoming voice_call uses APNs voip. If that user has no PushKit token, the server sends one ordinary APNs alert with the default call sound (base bundle topic, priority 10) instead; it never sends both channels for the same incoming call. voice_call_cancel uses the ordinary background channel because Apple requires every VoIP push to report a new CallKit call. Android and Web push behavior is unchanged.

coturn deployment

Each staging/production host runs the digest-pinned coturn service from docker-compose.production.yml. Configure:

• a distinct, single-line TURN_SECRET of at least 32 characters;
• TURN_HOST, COTURN_EXTERNAL_IP, and immutable COTURN_IMAGE_REF;
• DNS and a valid TLS certificate for the TURN hostname;
• TCP/UDP 3478, TCP 5349, and UDP 49152–65535 in the host/provider firewall.

The deploy pipeline calls scripts/deploy/render-coturn-config.sh, which writes the generated config with mode 0600. Production configuration fails closed if TURN is absent. Coturn accepts TLS 1.2+ only and enforces per-user, total allocation, and aggregate bandwidth limits. Relay cutover first runs the exact candidate image and TLS configuration on isolated ports while the existing container continues serving. Only a healthy candidate reaches the real-port cutover; the old container remains a rollback target until the replacement passes TCP/UDP 3478 and verified TLS 5349 health. Failure restores that exact container and emits a prominent warning without blocking an app image change. Central blackbox probes cover 3478 and 5349 in both environments. See infra/coturn/README.md for environment-specific names and exact limits.

For Speech Engine agent calls, the browser-side terminal operation is the ElevenLabs SDK endSession() handle owned by the call store. Minimizing or temporarily unmounting the call view does not clear that handle; the store clears it only with the call lifecycle. If the SDK termination cannot run, the server-issued connect/max-duration deadline remains the final cleanup bound.

Metrics and triage

Use these metrics together:

• arinova_voice_ring_answered_total: rings that crossed the atomic answer boundary. This is intentionally separate from terminal outcomes.
• arinova_voice_ring_outcome_total{outcome=...}: rejected, cancelled, missed, connect_failed, routing_unavailable, stale, server_restart, hangup, disconnect, and the separate H2A agent_disconnect terminal outcomes. Do not add answered to this terminal partition or sum the answered counter into terminal totals.
• arinova_voice_active_calls: process-local connected-call gauge.
• arinova_voice_call_duration_seconds: completed-call duration histogram.
• arinova_voice_routing_unavailable_total: signaling could not find the intended connection across the fleet.
• arinova_voice_turn_config_total{status="configured|missing"}: ICE config requests observed with or without TURN.
• arinova_voice_turn_connection_total{path="direct|relay"}: client-selected ICE path.
• arinova_voice_voip_without_standard_token_total: invalid iOS dual-token registrations observed when attempting an incoming call.

For a sudden connection-failure increase:

1. Confirm turn_config_total{status="missing"} is not increasing.
2. Resolve the current TURN DNS and verify the TLS certificate and 5349/TCP.
3. Check the coturn container and host firewall, including the relay UDP range.
4. Exercise GET /api/voice/ice-servers as an authenticated user and confirm all three TURN URLs and a future-dated username are returned.
5. Compare direct versus relay counts. No relay traffic under a symmetric-NAT test usually means coturn reachability/authentication is broken; a sharp relay-only shift suggests STUN or direct-network regression.
6. If routing-unavailable rises, inspect instance/Redis health and exact connection registration before investigating media.

For stuck busy users, inspect recent non-ended H2H rows, the stale sweeper, and startup recovery logs. Do not manually mark rows ended until the owner-instance and six-hour busy-window behavior have been checked.

Release validation

• Run the call-ring PR smoke journey: dial UI → incoming banner → answer → ActiveCall.
• Test an older browser tab/device accepting a multi-device ring.
• Test a 10-second network interruption and ICE restart.
• Test hotspot-to-home-network media and confirm a path="relay" sample.
• Verify group-originated 1:1 call logs appear only in the participants’ DM.
• Build the signed desktop target to validate microphone entitlement/plist.
• On iOS, validate killed-app PushKit delivery, immediate CallKit reporting, answer/reject signaling, and cancellation dismissal.