Arisa

An interactive 3D AI agent rendered in the browser, inspired by Grok's Ani. Users can speak or type to Arisa, and she responds with voice, synchronized lip movements, body animations, and emotion-driven facial expressions — all in real time.

How It Works

The user records audio or types a message. Audio is transcribed via ElevenLabs STT, then sent to xAI Grok which returns a text response along with an animation state and facial expression weights. The response is converted back to speech with ElevenLabs TTS, and lip-sync viseme data is extracted using Rhubarb. On the frontend, a 3D avatar rendered with Three.js and React Three Fiber plays the matching body animation, blends facial morph targets for the expression, and synchronizes mouth shapes to the audio.

Key Features

• Voice and text input — speak or type, Arisa understands both
• Real-time lip sync — mouth shapes driven by viseme data from Rhubarb
• LLM-controlled animations — Grok decides the body animation and facial expression for each response
• Emotion-aware responses — happy, sad, angry, relaxed, surprised — reflected in both voice tone and 3D expression
• Morph target blending — smooth transitions between facial expressions using blend shapes

Tech Stack

Layer	Technology
Frontend	React, React Three Fiber, Three.js, Tailwind CSS
Backend	Node.js, Express, TypeScript
LLM	xAI Grok
Voice	ElevenLabs (STT + TTS)
Lip Sync	Rhubarb
3D	FBX animations, morph target blending