02 — 2025 — present
Intelligent Robotic Pet
An ESP32-powered companion joining embedded hardware, cloud intelligence, and interactive behavior in a palm-sized system.
- Role
- Robotics Systems Engineer — solo build
- Stack
- ESP32-S3JavaWebSocketDeepSeekVolcengine TTSI2S / MAX98357AOLEDServo motors
Problem
Desk workers spend long hours in isolated focus. The question was whether a low-cost companion could hold natural voice interaction — on a microcontroller with a few hundred kilobytes of usable memory.
System
The ESP32 owns everything physical — microphone capture, audio playback, display, and motion — while language and speech run in the cloud behind a persistent WebSocket carried by a Java backend.
- Firmware — Wi-Fi, I2S capture and playback, OLED, servo control
- Transport — persistent WebSocket streaming both directions
- Backend — Java service orchestrating LLM and TTS calls
- Cloud — DeepSeek for dialogue, Volcengine for speech synthesis
My contribution
- 01Designed the embedded hardware architecture and the cloud-assisted split.
- 02Integrated the Java backend with ESP32 firmware over streaming WebSocket.
- 03Developed the speech interaction pipeline from capture to playback.
- 04Implemented embedded audio playback over I2S with a MAX98357A amplifier.
- 05Owned hardware prototyping and PCB planning.
Engineering decisions
AI on constrained hardware
Problem
The ESP32-S3 has limited compute and memory, so it cannot run wake-word detection and VAD while also hosting STT, LLM, and TTS locally.
Decision
Applied a hybrid edge-cloud architecture: the ESP32 handles wake-word detection, VAD, audio capture, OLED, and servo control; the cloud runs STT, LLM, and TTS; WebSocket carries bidirectional streaming audio and control messages.
Tradeoff
Robot behavior is strongly coupled to backend availability and network stability.
Outcome
Kept the device lightweight while preserving full voice interaction quality.
ESP32 hardware resource conflicts
Problem
The board has to drive the I2S microphone, OLED, Wi-Fi, and servos at the same time, but GPIO, LEDC, and real-time scheduling resources are limited. Servo control became unreliable under resource conflicts.
Decision
Integrated servo control into the current ESP-IDF firmware structure instead of running it as an independent control program.
Tradeoff
Tighter coupling inside the firmware, but shared timing, pin allocation, and task scheduling became easier to manage.
Outcome
Reduced resource contention and made servo behavior more predictable alongside audio and networking.
End-to-end voice latency
Problem
The first pipeline waited for complete speech synthesis before playback, making replies feel slow.
Decision
Moved to streaming audio so the ESP32 receives and plays PCM chunks concurrently.
Tradeoff
More complex buffer and underrun handling in firmware.
Outcome
Noticeably lower perceived latency and a far more natural conversation.
Demo
Results
System scale
System status
- Wi-Fi communication
- Java backend
- WebSocket streaming
- Streaming PCM playback
- ESP32 firmware integration
- Hardware prototype completed
Build status — not a measured benchmark.
Roadmap
- 01
Motion control and facial expression integration
- 02
PCB design and hardware miniaturization
- 03
Modular AI backend supporting multiple LLM / TTS providers
Lessons learned
- Hardware and software have to be designed together; most of the hard problems live at the interfaces between firmware, network, AI services, and physical interaction.
- Streaming beats batching whenever a human is waiting for the response.
Next project