AI Pet-Sitter Robot Service
Context-aware risk reasoning and proactive robotic intervention for pet safety.
Overview
Developed for the AI Rookie Competition, this project connects YOLOE-based perception and EXAONE risk reasoning with a mobile robot, a robotic arm, and voice warnings. Representative scenarios involve pets approaching grapes, chocolate, and garlic.
The system considers the pet's proximity and approach behavior to determine whether to continue monitoring, issue a warning, block access, or remove the object. A mobile app provides owners with risk alerts and information about the robot's response.
The project advanced to the competition finals.
Problem
When pets are left home alone, ordinary exploratory behavior can bring them into contact with hazardous objects or foods. Monitoring cameras allow owners to observe these situations, but timely intervention still depends on an owner noticing the risk and responding remotely.
Object detection alone is also insufficient: a pet and a hazardous item appearing in the same frame does not necessarily require intervention.
The system therefore considers spatial relationships and approach behavior to distinguish ordinary exploration from situations that call for a response.
My role
- Implemented LG EXAONE4.5 inference using structured JSON observations from the visual perception pipeline to assess risks and determine appropriate responses.
- Connected structured model outputs to robot action commands and voice-warning requests.
- Integrated NC VARCO custom TTS for owner-voice warnings, using predefined warning messages and caching generated audio for reuse.
- Applied asynchronous speech processing so that TTS latency or failure would not delay urgent robot intervention.
Approach
- YOLOE recognition results are combined with RGB-D distance information and formatted as JSON observations, including object categories, detection confidence, relative positions, and approach status.
- EXAONE uses these observations to produce structured risk assessments and response recommendations.
- The project describes the gap between a pet's exploratory behavior and its potentially harmful consequences as the “Intent Gap.”
- Response policies consider distance, direction, and persistence of approach to select monitoring, warnings, access blocking, or object retrieval.
- Spoken warnings are selected from predefined messages and generated through NC VARCO custom TTS.
System architecture
- Perception: YOLOE and an RGB-D camera provide object recognition and distance information.
- Risk reasoning: EXAONE receives structured observations and returns risk assessments with supporting explanations.
- Action policy: A policy layer maps the results to monitoring, warning, access-blocking, or object-retrieval commands.
- Physical response: A TurtleBot performs mobile intervention, while an OpenMANIPULATOR-X arm handles object retrieval.
- Speech and owner interface: NC VARCO provides owner-voice warnings, and a mobile app displays risk alerts and response information.
Technologies
Results
- Advanced to the main round of the AI Rookie Competition.