Deployed robotics | Online monitoring and interaction
Panbotica Robot Safety and Interaction Systems
I built an online workspace monitor with interrupt-driven motion protection and an edge-to-cloud voice interaction system for customer-facing robotic baristas.
Deployed systems
Summary
150
Robotic barista units
80%
Reduction in collision-related support calls
5 to 1
Support calls per week
Edge-to-cloud
Voice interaction system
The online workspace monitor and voice interaction system ran in customer-facing robotic barista workflows, where people and objects could enter the open workspace during service.
Safety problem
The open workspace
Panbotica's robotic barista fleet of 150 units operates around customers rather than behind fenced industrial cells. Cups, hands, bags, and other foreign objects can enter the arm workspace while a drink is being prepared. The monitor samples the live camera stream between arm-motion steps and sends an interrupt to the robot host controller when the workspace needs attention.
Responsibilities
My role
I worked on workspace validation and customer interaction systems. I trained and integrated the workspace detector, connected an external VLM to the validation pipeline, and mapped its decisions into robot-host-controller actions alongside the edge-to-cloud voice interaction system.
Runtime protection
Online workspace monitoring
YOLO and VLM validation around motion steps
The pipeline samples frames from the robot's live camera stream and sends them to a central perception service. A custom-trained YOLO model handles fast, structured recognition of expected workspace items and known obstruction classes. An external VLM API compares the current frame with a reference normal-state image to identify contextual anomalies outside YOLO's fixed class set.
- Custom-trained YOLO: checks expected object placement and returns known object classes, locations, and confidence values for the workspace.
- External VLM API: compares the current frame and reference image to flag unusual or misplaced objects that do not fit the fixed detector classes.
- Decision policy: combines both outputs and sends constrained action codes to the robot host controller.
- HOLD_AND_PROMPT: pauses the workflow, plays a spoken removal instruction, runs workspace revalidation, and retries the interrupted motion sequence.
- STOP_AND_BLOCK: interrupts execution and requires operator inspection and reset for a critical hazard.
No interrupt means normal execution continues. This is an asynchronous interrupt path into the host controller, rather than continuous collision avoidance.
Measured result
Deployment result
Collision-related support calls
Panbotica reported that collision-related support calls fell 80%, from five per week to one per week after deployment.
Deployment scope
The system ran in customer-facing operation across Panbotica's 150-unit fleet.
Voice interaction
Voice ordering and payment
Local audio, central reasoning, local response
I worked on a deployed edge-to-cloud system that converts spoken requests into defined robot actions and a payment handoff.
- Robot edge: the built-in microphone unit performs local automatic speech recognition (ASR), while the robot handles text-to-speech (TTS), speaker output, and display/control.
- Central server: the transcript packet is sent to the company's LLM service, which uses function calling to return defined actions instead of free-form text for the robot to interpret.
- Available functions cover voice ordering, FAQ responses, and POS payment QR-code display.
- Each conversational response remains tied to a defined robot or transaction action.
System diagrams
Safety and interaction flows
The system separates two online paths: camera-frame validation can interrupt robot execution, while local ASR sends transcript packets to the company's central LLM and returns defined voice, order, and payment actions to the robot.
