Preparing the published article.
How to Read a Humanoid Demo Video Without Being Misled
A frame-by-frame method for identifying cuts, spotters and missing evidence, and turning a clip into test requests.
A demonstration video is evidence that a behaviour happened at least once, under conditions you cannot see. Treat it as a prompt for questions, not as a result, and extract from it the specific things it can and cannot establish.
1. Read the environment before you watch the robot
Start by identifying what has been added to make the shot work. Look for painted lane markings, an empty aisle, high-contrast matting, taped hazard zones, overhead gantries or tethers, cable channels, and an unusual absence of pallets, cables and people. Note the floor: a legged platform carrying a load across a clean, level, dry surface is doing something different from one crossing a wet loading dock or a stairwell with a worn nosing. Check for lighting control, because glare, hard shadows and reflective floors are where perception fails. Count the people in frame and ask what each one is doing: a person walking behind with a hand near the robot is a spotter, and a spotter means the run was supervised. If a safety operator holds a pendant or an e-stop within reach, the autonomy demonstrated is bounded. These are legitimate setups, but they change what the video proves, and the video should be read with that boundary drawn in.
List what you can verify from the frame: floor type, obstacles present, people present, tethers, visible safety equipment, load carried, and where the operator stands. Anything on that list that the vendor describes differently in the accompanying text is a question to put in writing.
2. Identify cuts, takes and intervention points
Watch once for continuity, then watch again at half speed around the moments of risk: the approach to an object, the grasp, the stair descent and the placement. Cuts usually land right where a failure would occur. Look for changes in shadow direction, repositioned objects, a swapped battery pack, a moved chair, or a jump in the timestamp. Count the number of attempts shown per success; a clip that shows three failures and one success is more informative than one that shows only the success, because it tells you the realistic attempt count. Ask directly for an unedited single-take run of the same task, including the moments the robot is corrected. Ask how many operators appear in the full session and how many times the robot was touched. If the vendor cannot supply an unedited run, note that plainly in your evaluation file and weight the video accordingly. Then ask for frame rate and playback speed, because a legged gait looks smoother at a lower frame rate and a fast playback makes a slow behaviour seem brisk.
Keep a short table per clip: task shown, duration of the unedited sequence if supplied, number of attempts, visible operator interventions, cuts at points of risk, and what remains unverifiable. This table is the artefact you carry into a follow-up call; it turns a marketing email into a list of specific requests.
3. Convert what you saw into the tests you will run
Write three columns: demonstrated, inferred, unknown. Demonstrated means you saw it happen under stated conditions. Inferred means it is plausible but unshown, such as the same task with a different object or the same route at shift change. Unknown covers everything about reliability, endurance and cost. Then convert the unknowns into test requests for your own site: the same task with your objects, your floor and your lighting, run for a stated number of repetitions, with intervention and recovery time logged. Ask for the failure cases explicitly, because a vendor who can describe the three most common failure modes on your type of task is telling you something useful. Finally, request the measurement conditions alongside any figure quoted verbally, then plan to reproduce it on your floor with your own stopwatch and your own log sheet. The video's job is to generate that test list; it cannot substitute for it.
Photo credits: the images in this article are licensed reference photographs of the relevant robot class, credited with author and licence in each caption; they are not photographs of the specific equipment discussed. Licence and author details are recorded with each image.