Record & Replay: Motion Capture for A-Frame

No time for blog posts today? Skip the line, and get to the trendiest club of the Metaverse through the VIP access:

  1. Make sure you have a WebVR-enabled browser.
  2. Fix your hair.
  3. Hop right onto the dance floor.
  4. Share with the world your best moves - to the tune of the wonderful music made by JosSs!

VR devices, such as the Oculus Rift and the HTC Vive, have accurate systems to track the position and orientation of head and hands. On a-saturday-night, we wanted to create a fun experience around the idea of recording and reproducing tracking data. The mechanics are simple: Put your headset on, select an avatar, and dance. At the end of the countdown, and thanks to the magic of the Web, you will get a link like this one with the recorded dance that you can share instantly with anybody. The positions of the head and hands are sampled and persisted in a JSON format so they can be reproduced later.

Make your own motion-capture experience

With a-saturday-night we're also releasing a set of A-Frame components for anyone to record, replay and persist dancers' motion and interactions. The data can be saved as a JSON file and reused anywhere. There are plenty of interesting applications.

Interactive animation tools

You can sample the position and orientation of a VR controller, while the application is running, and apply the recorded data immediately to any entity in the scene. We can, for instance, build an interactive animation tool for game characters and become a virtual puppeteer.

Jim Henson

There will be two entities in the scene: one that records and the other replays.

  • One entity will generate the recording data. In the example above, it corresponds with one of the dancer's hand. We apply the motion-capture-recorder component to it and set recordingControls to true so that the recording/stopping will be driven by press-and-holding the trigger button on the controller.
<a-entity id="rightHand" motion-capture-recorder="hand: right; recordingControls: true"></a-entity>  

There are other properties that can be configured:

PropertyDescription
autoRecordSet the component to start recording upon page load.
recordingControlsControls for recording/stopping the motion capture.
visibleStrokeThe motion of the entity is visible while recording. Useful to provide feedback for interactive tools.
  • The other entity will replay the data recorded from the controller. We set up the motion-capture-replayer on it with the recorderEl property pointing to the entity that records the data:
<a-entity id="ghost" motion-capture-replayer="recorderEl: #rightHand; loop: true"></a-entity>  
Property Description
spectatorPosition The position of the spectator camera.
src The recording data can be hosted in a URL
recorderEl An entity with a motion-capture-recorder can be used as a source of the component. This allows for interactive cycles of recording/replaying.

Test automation and development

The motion-capture components allow to emulate the presence of a VR headset and controllers. We can build test automation for VR experiences. One can replay the recorded user behavior and assert the state of the entities at the end. This can happen without any user intervention at all.

We can also record user interactions and develop on the go when we're without our VR hardware and gear. One can iterate on the visual aspects and behaviors of the experience using the recorded user input.

Recording user interactions

To record the user interactions, just drop the avatar-recorder component in your scene:

<a-scene avatar-recorder></a-scene>  

The component will look for the camera and any entities using the tracked-controls component and apply the motion-capture-recorder to them. Remember to add an id to the controller entities, such that the recording information can be associated and the recording be replayed later.

The component can be configured in various ways:

PropertyDescription
autoRecord The component starts recording at page load.
autoPlay Replaying starts automatically when recording ends.
spectatorPlay The recording replays from a third-person perspective camera.
spectatorPosition The position of the spectator camera.
localStorage Whether to persist the recording to window.localStorage (default: true).
saveFile At the end of the recording, the dancer is prompted to download a JSON file with the data.
loop The recording replays in loops after finishing.

Replaying user interactions

To replay the user's recorded motion, add the component to the scene and pass the URL to the recording data:

 <a-scene avatar-replayer="src: assets/tracked-recording.json"></a-scene>

These are the different options that the component provides:

PropertyDescription
srcThe URL that points to the recorded data.
loopIf the recording will be replayed in a loop.
spectatorMode If the recording is replayed using a third-person camera.
spectatorPosition The position of the spectator camera.

Final words

I hope after reading this blog post you are as excited as we are with the realization that your VR device at home is also a super-accurate motion-capture system. We cannot wait to hear your feedback about the motion-capture API and see what you do with it.