WebVR coming to Servo: Architecture and latency optimizations

We are happy to announce that the first WebVR patches are landing in Servo.

For the impatients: You can download a highly experimental Servo binary compatible with HTC Vive. Switch on your headset and run servo.exe --resources-path resources webvr\room-scale.html

The current implementation supports the WebVR 1.2 spec that enables the API in contexts other than the main page thread, such as WebWorkers.

We’ve been working hard on an optimized render path for VR to achieve smooth FPS and the required less than 20ms of latency to avoid motion sickness. This is the overall architecture:

Rust-WebVR Library

The Rust WebVR implementation is a dependency-free library providing both the WebVR spec implementation and the integration with the vendor specific SDKs (OpenVR, Oculus …). Having it decoupled on its own component comes with multiple advantages:

• Fast develop-compile-test cycle. Compilation times are way faster than developing and testing in a full browser.
• Contributions are easier because developers don’t have to deal with the complexity of a browser code base.
• It can be used on any third party project: Room scale demo using vanilla Rust.

The API is inspired on the easy to use WebVR API but adapted to Rust design patterns. The VRService trait offers an entry point to access native SDKs like OpenVR and Oculus SDK. It allows to perform operations such as initialization, shutdown, event polling and VR Device discovery:

The VRDevice trait provides a way to interact with Virtual Reality headsets:

The integration with vendor specific SDKs (OpenVR, Oculus…) are built on top of the VRService and VRDevice traits. OpenVRService, for instance interfaces with Valve’s OpenVR API. While the code is written in Rust, native SDKs are usually implemented in C or C++. We use Rust FFI and rust-bindgen to generate Rust bindings from the native C/C++ header files.

MockService implements a mock VR device that can be used for testing or developing without having a physical headset available. You will be able to get some code done in the train or while attending a boring talk or meeting ;)

VRServiceManager is the main entry point to the rust-webvr library. It handles the life cycle and interfaces to all available VRService implementations. You can use cargo-features to register the default implementations or manually register your own ones. Here is an example of initialization in a vanilla Rust app:

WebVR integration in Servo

Performance and security are both top priorities in a Web browser. DOM Objects are allowed to use VRDevices but they neither own them or have any direct pointers to native objects. There are many reasons for this:

• JavaScript execution is untrusted and the input data might be malicious . That’s why the entire JavaScript thread must be isolated to its own sandboxed process.
• There could be many parallel JavaScript contexts requesting access to the same native VRDevice instance which could lead to data race conditions.
• WebVR Spec enforces privacy and security guidelines. For example a secondary tab is not allowed to read VRDisplay data or stop a VR presentation while the user is having a VR experience in the current tab.

The WebVRThread is a trusted component that fulfills all the performance and security requirements. It owns native VRDevices, handles their life cycle inside Servo and acts a doorman for untrusted VR requests from DOM Objects. Thanks to using Rustlang the implementation is guaranteed to be safe because ownership and thread safety rules are checked at compile-time. As other Servo components traits are splitted into a separate subcomponent to avoid cyclic dependencies in Servo.

In a nutshell, the WebVRThread waits for VR Commands from DOM objects and handles them in its trusted thread. This guarantees that there are not data racing conditions when receiving parallel requests from multiple JavaScript tabs. The back and forth communication is done using IPC-Channels. Here is how the main loop is implemented:

Not all VR Commands are initiated in the DOM. The WebVR spec defines some events that are fired when a VR Display is connected, disconnected, activated... The WebVR Thread polls this events from time to time and sends them to JavaScript. This is done using an Event Polling thread which wakes up the WebVRThread by sending a PollEvents message.

The current implementation tries to minimize the use of resources. The event polling thread is only created when there is at least one live JavaScript context using the WebVR APIs and shuts down it when the tab is closed. The WebVR thread is lazy loaded and only initializes the native VRServices the first time a tab uses the WebVR APIs. The WebVR implementation does not introduce any overhead on browser startup or tab creation times.

VRCompositor Commands (integration with WebRender)

WebRender handles all the GPU and WebGL rendering work in Servo. Some of the native VR SDK functions need to run in the same render thread to have access to the OpenGL context:

• Submitting pixels for each eye to the headset uses a OpenGL texture which can only be read by the driver from the render thread of the WebGL Context.
• Vsync and Sync poses calls must be done in the same render thread where the pixels are sent.

WebVRThread can’t run functions in the WebGL render thread. Webrender is the only component allowed to run functions in the WebGL render thread. The VRCompositor trait is implemented by the WebVRThread using shared VRDevice instance references. It sets up the VRCompositorHandler instance into Webrender when it’s initialized.

A VRDevice instance can be shared via agnostic traits because Webrender is a trusted component too. Rust’s borrow checker enforces multithreading safety rules making very easy to write secure code. A great thing about the language is that it’s also flexible, letting you circumvent the safety rules when performance is the top priority. In our case we use old school raw pointers instead of Arc<> and Mutex<> to share VRdevices between threads in order to optimize the render path by reducing the levels of indirection and locks. Multithreading won’t be a concern in our case because:

• VRDevice implementations are designed to allow calling compositor functions in another thread by using the Send + Sync traits provided in Rustlang.
• Thanks to the security rules implemented in the WebVRThread, when a VRDisplay is in a presenting loop no other JSContext is granted access to the VRDisplay. So really there aren’t multithreading race conditions.

To reduce latency we also have to minimize ipc-channel messages. By using a shared memory implementation Webrender is able to call VRCompositor functions directly. VR render calls originated from JavaScript like SubmitFrame are also optimized to minimize latency. When a JavaScript thread gains access to present to a headset, it receives a trusted IPCSender instance that allows to send messages back and forth to the Webrender channel without using WebVRThread as an intermediary. This avoids potential "JavaScript DDoS" attacks by design like a secondary JavaScript tab degrading performance by flooding the WebVRThread with messages while the JavaScript tab that is presenting to the headset.

These are the VR Compositor commands that an active VRDisplay DOM object is able to send to WebRender through the IPC-Channel:

You might wonder why a vector of bytes used to send the VRFrameData. This was a design decision to decouple Webrender and the WebVR implementation. This allows for a quicker pull request cycle and avoids dependency version conflicts. Rust-WebVR and WebVRThread can be updated, even adding new fields to the VRFrameData struct without requiring further changes in Webrender. IPC-Channel messages in Servo need to be serialized using serde-serialization, so the array of bytes is used as forward-serialization solution. Rust-webvr library implements the conversion from VRFrameData to bytes using a fast old school memory transmute memcpy.

DOM Objects (Inside Servo)

DOM Objects are the ones that communicate JavaScript code with native Rust code. They rely on all the components mentioned before:

• Structs defined in rust-webvr are used to map DOM data objects: VRFrameData, VRStageParameters, VRDisplayData and more. There are some data conversions between raw Rust float arrays and JavaScript typed arrays.
• WebVRThread is used via ipc-channels to perform operations such as discovering devices, fetching frame data, requesting or stopping presentation to a headset.
• The optimized Webrender path is used via ipc-channels when the WebVRThread grants present access to a VRDisplay.

The first step was to add WebIDL files files in order to auto generate some bindings. Servo requires a separate file for each object defined in WebVR. Code auto generation takes care of a lot of the boilerplate code and lets us focus on the logic specific to the API we want to expose to JavaScript.

A struct definition and trait methods implementation are required for each DOMObject defined in WebIDL files. This is what the struct for the VRDisplay DOMObject looks like:

A struct implementation for a DOMObject needs to follow some rules to ensure that the GC tracing works correctly. It requires interior mutability. The JS<DOMObjectRef> holder is used to store GC managed values in structs. On the other hand, Root<DOMObjectRef> holder must be used when dealing with GC managed values on the stack. These holders can be combined with other data types such as Heap, MutHeap, MutNullableHeap, DOMRefCell, Mutex and more depending on your nullability, mutability and multithreading requirements.

Typed arrays are used in order to share efficiently all the VRFrameData matrices and quaternions with JavaScript. To maximize performance and avoid Garbage Collection issues we added new Rust templates to create and update unique instances of typed arrays. These templates automatically call the correct SpiderMonkey C functions based on the type of a rust slice. This is the API:

The render loop at native headset frame rate is implemented using a dedicated thread. Every loop iteration syncs pose data with the headset, submits the pixels to the display using a shared OpenGL texture and waits for Vsync. It’s optimized to achieve very low latency and a stable frame rate. Both the requestAnimationFrame call of a VRDisplay in the JavaScript thread and the VRSyncPoses call in the Webrender thread are executed in parallel. This allows to get some JavaScript code executed ahead while the render thread is syncing the VRFrameData to be used for the current frame.

The current implementation is able to handle the scenarios where the JavaScript thread calls GetFrameData before the render thread finishes pose synchronization. In that case JavaScript thread waits until the data is available. It also handles the case when JavaScript doesn’t call GetFrameData. When that happens it automatically reads the pending VRFrameData to avoid overflowing the IPC-Channel buffers.

To get the best out of WebVR render path, GetFrameData must be called as late as possible in your JavaScript code and SubmitFrame as soon as possible after calling GetFrameData.

Conclusion

It's been a lot of fun seeing Servo WebVR implementation take shape from the early stage without a WebGL backend until it's able to run WebVR samples at 90 fps and low latency. We found Rustlang to be a perfect language for simplifying the development of a complex parallel architecture while matching the high performance, privacy and memory safety requirements in WebVR. In addition Cargo package manager is great and makes handling optional features and complex dependency trees very straightforward.

For us the next steps will be to implement the GamePad API extensions for tracked controllers and integrate more VR devices while we continue improving the WebVR API performance and stability. Stay tuned!