Quest’s extremely anticipated “Passthrough Digicam API” is now out there for all builders to experiment with, although they cannot but embody it in retailer app builds.
The brand new functionality was introduced at Meta Join 2024 in September as coming this yr. Now it is right here, as an experimental launch for Quest 3 and Quest 3S. Meaning any developer can experiment with it, and even distribute APKs utilizing it on platforms like SideQuest, however they cannot but embody it in Meta Horizon Retailer apps. Meta has taken this method for brand spanking new APIs a number of instances prior to now, and sometimes makes the function out there to be used in retailer apps inside a couple of months at most.
Choose builders have had early entry to experiment with the aptitude for some time now, and Meta will host Niantic, Creature, and Decision to debate it at GDC subsequent week.
What Is Passthrough Digicam Entry?
Whereas headsets like Quest 3 use cameras to allow you to see the actual world, till now solely the system software program obtained uncooked entry to those cameras. Third-party builders may use passthrough as a background, positive, however they did not really get entry to it. They as an alternative obtained higher-level knowledge derived by the system, comparable to hand and physique skeletal coordinates, a 3D mesh of your atmosphere with bounding packing containers for furnishings, and restricted object monitoring capabilities. That meant they could not run their very own laptop imaginative and prescient fashions, which severely restricted the augmentation capabilities of those headsets.
The exception was that on visionOS 2, Apple provides enterprise corporations uncooked entry to Imaginative and prescient Professional’s passthrough cameras for private inner apps, however this requires a particular licence from Apple and is restricted to “in a enterprise setting solely”.
For the “Passthrough Digicam API” to work, you because the consumer must grant the app permission to entry your headset cameras, simply as you’ll the microphone. If granted, the app will get entry to the forward-facing coloration cameras, together with metadata just like the lens intrinsics and headset pose, which it may leverage to run customized laptop imaginative and prescient fashions.
Examples of how apps may use this embody scanning and monitoring QR codes, detecting a recreation board on a desk so as to add digital characters and objects to it, detecting bodily objects for enterprise information experiences, or integrating the visible AI performance of cloud-hosted giant language fashions (LLMs). Builders are solely restricted by which real-time laptop imaginative and prescient fashions can run on the XR2 Gen 2 chipset performantly, or which cloud-hosted picture fashions they’re keen to pay for.
Meta software program engineer Roberto Coviello’s QuestCameraKit samples.
The passthrough digicam stream is supplied to the app with as much as 1280×960 decision per digicam at 30FPS, with a acknowledged latency of 40-60 milliseconds. Meaning it is not appropriate for monitoring fast-paced objects, comparable to customized controllers, nor for discerning fantastic options like small textual content.
Technically, at a base stage, there isn’t a particular Meta Quest Digicam Passthrough API, neither is it an extension to OpenXR. Builders do must request a Horizon OS particular Headset Cameras permission, however in any other case Quest’s passthrough digicam entry leverages Android’s present Camera2 API to additionally return the headset pose, obtained with OpenXR, and the Camera2 API is what builders of customized engines, or supply code for Unreal or Godot, use for it. This additionally means the identical code ought to work on Google’s upcoming Android XR platform, set to debut in Samsung’s standalone headset, with solely the permission request being totally different.
For Unity, builders can simply entry the cameras via Unity’s WebCamTexture API, which is how they already entry cellphone, pill, and PC cameras and webcams within the engine. A limitation right here, nonetheless, is that Unity’s WebCamTexture API solely helps one digicam at a time, not each.
Walkthrough from Meta software program engineer Roberto Coviello.
builders can discover Quest passthrough digicam entry documentation right here: Unity / Native Android.
Meta has printed 5 official Unity samples on GitHub: CameraViewer, CameraToWorld, BrightnessEstimation, MultiObjectDectection, ShaderSample. Meta software program engineer Roberto Coviello has individually printed QuestCameraKit on GitHub, a group of 5 additional samples: Coloration Picker, Object Detection with Unity Sentis, QR Code Monitoring with ZXing, Frosted Glass Shader, and OpenAI imaginative and prescient mannequin.
