A developer found out a intelligent option to entry Quest 3’s passthrough cameras to run customized pc imaginative and prescient fashions.
Meta doesn’t enable third-party builders to entry the cameras of Quest headsets, however Michael Gschwandtner, the co-founder and lead developer of the hand monitoring health app XRWorkout, posted a video to social media platforms exhibiting the outcomes of an attention-grabbing workaround to this restriction.
The demo video reveals the open-source MobileNet V2 object detection mannequin operating natively on Quest 3 at 0.75 Hz. This was chosen for practicality, reasonably than for efficiency causes.
Gschwandtner despatched UploadVR a take a look at construct and we independently verified that that is actual and works. He additionally solely revealed to UploadVR how precisely he pulled it off.
How It Works
Gschwandtner’s app construct creates a webview, a Chromium net browser occasion embedded throughout the app, which mechanically masses the oculus.com/casting URL which Quest headsets can solid their view to.
You then must undergo the complete technique of logging into your Meta account on this webview. When you’re achieved, you utilize the Quest system menu to provoke net casting. Uniquely, you are casting from the headset to an app inside the identical headset, to not a unique gadget.
Gschwandtner’s app then samples the pixel content material, the rendertexture, of the webview receiving the casted view, and from there can do with it no matter he likes. Within the demo instance, he is utilizing Google’s MediaPipe on-device SDK.
A key limitation right here is that the view will embody digital objects and interfaces too, not simply the digital camera view, that means it would solely work when a lot of the picture is passthrough. However Gschwandtner claims he has a “plan” for a workaround for this.
What Occurs Subsequent?
It is unclear how Meta will reply to Gschwandtner’s discovery. It is potential the corporate may make this method inoperable in a future model of Quest’s Horizon OS. Gschwandtner’s social media posts embody a public request to “hop on a name” with Meta CEO Mark Zuckerberg or CTO Andrew Bosworth to “present them” the method.
Final month Bosworth gave his ideas on the concept of deliberately giving Quest builders entry to the passthrough cameras throughout an interview, saying Meta will “maintain taking a look at” the problem.
As compared, for its Imaginative and prescient Professional headset Apple solely offers uncooked entry to enterprise firms, and just for personal inner apps.