Virtual + Augmented Reality is a problem that is being simultaneously tackled by multiple companies at the same time. We have many incumbents in the B2C space like the HTC Vive, various Oculus products, MS HoloLens and the Valve Index. Apple is also working on it’s own AR headset that might be launched sometime next year.

When we talk about VR/AR development, many challenges come to mind -

I’m sure there are many more questions that I missed out on but basically VR/AR is a big unsolved problem that Silicon Valley has set it’s eyes on. Amongst the key players, Meta has the most comprehensive approach - having bet the farm on being successful in this space. Last year they spent $10 billion on this problem, far more than any other big-tech firm. And recently, Meta held a technology showcase where they showed off various prototypes and talked about the key problems they were solving. In the showcase they talked about none of the big questions I mentioned earlier. If I hadn’t read about the showcase, I would imagine those are the problems different teams at Meta are working on. But what I saw made me think about how to tackle product design deeply.

Meta’s platform approach is trying to build both hardware and software for VR. The way they are trying to tackle hardware is breaking the task of building VR headset into a couple of approaches based on price point and form factor and then breaking down each approach into simple problem statements. For each problem statement, they have built a prototype. Let’s discuss the two main approaches they have taken -

This approach is championed by their Oculus Quest family of headsets which are the best selling VR headsets in the world right now. The Oculus Quest 2 is already in the market and since this show was about prototypes, Meta didn’t showcase any in this category. However, based on my experience of playing around with the Quest, I can elaborate on the problems Meta has solved with it-

The Quest is the first mainstream family of headsets that can function without any PC or smartphone. This is a major departure from other Meta/Oculus products like the Rift which needed to be connected to a PC and hence were cumbersome to use for long periods of time.

Additionally the cost of owning a PC that is capable of supporting VR alongside the cost of headset was very expensive (easily in the 2-3k USD range). This meant no-one apart from a very narrow niche of VR enthusiasts were able to afford them and that meant the software support was also like that of a nice product.

Headsets before Quest used outside-in tracking which required people to place trackers in various parts of a room to be able to track them. This approach was quite accurate and the accuracy could be further increased by adding more trackers. However this meant that that you could only use your headset in the room where r you set up the trackers and incase you didn’t have a suitable room - it wouldn’t work at all. Meta added inside-out tracking which used sensors on the headset it self to map the space around the user and track them. This is definitely less accurate approach but i reduces the number of parts n the setup and also lets the user to use their headset in any room they want.

While this is a subjective metric, the Quest 2 came with a slew of features that made it one of the most comfortable headsets for a while variety of head shapes and for folks with and without glasses. The adjustable headband and the special memory foam for the user’s face along with the moveable lenses made it extremely comfortable to use.

Now that we have a good handle on the problems Meta has solved with the Quest 2, we can talk about the new problems they are trying to tackle with these new prototypes

Reading text is a big part of most jobs and if Meta wants its headsets to be considered as serious work tools, then text is a big problem to solve. Project Butterscotch aims at bringing retina-quality high resolution screens to its consumer grade devices by increasing the current pixel density of the current Quest 2 up to 250%. The aim is to let uses with perfect vision read the 20/20 line. The current prototype offers 55 pixels per degree of vision, slightly short of Meta’s goal of 60. However, this no where close to being market ready. The current industry leader in this metric is Varro but they offer an impressive 65 pixels per degree of vision.

There is a lot of merit in making VR headsets thinner and lighter. The headset becomes more comfortable and people will use them longer. In case of AR, since people might want to wear them outside, size and weight is even more important. It uses a light-bending technique that allows completely flat lenses which is great. The engineering challenge behind is to build a self-contained light source using a laser that is unlike the olds used today. The challenge is to make it cost effective, safe and small enough.

One of the main challenges Meta talked about was achieving photorealism - making the virtual world undistinguishable from the real one. One of the challenges in doing that is the visual fidelity of the display - not in terms of resolution (which is being tackled by project Butterscotch) - but in terms of dynamic range (think situations with very bright objects like looking at the sun). This is done by have a locally dimmable display that can achieve really high brightness. The current prototype can support 20000 nits (for reference, most expensive TVs do 1000 nits and the iPhone does 1300 nits at peak brightness). However it is no-where close to being market ready.

If you want VR to feel natural, the virtual world has to have depth to it. It can’t be just two dimensions. Halfdome has been through several iterations - the first iteration was a series of mechanical lenses that moved to create a sense of depth. The new version, which is supposed to be market ready is supported by an array of liquid crystal lenses.

Supposed to be released sometime later this year, Project Cambria is going to be Meta’s premium stand-alone headset. It is supposed to support both VR and mixed reality with the help of camera on it’s exterior. While the Quest 2 was priced to be more approachable and had appropriate constrains for the price, this is going to be the best Meta can offer currently.

This is not a real prototype but a though experiment about what an ideal VR and mixed reality headset would look like. It involves using elements from all the prototypes above and adding additional features like adding an outward facing screen to show the user’s eyes. Ideally, it would be no bigger than a pair of ski goggles/

When I usually read about VR and how companies are planning out their vision - they are usually building out technologies in search of problems. This is where MagicLeap and Google Glasses fail. Meta’s approach differs from this in that respect. All the technologies they are pursuing are in response to very specific user problems. They are not trying to answer the big questions about what the nature of the virtual world - if they were doing that - they would certainly come up short. Just look at the last platform that became mainstream - the4 smartphone. No one thought it would be used to call cabs, play console quality games and be a movie making machine. A lot of those use-cases came to be because the underlying hardware and software platforms were robust and easy to use. Essentially everything - your PC, smartphone and a VR headset are a natural evolution of the computer and the more easy and pleasurable to use they are the more likely people will use them for more and more things in their life. It also highlights how the optimal product design should be → Identify the current user frustrations with your product

→ Ideate theoretical solutions for the problems

→ Associate solutions with technology that you would need for them

→ Iterate on the technology until it’s ready in an approachable form factor and usable

→ Explore multiple price-points so that you can bring costlier and more cutting edge stuff to market