An Eye to the Future: 5 Innovative Applications of Mixed Reality

The rumors are gaining momentum – in theory, Apple is preparing mixed reality glasses to give this technology a definitive boost. So far, the most prominent examples of mixed reality have been confined mainly to work environments. However, it seems that we are on the verge of a qualitative leap.

In this article, we will address the following issues:

• What the future has in store: five examples of mixed reality
• Requirements for an optimal mixed reality experience
• Latest technology breakthroughs

As a quick reminder, mixed reality differs from augmented reality in that, in addition to mixing real and virtual images, it offers the possibility of interacting with virtual elements. For instance, in a video game, a character could perceive the greeting of our physical hand. Or a virtual sculpture would react to the movement of our hands when we turn it over or rotate it.

 

 

What the future has in store: five examples of mixed reality

HoloLens glasses have made great strides in corporate applications, but they are still relatively bulky, so they are not yet everyday devices.

However, new technologies will probably enable the emergence of mixed reality applications. Only time will tell what remains hype and what reaches mass adoption.

1. Giant screens. Imagine if your glasses could show you a giant movie screen on your bus ride. Thanks to direct retinal projection, discussed at the end of the article, transparent glasses could offer a cinematic experience close to a conventional movie theater. They're not there yet, but the maker of the GLOW glasses provides the ability to select the movie with a hand gesture – a la Minority Report – and project it directly onto the lenses of the device.
2. A new generation of video games. Unless you have been living under a rock, you will be familiar with Pokemon Go. This video game offers augmented reality functionalities, but its developers wanted to go further. Thus, in its most recent demo, you can see the possibility of a user interacting with virtual creatures utilizing their hands. It seems that Apple is also interested in these applications.
3. Education. One of the areas destined to benefit from mixed reality is education. A recent example would be HoloMusic XP, a tool that helps students learn to play the piano through mixed reality and gamification. Whether through virtual, augmented, or mixed reality, an improvement in information retention, motivation and concentration is expected, as indicated by a recent study.
4. Equipment repair in extreme environments. In environments such as outer space, it is not possible to move an entire team of engineers to perform maintenance on complex devices. So in 2020, NASA turned to mixed reality for one of its astronauts to upgrade the Cold Atom Lab, a quantum laboratory aboard the International Space Station. On our planet, it is already being used to control desalination plants, water purification plants, and wind farms.
5. Geospatial simulation. Project Anywhere XR is an initiative to visualize complex maps in 3D. This includes everything from orographic formations to urban environments. Besides the mixed reality glasses, the crucial aspect is that all content is sourced from the cloud in real-time.

What is needed for an optimal mixed reality experience?

In addition to portability, which is as basic a requirement as two wheels are for a bicycle, mixed reality must address several critical issues. A failure in any of these can break immersion and lead to unwanted effects like dizziness or confusion.     

1. Minimum latency. The slightest mismatch between the controller's instructions and the displayed image can ruin the experience. This is known as latency. For example, Apple's new glasses are rumored to be paired to the phone with Wi-Fi 6E technology, which offers minimal latency.
2. Advanced environment recognition. To integrate elements in a scene, it is necessary to have a complete recognition or mapping of the environment. To achieve this, apart from cameras, it is ideal to have depth sensors such as LiDAR, which obtain a laser-scanned and three-dimensional image of a room. Finally, AI algorithms are required to process all this data.
3. Ability to use a control interface. Unlike augmented reality, in the mixed reality examples we have seen, some form of technology is required to interact with virtual objects. Cameras and depth sensors can identify the user's hands, but mixed reality glasses would most likely need additional controllers. An alternative would be motion-sensing rings such as those patented by Apple.
4. Ability to integrate virtual objects. This is one example of the challenges facing mixed reality. If you want to wear glasses with this technology for prolonged periods, all elements must be integrated into the scene. Ultimately, that means that their proportions and focus are consistent with the nearby physical objects. They must also take eye movement into account.

Latest breakthroughs in mixed and augmented reality technologies

Of the four challenges we have mentioned, the last one is probably the most complex. Fortunately, technological innovations are already being developed that could provide the solution. As noted, the problem is not so much that a virtual object may be displayed more out of focus as distance increases.

After all, if the room has been mapped three-dimensionally, it is not difficult to assign characteristics according to the represented object. No, the problem is that the virtual object behaves logically according to the focus of the retina and the movement of the eyes.

This is where one of the most fascinating technological breakthroughs of recent years comes into play: retinal projection. This technology, which is still in its infancy, projects images directly onto the retina instead of screens. This enables a seamless integration of images into the environment, which will benefit both mixed reality and augmented reality glasses.

In addition to the patent recently obtained by Apple, some prototypes offer functionalities in this sense. One of them would be the Light Drive glasses presented by Bosch at the 2020 CES technology fair. This model, which has not yet reached the market, projects images in the form of laser beams onto the retina.  

Another of the technologies likely to be used in mixed reality glasses will be the one developed by the Swiss company CREAL. As can be seen in this video, their device projects a hologram known as a light field onto the lenses.

In this case, it involves projecting six thousand images per second to each eye at different angles and depths, emulating the behavior of light. Thus, the eye can change the focus of the object depending on the part of the scene it is looking at.

Regardless of the system that prevails, it is most likely that this type of glasses will depend on the computing power of a paired device to remain light and manageable. One of the examples of mixed reality in that sense is the Nreal glasses, which are linked to an Android phone.

With this approach, longer battery life and 5G connectivity can be offered, which will be crucial for the mass adoption of these devices. However, that will be only an intermediate phase. Ultimately, mixed reality glasses are expected to give way to smart contact lenses like the prototype presented at CES 2021.  

Sources: Forbes, BBC, Ieeexplore, https://scitechdaily.com/how-astronauts-upgraded-a-complex-experiment-in-space-video/