Blog: Augmented Reality Fieldview 2020 – Part II

Context

What is the current state of the human endeavor in digitally augmenting our world? Moreover, in considering where we are today, how have we arrived at this point, and where will our efforts lead? To understand the augmented reality (AR) ecosystem in 2020, we must rewind a decade and project half as far into the future.

Before we dive into the state of AR in 2020, we must first have a mutual understanding of what AR is, and equally importantly, what it is not. Herein, we will define these terms as follows:

Virtual Reality (VR) : Using traditional displays and rendered environments to simulate other world

Augmented Reality (AR): Using transparent displays to overlay additional layers on the real world

Extended/Cross Reality (XR): Using a synthesis of either or both approaches. In this discussion, we will break the development of AR into three distinct epochs. Each of these phases of development are marked with distinct features and are additive over time. While there are a series of small and medium sized players that ebb and flow, throughout it all are a handful of whales that have directed the entire ecosystem in profound ways, the culmination of which we will begin to experience in the new decade.

The three eras of AR growth and proliferation can be described comparably to the trajectory of an individual startup, only extrapolated to an industry scale. In the fledgeling stage of seeding (2010-2019), the incumbents surveyed the field, made their initial bets, and formulated long term strategies; simultaneously a handful of unknown and untested ventures arose of their own accord. Once the initial field was cemented, the second phase of development (2020-2022) could begin, wherein the existing players began to cross pollinate through the transfer of personnel, flow of intellectual property, and acquisition or merger of smaller entities. In the ultimate stage, we will finally reap the benefits of synthesizing our analog and digital worlds when these structures come to fruition (2023-2025); brought on by a final charge of rapid Pac-Manning of smaller corporations and commercial deployment to large parts of the domestic, and then global, populations.

Era II – Cross Pollination

The first decade of AR development (2010-2019), the seeding phase covered in Part I, was focused on entrenchment and solidification of defensible positions. At the turn of the new decade, the next era is one of motion, exchange, and consumption between these established positions. While each of the major players have remained largely steadfast, certain constituents and components, such as personnel, customers, sensors, inputs, intellectual property (IP), and methodologies have flowed across organizations. In short, these circulations fall into the category of either mind (employees, board members, customers, IP, strategies, methods) or machine (hardware, software, inputs, sensors). These flows can be seen in particularly high resolution through the mergers and acquisitions (M&A) in the AR space. 2020 has already had some particularly pronounced movements, on individual and collective scales, by volition and by mandate, across the AR ecosystem.

Minds

Companies are vessels into which people can pour their resources, toil, and ambitions, but underpinning it all are the selfsame human entities. This facet of the AR industry is essential to grasp and is a core factor of the market today. AR developer talent is sorely lacking and true expertise is few and far between. This has led to an ossification of talent at specific organizations and not much motility outside of that. However, in late 2019 and early 2020, this hard trend began to soften, first a little and then all at once.

Nowhere is the flow of people more apparent than at Magic Leap. Historically, Magic Leap has been a magnet for talent both within the AR hardware industry and in the related verticals of cinema, special effects, and UI/UX. Because of the totalizing dream they espouse, Magic Leap was able to draw the likes of Lucas Film and ILM co-founder John Gaeta and eminent game designer Graeme Devine for creative leadership roles. While historically Magic Leap has drawn in ever more professionals, recently its magnetic nature has inverted, and rather than serve as an attractor, the company has started to push out personnel. Starting from a macro perspective, the company has undergone significant restructuring since the start of the year. Given the change in focus resulting from the lack of consumer commitment to the Magic Leap Creator One Edition, the company has re-charted a course toward the enterprise markets. This means that a lot of teams and positions that were developed for a consumer facing product are no longer necessary; in particular for groups focused on consumer content, marketing, and design. Additionally, the COVID-19 pandemic offered a convenient coverall excuse for trimming, and the combined forces resulted in a 50% workforce reduction, totaling around 1,000 individuals by May 2020. Interestingly, because of the aforementioned small talent pool, other AR focused firms were quick to pick up the highly experienced and suddenly available Magic Leap team. This article and graphic from Protocol does an excellent job of capturing the diaspora. The results of this personnel cross pollination are profound and have yet to be felt by the market. Additionally, many former Leapers will go on to find jobs outside of the major tech firms and may even form startups of their own, which in turn will help flesh out the AR ecosystem with new perspectives and approaches.

(https://www.protocol.com/who-is-hiring-magic-leap-employees)

In service of the change in strategic vision and move from consumer to enterprise, Magic Leap has also undertaken specific and significant changes in personnel over the last six months that are certainly worthy of note. The largest of these transitions has been the abdication of Rony Abovitz from the CEO role, thereby ending the decade long tenure of the founding father figure. Importantly, Rony is being replaced by a hyper-capable executive from Microsoft, Peggy Johnson. Having worked directly and indirectly with the Hololens team on enterprise applications, it makes perfect sense for Magic Leap to onboard this sort of expertise so they can better embody their strategic pivot across the entire organization. Additionally, Peggy’s deep operating knowledge should help accelerate and solidify Magic Leap’s metamorphosis from startup to establishment player.

Rewinding from 2020, there have also been a series of important flows of people that are exemplars of cross pollination, but do not fall explicitly within this second era. Certainly, much of the M&A activity from 2010-2019, our seeding period, could also be defined as cross pollination. This is particularly true for acqui-hires, wherein a firm is purchased for its human expertise more than its IP portfolio or existing hardware and software. Much activity in the AR space has been in this vein, and Apple in particular has been headhunting for AR talent unlike any other. This can be seen in the charts above, wherein Apple gobbled up 30%+ of former Magic Leapers who left due to restructuring. This is also echoed in the acquisition streak discussed in Part I, and is most obvious in the acquisition of Sensomotoric Instruments (SMI) for their eye tracking experts and IP. While M&A activity has been bubbling under the surface for the past 5+ years, it’s pace is accelerating and will continue at ever more breakneck speeds over the coming years.

In addition to the physical persons that move across organizations over time are the ideas and methods those persons employ to realize their visions. The most notable and formalized of these practices are codified in intellectual property, and more specifically in issued patents. Patents serve two equally important purposes, to enable use of a technology and to block others from doing so. IP is usually locked within an organization, and can only be taken up if an entire organization is consumed by a larger entity or if it is picked from the bones of a dead start up. Many acquisitions are equally focused on getting the existing engineering talent as well as the legacy paper trail of ideas they’ve generated in patent literature. This is true when an organization is attempting to enter a new field or put further emphasis on an area where it doesn’t have much historical expertise, and is especially true when branching into niche fields like eye tracking, device inputs, and novel sensors. In the case of picking the bones of a corporate corpse, there have been a handful of notable cases in the last year such as the collapse of Meta AR, Osterhout Design Group (ODG), and Daqri. In the case of ODG, their patent portfolio of more than 100 issued patents went up for public sale via an auctioning service. Interestingly, they had also sold some foundational AR patents to Microsoft prior to the Hololens I development, which may have contributed to their difficulty in the marketplace. The dissemination of this much concentrated know-how offers upstart players a chance to leap frog ahead and gives incumbents a chance to branch into new methodologies and approaches.

Machines

In addition to the evolution of AR displays and form factors which were discussed at length in Part I, there are two other essential components of AR that have been evolving and will come to the fore. Sensors and inputs are essential for a complete ecosystem, and just as the major players have elected different strategies for display types and form factors, so too do they have major differences in their respective sensor suites and input devices. Before we can grapple with what the near term future will hold, it is important to ground our understanding in the state of these systems today.

In terms of major differences in input methodologies, nowhere is the rift more stark than in the different routes taken by Magic Leap and Microsoft for their device inputs. The Magic Leap Creator One shipped with a controller while the Hololens I and Hololens II did not. The controller for the Magic Leap is tracked by a small module on the right side of the headset and has six degrees of freedom. Magic Leap also has developer and beta options for hand inputs which allow the headset to use its outward facing cameras and point cloud to interpret user hand motions such as swiping or tapping. This is in stark juxtaposition to the Hololens which has never shipped with a controller, and instead has by default operated with robust hand and gesture inference. To open the home menu on each device is a markedly different experience; on Magic Leap, it requires a press of a button on the controller, while on Hololens it requires a tap on the wrist. Additionally, Microsoft has heavily pursued voice as an alternative method of input and has baked in the Cortana assistant to these ends.

While both Microsoft and Magic Leap have developed their respective special sauces for device input with internal teams, other firms have elected to look outside their own walls for varied approaches. In the parallel VR space, 2019 saw the acquisition of CTRL Labs by Facebook for as much as $1B USD. This was hugely significant for the neuro space as a whole, as it validated the importance of brain machine interfacing and measurement in the markets. Equally important was the technology acquired and the interoperability established between CTRL Labs and Facebook’s Oculus division.

So how do these different inputs and sensors impact the end user? Starting with similarities, both head mounted displays (HMDs) by Magic Leap and Microsoft have accelerometers, outward facing cameras, outward facing infrared dot projectors, outward facing infrared dot receivers, and inward facing infrared cameras. These can inform on a variety of things including head position, head pose, acceleration, gait, eye position, pupillometry, and relative position to objects and the world. Historically, medical devices have been a single sensor ordeal, but with these new AR platforms, suddenly software applications have access to a dearth of varied sensor outputs and can exist as sensor fusion devices as a default.

While AR devices are highly sophisticated already, the sensor possibilities are only beginning to be explored. As technology advances, existing sensors will become far more robust as constraints like power consumption, heat output, and sensitivity are refined or overcome. In the case of eye tracking specifically, in addition to general miniaturization, it is likely we will see ever increasing frame rates in these mobile sensors that may eventually rival even the most powerful desktop systems of today. Equally important to making improvements on the extant components is the inclusion of new and novel sensors in these HMDs. While the major players haven’t taken them up yet, medically significant sensors are beginning to proliferate the VR market, and their translation to the AR market is only a question of time. Natively integrated electroencephalography (EEGs) in consumer grade HMDs are predicated by LooxidVR. EEGs have the potential to be used for everything from brain computer interfaces (BCI), to brain health assessments, and beyond. Other sensors, like functional near infrared spectroscopy (fNIRs), may be useful to integrate further down the line to take measurements of cerebral blood flow for either medical or recreational ends.

In addition to the inclusion of better and newer sensors and inputs, it is also important to emphasize that the underlying hardware is still of paramount importance and is also a candidate for cross pollination. The most glaring example of this in 2020 has been the acquisition of North by Google. North was a maker of lightweight AR spectacles, called Focals. Importantly, these devices have a much slimmer and unobtrusive silhouette than either the Hololens or Magic Leap. It seems that the next generation of Google Glass may indeed be built on this existing infrastructure and could be paired with a modified version of the Android OS for stellar functionality and broad reach. It is also important to recall that North, formerly Thalmic Labs, was the maker of the Myo armband, which used the electrical impulses of muscle contractions to move a digital cursor and do basic operations like clicking, scrolling, etc. Interestingly, North sold its Myo division to CTRL Labs in mid 2019, which in turn made the company an attractive target for Facebook to pursue and roll up with their Oculus division. By purchasing North, Google is staking its AR position similar to Apple’s; with a focus on lightweight, universal, and attractive AR devices for everyday consumer use.

While the circulation of people and products through the AR industry has been bubbling gently over the past decade, the accelerating pace and fervor demonstrated over the last year have shown this activity is building towards a crescendo. Firms have become internally stagnant and increasingly must look outside for novel ideas and approaches. This trend of ever more cross pollination will continue to accelerate over the next handful of years as the industry manifests the dream of true AR for consumer, enterprise, and military markets. This period of tumult is unpredictable and exciting, as many small firms are snapped up by the larger players and the legacy powers re-examine their respective stances and approaches in an attempt to gain the upper hand and associated future market dominance. All of this labor; the trimming, the acquisitions, and the pivots, will come together in a moment of fruition. When this moment comes, humans will invert their fundamental relationship to the digital. Rather than opting into digital worlds by pulling out our laptop, tablet, or phone, we will now have to opt out of the digital realm by removing our lenses.