As the spread of COVID-19 restricts people's mobility and face-to-face communication, various activities such as company meetings and sales activities are increasingly being conducted remotely. The sports and entertainment industries are not exceptions to this trend, and there has been a dramatic increase in the number of spectator-less sports broadcasts and live-streamed events.
Today, we will be introducing the Spectator Assistance System, which allows remote spectators of these events to share in their excitement despite the distance.

What is the Spectator Assistance System?

NTT Service Evolution Laboratories has been pursuing research to measure and quantify all aspects of events that occur in venue spaces in order to transmit and reproduce the atmosphere and enthusiasm of the actual venue for remote viewers at live viewing locations. However, circumstances have been changed dramatically by the COVID-19 pandemic. It is now difficult to have large groups of spectators gather at large venues such as live viewing locations.
To address this, we began work on the Spectator Assistance System (Fig. 1) to enable viewers to experience the intensity and special atmosphere of a stadium or live venue from their home or other small remote venue.

The Three Technologies Underlying the Spectator Assistance System

The Spectator Assistance System consists of three primary technologies. A brief overview of each technology is given below.

Remote Sensing Technology: "Understanding" audience reactions and the atmosphere of the venue

The first technology is designed to capture audience reactions and the atmosphere of the venue in real time.
For example, different people are excited by different aspects of a given sporting event: some people focus on specific players, while others view the entire match as a whole.
At a musical event such as a concert, some people get worked up during the performance itself, while others let their emotions out after the performance is over. People experience their emotional reactions at different times.
With this in mind, voice-capture devices such as smart speakers, cameras in TVs, and wearable devices such as smart watches are used to measure and quantify people's emotions to understand exactly how engaged viewers currently are and what part of the broadcast they are focused on.

Ultra Reality Field Creation Technology: Creating virtual spaces

This technology is used to create virtual viewing spaces and place each individual viewer into them.
Bringing together excited viewers makes it possible to create a viewing space that enhances enthusiasm. Conversely, you can boost interest and engagement by placing less-interested viewers in an enthusiastic space.
By enhancing aspects that increase the excitement of remote viewing, such as cheering, enthusiasm, buzz, and motion to match the excitement of the scene, we can create a virtual space that gives the impression of tens or hundreds of thousands of people watching, even if the event really only accommodates several thousand people.

Field Reproduction Technology: Enhancing the sense of unity among viewers

This technology is used to enhance the sense of unity between viewers in a virtual space.
Specifically, this system feeds back the images, sounds, shadows, etc. of other viewers without any delay, creating an environment that makes viewers in remote locations can feel as if they are cheering along next to one another, encouraging interaction.
As live viewing in large venues is restricted, there are high expectations for technologies that can enhance enthusiasm, unity, and interactivity even in small, isolated environments such as the home. With this in mind, NTT Service Evolution Laboratories has constructed an experimental facility for the use case of watching sports in the living room at home, and tests are being conducted on viewer impact and the best feedback methods. (Fig. 2)

Technology development related to Spectator Assistance

NTT is also researching methods of enhancing video realism to achieve a spectator experience that conveys a shared sense of excitement.
As an example, NTT is promoting automation, streamlining, and other systems development directed toward the commercialization of HiddenStereo, a 3D stereo image generation technology that renders 2D images clearly even to the naked eye.
One of the features of this technology is the ability to semi-automatically generate 3D images from existing 2D images. It does this by first generating high-precision depth data (depth map) using edge-preserving smoothing to combine the results of depth estimates and region segmentation derived from 2D images captured by a single lens. (Fig. 3) In addition, depending on the scene, corrections are made such as limiting the range of depth and emphasizing important subjects in order optimize the image for an effective and natural three-dimensional effect.

With these technologies, it is possible to add 3D effects to past live music performances and theatrical performances that were filmed in 2D, as well as to add 3D effects to 2D works such as paintings and posters. They also enable barrier-free 3D viewing, in which each viewer in the room can freely choose how they want to enjoy the show.

NTT will continue to strive to create experiences that are no less exciting than watching in a stadium or concert hall, where you can feel as if you are watching and experiencing an event together with many other spectators.

*The names of the laboratories mentioned in the article may have changed since the time of writing/interview.