Virtual videos, real holiday experiences

By Stuart Pallister

SMU Office of Research & Tech Transfer – A Singapore Management University researcher is investigating the potential of how we work together when navigating 360-degree videos. Associate Professor of Computer Science, Tony Tang, and his co-researchers in Canada and India developed a project called Tourgether360, which used 360-degree videos of tourist locations in Italy and Japan, to discover how participants collaborate with each other when navigating these spaces.

Their paper, ‘Tourgether360: Collaborative Exploration of 360° Videos using Pseudo-Spatial Navigation’, states that collaborating with others when exploring 360-degree videos is ‘challenging’ because you are not aware of the other person’s viewing activities.

“Our motivation was really to think about how one might engage with next-generation tour taking,” Professor Tang told the Office of Research & Tech Transfer.  

“When I move somewhere new, I want to show my mother where I’m living. I can do a walking tour over Skype with her and my father, but invariably, this ends up being a shouting match! I am never pointing at what she wants to see, or my father wants to look at something that she doesn’t care about, and vice versa. The problem is that everyone wants to look at different things.”

Video tour, video game?

Professor Tang and his collaborators created Tourgether360, which changes how we interact with omnidirectional videos, which are captured by multiple cameras whose video feeds are stitched together. In Tourgether360, the 3D architecture is reconstructed from the video feed, which allows viewers to navigate and explore the video as if it were a level in a video game. Little wonder that Tourgether360 was built using the Unity game engine, which powers many contemporary video games, including Pokemon Go and Fortnite. Tourgether360 ‘enhances social exploration’ of 360-degree videos by providing collaborators with an overhead ‘architectural minimap’ of the space explored in these videos, which corresponds to a shared 3D perspective of that particular location.

‘We found that participants adopted the Tourgether360 approach with ease and enjoyed the shared social aspects of the experience,’ the paper continues. ‘Participants reported finding the experience similar to an interactive social video game.’

“With the Tourgether360 tool, we asked the question: what if we consider a tour video as a video game space?,” Professor Tang said. “If we did that – and represented people like video game characters – would it be easier for them to navigate, explore, and make sense of the piece of media or the space? That motivated the work. The quick answer is that people liked it. It’s quite straightforward to navigate and is engaging to play with.”

In a previous study some five years ago, they had asked participants to explore 360-degree videos using iPads together but “as you might expect, one of the challenges is that people are at different parts of the video, looking at different things, so it was really challenging for them to synchronise and communicate about what they were seeing.”

The Tourgether360 project used computers rather than iPads, and the next step will be to move to VR headsets. “The next iteration of this particular project we’d like to explore, is what it’s like when you’re wearing headsets, which are more immersive.”

Project findings and future uses

Along with Simultaneous Localization and Mapping (SLAM) technology that synchronises what users see on the video with their location on the map, the Tourgether360 project provided each participant with an avatar when exploring, say, the area around Florence Cathedral in Italy and the Asakusa Shrine Complex in Japan. “The novelty was what was essentially taking a public domain video and mapping it into a 3-D environment, so it becomes like a video game and that’s what’s fun about it.”

According to the study, all but three of the 16 participants preferred to communicate using spatial references (e.g., “Do you see a small temple?”) rather than time-based ones (“I’m at one minute and twelve seconds”).

The paper concludes that ‘Tourgether360 provides a way for collaborators to explore and navigate 360° tour videos together with the metaphor of a shared space.’

Professor Tang acknowledges that, compared to avatars in video games, theirs are “super simplified.” In fact, the avatars – which are described in the paper as viewing capsules which can position and mark points of interest – currently look like floating digital eyeballs as the tool is “bare bones” at this stage.

“We do have some other work where we’re really interested in how we drive avatars and how people use their bodies to express themselves. One of the things we know right now is that the way VR headsets capture movement is fairly limited in terms of the expressive gestures we use every day.”

That said, the commercial potential of this sort of tool could be immense as it could be developed by the tourism industry for holiday planning. “We can use these techniques to give people those sorts of cultural experiences to see what life is like in different places without actually having to go.”

“It doesn’t solve every problem in vacation planning, but it helps solve certain kinds of problems when it comes to talking about, discussing and thinking about space.”

Professor Tang and a colleague from SMU are now “playing with the next version of the tool.” This will extend the tool’s accessibility to the partially sighted. “We’re thinking about audio descriptions for people with low vision. That’s the most immediate thing we’re pursuing.”

“The other thing, more broadly, that we’re thinking a lot about, is what does interacting with a tourism-based interactive agent look like? Maybe a tour guide that is digital somehow?”

Or perhaps a digital concierge?

“That’s a great way to think about it. I think about my research as being about how do we build tools that allow people to help each other and work with one another.”

Professor Tang's paper, 'Tourgether 360: Collaborative Exploration of 360° Videos using Pseudo-Spatial Navigation' was published recently at an Association of Computing Machinery (ACM) conference on Human-Computer Interaction "which explores tools that allow people to do things together." His co-researchers were Lev Poretski and Li Jiannan of the University of Toronto, as well as Kartikaeya Kumar of IIT Guwahati in India.

Back to Research@SMU February 2023 Issue