Social Augmented Reality

THIS MATERIAL ©2008–2022 SALLY A. APPLIN AND MICHAEL D. FISCHER. ALL RIGHTS RESERVED.

"Social Augmented Reality":

• Augmented World Expo (AWE 2015), June 2015, Santa Clara, CA

'Social Augmented Reality: What it is. How to get it. Towards a Multiuser Social AR Experience.' Video and Transcript

VIDEO

Full Length VIdeo: Video Complete Talk: given by Sally Applin with Michael Fischer • Social Augmented Reality: What it is. How to get it.Towards a Multiuser Social AR Experience. • AWE 2015

TRANSCRIPT:

Sally Applin Michael Fischer

Social Augmented Reality:

What It Is, How to Get it.

Towards a Multiuser Social AR Experience.

Sally Applin

I’m going to talk about social augmented reality, what it is, how to get it. And looking at working towards a multi-user social AR experience. This is from a publication that came out in the IEEE consumer electronics magazine, the most recent issue, in April. We have a full article and I’m giving a snapshot of that.

We look at social augmented reality, we’re trying to figure out what are cooperation and sociability? What makes people social? And what are the outcomes for mobile that we’re seeing that are changing the way we are social? We’re going to look at PolySocial Reality, and the design problem of heterogeneity, which is what we’re getting when we’re looking at all of these people using all of these devices.

The goal is to develop cooperation and designing shared experiences and connecting people through those shared experiences. And we’re going to move through Magic Bus towards what I like to call “social blue brain,” which will make sense as we move forward.

What is social augmented reality? It has those elements of sociability, cooperation and shared experience. Cooperation is this. If we look at the highway and we look at the windfarm behind it, that is a representation of human’s massive amount of cooperative effort. Metal mining, manufacturing, engineering, all the social structures that came together to make the road system, to make the things that we use and don’t even think about. We came here in cars. Cars were collaborations, between humans, between humans and robots, between robots and robots. All that is a result of cooperation.

Cooperation requires the shared information. We can’t cooperate unless we share information with each other. If we’ve developed systems where we’re not able to do that, it’s harder for us to cooperate. I hate to be a downer about it, but if we don’t cooperate we die. We really do need each other.

We help each other in different ways. This is one way that we’re social and we cooperate. This is another. The bakery doesn’t exist without that people that—baked goods can’t exist without the people that make them and the raw materials that are made by others. These women working on their robot, that robot’s not going to work unless they come together, collaborate, share information to work together.

Social is also shared experience. He is sharing an experience with his camel. It may not be an experience that both of them could communicate about yet. But we’re also looking at human-animal communication.

Also, something like this hot air balloon where people are sharing an experience together, or the kids playing with Legos. They each may be individually building something, but they’re actually having a shared experience of that Lego environment as they’re hanging out at Maker Faire making things.

The current landscape is social. But there’s a lot less shared experience in the local locale. We’re being very social. Everybody in this picture is actually connecting to people. But they’re not connecting to people in their presence, they’re connecting to people in a distance presence. And that sociability that’s remote and not local changes our local experience.

When you’re on the street looking at people, they might look like they’re doing something like this. They’re not social, they’re not part of us. And certainly that’s what came out when we were starting to see mobile permeate in the last six, seven, eight years.

What they’re really doing is they’re looking at lots of different things. When you see someone on the street looking at a mobile, they’re not just looking at their mobile. They’re looking at maybe lots of different things and their mobile might be looking at other things on their behalf. You check in at Foursquare. That’s going to send messages to people that may send messages back. Or if you’re sending something on Facebook, it’s a broadcast medium, you may be actively texting to people that are cycling back. There’s a lot of messages that are going on that are multiply social. It’s not a single synchronous sociability in mobile. It’s multiple.

We call that PolySocial Reality. PolySocial Reality, or POSR, is the idea that we have this system that we’re currently in that everybody’s in, where each message is counted. If we just take for granted every single message—we know people are sending lots of them. We know their machines are sending lots of them. We know that they are showing up in synchronous messages that we’re connecting in real time and we’re doing messages asynchronously, we’re sending messages that people may read and respond to later. If you take the whole universe of messages, that’s what we’re calling PolySocial Reality.

And POSR has a really complex interactive environment. This is one photo of one group using devices. And if you think about all the multiple messaging going on just in that snapshot and you take the aggregate of all the people doing these things, you can see that it grows. And it becomes a very complex problem.

Just to recap, PolySocial Reality describes the aggregate of multiplexed, multiple, synchronous or asynchronous individual data creations. And they can be things just we speak or the things our devices are generating or the things that we generate. And it has outcomes. And some of the outcomes are great. We can expand our networks, we expand our connections. The people that we can socialize with are maybe around the globe and we’re very close to them instantly. It’s great.

But it also creates these individual experiences that fragment us in the local locale. And asynchronicity has become this adaptive strategy. People use asynchronicity to change the way they’re managing their time. People may defer to late at night to write messages when they know people won’t write back. Or they may have so many messages that they just pitch it and wait and they think, oh, well, the most important people, they’ll write back. And that may or may not happen. If our messages that are asynchronous don’t overlap enough, we don’t get that communication. If we don’t get communication, we can’t have cooperation, because we’re no longer sharing a story or a narrative. It’s really critical that we actually do connect with our messages. And because of PolySocial Reality and all these messages and all this asynchronicity and synchronicity happening at the same time, stories can be fragmented.

We’re going to have soon to be a more complex environment. It’s going to get worse. Or better, depending. People are going to have another layer of complexity on top of the PolySocial Reality we’re already experiencing. We’re going to have AR, another layer on top of our messaging, as we’re starting to have. And we’re also going to have the IoT and other things that are generating more messages for us.

We look at heterogeneity and sociability. We’re looking at heterogeneity as a complexity. Dissimilar, diverse, complicated, mixed. We look at AR hardware. Not everyone’s going to have the same kind. Some people might have head-mounted, people might be looking through phones. It’s going to be different. People have different apps. They’re in different locations. There’s different network structures. People are different genders, different cultures, different ethnicities, and they’re using different devices, different apps. There’s a lot of heterogeneity. And when we talk about designing for AR, we’re designing heterogeneity and sociability.

And sociability’s really difficult if the AR in devices is only being carried by you and those around you, not everyone. The fact that there is a lot of heterogeneity is a critical design problem.

When we look at social augmented reality, we’re really looking at stories. And stories are narratives. And narratives are how people represent and control information about ourselves, how we do that, and how we control information about people—we tell stories. And it contributes to our orientation of ourselves in the world at that time, by making it a narrative, even a small narrative, of how we understand the world. It’s how we develop our own experience and share that with others.

How do we get to social augmented reality? How do we make it social? We can create shared experiences around a story and narrative. A lot of the talks today talked about this. You can do it through sensors in an active AR environment, games with multiplayer components, projection that enables AR without a sealed headset, if you open up that headset, you can be social with people. This is an intelligent AR environment where the signup dates as the bus moves through town, the sensors trigger and updates the sign. Sorry. There you go. This is more of a could be called an augmentation, it could be a smart environment or an IoT, but people have the ability to go around the sign and talk and share with their heads up. They’re not down, they’re not fragmented. They can still be in the same locale and social.

We look at castAR, what castAR has to offer. It’s a contained environment. It has that retro-reflective mat with glasses, but the glasses are open and you can see through them. In castAR’s model, the AR is projected out, the projection is out, not on your eyes, so that you can actually experience the AR and turn your head and have a conversation with someone and then go right back to it. It’s really great how they do that. And you can also be social while you use castAR and share in that way. But the glasses are also semi-transparent so you could see who you’re talking to as you do it, or who you’re playing with, and have conversations. It’s not immersive. It’s not cut off. It’s social.

And this is a sample as to what you look like, what you’re seeing as you’re looking through those glasses. It’s a shared AR environment, multiple people can play.

An interactive public environment can also do this. It has, this particular example is a place where people can gather and interact with something. The bus sign we saw was projected one way. But in this particular case, you can actually, the people can interact and see results on their own.

My very favorite is the Magic Bus. I love the Magic Bus and I think the Magic Bus is a really great example of a shared AR story. The Magic Bus is a retrofitted school bus in San Francisco and it takes people on a tour of San Francisco. And as people ride in the Magic Bus, it plays videos and stories and music of the period that they are trying to illustrate on the way to the point. It plays the story on the pathway. As you’re going, it’s telling you something and then, when you get there, the shades go up and then you see where you’ve arrived to. The narrative is in the pathway with Magic Bus.

We want to go beyond Magic Bus. People make pathways as they move from point to point. Right now, AR’s trigger-based in terms of points. But stories go beyond that. Stories are about, well, I was going to this place and all this stuff happened. We’re suggesting, how do we create—what we’re interested in having people think about is how do you create pathway-based, multi-point AR? The goal is to get more information to form that storyline. What’s a narrative accompaniment to AR? What’s the soundtrack of AR? What are triggers that can be beyond point triggers, but more pathway triggers?

Blue Brain does site-specific sound sculptures. And what they’ve done is, in Washington, D.C. at the National Mall and also in Central Park, they’ve set up sound sculptures that, as you go to different regions in an area within the park, you hear a soundtrack when you use their app. As you move this way, you’ll hear something or that way—you hear a soundtrack. And your soundtrack is your own narrative as you move through. And people move through differently, so it’s not scripted. People have different ways to do it, but they are making up a narrative composed of these many pathways.

We’re interested in social Blue Brain. Individually, extended AR is a great experience, but it’s not really sharable. How do you give people ways to share those multi-point pathway narratives, via social media, trusted network spaces? What are ways we can change the mode of navigation triggers from these location triggers to these composite pathways?

In conclusion, in talking about designing for sociability and cooperation, we’re interested in what you can design that provides hooks for shared experience, so we don’t have that street where everyone’s involved in AR, but they’re connected to people that are really far away. That’s great, but the local locale needs people to cooperate and socialize to be able to function. Explore playing with time. What are people about to do? What have they just experienced? How does that fit into your narrative? The goal is to connect people through a shared experience and this is going to produce that shared cooperation that we need and that better understanding which we need, which is going to support society.

Even though a composite pathway may seem like it’s just filled with all kinds of people’s experience, if enough people travel that and share that, it builds a society that shares a common history. And that’s something that we’re moving away from.

I always close with this slide and it’s one of my favorites. This is from Burning Man. And this is what I hope AR can be. Look at this cooperative structure. Look at how they had to all come together to make this. It’s colorful, it’s a representation of an awful lot of cooperation. People are enjoying it. They’re having a shared experience. And they’re also, as you can see by the lights, individually participating and participating together. And, in Burning Man, people do have pathway experiences to get to events and to think about the path as well as the point I think is the message I’d like to leave you with.

Thank you.

[applause]

Moderator We have time for a couple of questions for Sally. Are there any questions?

Question Mentioning the social piece of augmented reality, really when we’re talking about social, you’re talking about, you mentioned the narrative a few times. I’m wondering, with some of the challenges that I face is creating that dialogue, of putting out information and receiving it back and how you would encourage that with this program.

Sally Applin Could you clarify specifically a little bit more about…?

Question When the individuals that I work for are putting out information, let’s say in a social media environment, it’s really to … rather than just pushing out information, and just essentially receiving unscripted information back, it’s really generating that dialogue and then watching the circles grow from there. So I’m just wondering how that might apply here, if at all.

Sally Applin Sure, I think it’s… I think it could be similar. So we’re looking at, as you move through the physical world using AR, in particular, rather than being stationary, there are pathways that you follow to get from place to place. And as you are on that path from place to place, other people have traveled there and had experiences there and they may want to share something about that.

And it could be quite a glut if everyone just filled a pathway with everything. But you could have a pathway that’s filled way with your trusted network, their memories or shared experiences.

Okay, and then those, depending on, in a structure where people could do that—right now, media doesn’t, but where they could—those things could be shared again outside the trusted network.

Question Thank you.

Moderator Sally Applin, everyone.

Sally Applin Thanks!

End of audio