Nonverbal Communication in Social VR: Recent Academic Research

Gestures (like this peace sign) are an example of nonverbal communication (Photo by Dan Burton on Unsplash)

In the real world, much of our communication is non-verbal: facial expression, gaze, gestures, body movements, even spatial distance (proxemics).

While older, flat-screen virtual worlds such as Second Life are somewhat limited in the forms of nonverbal communication available (most people rely on text or voice chat), modern VR equipment and social VR platforms allow for more options:

  • Hand/finger movement: most VR headsets have hand controllers; the Valve Index has Knuckles hand controllers which allow you to move your fingers as well as your hands;
  • Body movement: the Vive pucks can be attached to your waist, hips, feet, and other parts of your body to track their movement in real time;
  • Eye movements/gaze: for example, the Vive Pro Eye VR headset can track the blinking and movement of the eyes;
  • Facial expression: add-ons such as the Vive Facial Tracker (which attaches to your VR headset) allow you to convey lower face and mouth movements on your avatar.

In addition, many social VR platforms also employ emoticons, which can be pulled up via a menu and displayed over the head of the avatar (e.g. the applause emoji in AltspaceVR), as well as full-body pre-recorded animations (e.g. doing a backflip in VRChat). The use of all these tools, in combination or alone, allows users in social VR to approach the level of non-verbal communication found in real life, provided they have the right equipment and are on a platform which supports that equipment (e.g. NeosVR, where you can combine all these into an avatar which faithfully mimics your facial and body movements).

Two recently published research papers investigate nonverbal communication on social VR platforms, adding to the growing academic literature on social VR. (I am happy to see that social VR is starting to become a topic of academic research!)


Maloney, D., Freeman, G., & Wohn, D. Y. (2020). “Talking without a Voice”: Understanding Non-Verbal Communication in Social Virtual Reality. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2). https://doi.org/10.1145/3415246

Unfortunately, there is no open-access version of this conference proceeding available; you’ll have to obtain a copy from your local academic or public library. This paper, by Divine Maloney and Guo Freeman of Clemson University and Donghee Yvette Wohn of the New Jersey Institute of Technology, consists of two parts:

  • conducting unobtrusive observations of 61 public events held in AltspaceVR over the span of four weeks, to see what non-verbal interactions were being used naturally on the platform; and
  • interviewing 30 users of social VR platforms (of which I was one!), where the paper’s authors read through the transcribed interview data to acquire a picture with regards how social VR users used, perceived, and experienced non-verbal communication for further analysis.

In the first study of the two, the authors noted the following different kinds of nonverbal communication:

  • the use of movement to indicate that someone was paying attention. These included nodding behaviors and moving the body or head toward the person or object that was subject of attention;
  • the use of applause to indicate approval;
  • pointing and patting one’s own chest as a form of directing attention either at a remote object/person or oneself;
  • and behaviours such as waving, dancing, and kissing, which were mostly used in social grooming contexts (dancing was also used as entertainment);
  • and finally, the behaviour of trolls: interpersonal provocation and social disruptions.

With respect to the thirty interviewed conducted, they were analyzed as follows to answer two research questions:

Using quotes from users’ own accounts, in this section we present our findings as two parts. First, to answer RQ2 (How do people perceive and understand non-verbal communication in social VR?), we identified three common themes that demonstrated how users perceive and understand non-verbal communication in social VR: as more immersive and embodied interactions for body language; as a similar form of communication to offline face-to-face interaction in terms of spatial behavior, hand behavior, and facial expressions; and as a natural way to initiate communication with online strangers.

Second, to answer RQ3 (How, if at all, does non-verbal communication affect interaction outcomes in social VR?), we described the social consequences of interacting through non-verbal communication in social VR for various user groups, including marginalized users such as cis women, trans women, and disabled users. We specially highlighted how non-verbal communication in social VR afforded privacy and social comfort as well as acted as a protection for marginalized users.

Unsurprisingly, the researchers discovered that most participants considered non-verbal communication to be a positive aspect in their social VR experience. Those surveyed highly praised body tracking (either just the hands and head, or ins ome cases the whole body), as it allowed for a more immersive and embodied form of non-verbal communication than those in traditional, flatscreen virtual worlds.

In addition to supporting more immersive and embodied interactions for body language, participants also considered non-verbal communication in social VR similar to offline face-to-face interaction in terms of spatial behavior, hand behavior, and facial expressions. This familiarity and naturalness greatly contributed to their generally positive perceptions.

Participants also viewed non-verbal communication in social VR as positive and effective because it became a less invasive way to start interactions with online strangers (e.g. waving hello at someone you’ve just met). Nonverbal communication also afforded some users a sense of privacy and social comfort, and in some cases, became an effective protection for them to avoid unwanted interactions, attention, and behaviors (especially with LGBTQ people and women).

The paper made three design recommendations for improved nonverbal communication in social VR platforms: providing support for facial tracking (which is already on its way with products like the Vive Facial Tracker); supporting more accurate hand and finger tracking (again, already underway with the Knuckles controllers for the Valve Index); and enabling alternative modes of control, especially for users with physical disabilities. While most of the study participants highly praised full body tracking in social VR, disabled users in fact complained about this feature and demanded alternatives.

The conference paper concludes:

Recently, commercial social VR applications have emerged as increasingly popular digital social spaces that afford more naturally embodied interaction. How do these novel systems shape the role of non-verbal communication in our online social lives? Our investigation has yielded three key findings. First, offline non-verbal communication modalities are being used in social VR and can simulate experiences that are similar to offline face-to-face interactions. Second, non-verbal communication in social VR is perceived overall positive. Third, non-verbal interactions affect social interaction consequences in social VR by providing privacy control, social comfort, and protection for marginalized users.


Tanenbaum, T. J., Hartoonian, N., & Bryan, J. (2020). “How do I make this thing smile?”: An Inventory of Expressive Nonverbal Communication in Commercial Social Virtual Reality Platforms. Conference on Human Factors in Computing Systems – Proceedings, 1–13. https://doi.org/10.1145/3313831.3376606

This paper is available free to all via Open Access. In this conference proceeding, Theresa Jean Tanenbaum, Nazely Hartoonian, and Jeffrey Bryan of the Transformative Play Lab at the Department of Informatics at the University of California, Irvine, did a study of ten social VR platforms:

  • VRChat
  • AltspaceVR
  • High Fidelity (which shut down in January of 2020)
  • Sansar
  • TheWave VR (this social VR platform shut down in early 2021)
  • vTime XR
  • Rec Room
  • Facebook Spaces (since shut down and replaced by Facebook Horizon)
  • Anyland
  • EmbodyMe

For each platform, investigators answered the following eight questions:

  1. Can the user control facial expressions, and if so, how? (Pre-baked emotes, puppeteering, etc.)
  2. Can the user control body language, and if so, how? (Pre-baked emotes, puppeteering, postures. etc.)
  3. Can the user control proxemic spacing (avatar position), and if so, how? (Teleport, hotspots, real world positioning, etc.) How is collision handled between avatars? (Do they overlap, push each other, etc.)
  4. How is voice communication handled? Is audio spatialized, do lips move, is there a speaker indicator, etc.
  5. How is eye fixation/gaze handled? (Do avatars lock and maintain gaze, is targeting gaze automatic, or intentional, or some sort of hybrid, do eyes blink, saccade, etc.)
  6. Are different emotions/moods/affects supported, and how are they implemented? (Are different affective states possible, and do they combine with other nonverbal communications, etc.)
  7. Can avatars interact physically, and if so, how? (Hugging, holding hands, dancing, etc.) What degree of negotiation/consent is needed for multi- avatar interactions? (One-party, two-party, none at all?)
  8. Are there any other kinds of nonverbal communication possible in the system that have not be described in the answers to the above questions?

The results were a rather complete inventory of nonverbal communication in social VR, with the goal to catalogue common design elements for avatar expression and identify gaps and opportunities for future design innovation. Here is the table from the paper (which can be viewed in full size at the top of page 6 of the document).

An inventory of non-verbal communication in ten social VR platforms (source)

VR development is proliferating rapidly, but very few interaction design strategies have become standardized…

We view this inventory as a first step towards establishing a more comprehensive guide to the commercial design space of NVC [non-verbal communication] in VR. As a design tool this has two immediate implications for designers. First, it provides a menu of common (and less common) design strategies, and their variations, from which designers may choose when determining how to approach supporting any given kind of NVC within their platform. Second, it calls attention to a set of important social signals and NVC elements that designers must take into consideration when designing for Social VR. By grounding this data in the most commonly used commercial systems, our framework can help designers anticipate the likelihood that a potential user will be acquainted with a given interaction schema, so that they may provide appropriate guidance and support.

Our dataset also highlights some surprising gaps within the current feature space for expressive NVC. While much social signaling relies upon control of facial expression, we found that the designed affordances for this aspect of NVC to be mired in interaction paradigms inherited from virtual worlds. Facial expression control is often hidden within multiple layers of menus (as in the case of vTime), cannot be isolated from more complex emotes (as in the case of VR Chat), hidden behind opaque controller movement (as in Facebook Spaces), or unsupported entirely. In particular, we found that with the exception of dynamic lip-sync, there were no systems with a design that would allow a user to directly control the face of their avatar through a range of emotions while simultaneously engaging in other forms of socialization.

The authors go on to say that they observed no capacity in any of the systems to recombine and blend the various forms of nonverbal communication, such as can be done in the real world:

As we saw in our consideration of the foundations of NVC in general, and Laban Movement Analysis in particular, much NVC operates by layering together multiple social signals that modify, contextualize, and reinforce other social signals. Consider, for instance, that it is possible to smile regretfully, laugh maliciously, and weep with joy. People are capable of using their posture to
indicate excitement, hesitation, protectiveness, and many other emotional states, all while performing more overt discourse acts that inherit meaning from the gestalt of the communicative context.

The conference paper concludes:

As is evident in the scholarly work around social VR, improving the design space for NVC in VR has the potential to facilitate deeper social connection between people in virtual reality. We also argue that certain kinds of participatory entertainment such as virtual performance will benefit greatly from a more robust interaction design space for emotional expression through digital avatars. We’ve identified both common and obscure design strategies for NVC in VR, including design conventions for movement and proxemic spacing, facial control, gesture and posture, and several strategies unique to avatar mediated socialization online. Drawing on previous literature around NVC in virtual worlds, we have identified some significant challenges and opportunities for designers and scholars concerned with the future of socialization in virtual environments. Specifically, we identify facial expression control, and unconscious body posture as two critical social signals that are currently poorly supported within today’s commercial social VR platforms.

It is interesting to note that both papers cite the need to properly convey facial expressions as key to expanding the ability of avatars in social VR to convey non-verbal communication!

Editorial: Some Random Thoughts on the Morning After the Facebook Horizon Announcement

I don’t have a lot of time today, but I wanted to write up a few more thoughts I had about yesterday’s Facebook Horizon announcement.


First: Linden Lab knew something was up. The timing of their splashy revamp of Sansar (the day before the OC6 keynote) was no accident. They wanted to get some media air time before Facebook came in and sucked all the oxygen out of the room! The new promotional video for Sansar is slick, savvy, and obviously designed to entice the curious newcomer:

The big Sansar news on Sept. 24th was the announcement of partnerships with major corporations including Sanrio, Levi’s, and Spinnin’ Records. But there was also a redo of the Sanar logo (switched from red to aqua blue), and some slick new promotional imagery designed to appeal to gamers:

Agent Primus and Agent Forma: Your world is waiting!

Now, many of the features VentureBeat touted about Sansar in their article about the relaunch were not exactly news to long-time Sansarians, or to avid readers of my blog: the new Avatar 2.0 avatars with facial deformation features, the Nexus, etc. But we oldbies are not the target audience here; Linden Lab is now pulling out all the stops to get maximum marketshare and mindshare in a social VR marketplace that suddenly got a lot more competitive. It’s a smart move.

They’ve also got lots of branded events coming up in the next little while to entice people to come visit.

I think it is probably safe to say that many other social VR platforms and virtual worlds were completely caught off guard by yesterday’s detailed announcement of Facebook Horizon, and they will need to take some time to adjust to the new reality, the “new normal”.


Now, back to the main event: Facebook Horizon.

Screen capture from Mark Zuckerberg’s OC6 Keynote Yesterday

You will remember that I left Facebook and Instagram as my New Year’s resolution at the end of 2018, in protest over what I saw as a number of very serious data security and privacy issues with the social network:

I am fed up. I have had enough. And I am fighting back the only way I can: by shutting down my Facebook and Instagram accounts and deleting all of the data that Facebook has gathered on me.

Well, last night I came crawling back to Facebook. But the company did indeed keep its promise: it did not offer to reconnect me with 13 years of data it had on me, which I had asked them to delete. However, Facebook still knows it’s me: many of the initial friend suggestions it made were people who I had been connected to on the previous incarnation of my account. (I politely declined all of them. I am doing things totally differently this time around.)

As it happens, I have used the same email address for both my Oculus hardware account and for my deleted-and-now-reinstated Facebook social network account. I have no idea if Facebook is going to keep those two accounts separate, or try to merge them sometime in the future. In fact, there’s still no concrete evidence to support the thesis that you have to have an account on the Facebook social network in order to use Horizon.

So now I have a shiny new empty Facebook account, but I am approaching this fully forearmed with the knowledge that Facebook will strip mine the hell out of any data I provide, as well as the knowledge that my data can (and in fact, already has been) weaponized by companies such as Cambridge Analytica and used against me.

I am willing to come back, but I am damn wary. And I have essentially locked down everything I can using Facebook’s own security and privacy settings, as well as installing and setting up the excellent F.B. (Fluff Buster) Purity web browser extension. Please note that Facebook does not like F.B. Purity, and will actively block any mention of F.B. Purity on its social network; I only learned about this tool through word of mouth, from other Facebook users. You can read more about it via the link I posted above.

Also, I have registered for the closed beta test of the new Facebook Horizon platform, which starts sometime in early 2020. The application process asked for my Oculus account information I provided when I first purchased and set up my Oculus Rift headset in January 2017, followed by the purchase and set up of my Oculus Quest in May 2019. (I have read on Reddit that Facebook will accept either a Facebook social network account or an Oculus account, but I did not see any option to enter the former, only the latter. Perhaps I missed something.)

All the short registration form asked me for was my gender (male, female, or something custom), what experience I had building content for social VR and virtual worlds (and what tools I used), and whether I lead, moderate, or administer an online community (such as Reddit, Facebook Groups, Discord, Twitch, etc.). I did tell them that I was an influential blogger who writes a blog about social VR, virtual worlds, and the metaverse, which gets anywhere between 600 and 6,000 views per day. (I forgot to tell them that I also have a popular Discord discussion forum associated with my blog. Oh well.)

The worst that can happen is that Facebook decides I am not worthy to enter the closed beta test, in which case I will need to examine my options. Also, Facebook may ask beta testers to sign an NDA (non-disclosure agreement), which is fairly common in these sorts of cases. For example, even though I was accepted into the Sansar closed alpha/beta testing back in December of 2016, I was not allowed to blog about anything I saw in-world until the open beta launch on July 31, 2019—and I also had to receive explicit permission from Linden Lab to post pictures taken before that date on my blog, as a sort of history of Sansar’s early development. We may face the same situation with Facebook Horizon. We’ll see. It’s still very early days.


It wasn’t until late last night that I realized several unusual things about the veritable flurry of announcements made at OC6.

First: Facebook still does not have a single social VR platform to bring together Oculus Go, Oculus Quest, and Oculus Rift users! Facebook Horizon is only intended for the Oculus Rift and the Oculus Quest. What this means for Oculus Go users is unclear.

Second, Facebook has announced that they will be shutting down both Facebook Spaces and Oculus Rooms on October 25th, 2019, presumably to free up staff who will be deployed to work on Facebook Horizon.

I will not be sorry to see Facebook Spaces go; as I have said before on this blog when making predictions in December 2018:

…we might yet see the launch of a new social VR platform backed by Facebook, after they decide to ditch the lamentable Facebook Spaces once and for all. Maybe it will be based on Oculus Rooms; maybe it will be something completely different. But despite my negative feelings about the social networking side of Facebook, they still have the hardware (Oculus), the money, and the reach to be a game-changer in social VR. (Just not with Facebook Spaces. At this point, they should just kill the project and start over. Any improvements will be like putting lipstick on a pig.)

Yep. Take that mutt out back behind the woodshed and put it out of its misery… I really doubt anybody was using Facebook Spaces anyway, despite courageous efforts by people such as Navah Berg to promote it. And I’m quite sure that the entire episode was instructive to Facebook about what works and doesn’t work in a social VR platform.

And Oculus Rooms was only for Gear VR and the Oculus Go, which means that they will have no Facebook-branded social VR platform at all. This is, of course, an opportunity to other, third-party platforms which support Gear VR and Oculus Go, such as AltspaceVR, Rec Room, Bigscreen, and vTime XR.

Interesting times ahead!

Does Facebook/Oculus Need to Get Its Act Together On Social VR? A Thread on Reddit

As most of you already know, I quit Facebook as my New Year’s resolution, and I am still quite happy with my decision.

One of the places where I have spent more time since leaving Facebook has been Reddit, which is home to many thriving communities such as the Oculus subReddit, with over 140,000 subscribers who discuss and debate various issues related to Oculus VR hardware and software. (Sometimes I cross-post one of my blogposts there.)

Mark Zuckerberg presents the Oculus lineup (image taken from The Ghost Howls blog)

Yesterday, someone posted a lengthy item titled Opinion: Oculus needs to get its sh*t together in social, which I have only quoted in part (so please go over there and read the whole thing):

First, it’s ridiculous that cross-communication between the Go and Rift communities is so difficult, especially with Quest coming. VR social presence should be an underlying infrastructure that the whole Oculus ecosystem can plug into… Oculus: don’t split the VR community simply because of which of your own platforms they buy. Make communication easy, automatic, and built in.

Second, where the hell is the metaverse/Oasis/shared world? I know, VR Chat, AltspaceVR, Rec Room, High Fidelity, etc. …but why has Oculus–let alone Facebook!–abdicated this whole sector to third parties?

Now, this poster has raised a valid point. Why is social VR across all the Oculus hardware such a disorganized mess? Currently, Facebook offers Oculus Rooms for the Oculus Go users and Facebook Spaces for Oculus Rift users. Is Facebook going to use the release of the Oculus Quest (which I predict will be popular) to try and clean up this situation?

Some of the comments to this poster are worth quoting (again, I am going to take it easy with the quoting, taking into account the feedback I have recently received). I did go in and write a lengthy response, outlining the situation as I see it, and pointing out that there were already many social VR platforms which have been in development for several years (Sansar, High Fidelity, Sinespace, etc.).

When I said that High Fidelity and VRChat were planning to support the upcoming standalone Oculus Quest VR headset, one person responded:

Sadly, I don’t think VRChat’s gonna support Quest. It’s just not compatible with mobile CPUs. Hell, it brings modern up-to-date PC’s to a standstill with too many people. I very much doubt the Snapdragon 835 can handle all the custom shaders, avatars, IK, etc. The team would basically need to do a full rewrite. And that’s unlikely unless the team was way bigger.

Someone else said:

Do you really WANT Facebook also own the social VR “metaverse”? That seems like a really terrible idea in general to have a monopoly control so many things. Especially one that has proven dozens of times that they could care less about our privacy in respect to their profits…

I for one HOPE Oculus fails WILDLY on social VR. The alternative would be the worst-case dystopian future for VR and likely disasterous for humanity as a whole in the long-term.

To which I say, Amen and Hallelujah. However, Facebook has the deep pockets (lots of money) and the hardware (Oculus) to totally up-end the current, nascent social VR market, if they finally get their act together and choose to do so.

And finally, one person said:

To me it comes down to this—people want Second Life in VR format. It’s true. A metaverse where you can build or be anything you want…but in VR. We know that SL’s infrastructure cannot be upgraded to do that. High Fidelity, even with the founder of SL, isn’t cutting it. And neither is Sansar, Linden Lab’s actual VR offering.

I believe there are two main issues impeding them. 1. Instances – every platform nowadays does instance based “world” creation. Worlds are not permanent, player join numbers are limited, and the worlds are not visually connected to each other. People want permanence! To be able to wander from one place to the next aimlessly. This isn’t just an issue limited to VR of course (hello WoW). The reasoning behind this is that it’s much easier on the server hardware. Personally, I’d deal with some lag to be able to participate in a true open world environment. And

2. Adult content – All of the platforms are scared as f*** here. They don’t want their brand to be ‘marked’ by that, they don’t want to have to figure out how to police it, etc etc. That stuff is not going away. Whether they admit it or not, adult content has kept Second Life alive and thriving for 15 years (they still have ~50k concurrent users). Bottom line—deal with it. Embrace it. It’s going to happen with or without you.

This wide-ranging and fascinating discussion is the kind of thing that Reddit has become well-known for, and I would encourage you to go over there and read all or most of it for yourself, and perhaps add your own comments.

Editorial: My Social VR/Virtual World Predictions for 2019

Have you joined the RyanSchultz.com Discord yet? Come join 170 avid users of various metaverse platforms, and discuss social VR and virtual world predictions for 2019! More details here


wyron-a-561710-unsplash

Time to peer into that crystal ball and make some predictions!

First: Second Life is going to continue to coast along, baffling the mainstream news media and the general public with its vitality and longevity. It will continue to be a reliable cash cow for Linden Lab as they put a portion of that profit into building Sansar. And I also predict that the ability to change your first and last names in SL will prove very popular—and also very lucrative for Linden Lab! Remember, they’ve got seven years of pent-up demand for this feature. (I have a couple of avatars myself that I’d like to rename.)

Second: An unexpected but potentially ground-breaking development in OpenSim was the announcement of the release of a virtual reality OpenSim viewer to the open source community at the 2018 OpenSim Community Conference. There’s still lots of technical work left to do, but if they can successfully pull this off, it could mean a new era for OpenSim.

Third: I confidently predict that one or more blockchain-based virtual worlds are going to fold. Not Decentraland; there’s too much money tied up in that one to fail. But several cryptocurrency-based virtual worlds are starting to look like trainwrecks of epic proportions (and I’m looking at you, Staramaba Spaces/Materia.One). Somebody still needs to explain to me why people will want to pay to hang out with 3D-scanned replicas of Paris Hilton and Hulk Hogan. The business model makes absolutely no sense to me. Another one that I think is going to struggle in 2019 is Mark Space.

Fourth: I also predict that one or more adult/sex-oriented virtual worlds are going to fail (yes, I’m looking at you, Oasis). I’ve already gone into the reasons why even the best of them are going to find it hard to compete against the entrenched front-runner, Second Life.

Fifth: High Fidelity and Sansar will continue their friendly rivalry as both social VR platforms hold splashy events in the new year. (I’m really sorry I missed the recent preview of Queen Nefertari’s tomb in HiFi, but it looks as though there will be many other such opportunities in 2019.) And High Fidelity will continue to boast of new records in avatar capacity at well-attended events (it certainly helps that they’ve got those venture-capital dollars to spend, to offer monetary enticements for users to pile on for stress testing).

Sixth: the Oculus Quest VR headset will ignite the long-awaited boom in virtual reality that the analysts have been predicting for years. There; I’ve said it! And those social VR platforms which support Oculus Quest users will benefit.

Seventh: Linden Lab’s launch of Sansar on Steam will likely have only a modest impact on overall usage of the platform. I’m truly sorry to have to write this prediction, because I love Sansar, but we’ve got statistics we can check, and they are not looking terribly encouraging at the moment. And where is the “significant ad spend” that was promised at one of the in-world product meetups back in November? Now that they’ve pulled the trigger and launched on Steam, it’s time to promote the hell out of Sansar, using every means at Linden Lab’s disposal. Paying bounties to Twitch livestreamers is not enough.

And Facebook? If they thought 2018 was a bad year, I predict that we’re going to see even more scandals uncovered in 2019 by news organizations such as the New York Times. And more people (like me) will decide that they’ve had enough of being sold to other corporations and data-mined to within an inch of their lives, and jump ship. The public relations people at Facebook are going to face a lot of sleepless nights…

And, still on the same topic, we might yet see the launch of a new social VR platform backed by Facebook, after they decide to ditch the lamentable Facebook Spaces once and for all. Maybe it will be based on Oculus Rooms; maybe it will be something completely different. But despite my negative feelings about the social networking side of Facebook, they still have the hardware (Oculus), the money, and the reach to be a game-changer in social VR. (Just not with Facebook Spaces. At this point, they should just kill the project and start over. Any improvements will be like putting lipstick on a pig.)

Finally, I predict that the RyanSchultz.com blog will head off into new and rather unexpected directions (that is, if the past 12 months’ activity is any indication!). I never expected to cover blockchain-based virtual worlds, or Second Life freebies; they just kind of happened.  Expect more of the same in 2019, as various new topics catch my interest.