Nonverbal Communication in Social VR: Recent Academic Research

Gestures (like this peace sign) are an example of nonverbal communication (Photo by Dan Burton on Unsplash)

In the real world, much of our communication is non-verbal: facial expression, gaze, gestures, body movements, even spatial distance (proxemics).

While older, flat-screen virtual worlds such as Second Life are somewhat limited in the forms of nonverbal communication available (most people rely on text or voice chat), modern VR equipment and social VR platforms allow for more options:

  • Hand/finger movement: most VR headsets have hand controllers; the Valve Index has Knuckles hand controllers which allow you to move your fingers as well as your hands;
  • Body movement: the Vive pucks can be attached to your waist, hips, feet, and other parts of your body to track their movement in real time;
  • Eye movements/gaze: for example, the Vive Pro Eye VR headset can track the blinking and movement of the eyes;
  • Facial expression: add-ons such as the Vive Facial Tracker (which attaches to your VR headset) allow you to convey lower face and mouth movements on your avatar.

In addition, many social VR platforms also employ emoticons, which can be pulled up via a menu and displayed over the head of the avatar (e.g. the applause emoji in AltspaceVR), as well as full-body pre-recorded animations (e.g. doing a backflip in VRChat). The use of all these tools, in combination or alone, allows users in social VR to approach the level of non-verbal communication found in real life, provided they have the right equipment and are on a platform which supports that equipment (e.g. NeosVR, where you can combine all these into an avatar which faithfully mimics your facial and body movements).

Two recently published research papers investigate nonverbal communication on social VR platforms, adding to the growing academic literature on social VR. (I am happy to see that social VR is starting to become a topic of academic research!)


Maloney, D., Freeman, G., & Wohn, D. Y. (2020). “Talking without a Voice”: Understanding Non-Verbal Communication in Social Virtual Reality. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2). https://doi.org/10.1145/3415246

Unfortunately, there is no open-access version of this conference proceeding available; you’ll have to obtain a copy from your local academic or public library. This paper, by Divine Maloney and Guo Freeman of Clemson University and Donghee Yvette Wohn of the New Jersey Institute of Technology, consists of two parts:

  • conducting unobtrusive observations of 61 public events held in AltspaceVR over the span of four weeks, to see what non-verbal interactions were being used naturally on the platform; and
  • interviewing 30 users of social VR platforms (of which I was one!), where the paper’s authors read through the transcribed interview data to acquire a picture with regards how social VR users used, perceived, and experienced non-verbal communication for further analysis.

In the first study of the two, the authors noted the following different kinds of nonverbal communication:

  • the use of movement to indicate that someone was paying attention. These included nodding behaviors and moving the body or head toward the person or object that was subject of attention;
  • the use of applause to indicate approval;
  • pointing and patting one’s own chest as a form of directing attention either at a remote object/person or oneself;
  • and behaviours such as waving, dancing, and kissing, which were mostly used in social grooming contexts (dancing was also used as entertainment);
  • and finally, the behaviour of trolls: interpersonal provocation and social disruptions.

With respect to the thirty interviewed conducted, they were analyzed as follows to answer two research questions:

Using quotes from users’ own accounts, in this section we present our findings as two parts. First, to answer RQ2 (How do people perceive and understand non-verbal communication in social VR?), we identified three common themes that demonstrated how users perceive and understand non-verbal communication in social VR: as more immersive and embodied interactions for body language; as a similar form of communication to offline face-to-face interaction in terms of spatial behavior, hand behavior, and facial expressions; and as a natural way to initiate communication with online strangers.

Second, to answer RQ3 (How, if at all, does non-verbal communication affect interaction outcomes in social VR?), we described the social consequences of interacting through non-verbal communication in social VR for various user groups, including marginalized users such as cis women, trans women, and disabled users. We specially highlighted how non-verbal communication in social VR afforded privacy and social comfort as well as acted as a protection for marginalized users.

Unsurprisingly, the researchers discovered that most participants considered non-verbal communication to be a positive aspect in their social VR experience. Those surveyed highly praised body tracking (either just the hands and head, or ins ome cases the whole body), as it allowed for a more immersive and embodied form of non-verbal communication than those in traditional, flatscreen virtual worlds.

In addition to supporting more immersive and embodied interactions for body language, participants also considered non-verbal communication in social VR similar to offline face-to-face interaction in terms of spatial behavior, hand behavior, and facial expressions. This familiarity and naturalness greatly contributed to their generally positive perceptions.

Participants also viewed non-verbal communication in social VR as positive and effective because it became a less invasive way to start interactions with online strangers (e.g. waving hello at someone you’ve just met). Nonverbal communication also afforded some users a sense of privacy and social comfort, and in some cases, became an effective protection for them to avoid unwanted interactions, attention, and behaviors (especially with LGBTQ people and women).

The paper made three design recommendations for improved nonverbal communication in social VR platforms: providing support for facial tracking (which is already on its way with products like the Vive Facial Tracker); supporting more accurate hand and finger tracking (again, already underway with the Knuckles controllers for the Valve Index); and enabling alternative modes of control, especially for users with physical disabilities. While most of the study participants highly praised full body tracking in social VR, disabled users in fact complained about this feature and demanded alternatives.

The conference paper concludes:

Recently, commercial social VR applications have emerged as increasingly popular digital social spaces that afford more naturally embodied interaction. How do these novel systems shape the role of non-verbal communication in our online social lives? Our investigation has yielded three key findings. First, offline non-verbal communication modalities are being used in social VR and can simulate experiences that are similar to offline face-to-face interactions. Second, non-verbal communication in social VR is perceived overall positive. Third, non-verbal interactions affect social interaction consequences in social VR by providing privacy control, social comfort, and protection for marginalized users.


Tanenbaum, T. J., Hartoonian, N., & Bryan, J. (2020). “How do I make this thing smile?”: An Inventory of Expressive Nonverbal Communication in Commercial Social Virtual Reality Platforms. Conference on Human Factors in Computing Systems – Proceedings, 1–13. https://doi.org/10.1145/3313831.3376606

This paper is available free to all via Open Access. In this conference proceeding, Theresa Jean Tanenbaum, Nazely Hartoonian, and Jeffrey Bryan of the Transformative Play Lab at the Department of Informatics at the University of California, Irvine, did a study of ten social VR platforms:

  • VRChat
  • AltspaceVR
  • High Fidelity (which shut down in January of 2020)
  • Sansar
  • TheWave VR (this social VR platform shut down in early 2021)
  • vTime XR
  • Rec Room
  • Facebook Spaces (since shut down and replaced by Facebook Horizon)
  • Anyland
  • EmbodyMe

For each platform, investigators answered the following eight questions:

  1. Can the user control facial expressions, and if so, how? (Pre-baked emotes, puppeteering, etc.)
  2. Can the user control body language, and if so, how? (Pre-baked emotes, puppeteering, postures. etc.)
  3. Can the user control proxemic spacing (avatar position), and if so, how? (Teleport, hotspots, real world positioning, etc.) How is collision handled between avatars? (Do they overlap, push each other, etc.)
  4. How is voice communication handled? Is audio spatialized, do lips move, is there a speaker indicator, etc.
  5. How is eye fixation/gaze handled? (Do avatars lock and maintain gaze, is targeting gaze automatic, or intentional, or some sort of hybrid, do eyes blink, saccade, etc.)
  6. Are different emotions/moods/affects supported, and how are they implemented? (Are different affective states possible, and do they combine with other nonverbal communications, etc.)
  7. Can avatars interact physically, and if so, how? (Hugging, holding hands, dancing, etc.) What degree of negotiation/consent is needed for multi- avatar interactions? (One-party, two-party, none at all?)
  8. Are there any other kinds of nonverbal communication possible in the system that have not be described in the answers to the above questions?

The results were a rather complete inventory of nonverbal communication in social VR, with the goal to catalogue common design elements for avatar expression and identify gaps and opportunities for future design innovation. Here is the table from the paper (which can be viewed in full size at the top of page 6 of the document).

An inventory of non-verbal communication in ten social VR platforms (source)

VR development is proliferating rapidly, but very few interaction design strategies have become standardized…

We view this inventory as a first step towards establishing a more comprehensive guide to the commercial design space of NVC [non-verbal communication] in VR. As a design tool this has two immediate implications for designers. First, it provides a menu of common (and less common) design strategies, and their variations, from which designers may choose when determining how to approach supporting any given kind of NVC within their platform. Second, it calls attention to a set of important social signals and NVC elements that designers must take into consideration when designing for Social VR. By grounding this data in the most commonly used commercial systems, our framework can help designers anticipate the likelihood that a potential user will be acquainted with a given interaction schema, so that they may provide appropriate guidance and support.

Our dataset also highlights some surprising gaps within the current feature space for expressive NVC. While much social signaling relies upon control of facial expression, we found that the designed affordances for this aspect of NVC to be mired in interaction paradigms inherited from virtual worlds. Facial expression control is often hidden within multiple layers of menus (as in the case of vTime), cannot be isolated from more complex emotes (as in the case of VR Chat), hidden behind opaque controller movement (as in Facebook Spaces), or unsupported entirely. In particular, we found that with the exception of dynamic lip-sync, there were no systems with a design that would allow a user to directly control the face of their avatar through a range of emotions while simultaneously engaging in other forms of socialization.

The authors go on to say that they observed no capacity in any of the systems to recombine and blend the various forms of nonverbal communication, such as can be done in the real world:

As we saw in our consideration of the foundations of NVC in general, and Laban Movement Analysis in particular, much NVC operates by layering together multiple social signals that modify, contextualize, and reinforce other social signals. Consider, for instance, that it is possible to smile regretfully, laugh maliciously, and weep with joy. People are capable of using their posture to
indicate excitement, hesitation, protectiveness, and many other emotional states, all while performing more overt discourse acts that inherit meaning from the gestalt of the communicative context.

The conference paper concludes:

As is evident in the scholarly work around social VR, improving the design space for NVC in VR has the potential to facilitate deeper social connection between people in virtual reality. We also argue that certain kinds of participatory entertainment such as virtual performance will benefit greatly from a more robust interaction design space for emotional expression through digital avatars. We’ve identified both common and obscure design strategies for NVC in VR, including design conventions for movement and proxemic spacing, facial control, gesture and posture, and several strategies unique to avatar mediated socialization online. Drawing on previous literature around NVC in virtual worlds, we have identified some significant challenges and opportunities for designers and scholars concerned with the future of socialization in virtual environments. Specifically, we identify facial expression control, and unconscious body posture as two critical social signals that are currently poorly supported within today’s commercial social VR platforms.

It is interesting to note that both papers cite the need to properly convey facial expressions as key to expanding the ability of avatars in social VR to convey non-verbal communication!

vTime XR Has Been Downloaded One Million Times Since Its Launch in 2015

There’s an interesting article about the Liverpool-based social VR company vTime XR, which appeared last October on the British website BusinessLive (I guess I must have missed it!).

Titled Inside Liverpool firm vTime – the tech business behind world’s first cross-reality social network, the article provides some statistics on how popular vTime XR has proven to be since its launch in 2015:

…earlier this year, an update to the app meant a name change from vTime to vTime XR. That saw it become the world’s first cross-reality social experience – meaning users can meet, chat and share with others in AR, VR or 2D mode.

With VR, the app’s users stay fully-immersed inside the locations using a headset, while AR mode allows them to place a live, 360-degree model of the destination on any real-world, flat surface.

That creates a shared virtual space by using VR headsets or a smartphone, with the app allowing any users to meet – regardless of which mode they are using.

The product has been downloaded around 1 million times, with users in 190 countries across the world, and [the software] available in eight formats.

vTime XR is an example of a simple but well-done single-purpose VR platform. Its sole purpose is to allow you to connect and converse with your friends, coworkers—even complete strangers!—in groups of up to four people at a time. It has deliberately focused on that core use (conversation), adding support for new platforms over time. Because it is so easily rendered, it can be used by just about anybody: people on mobile devices, cellphone-based VR, all the way up to the Oculus Rift VR headset. (Surprisingly, it’s not available for the Oculus Quest yet.)

Congratulations to vTime XR for reaching the significant milestone of one million downloads!

Social VR App vTime Relaunches as vTime XR, Now Supports both Virtual Reality and “Augmented Reality”

The app formerly known as vTime is now called vTime XR, and (in a bit of surprising news) it now offers a so-called “augmented reality” mode. I don’t really consider it true augmented reality, since it doesn’t work with an AR headset like the Magic Leap One or the Microsoft Hololens. (It’s more like Pokémon Go, where you’re looking through your cellphone.)

vTime XR also supports a variety of VR headsets and 2D/flatscreen viewing, which the company whimsically calls “Magic Window mode”.

Here’s the new promo video:

I notice they’ve also added a lot of new hand gestures that weren’t there before. The slickly-produced video almost obscures the fact that your avatar is still locked in place in one of four seats in all of the vTime environments (in this it is similar to Facebook Spaces, in that you are limited to a maximum of four avatars in one experience).

However, as the video demonstrates, vTime XR offers you a lot of different, beautiful scenes for you to meet up with your friends (or, far more likely, other people you don’t know who happen to be using vTime XR at the same time you are).

vTime XR is an interesting update to vTime, which you might want to take a look at. Or not, considering how much more you can get from other social VR platforms that actually let you move around freely.

However, vTime XR does support pretty much any computer hardware you have, including cellphone-based VR like Google Daydream:

UPDATED: Which Social VR Platforms and Virtual Worlds Will Benefit from the Upcoming Standalone VR Headset Oculus Quest?

#ff0000;">Did you know that you can help support my blog (as well as the upcoming Metaverse Newscast show), and get great rewards in return? #ff0000;" href="https://ryanschultz.com/2018/11/22/new-i-have-joined-patreon/" target="_blank" rel="noopener">Here’s how.


Oculus Quest.jpeg

As many of you already know, Oculus is releasing a new, standalone VR headset, the Oculus Quest, sometime this coming spring, 2019. Priced at just US$399, it is sure to be a popular option for people who are interested in VR, but who don’t want to purchase a more expensive VR headset solution like the HTC Vive and Oculus Rift.

The Oculus Rift is meant to fill the space in the Oculus product line-up between their entry-level, lower-powered standalone VR headset, the Oculus Go, and the Oculus Rift, a VR headset with Touch controllers which requires a high-end Windows gaming-level PC with a good graphics card to run. (Unfortunately, there is, as yet, no satisfactory native virtual reality hardware solution for Apple Mac users, although there are native Mac desktop clients for virtual worlds such as High Fidelity and Sinespace.)

Oculus Line.jpg

If the Oculus Quest becomes very popular, those social VR platforms which can run on the Quest hardware may gain an advantage over those which require a full-blown VR headset and a higher-end computer.

I think it’s safe to assume that Facebook/Oculus properties such as Facebook Spaces and Oculus Rooms (or at least some version of them) will be available for the Oculus Quest on its launch date. Social VR platforms with simpler avatars and spaces, which already run on the Oculus Go (like AltspaceVR, Bigscreen, and vTime) will probably also be available for the Quest.

Surprisingly, Rec Room, TheWaveVR, and VRChat are not among the social VR programs that are currently available for the Oculus Go ( I searched for them on the Oculus Go apps store and could not find any mention of them.) It remains to be seen if the companies behind those three products will release versions which will run on the more powerful Oculus Quest.

In a discussion thread over on the official High Fidelity user forums, HiFi CEO Philip Rosedale stated back in October:

We are definitely going to get High Fidelity running on as many standalone devices as we can, and we love the Quest. VR will not find a large audience until the Quest and other devices (like the Mirage and Vive Focus) become widely available.

Talking to Oculus about the process now… stay tuned.

When asked for to provide a more recent update, Philip added:

Yes, we are working on the Quest, and hope to have High Fidelity ready to run on it for launch! Very high quality device.

I also don’t know what Sinespace’s exact plans are for the Oculus Quest, but Adan Frisby, their lead developer, said on a Facebook comment when I cross-posted this blogpost over there:

We’ll be fine with it too – anyone doing Android support will have an easier time of it.

So it looks like High Fidelity and Sinespace will indeed both be working with the Oculus Quest, if not right at launch date, then shortly thereafter. This gives them both an advantage over Linden Lab’s Sansar, which very likely will not be able to work with the Quest. There’s still a lot of data that has to get sent to and from a VR headset to properly render Sansar experiences (especially for any experience which has global illumination enabled), which would probably completely overload any standalone headset.

As I often say: interesting times ahead! Let’s hope that the Oculus Quest makes a big splash and brings even more people into VR. A rising tide lifts all boats, and many social VR platforms would benefit from greater consumer awareness and uptake of virtual reality in general. And I promise to cover all of it as it happens on this blog!

rawpixel-651331-unsplash
Photo by rawpixel on Unsplash

UPDATE Dec. 14th: Adeon Writer posted the following to the VirtualVerse Discord server (VirtualVerse is the successor to the long-running SLUniverse forums):

VRChat was just announced for the Oculus Store. While it already worked with Oculus on Steam, [the] OculusSDK version of VRChat means it will almost certainly be ported to Oculus Quest when it comes out, making it the first metaverse-style game available for wireless/unteathered/portable VR.

Thanks, Adeon!

UPDATE Feb. 11th: Since this blogpost was written, I have had someone tell me the following about VRChat:

Sadly, I don’t think VRChat’s gonna support Quest. It’s just not compatible with mobile CPUs. Hell, it brings modern up-to-date PC’s to a standstill with too many people. I very much doubt the Snapdragon 835 can handle all the custom shaders, avatars, IK, etc. The team would basically need to do a full rewrite. And that’s unlikely unless the team was way bigger.

It does sound as though VRChat would have to be pared down significantly in order to run on the Oculus Quest, if at all.

I also noticed that I have received a lot of traffic to this blogpost due to this post on the OculusQuest subReddit (which I had never heard of before today). If anybody over there has any inside information on social VR/virtual worlds that will launch with the Quest, I’d certainly love to hear about it! Thanks.