Nonverbal Communication in Social VR: Recent Academic Research

Gestures (like this peace sign) are an example of nonverbal communication (Photo by Dan Burton on Unsplash)

In the real world, much of our communication is non-verbal: facial expression, gaze, gestures, body movements, even spatial distance (proxemics).

While older, flat-screen virtual worlds such as Second Life are somewhat limited in the forms of nonverbal communication available (most people rely on text or voice chat), modern VR equipment and social VR platforms allow for more options:

  • Hand/finger movement: most VR headsets have hand controllers; the Valve Index has Knuckles hand controllers which allow you to move your fingers as well as your hands;
  • Body movement: the Vive pucks can be attached to your waist, hips, feet, and other parts of your body to track their movement in real time;
  • Eye movements/gaze: for example, the Vive Pro Eye VR headset can track the blinking and movement of the eyes;
  • Facial expression: add-ons such as the Vive Facial Tracker (which attaches to your VR headset) allow you to convey lower face and mouth movements on your avatar.

In addition, many social VR platforms also employ emoticons, which can be pulled up via a menu and displayed over the head of the avatar (e.g. the applause emoji in AltspaceVR), as well as full-body pre-recorded animations (e.g. doing a backflip in VRChat). The use of all these tools, in combination or alone, allows users in social VR to approach the level of non-verbal communication found in real life, provided they have the right equipment and are on a platform which supports that equipment (e.g. NeosVR, where you can combine all these into an avatar which faithfully mimics your facial and body movements).

Two recently published research papers investigate nonverbal communication on social VR platforms, adding to the growing academic literature on social VR. (I am happy to see that social VR is starting to become a topic of academic research!)


Maloney, D., Freeman, G., & Wohn, D. Y. (2020). “Talking without a Voice”: Understanding Non-Verbal Communication in Social Virtual Reality. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2). https://doi.org/10.1145/3415246

Unfortunately, there is no open-access version of this conference proceeding available; you’ll have to obtain a copy from your local academic or public library. This paper, by Divine Maloney and Guo Freeman of Clemson University and Donghee Yvette Wohn of the New Jersey Institute of Technology, consists of two parts:

  • conducting unobtrusive observations of 61 public events held in AltspaceVR over the span of four weeks, to see what non-verbal interactions were being used naturally on the platform; and
  • interviewing 30 users of social VR platforms (of which I was one!), where the paper’s authors read through the transcribed interview data to acquire a picture with regards how social VR users used, perceived, and experienced non-verbal communication for further analysis.

In the first study of the two, the authors noted the following different kinds of nonverbal communication:

  • the use of movement to indicate that someone was paying attention. These included nodding behaviors and moving the body or head toward the person or object that was subject of attention;
  • the use of applause to indicate approval;
  • pointing and patting one’s own chest as a form of directing attention either at a remote object/person or oneself;
  • and behaviours such as waving, dancing, and kissing, which were mostly used in social grooming contexts (dancing was also used as entertainment);
  • and finally, the behaviour of trolls: interpersonal provocation and social disruptions.

With respect to the thirty interviewed conducted, they were analyzed as follows to answer two research questions:

Using quotes from users’ own accounts, in this section we present our findings as two parts. First, to answer RQ2 (How do people perceive and understand non-verbal communication in social VR?), we identified three common themes that demonstrated how users perceive and understand non-verbal communication in social VR: as more immersive and embodied interactions for body language; as a similar form of communication to offline face-to-face interaction in terms of spatial behavior, hand behavior, and facial expressions; and as a natural way to initiate communication with online strangers.

Second, to answer RQ3 (How, if at all, does non-verbal communication affect interaction outcomes in social VR?), we described the social consequences of interacting through non-verbal communication in social VR for various user groups, including marginalized users such as cis women, trans women, and disabled users. We specially highlighted how non-verbal communication in social VR afforded privacy and social comfort as well as acted as a protection for marginalized users.

Unsurprisingly, the researchers discovered that most participants considered non-verbal communication to be a positive aspect in their social VR experience. Those surveyed highly praised body tracking (either just the hands and head, or ins ome cases the whole body), as it allowed for a more immersive and embodied form of non-verbal communication than those in traditional, flatscreen virtual worlds.

In addition to supporting more immersive and embodied interactions for body language, participants also considered non-verbal communication in social VR similar to offline face-to-face interaction in terms of spatial behavior, hand behavior, and facial expressions. This familiarity and naturalness greatly contributed to their generally positive perceptions.

Participants also viewed non-verbal communication in social VR as positive and effective because it became a less invasive way to start interactions with online strangers (e.g. waving hello at someone you’ve just met). Nonverbal communication also afforded some users a sense of privacy and social comfort, and in some cases, became an effective protection for them to avoid unwanted interactions, attention, and behaviors (especially with LGBTQ people and women).

The paper made three design recommendations for improved nonverbal communication in social VR platforms: providing support for facial tracking (which is already on its way with products like the Vive Facial Tracker); supporting more accurate hand and finger tracking (again, already underway with the Knuckles controllers for the Valve Index); and enabling alternative modes of control, especially for users with physical disabilities. While most of the study participants highly praised full body tracking in social VR, disabled users in fact complained about this feature and demanded alternatives.

The conference paper concludes:

Recently, commercial social VR applications have emerged as increasingly popular digital social spaces that afford more naturally embodied interaction. How do these novel systems shape the role of non-verbal communication in our online social lives? Our investigation has yielded three key findings. First, offline non-verbal communication modalities are being used in social VR and can simulate experiences that are similar to offline face-to-face interactions. Second, non-verbal communication in social VR is perceived overall positive. Third, non-verbal interactions affect social interaction consequences in social VR by providing privacy control, social comfort, and protection for marginalized users.


Tanenbaum, T. J., Hartoonian, N., & Bryan, J. (2020). “How do I make this thing smile?”: An Inventory of Expressive Nonverbal Communication in Commercial Social Virtual Reality Platforms. Conference on Human Factors in Computing Systems – Proceedings, 1–13. https://doi.org/10.1145/3313831.3376606

This paper is available free to all via Open Access. In this conference proceeding, Theresa Jean Tanenbaum, Nazely Hartoonian, and Jeffrey Bryan of the Transformative Play Lab at the Department of Informatics at the University of California, Irvine, did a study of ten social VR platforms:

  • VRChat
  • AltspaceVR
  • High Fidelity (which shut down in January of 2020)
  • Sansar
  • TheWave VR (this social VR platform shut down in early 2021)
  • vTime XR
  • Rec Room
  • Facebook Spaces (since shut down and replaced by Facebook Horizon)
  • Anyland
  • EmbodyMe

For each platform, investigators answered the following eight questions:

  1. Can the user control facial expressions, and if so, how? (Pre-baked emotes, puppeteering, etc.)
  2. Can the user control body language, and if so, how? (Pre-baked emotes, puppeteering, postures. etc.)
  3. Can the user control proxemic spacing (avatar position), and if so, how? (Teleport, hotspots, real world positioning, etc.) How is collision handled between avatars? (Do they overlap, push each other, etc.)
  4. How is voice communication handled? Is audio spatialized, do lips move, is there a speaker indicator, etc.
  5. How is eye fixation/gaze handled? (Do avatars lock and maintain gaze, is targeting gaze automatic, or intentional, or some sort of hybrid, do eyes blink, saccade, etc.)
  6. Are different emotions/moods/affects supported, and how are they implemented? (Are different affective states possible, and do they combine with other nonverbal communications, etc.)
  7. Can avatars interact physically, and if so, how? (Hugging, holding hands, dancing, etc.) What degree of negotiation/consent is needed for multi- avatar interactions? (One-party, two-party, none at all?)
  8. Are there any other kinds of nonverbal communication possible in the system that have not be described in the answers to the above questions?

The results were a rather complete inventory of nonverbal communication in social VR, with the goal to catalogue common design elements for avatar expression and identify gaps and opportunities for future design innovation. Here is the table from the paper (which can be viewed in full size at the top of page 6 of the document).

An inventory of non-verbal communication in ten social VR platforms (source)

VR development is proliferating rapidly, but very few interaction design strategies have become standardized…

We view this inventory as a first step towards establishing a more comprehensive guide to the commercial design space of NVC [non-verbal communication] in VR. As a design tool this has two immediate implications for designers. First, it provides a menu of common (and less common) design strategies, and their variations, from which designers may choose when determining how to approach supporting any given kind of NVC within their platform. Second, it calls attention to a set of important social signals and NVC elements that designers must take into consideration when designing for Social VR. By grounding this data in the most commonly used commercial systems, our framework can help designers anticipate the likelihood that a potential user will be acquainted with a given interaction schema, so that they may provide appropriate guidance and support.

Our dataset also highlights some surprising gaps within the current feature space for expressive NVC. While much social signaling relies upon control of facial expression, we found that the designed affordances for this aspect of NVC to be mired in interaction paradigms inherited from virtual worlds. Facial expression control is often hidden within multiple layers of menus (as in the case of vTime), cannot be isolated from more complex emotes (as in the case of VR Chat), hidden behind opaque controller movement (as in Facebook Spaces), or unsupported entirely. In particular, we found that with the exception of dynamic lip-sync, there were no systems with a design that would allow a user to directly control the face of their avatar through a range of emotions while simultaneously engaging in other forms of socialization.

The authors go on to say that they observed no capacity in any of the systems to recombine and blend the various forms of nonverbal communication, such as can be done in the real world:

As we saw in our consideration of the foundations of NVC in general, and Laban Movement Analysis in particular, much NVC operates by layering together multiple social signals that modify, contextualize, and reinforce other social signals. Consider, for instance, that it is possible to smile regretfully, laugh maliciously, and weep with joy. People are capable of using their posture to
indicate excitement, hesitation, protectiveness, and many other emotional states, all while performing more overt discourse acts that inherit meaning from the gestalt of the communicative context.

The conference paper concludes:

As is evident in the scholarly work around social VR, improving the design space for NVC in VR has the potential to facilitate deeper social connection between people in virtual reality. We also argue that certain kinds of participatory entertainment such as virtual performance will benefit greatly from a more robust interaction design space for emotional expression through digital avatars. We’ve identified both common and obscure design strategies for NVC in VR, including design conventions for movement and proxemic spacing, facial control, gesture and posture, and several strategies unique to avatar mediated socialization online. Drawing on previous literature around NVC in virtual worlds, we have identified some significant challenges and opportunities for designers and scholars concerned with the future of socialization in virtual environments. Specifically, we identify facial expression control, and unconscious body posture as two critical social signals that are currently poorly supported within today’s commercial social VR platforms.

It is interesting to note that both papers cite the need to properly convey facial expressions as key to expanding the ability of avatars in social VR to convey non-verbal communication!

To Teleport or Not to Teleport: Teleporting Versus Walking in the Metaverse

Ever wish you could teleport in real life?
(Photo by Chris Briggs on Unsplash)

Earlier this week, I had a guided tour of the blockchain-based social VR platform Somnium Space, where I was informed by my tour guide that the virtual world had just implemented teleporting. Scattered throughout the one large, contiguous virtual landscape which comprises Somnium Space were teleporter hubs, where you could pull up a map, click on the teleporter hub you wanted to travel to, press a button, et voilà! You were instantly transported to your destination.

A teleporter hub in the central city square of Somnium Space (at night)
The red arrows indicate the location of teleporter hubs on the map

What makes Somnium Space unusual among metaverse platforms is that you cannot simply teleport from one place to another distant location; you either must make use of the provided teleporters, or walk/run/fly/swim to your destination. (Of course, you can certainly “short hop” using a limited form of teleporting, but that is only for shorter distances, not for instantly getting from one end of a large, contiguous landmass to another.)

In other words, the teleporter hubs of the Somnium Transportation System are set up much like a modern urban subway system, where you can only travel to a particular, pre-built subway station that is situated the nearest to your intended destination, and then walk the rest of the way. Many people might remember that in the very earliest days of Second Life, there were also teleporter hubs in the days before avatars could instantly teleport themselves from one location to another!

Another thing that sets Somnium Space apart from other social VR platforms is that there are only going to be so many “public” teleporter hubs. In face, some of these hubs are going to be auctioned off as NFTs (Non-Fungible Tokens), and the successful bidders with such a teleporter hub on their properties will be able to charge a cryptocurrency fee in order to use their teleporters! (In other words, they would operate much the same as a real-life toll road or highway.)

Closely intertwined with the idea of teleporting vs. walking is the layout of a metaverse platform. Is it one large contiguous landmass, like Somnium Space, Decentraland, Cryptovoxels, and (to a certain extent) Second Life? Or is it a collection of smaller worlds, like VRChat, Rec Room, Sansar, and Sinespace? If it is the former, then means of transportation (and ease of access to transportation) becomes more important. If it is the latter, then another tool which many of the newer social VR platforms offer is the ability to create a portal—either temporary or permanent— between two worlds. (Of course, you could consider a teleporter hub a portal.)

So, keeping all this in mind (particularly the distinction between SHORT HOP teleporting and teleporting to a DISTANT location), we can create a chart outlining the transportation affordances of the various metaverse platforms:

Name of Platform (Layout)Walk/Run? *Distance
Teleport?
**
Create Portals?
Second Life (mostly one contiguous landmass, with private islands)YESYESYES
Sinespace (separate worlds)YESNOYES
Sansar (separate worlds)YESNO (but you can create teleport hubs)YES
VRChat (separate worlds)YESNOYES
Rec Room (separate worlds)YESNOYES
AltspaceVR (separate worlds)YESNOYES
NeosVR (separate worlds)YESNOYES
Cryptovoxels (one contiguous landmass with some islands) YESNO (you can add coordinates to a URL, though)YES
Decentraland (one contiguous landmass) YESYES (/goto X,Y)NO
Somnium Space (one contiguous landmass)YESNO (but there are teleport hubs)NO (unless you count teleport hubs)
* – Can a user walk/run/fly/swim from one location to another? This includes SHORT HOP teleporting.
** – Can a user personally choose to teleport from one location to a second, DISTANT location?
† – Can a user create a temporary or permanent portal from one location to another?

Obviously, all metaverse platforms offer some form of personal locomotion for your avatar (walk, run, fly, swim, short-hop teleporting, etc.). This is standard.

It is also clear from this table that the metaverse platforms which consist of many smaller worlds (Sinespace, Sansar, VRChat, Rec Room, AltspaceVR, and NeosVR) all prefer the creation of temporary and permanent portals to allowing users to teleport great distances on their own steam. On the other hand, all the social VR platforms and virtual worlds which consist of one contiguous landmass tend to allow some form of teleportation across great distances.

You will notice that Cryptovoxels uses a rather brute-force method of “teleporting”, which consists of appending the coordinates to the end of the URL you enter into your web browser client (which are much the same as the coordinates which form part of the SLURLs used in Second Life, but not nearly as convenient in my opinion).

Transportation affordances are yet another way to classify metaverse platforms in my continuing effort to create a taxonomy of social VR platforms and virtual worlds.

So, what do you think? Have I made an error in my table? Do you have an opinion about the benefits of teleporting and portals versus walking around and exploring the landscape? I’d love to hear your opinions, so please leave a comment, thank you!

The Perks of Virtual World/Social VR Premium Memberships: Are They Worth It? What Do You Get?

Second Life (which I still consider to be the perfect model of the mature, fully-evolved virtual world that the companies creating the newer social VR platforms would be wise to study) has two levels of membership: Basic (free), and Premium. How Premium membership in Second Life works: for US$99 a year (or $32.97 quarterly, or $11.99 monthly), you get a set of benefits and perks over free, Basic user accounts:

Second Life Premium Membership (source)

VRChat is another platform that decided to offer a comparably-priced paid premium membership level last December, called VRChat Plus (which I first wrote about here). Now, upon first reading of the perks such a membership would offer me (see below), I was less than impressed (probably because I have been spoiled by all the goodies Second Life Premium memberships offer me in comparison).

Among the (relatively) small number of features for VRChat Plus users is the ability to set a user icon to display in a circle next to your user name:

But in conversation with Voices of VR podcaster Kent Bye last night via Zoom, he raised a point that I had hitherto failed to consider, Given my well-documented, one-man, scorched-earth campaign against Facebook and Oculus for, among other things, forcing Oculus headset users to get Facebook accounts and their toxic advertising-based business model which scrapes and strip-mines users’ personal data, why would I not support an alternative way for VRChat to earn a profit?

I stopped to think of what VRChat would be like with Facebook-like advertising, and I positively shuddered in revulsion. So this evening, I pulled out my credit card and ponied up for a VRChat Plus membership (US$99.99), so I now have the familiar “red Ryan” logo displayed next to my username in world (which has sort of become an icon for my brand, as I use it everywhere else, too). If it helps other users in VRChat recognize who I am, then I think it’s worthwhile.

My familiar “red Ryan” user icon

So, I have decided to do a quick survey of the major social VR and virtual world platforms, and find out whether or not they offer a paid premium service, and if so, what you get for your money.

Second Life

My alt Moesha Heartsong, sitting on the porch of her lovely Victorian Linden Home on the continent of Bellisseria (one of the many nice perks you get with your Second Life Premium membership)

Second Life Premium membership (currently priced at US$99 a year) offers you the following benefits:

  • A weekly L$300 stipend (basically enough to buy a nice outfit or pair of shoes for your avatar every week)
  • A L$1,000 sign-up bonus for first-time Premium users (can only be used once)
  • Priority entry when regions/sims are full of avatars (in other words, if a Basic user and a Premium user both try to get into a packed sim at the same time, the Premium user gets priority; this comes in handy at crowded shopping events, and I have made use of this perk often!)
  • A 1024m² virtual land allotment for use towards a nice starter Linden Home or a parcel on the Second Life mainland; this is another benefit I do take advantage of!
  • Expanded live-chat customer support (which I have used on occasion!)
  • Premium virtual gifts (frankly, kinda useless to me)
  • Exclusive access to Premium areas and experiences (such as building sandboxes)
  • Increased cap on missed IMs (which I never use)
  • Increased group membership limits (I make use of my groups ALL THE TIME! A freebie fashionista can NEVER have too many free group slots for store groups, freebie groups, etc. Basic accounts have 42 group slots, but Premium has 70;)
  • Voice morphing (never used it, myself; most SL users never use voice, anyways)
  • UPDATE 11:36 p.m.: Animesh (animated mesh) creator Medhue tells me that SL Premium members can attach two animesh items (e.g. pets such as Medhue’s delightful animesh cihuahua), while Basic members can only attach one.

Basically, I have three Premium accounts, with two lovely Linden Homes between them (which I think is the major benefit of a Premium membership). More group space and priority access to overcrowded sims are also perks I tend to use a lot.

Sansar

Sansar offers three levels of premium subscriptions (unchanged from when Linden lab owned the platform), which give you:

  • A 45-day free trial of the Marvelous Designer software (used to create avatar clothing in Sansar)
  • Purchase discounts on Marvelous Designer for when you do decide to buy it
  • An increase in the number of Sansar worlds you can create (frankly, I’m not sure most people bother beyond the free Basic account, which lets you create up to 25 worlds)
  • Expedited user support options

Sinespace

The Unity-based Sinespace virtual world/social VR platform, created by Sine Wave Entertainment, offers a truly overwhelming number of Premium levels to choose from:

Premium users can create larger regions/worlds, have a larger number of regions active at one time, and get priority support and user-created content processing and approval, among other benefits.

AltspaceVR

Surprisingly, Microsoft-owned AltspaceVR doesn’t seem to offer any premium accounts (that may change in the future, though).

VRChat

VRChat Plus offers you the following perks (with more promised soon):

  • A nameplate icon: With VRChat+, you can personalize your nameplate with an icon you create! Snap a pic in VRChat or upload your own image on our website.
  • You can send a picture with an invitation to a friend to join you at your location
  • Free slots for up to 100 favourite avatars (as opposed to 25 for basic users)
  • “A limited edition VRCat Badge to display on your profile” (Really? Really?!??)
  • A higher trust ranking in VRChat’s Safety and Trust System

As I said up top, this list is a bit sparse, especially compared to what Second Life offers (and yes, you can be an anime girl in SL, just as easily as you can in VRChat!), but of course, there’s zero VR support in Second Life.

Rec Room

Rec Room offers something called Rec Room Plus at US$7.99 a month, which includes the following benefits:

  • You get 6000 tokens (r6000) monthly, delivered in installments of r1500 per week
  • One four-star gift box per week
  • A 10% discount in Rec Room stores that accept tokens
  • Exclusive access to the RR+ section of the item store
  • 100 saved outfit slots
  • The ability to sell premium inventions/keys for tokens

NeosVR

NeosVR uses Patreon levels to hand out perks to various levels of paying users (more info). For example, at my current “Blade Runner” level ($6 per month), I get:

  • Access to private channels on the official Discord Server
  • Patreon supporter badge in Neos
  • Early access to Linux builds
  • Early Access to Patreon only content (exclusive experiences, work in progress experiences before they’re public)
  • A Neos Mini account with 25 GB of storage
  • Your name in the stars! (your name will appear in the sky in the Neos hub)
  • 30 Neos Credits (NCR) monthly, accumulates

(Note that there is an even less expensive level, the “Agent Smith” level, at just $1 a month. Please check out the NeosVR Patreon page for more details.)

ENGAGE

The ENGAGE educational/corporate/conference social VR platform offers a free, “lite” version, and a premium, “plus” version for €4.99 a month, which gives you space to save your presentations, among other benefits. (They also offer enterprise and educational rates on request.)

Blockchain-Based Virtual Worlds (Cryptovoxels, Decentraland, and Somnium Space)

Of course, the various blockchain-based virtual worlds sell everything using whatever cryptocurrencies they support (for example, a custom, non-randomly-generated avatar username in Decentraland will set you back 100 MANA, Decentraland’s in-world cryptocurrency (which is about US$36 at current exchange rates). It’s just a completely different model than the “freemium” ones offered above.


Thanks to Kent Bye for giving me the idea for this blogpost!

Social VR Research Alert: You Can Participate in a Clemson University Research Survey of LGBTQ+ Users of Social VR Platforms

Back in October of 2019, I wrote a blogpost about a research study being conducted by Clemson University on the use of social VR. Well, Clemson University’s Gaming and Mediated Experince (CU GAME) Lab, led by Dr. Guo Freeman in their School of Computing, is conducting a survey of LBGTQ+ (Lesbian, Bisexual, Gay, Transgender, Queer, etc.) users of social VR platforms—including conducting interviews in AltspaceVR, Rec Room and VRChat, if you wish!

If you are a member of the LGBTQ+ community, and you are interested in being interviewed for 60 to 90 minutes about your experiences in social VR, particularly with respect to self-presentation and social support, then you are invited to fill out this online form (more information about the research study can be found here). The form states:

We are a group of academic researchers at Clemson University who are conducting a research project about social VR. We are interested in interviewing individuals who identify as LGBTQ+, and understanding their experiences.

No personally identifiable data will be asked or collected, but we’ll ask general demographics questions (age, location, race, etc). You do not have to answer any questions that you do not feel comfortable answering.

If you have experienced any social VR platforms / applications / environments (AltspaceVR, Rec Room, VRChat, etc.) and are willing to be interviewed, please fill out the form … and we will contact you for more details about this research project.

Here is the link for a document with more information about the study.

Feel free to email us at dacena@clemson.edu if you have any questions.

Interviews are to be scheduled during the month February, and can be done via telephone call, Discord (text or voice chat), Zoom (voice or video chat), or even on the social VR platforms AltspaceVR, Rec Room, or VRChat!

If you are interested, here is a the website (including a list of current research publications) by the Clemson University GAME Lab.

Are you a member of the LGBTQ community and use one or more social VR platforms? Clemson University wants to interview you! (Photo by Sharon McCutcheon on Unsplash)