The Institute for the Future Issues a Report Identifying Leading-Edge Behaviours in Social VR

From time to time on this blog, I have covered academic research involving virtual reality in general (here and here) and social VR in particular (here), and it is wonderful to see a new field of research take shape around social virtual reality. One of those researchers is called, aptly enough, the Institute for the Future:

The Institute for the Future (IFTF) is a Palo Alto, California–based not-for-profit think tank. It was established, in 1968, as a spin-off from the RAND Corporation to help organizations plan for the long-term future, a subject known as futures studies. They describe themselves as:

For over 50 years, businesses, governments, and social impact organizations have depended upon IFTF global forecasts, custom research, and foresight training to navigate complex change and develop world-ready strategies. IFTF methodologies and toolsets yield coherent views of transformative possibilities across all sectors that together support a more sustainable future.

IFTF has just released a report on social VR titled Leading-Edge Behaviors from the New World of Social VR:

They describe their study as follows:

On a handful of platforms around the world, a small group of pioneers are hanging out in 3D environments in 3D bodies. They are willing to endure technical challenges, limited content, and lack of standard practices and etiquette to be the first to inhabit and explore new shared virtual worlds. IFTF spent eight months exploring their environments, communities, and practices. Their experiments provide early signals that point to a future where we each have a personal digital body, and content can be experienced in full 3D space. This will have profound implications for how we socialize, learn, work, engage with content, and take care of ourselves.

This study contains 10 Leading-Edge Behaviors: emerging and innovative user practices likely to play out more broadly over the next few years. Leading-edge Behaviors inspire and inform new products, experience and service ideas, and reveal emerging opportunities and implications. Leading-Edge Behaviors are created through a blend of expert interviews, observational and ethnographic research, and horizon scanning with people pushing the edges of new applications, devices, and platforms.

In fact, I was interviewed a couple of years ago by Lyn Jeffery, a cultural anthropologist who works for IFTF, about my own experiences in social VR (which I believe predates the research in this new report).

I was very surprised and pleased to see my own definition of social VR appearing in this report! (Hey ma, I’m famous!)

Here are the 10 leading-edge behaviours in social VR identified in this report:

This is report which pulls together insights and information from a wide variety of sources in an attempt to identify future trends, and it is well worth your time to read through it in detail. If you are interested in this report, you can find it here (the link is a Adobe Acrobat PDF format slide deck). The report includes links to many other resources, making it a good starting point for your own investigations.

I’d like to thank Lyn Jeffery and the entire team at IFTF for releasing this report to the public for free. It’s a truly valuable contribution to the nascent field of research into various aspects of social VR.

Facebook Demos Highly Realistic Avatar Facial Animation

My Twitter feed keeps delivering news nuggets this week! This is an update to a blogpost I had written earlier this year on this technology.

Facebook Reality Labs has published a research article in the journal ACM Transactions on Graphics, which shows cutting-edge avatar facial animation using multiple cameras attached to a VR headset, and a new multiview image processing technique. (The full paper is free to download from the link above.) The researchers also gave a presentation at the SIGGRAPH 2019 computer graphics conference in Los Angeles.

The results are impressive, giving an avatar human-driven, lifelike animations not only of the lower face but also the upper face, which of course if covered by the VR headset:

This is light years ahead of current avatar facial animation technology, such as the avatar facial driver in Sinespace, which operates using your webcam. Imagine being able to conduct a conversation in VR where you can convey the full gamut of facial expressions while you are talking! This is a potential gamechanger for sure. It’s not clear when we can expect to see this technology actually applied to Oculus VR hardware, however. It might still be many years away. But it is exciting!

Virtual Memory Palaces: Combining an Ancient Memory Technique with Modern VR

Cicero (Wikipedia)

The method of loci (also referred to as memory palaces, memory places, or memory spaces) is a technique for remembering information which dates back to the ancient Greek poet Simonides (who lived circa 556 — circa 468 B.C.).

According to the Stanford Encyclopedia of Philosophy:

According to a legend passed on by Cicero (106-43 B.C.E.), the discovery occurred at a banquet in Thessaly which Simonides attended in order to present a lyric poem written in praise of the host. Simonides was called outside shortly after his performance, and during his absence the roof of the banqueting hall suddenly collapsed, crushing the other diners, and mangling many of their corpses beyond recognition. Simonides, however, found he was able to identify the bodies (important for proper burial) by consulting his visual memory image of the people sitting around the banqueting table, which enabled him to identify the corpses according to where they were found. From this experience,

[Simonides] inferred that persons desiring to train this faculty [of memory] must select places and form mental images of the things they wish to remember and store those images in the places, so that the order of the places will preserve the order of the things, and the images of the things will denote the things themselves, and we shall employ the places and images respectively as a wax writing-tablet and the letters written on it. (Cicero, De Oratore, II, lxxxvi – translation: Sutton & Rackham, 1942).

Supposedly, this was the origin of the mnemonic technique known as the method of loci, described by Roman rhetoricians such as Cicero and Quintilian (c.35-c.95 C.E.), and widely employed, in various forms, by orators and others from classical, through medieval, and up until early modern times.

Memory palaces help people remember information by taking advantage of the brain’s natural ability to spatially organize thoughts and concepts. The user associates information with objects and locations in a three-dimensional environment. Here’s a summary of the technique if you are interested in learning it.

A recent academic research paper by Eric Krokos, Catherine Plaisant, and Amitabh Varshney at the Univerity of Maryland, published in the journal Virtual Reality, has shown that people accessing virtual memory palaces in virtual reality (i.e. using a VR headset) were able to remember things better compared to people accessing the same memory palace using a flat computer desktop and mouse. (The research paper is Open Access, which means anybody on the Internet can download and read it without having to pay a publisher, in this case, Springer. You can access the paper for free using the link above, just click on the blue Download PDF button in the upper right-hand corner.)

I was pleased to discover this article, as the experiment was very similar to one that I wanted to conduct in my wildly overambitious research proposal, which unfortunately I had to suspend work on for various practical reasons. (I am currently writing up a paper about my experience. I am now working on developing a new research project involving virtual reality and libraries.)

In the University of Maryland study, testing was done using pre-constructed medieval town and palace environments purchased through TurboSquid, and 42 pictures of the faces of famous people. The pictures of the faces were hand-positioned in the 3D environment. Study participants were given a list of the pictures and names to study before the experiment. Next, each participant was given access to the memory palace, using either a VR headset or a or a desktop monitor with a mouse. Then, they had 5 minutes to study the memory palace with 21 pictures scattered throughout it. Then, in the recall phase, the pictures of the faces were swapped out with numbers, and participants were asked to give a name and level of confidence for their recalled faces for each numbered position. Each study participant was tested in both the town and the palace scenes, and both in VR and desktop mode. (If a face was shown in one set, it was not repeated in the second set. The 21 faces were presented all at once, and users were able to view and memorize the faces in any order of their choosing.)

Images Taken from the Research Paper

Statistical analysis of the experimental results supported the study’s hypothesis that a virtual memory palace experienced in an immersive VR headset led to a more accurate recall than on a mouse-controlled desktop display. Study participants in VR headsets also had a higher level of confidence in their answers than desktop users. A news report on the research project from the University of Maryland website states:

The results showed an 8.8 percent improvement overall in recall accuracy using the VR headsets, a statistically significant number according to the research team.

In post-study questionnaires, all 40 participants said that they were completely comfortable—and adept—in navigating a desktop computer to access information, yet all but two said they preferred the immersive VR environment as a potential learning platform. The questionnaire also found that only two people said they felt “uncomfortable” using VR.

Many of the participants said the immersive “presence” while using VR allowed them to focus better. This was reflected in the research results: 40 percent of the participants scored at least 10 percent higher in recall ability using VR over the desktop display.

In fact, there is already a VR app available which allows you to construct memory palaces and explore them in your VR headset. The product is called Munx VR:

Munx is a VR platform for building memory palaces to learn huge amounts in short time and with full retention. By combining medieval memory techniques with modern technology, we are redefining the way we learn, understand, and retain information in our minds. Imagine knowing the periodic table in the same way you know the layout of your living room, or being able to recall a president or ruler with the same effortlessness of reaching for a mug when making a cup of tea.

If you’re interested, you can get Munx VR on Steam (it’s free). Here’s a promotional video for the product:

Taking a Look at the Current Academic Research on Social Virtual Reality (Part I of a Series)

Photo by Martin Adams on Unsplash

One of the things I most enjoy about being an academic librarian is having access to all the research databases to which my university subscribes—and the know-how to search them effectively and efficiently! Every so often I like to do a deep-dive into the research to see what’s new in the world of virtual worlds and virtual reality.

In this blogpost, I wish to highlight some recent academic work which looks at the rapidly evolving world of commercial social VR. This is the first of what I expect will be a series on social VR research.

A trio of researchers at the University of California, Santa Cruz (Dr. Katherine Isbister, Professor in Computational Media at the Jack Baskin School of Engineering; her Ph.D. student Anya Kolesnichenko; and post-doc Joshua McVeigh-Schultz, who recently left UCSC and accepted a professorship at San Francisco State University) have been publishing a number of research papers at various computer conferences on social virtual reality. Here are citations to three of their papers, which you can obtain from your local academic or public library:

McVeigh-Schultz, J., Márquez Segura, E., Merrill, N., & Isbister, K. (2018). What’s It Mean to “Be Social” in VR?: Mapping the Social VR Design Ecology. In DIS ’18 Companion: Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems (pp. 289–294). https://doi.org/10.1145/3197391.3205451

McVeigh-Schultz, J., Kolesnichenko, A., & Isbister, K. (2019). Shaping Pro-Social Interaction in VR: An Emerging Design Framework. In CHI ’19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3290605.3300794

Kolesnichenko, A., Mcveigh-Schultz, J., & Isbister, K. (2019). Understanding Emerging Design Practices for Avatar Systems in the Commercial Social VR Ecology. In DIS ’19 Proceedings of the 2019 on Designing Interactive Systems Conference (pp. 241–252). https://doi.org/10.1145/3322276.3322352

In addition to these papers, which I strongly urge platform developers, as well as anybody who is interested in social VR, to obtain and read, Dr. Isbister and Dr. McVeigh-Schultz have given presentations which you can access on YouTube. The following is a presentation made this past March by Joshua McVeigh-Schultz at the 2019 Virtual Reality Developers Conference (part of the Game Developers Conference):

And the second is a presentation by both Dr. Isbister and Dr. McVeigh-Schultz given at the Mozilla Emerging Technology Speaker Series (the Mozilla Foundation supported some of their research with a grant).

Using a method of design-oriented autobiographical landscape research to examine existing commercial social VR platforms, the researchers attempted to identify key issues and concerns for future social VR design, and areas for possible future research. In-depth interviews were conducted with designers, developers, and other experts involved in the creation of social VR applications such as Rec Room, AltspaceVR, Facebook Spaces, VRChat, Mozilla Hubs, Anyland, and High Fidelity.

One interesting finding: the researchers discovered a clear design distinction between large open platforms where one was likely to encounter strangers (Rec Room, VRChat, High Fidelity, etc.) and non-open platforms where you connected mostly with people you already knew (Mozilla Hubs, Facebook Spaces):

Especially among the large open platforms where a user is likely to encounter strangers, we found convergence around broad design themes involving the role of place and space, community engagement, moderation, social catalysts and activity structures, social mechanics of friending/muting/blocking etc., and other embodied affordances including a range of communication modalities. However, we also observed fairly substantial divergence in terms of particular mechanics underlying these broad design areas. Furthermore, for non-open-world platforms such as Facebook Spaces and Mozilla Hubs, we noted less focus on supporting safety and security, and more investment in design goals associated with: supporting device interoperability, empowering a range of participation modalities, and bridging social encounters in VR with the outside world.

—McVeigh-Schultz, J. Shaping Pro-Social Interaction in VR

What’s exciting to me is that these people are conducting research in the exact same area where my own interests lie! (I’ve found my kind of people!!!) And since I have decided to suspend my previous VR research project for being much too broad and overambitious, discovering this published work gives me some much-needed food for thought on possible future research projects which I could decide to pursue at my university.