Facebook Demos Highly Realistic Avatar Facial Animation

My Twitter feed keeps delivering news nuggets this week! This is an update to a blogpost I had written earlier this year on this technology.

Facebook Reality Labs has published a research article in the journal ACM Transactions on Graphics, which shows cutting-edge avatar facial animation using multiple cameras attached to a VR headset, and a new multiview image processing technique. (The full paper is free to download from the link above.) The researchers also gave a presentation at the SIGGRAPH 2019 computer graphics conference in Los Angeles.

The results are impressive, giving an avatar human-driven, lifelike animations not only of the lower face but also the upper face, which of course if covered by the VR headset:

This is light years ahead of current avatar facial animation technology, such as the avatar facial driver in Sinespace, which operates using your webcam. Imagine being able to conduct a conversation in VR where you can convey the full gamut of facial expressions while you are talking! This is a potential gamechanger for sure. It’s not clear when we can expect to see this technology actually applied to Oculus VR hardware, however. It might still be many years away. But it is exciting!

Virtual Memory Palaces: Combining an Ancient Memory Technique with Modern VR

Cicero (Wikipedia)

The method of loci (also referred to as memory palaces, memory places, or memory spaces) is a technique for remembering information which dates back to the ancient Greek poet Simonides (who lived circa 556 — circa 468 B.C.).

According to the Stanford Encyclopedia of Philosophy:

According to a legend passed on by Cicero (106-43 B.C.E.), the discovery occurred at a banquet in Thessaly which Simonides attended in order to present a lyric poem written in praise of the host. Simonides was called outside shortly after his performance, and during his absence the roof of the banqueting hall suddenly collapsed, crushing the other diners, and mangling many of their corpses beyond recognition. Simonides, however, found he was able to identify the bodies (important for proper burial) by consulting his visual memory image of the people sitting around the banqueting table, which enabled him to identify the corpses according to where they were found. From this experience,

[Simonides] inferred that persons desiring to train this faculty [of memory] must select places and form mental images of the things they wish to remember and store those images in the places, so that the order of the places will preserve the order of the things, and the images of the things will denote the things themselves, and we shall employ the places and images respectively as a wax writing-tablet and the letters written on it. (Cicero, De Oratore, II, lxxxvi – translation: Sutton & Rackham, 1942).

Supposedly, this was the origin of the mnemonic technique known as the method of loci, described by Roman rhetoricians such as Cicero and Quintilian (c.35-c.95 C.E.), and widely employed, in various forms, by orators and others from classical, through medieval, and up until early modern times.

Memory palaces help people remember information by taking advantage of the brain’s natural ability to spatially organize thoughts and concepts. The user associates information with objects and locations in a three-dimensional environment. Here’s a summary of the technique if you are interested in learning it.

A recent academic research paper by Eric Krokos, Catherine Plaisant, and Amitabh Varshney at the Univerity of Maryland, published in the journal Virtual Reality, has shown that people accessing virtual memory palaces in virtual reality (i.e. using a VR headset) were able to remember things better compared to people accessing the same memory palace using a flat computer desktop and mouse. (The research paper is Open Access, which means anybody on the Internet can download and read it without having to pay a publisher, in this case, Springer. You can access the paper for free using the link above, just click on the blue Download PDF button in the upper right-hand corner.)

I was pleased to discover this article, as the experiment was very similar to one that I wanted to conduct in my wildly overambitious research proposal, which unfortunately I had to suspend work on for various practical reasons. (I am currently writing up a paper about my experience. I am now working on developing a new research project involving virtual reality and libraries.)

In the University of Maryland study, testing was done using pre-constructed medieval town and palace environments purchased through TurboSquid, and 42 pictures of the faces of famous people. The pictures of the faces were hand-positioned in the 3D environment. Study participants were given a list of the pictures and names to study before the experiment. Next, each participant was given access to the memory palace, using either a VR headset or a or a desktop monitor with a mouse. Then, they had 5 minutes to study the memory palace with 21 pictures scattered throughout it. Then, in the recall phase, the pictures of the faces were swapped out with numbers, and participants were asked to give a name and level of confidence for their recalled faces for each numbered position. Each study participant was tested in both the town and the palace scenes, and both in VR and desktop mode. (If a face was shown in one set, it was not repeated in the second set. The 21 faces were presented all at once, and users were able to view and memorize the faces in any order of their choosing.)

Images Taken from the Research Paper

Statistical analysis of the experimental results supported the study’s hypothesis that a virtual memory palace experienced in an immersive VR headset led to a more accurate recall than on a mouse-controlled desktop display. Study participants in VR headsets also had a higher level of confidence in their answers than desktop users. A news report on the research project from the University of Maryland website states:

The results showed an 8.8 percent improvement overall in recall accuracy using the VR headsets, a statistically significant number according to the research team.

In post-study questionnaires, all 40 participants said that they were completely comfortable—and adept—in navigating a desktop computer to access information, yet all but two said they preferred the immersive VR environment as a potential learning platform. The questionnaire also found that only two people said they felt “uncomfortable” using VR.

Many of the participants said the immersive “presence” while using VR allowed them to focus better. This was reflected in the research results: 40 percent of the participants scored at least 10 percent higher in recall ability using VR over the desktop display.

In fact, there is already a VR app available which allows you to construct memory palaces and explore them in your VR headset. The product is called Munx VR:

Munx is a VR platform for building memory palaces to learn huge amounts in short time and with full retention. By combining medieval memory techniques with modern technology, we are redefining the way we learn, understand, and retain information in our minds. Imagine knowing the periodic table in the same way you know the layout of your living room, or being able to recall a president or ruler with the same effortlessness of reaching for a mug when making a cup of tea.

If you’re interested, you can get Munx VR on Steam (it’s free). Here’s a promotional video for the product:

Taking a Look at Current Academic Research on Social Virtual Reality (Part II): Treating Substance Abuse Disorders with VR

ENDGAME is a long-running talkshow set on the social VR platform of VRChat:

Don’t panic! The end is always nigh for the old world. Endgame is a talk show / group discussion that takes place in the Metaverse (the social layer of virtual reality). Its focus is on technology and the future. Change is inevitable for all species; will we eventually destroy ourselves, or are we destined to transcend what we currently are? Get involved in the conversation by joining us every other Wednesday at 7pm PST / 10pm EST in VRChat. Or you can watch our livestream and previous recordings here at our Youtube channel. Come share your thoughts about the future of the world we share together.

One of the three regular hosts of ENDGAME is Noah Robinson (a.k.a. Psych; Twitter; LinkedIn), a clinical psychology doctoral student at Vanderbilt University and the founder and CEO of Very Real Help, a compnay devoted to building a new Internet-based clinical research platform “that can both treat and inform our understanding of psychopathology”.

In a recent episode of ENDGAME, Noah gave an hour-long presentation of his academic research on how social VR could be used in the treatment of people with substance abuse disorders. Here’s a two-minute YouTube video overview of his research:

Noah’s research interest focuses on exploring how VR social networks can be used as telehealth interventions to treat mental health disorders. He stresses that VR will not replace, but rather supplement, in-person therapy, and that there are still not nearly enough counselors to provide services to the millions of people suffering from addictions in the United States and around the world. Especially after getting out of rehab, patients experience a lot of distress and anhedonia, which tends to lead to relapse. Noah suggests that VR may be particularly effective to prevent relapse in patients with substance abuse disorders.

Here is Noah Robinson’s presentation in full:

Although there is already plenty of anecdotal evidence that virtual reality has a positive impact on mental health disorders (including my own personal experience), there is still a strong need to collect and analyze data in well-designed academic research projects. Noah’s research is fascinating and exciting to me, and I look forward to any papers that are published as a result of his pioneering work.