Facebook Demos Highly Realistic Avatar Facial Animation

My Twitter feed keeps delivering news nuggets this week! This is an update to a blogpost I had written earlier this year on this technology.

Facebook Reality Labs has published a research article in the journal ACM Transactions on Graphics, which shows cutting-edge avatar facial animation using multiple cameras attached to a VR headset, and a new multiview image processing technique. (The full paper is free to download from the link above.) The researchers also gave a presentation at the SIGGRAPH 2019 computer graphics conference in Los Angeles.

The results are impressive, giving an avatar human-driven, lifelike animations not only of the lower face but also the upper face, which of course if covered by the VR headset:

This is light years ahead of current avatar facial animation technology, such as the avatar facial driver in Sinespace, which operates using your webcam. Imagine being able to conduct a conversation in VR where you can convey the full gamut of facial expressions while you are talking! This is a potential gamechanger for sure. It’s not clear when we can expect to see this technology actually applied to Oculus VR hardware, however. It might still be many years away. But it is exciting!


Virtual Memory Palaces: Combining an Ancient Memory Technique with Modern VR

Cicero (Wikipedia)

The method of loci (also referred to as memory palaces, memory places, or memory spaces) is a technique for remembering information which dates back to the ancient Greek poet Simonides (who lived circa 556 — circa 468 B.C.).

According to the Stanford Encyclopedia of Philosophy:

According to a legend passed on by Cicero (106-43 B.C.E.), the discovery occurred at a banquet in Thessaly which Simonides attended in order to present a lyric poem written in praise of the host. Simonides was called outside shortly after his performance, and during his absence the roof of the banqueting hall suddenly collapsed, crushing the other diners, and mangling many of their corpses beyond recognition. Simonides, however, found he was able to identify the bodies (important for proper burial) by consulting his visual memory image of the people sitting around the banqueting table, which enabled him to identify the corpses according to where they were found. From this experience,

[Simonides] inferred that persons desiring to train this faculty [of memory] must select places and form mental images of the things they wish to remember and store those images in the places, so that the order of the places will preserve the order of the things, and the images of the things will denote the things themselves, and we shall employ the places and images respectively as a wax writing-tablet and the letters written on it. (Cicero, De Oratore, II, lxxxvi – translation: Sutton & Rackham, 1942).

Supposedly, this was the origin of the mnemonic technique known as the method of loci, described by Roman rhetoricians such as Cicero and Quintilian (c.35-c.95 C.E.), and widely employed, in various forms, by orators and others from classical, through medieval, and up until early modern times.

Memory palaces help people remember information by taking advantage of the brain’s natural ability to spatially organize thoughts and concepts. The user associates information with objects and locations in a three-dimensional environment. Here’s a summary of the technique if you are interested in learning it.

A recent academic research paper by Eric Krokos, Catherine Plaisant, and Amitabh Varshney at the Univerity of Maryland, published in the journal Virtual Reality, has shown that people accessing virtual memory palaces in virtual reality (i.e. using a VR headset) were able to remember things better compared to people accessing the same memory palace using a flat computer desktop and mouse. (The research paper is Open Access, which means anybody on the Internet can download and read it without having to pay a publisher, in this case, Springer. You can access the paper for free using the link above, just click on the blue Download PDF button in the upper right-hand corner.)

I was pleased to discover this article, as the experiment was very similar to one that I wanted to conduct in my wildly overambitious research proposal, which unfortunately I had to suspend work on for various practical reasons. (I am currently writing up a paper about my experience. I am now working on developing a new research project involving virtual reality and libraries.)

In the University of Maryland study, testing was done using pre-constructed medieval town and palace environments purchased through TurboSquid, and 42 pictures of the faces of famous people. The pictures of the faces were hand-positioned in the 3D environment. Study participants were given a list of the pictures and names to study before the experiment. Next, each participant was given access to the memory palace, using either a VR headset or a or a desktop monitor with a mouse. Then, they had 5 minutes to study the memory palace with 21 pictures scattered throughout it. Then, in the recall phase, the pictures of the faces were swapped out with numbers, and participants were asked to give a name and level of confidence for their recalled faces for each numbered position. Each study participant was tested in both the town and the palace scenes, and both in VR and desktop mode. (If a face was shown in one set, it was not repeated in the second set. The 21 faces were presented all at once, and users were able to view and memorize the faces in any order of their choosing.)

Images Taken from the Research Paper

Statistical analysis of the experimental results supported the study’s hypothesis that a virtual memory palace experienced in an immersive VR headset led to a more accurate recall than on a mouse-controlled desktop display. Study participants in VR headsets also had a higher level of confidence in their answers than desktop users. A news report on the research project from the University of Maryland website states:

The results showed an 8.8 percent improvement overall in recall accuracy using the VR headsets, a statistically significant number according to the research team.

In post-study questionnaires, all 40 participants said that they were completely comfortable—and adept—in navigating a desktop computer to access information, yet all but two said they preferred the immersive VR environment as a potential learning platform. The questionnaire also found that only two people said they felt “uncomfortable” using VR.

Many of the participants said the immersive “presence” while using VR allowed them to focus better. This was reflected in the research results: 40 percent of the participants scored at least 10 percent higher in recall ability using VR over the desktop display.

In fact, there is already a VR app available which allows you to construct memory palaces and explore them in your VR headset. The product is called Munx VR:

Munx is a VR platform for building memory palaces to learn huge amounts in short time and with full retention. By combining medieval memory techniques with modern technology, we are redefining the way we learn, understand, and retain information in our minds. Imagine knowing the periodic table in the same way you know the layout of your living room, or being able to recall a president or ruler with the same effortlessness of reaching for a mug when making a cup of tea.

If you’re interested, you can get Munx VR on Steam (it’s free). Here’s a promotional video for the product:

Philip Rosedale Talks About the New Direction for High Fidelity

Philip Rosedale is always a great interview: insightful, engaged, and articulate. Here’s a perfect example, a recent 11-minute interview with GameReactor (a European videogame magazine) at the Gamelab 2019 conference in Barcelona, where he talked about his favourite topic, the metaverse, and the new direction for High Fidelity as a platform for remote workteams:

He argues that the change in the medium and the technology with virtual reality is so profound that it’s unlikely that the same big companies will dominate it, thus creating business opportunities for new companies (like HiFi!). He compares the shift from flatscreen computer use to virtual reality as being similar to the change from radio to television in the last century.

Image from IEEE Spectrum

Philip Rosedale is a true pioneer and visionary, without whom we literally would not have the metaverse landscape that I love to blog about! Even though I am still somewhat annoyed at how High Fidelity chose to handle the sudden pivot away from their original consumer audience, I can certainly understand and appreciate the company’s need to establish a beachhead in one area (remote business teams) and then use that as a base to expand into other areas. VR needs more time to mature. As he says in this interview, HiFi was early to the game. The pivot was the best possible corporate strategy to keep the company moving ahead and generating revenue while waiting for millions of consumers to adopt VR (and eventually, they will).

I do admire Philip and I wish him and his team at High Fidelity the very best (even if I do deliver the occasional critical editorial on this blog).

Philip Rosedale Talks About His 30 Years of Experiences with Virtual Worlds and Social VR

Philip Rosedale

Numerous people have posted the following YouTube video to various social media and community forums in the past few days: a classroom presentation by Philip Rosedale at the University of Washington in Seattle on May 21st, 2019, as part of their Reality Lab Lectures series.

Philip is a pioneer and a visionary, and he is an engaging speaker, leading his audience through a history of how he became enamored and involved with virtual worlds and virtual reality, and how he built Second Life, founding Linden Lab in 1999, and then, in 2012, starting his new company High Fidelity. You need to watch this; it’s great! (There are a few minor sound issues with the video.)

In response to a student question, he talks about how High Fidelity is working on an app where you can take a single photo of a person and create a 3D avatar from that (at the 43:30 mark). I love this idea (especially since I happen to live a long way away from the closest Doob full-body scanner!), and I hope that HiFi has not dropped this project in their recent pivot to the remote business teams market.

He also says that they already have a version of High Fidelity that runs on the Oculus Quest (at the 1:00:25 mark), but he’s not sure when they will release it. The company may decide to allow people to sideload the app, which would get around having an official release on the Oculus Store.