Facebook Reality Labs Gives Us a Preview of What Your Avatar Could Look Like in the Future

Have you ever wondered what your virtual-world avatar could look like, 10 to 20 years from now?

A recently published article in WIRED covers the work of Facebook Reality Labs, which is developing stunningly lifelike virtual reality avatars, called codec avatars, which can recreate the full gamut of facial expressions:

Examples of Facebook Reality Labs’ Codec Avatars

For years now, people have been interacting in virtual reality via avatars, computer-generated characters that represent us. Because VR headsets and hand controllers are trackable, our real-life head and hand movements carry into those virtual conversations, the unconscious mannerisms adding crucial texture. Yet even as our virtual interactions have become more naturalistic, technical constraints have forced them to remain visually simple. Social VR apps like Rec Room and AltspaceVR abstract us into caricatures, with expressions that rarely (if ever) map to what we’re really doing with our faces. Facebook’s Spaces is able to generate a reasonable cartoon approximation of you from your social media photos but depends on buttons and thumb-sticks to trigger certain expressions. Even a more technically demanding platform like High Fidelity, which allows you to import a scanned 3D model of yourself, is a long way from being able to make an avatar feel like you.

That’s why I’m here in Pittsburgh on a ridiculously cold, early March morning inside a building very few outsiders have ever stepped foot in. Yaser Sheik and his team are finally ready to let me in on what they’ve been working on since they first rented a tiny office in the city’s East Liberty neighborhood. (They’ve since moved to a larger space on the Carnegie Mellon campus, with plans to expand again in the next year or two.) Codec Avatars, as Facebook Reality Labs calls them, are the result of a process that uses machine learning to collect, learn, and re-create human social expression. They’re also nowhere near being ready for the public. At best, they’re years away—if they end up being something that Facebook deploys at all. But the FRL team is ready to get this conversation started. “It’ll be big if we can get this finished,” Sheik says with the not-at-all contained smile of a man who has no doubts they’ll get it finished. “We want to get it out. We want to talk about it.”

The results (which you can see more of in the photos and videos in the WIRED article) are impressive, but they require a huge amount of data capture beforehand: 180 gigabytes of data every second! So don’t expect this to be coming out anytime soon. But it is a fascinating glimpse of the future.

Would you want your avatar in a virtual world to look exactly like you, and have their face move exactly like your face, with all your unique expressions? Some people would find this creepy. Others would embrace it. Many people would probably prefer to have an avatar who looks nothing like their real-life selves. What do you think of Facebook’s research? Please feel free to leave a comment on this blogpost, thanks!

Advertisements

A Look at the New Customization Features in Oculus Home

Some Oculus Home Decorations, Including a Mini Model of the Solar System

I must confess that I haven’t been paying much attention to Oculus Home, which I always saw just as a space to organize your purchased VR games and experiences. But Facebook/Oculus has been slowly but steadily building Oculus Home to the point where we can now consider it a true social VR platform. The VR news site UploadVR reports:

The latest update to Oculus Home adds custom environments support…

Support for user-generated objects was added back in June. A subsequent update even added animation support. And later in the month, the platform added real-time social, allowing up to 7 friends to visit your home and see those custom objects.

But until now the actual home geometry was the same for all users. The background could be changed between hills, space, a bay, or a future city, but the home itself could not.

David Hall (a long-time Sansar resident who has built a number of wonderful experiences including the Lord of the Rings-inspired Dwarven City and the futuristic Avalon social hub) has posted the following video to his YouTube channel. It is a livestream of him showing several of his own homes in Oculus Home, plus a tour of several other people’s homes:

I love the transition effects when David moves from one home to another, the way one home rezzes out of existence, and the new one rezzes in!

Oculus Home already includes functioning weapons such as bows and guns, as well as some cool animated content like a miniature model of the solar system you can see in a couple of visited homes in this video, and in the picture at the top of this blogpost.

I have now added Oculus Home to my list of social VR/virtual worlds.

Leaked Mark Zuckerberg Email Reveals His Thoughts on How Facebook Could Dominate Virtual Reality

There’s an extremely interesting article on the Road to VR website, that’s worth a read. Apparently, an author writing a book about the history of VR has obtained a copy of a lengthy 2015 email that Mark Zuckerberg of Facebook sent to his senior executives. (The email has been confirmed by Road to VR, a major news reporting website on VR, as authentic.)

Titled Leaked Zuckerberg Email Reveals Facebook’s XR Strategy, ‘Our goal is not only to win, but to accelerate its arrival’, the article quotes Mark at length about the strategy behind Facebook buying Oculus for 2 billion dollars in 2014, and what he feels Facebook needs to do to dominate in virtual reality.

Remember this prediction I made last April?

What I predict will happen, over the next two years, is that one of the Big Five computer companies:
– Alphabet/Google
– Amazon
– Apple
– Facebook/Oculus
– Microsoft

Is either going to launch their own social VR/virtual world/metaverse product, OR is going to buy one of the Big Four metaverse-building companies:
– High Fidelity
– Linden Lab (Second Life and Sansar)
– Sine Wave Entertainment (Sinespace)
– VRChat

Well, keep that in mind, and read this direct quote from Mark’s email:

The key [VR] apps are what you’d expect: social communication and media consumption, especially immersive video. Gaming is critical but is more hits-driven and ephemeral, so owning the key games seems less important than simply making sure they exist on our platform. I expect everyone will use social communication and media consumption tools, and that we’ll build a large business if we are successful in these spaces. We will need a large investment and dedicated strategy to build the best services in these spaces. For now, though, I’ll just assert that building social services is our core competence, so I’ll save elaborating further on that for another day.

The platform vision is around key services that many apps use: identity, content and avatar makerplace, app distribution store, ads, payments and other social functionality. These services share the common properties of network effects, scarcity and therefore monetization potential. The more developers who use our content marketplace or app store or payments system, the better they become and the more effectively we can make money.

It seems pretty clear to me, from this email, that Facebook is planning to step into social VR in a major way. Mark sees “social communication and media consumption” as the killer app for VR, and they want to be a part of it. Facebook, for better or for worse, wants to dominate social virtual reality the way they already dominate social media.

I expect that we will all begin to get some idea of what Facebook has planned with the launch of the standalone Oculus Quest VR headset later this year. Watch carefully for any parallel announcement regarding the next-generation Oculus Home, Oculus Rooms, Facebook Spaces, or a completely rebranded platform where Oculus Go, Quest, and Rift users can meet up with each other. I’m convinced, now more than ever, that it’s going to happen.

Does Facebook/Oculus Need to Get Its Act Together On Social VR? A Thread on Reddit

As most of you already know, I quit Facebook as my New Year’s resolution, and I am still quite happy with my decision.

One of the places where I have spent more time since leaving Facebook has been Reddit, which is home to many thriving communities such as the Oculus subReddit, with over 140,000 subscribers who discuss and debate various issues related to Oculus VR hardware and software. (Sometimes I cross-post one of my blogposts there.)

Mark Zuckerberg presents the Oculus lineup (image taken from The Ghost Howls blog)

Yesterday, someone posted a lengthy item titled Opinion: Oculus needs to get its sh*t together in social, which I have only quoted in part (so please go over there and read the whole thing):

First, it’s ridiculous that cross-communication between the Go and Rift communities is so difficult, especially with Quest coming. VR social presence should be an underlying infrastructure that the whole Oculus ecosystem can plug into… Oculus: don’t split the VR community simply because of which of your own platforms they buy. Make communication easy, automatic, and built in.

Second, where the hell is the metaverse/Oasis/shared world? I know, VR Chat, AltspaceVR, Rec Room, High Fidelity, etc. …but why has Oculus–let alone Facebook!–abdicated this whole sector to third parties?

Now, this poster has raised a valid point. Why is social VR across all the Oculus hardware such a disorganized mess? Currently, Facebook offers Oculus Rooms for the Oculus Go users and Facebook Spaces for Oculus Rift users. Is Facebook going to use the release of the Oculus Quest (which I predict will be popular) to try and clean up this situation?

Some of the comments to this poster are worth quoting (again, I am going to take it easy with the quoting, taking into account the feedback I have recently received). I did go in and write a lengthy response, outlining the situation as I see it, and pointing out that there were already many social VR platforms which have been in development for several years (Sansar, High Fidelity, Sinespace, etc.).

When I said that High Fidelity and VRChat were planning to support the upcoming standalone Oculus Quest VR headset, one person responded:

Sadly, I don’t think VRChat’s gonna support Quest. It’s just not compatible with mobile CPUs. Hell, it brings modern up-to-date PC’s to a standstill with too many people. I very much doubt the Snapdragon 835 can handle all the custom shaders, avatars, IK, etc. The team would basically need to do a full rewrite. And that’s unlikely unless the team was way bigger.

Someone else said:

Do you really WANT Facebook also own the social VR “metaverse”? That seems like a really terrible idea in general to have a monopoly control so many things. Especially one that has proven dozens of times that they could care less about our privacy in respect to their profits…

I for one HOPE Oculus fails WILDLY on social VR. The alternative would be the worst-case dystopian future for VR and likely disasterous for humanity as a whole in the long-term.

To which I say, Amen and Hallelujah. However, Facebook has the deep pockets (lots of money) and the hardware (Oculus) to totally up-end the current, nascent social VR market, if they finally get their act together and choose to do so.

And finally, one person said:

To me it comes down to this—people want Second Life in VR format. It’s true. A metaverse where you can build or be anything you want…but in VR. We know that SL’s infrastructure cannot be upgraded to do that. High Fidelity, even with the founder of SL, isn’t cutting it. And neither is Sansar, Linden Lab’s actual VR offering.

I believe there are two main issues impeding them. 1. Instances – every platform nowadays does instance based “world” creation. Worlds are not permanent, player join numbers are limited, and the worlds are not visually connected to each other. People want permanence! To be able to wander from one place to the next aimlessly. This isn’t just an issue limited to VR of course (hello WoW). The reasoning behind this is that it’s much easier on the server hardware. Personally, I’d deal with some lag to be able to participate in a true open world environment. And

2. Adult content – All of the platforms are scared as f*** here. They don’t want their brand to be ‘marked’ by that, they don’t want to have to figure out how to police it, etc etc. That stuff is not going away. Whether they admit it or not, adult content has kept Second Life alive and thriving for 15 years (they still have ~50k concurrent users). Bottom line—deal with it. Embrace it. It’s going to happen with or without you.

This wide-ranging and fascinating discussion is the kind of thing that Reddit has become well-known for, and I would encourage you to go over there and read all or most of it for yourself, and perhaps add your own comments.