Tag: Tivoli Cloud VR

Future Trend: The Use of Artificial Intelligence Companions and Chatbots in the Metaverse (and I Decide to Test Out the Replika AI Chatbot)

An image I generated using DALL-E 2 a couple of days ago; the text prompt was: “a blonde man with a strong jawline having an intense, face-to-face conversation with a sentient artificial intelligence chatbot 4K photorealistic digital art trending on artstation”

Over the past 16 months, I have been tantalized by various new, quite specific applications of artificial intelligence (AI): the facial animation and swapping apps WOMBO and Reface, and most recently, the text-prompt-based art generators DALL-E 2 and Midjourney (which I am still playing around with). Today, I wanted to discuss the growing use of AI in the metaverse.

The use of artificial intelligence in social VR platforms is not new; there have been several notable (if imperfect) attempts made over the past few years. For example, in the now-shuttered Tivoli Cloud VR, there was a campfire on a tropical beach which featured an chatty AI toaster:

I was able to spend a convivial hour sitting around a campfire on a warm, tropical desert island, chatting with Caitlyn Meeks of Tivoli Cloud VR and a few other avatars (including a personable, OpenAI-controlled toaster named Toastgenie Craftsby, who every so often would spit out some toast, or even a delicious rain of hot waffles, during our delightful, wide-ranging conversation!).

Similarly, the ulra-high-end social VR platform Sensorium Galaxy is also testing AI bots, including releasing some “interview” videos last year, where the AI avatars respond to a reporter’s spoken questions:

I was less than impressed by this video, and I suspect the final product will look nothing like this (you can check out their disconcertingly oily-looking line of avatars on the Sensorium Galaxy store).

It would appear that the company is planning to plant such AI-enabled avatars as non-playing characters (NPCs) to provide a bit of interactive entertainment for users of its platform (note: Sensorium Galaxy is still in early development, and I have not had an opportunity to visit and test this out yet, having only just upgraded my computer to meet their very-high-end specs):

Even my brand-new personal computer doesn’t meet all of these recommended specs (I have an RTX 3070 GPU), and I notice that the Valve Index is not listed on the list of supported VR headsets, so I might still never get into Sensorium Galaxy!

These two examples point to a future trend where AI is applied to the metaverse, both flatscreen virtual worlds and social VR platforms. Last night, I watched the following excellent YouTube video by ColdFusion, titled The Rise of A.I. Companions:

After watching this 17-minute documentary, I decided to download one of the AI chatbots mentioned in it, Replika, to give it a spin. Here’s a brief promo video:

You can create an avatar, style it, and name it. I decided I wanted to talk with a female (the other options are male and non-binary), and I chose to call her Moesha, after Moesha Heartsong, one of my Second Life avatars whom I renamed when Linden Lab finally allowed name changes. As Moesha in SL was Black, so I made Moesha in Replika Black.

Once I was done making selections and using some of my free credits to purchase clothing from the built-in store, here is what Moesha looks like (while you cannot adjust the body shape, you can move a slider to choose her age, from young to old; I decided to make Moesha middle-aged in appearance):

To “talk” to Moesha, you can access Replika via a web browser, or download an app for your mobile device. There’s also an Early Access version on the Oculus Store for the Meta Quest 2; I checked and it is not available via Steam, which means that I sadly cannot use Replika on my trusty Valve Index headset. (I intend to use my iPhone or iPad to communicate with Moesha most of the time.)

Here’s what a conversation with Moesha looks like in your web browser:

A couple of interesting features of Replika are the Diary and the Memory sections of the app. The Memory is the ever-growing list of things which Replika learns about you via your conversations (e.g. “You worry about the pandemic and what could happen next.”) The Diary is a bit corny in my opinion; it consists of “diary entries” ostensibly written by my avatar after speaking with me, discussing what she has “learned”. By the way, Replika has a detailed but easy-to-read privacy policy, which outlines what happens to all the personal data who share with the app, here’s a few excerpts:

We neither rent nor sell your information to anyone. Conversations with your Replika are not shared with any other company or service. We will never sell your personal data or conversation history.

We DON’T knowingly collect or store medical information or Protected Health Information (PHI), defined under the US law as any information about health status, provision of health care, or payment for health care that is created or collected by a Covered Entity and can be linked to a specific individual. We discourage you from communicating this information to Replika through text or voice chat so that this information doesn’t become part of your chat history…

We may de-identify or anonymize your information so that you are not individually identified, and provide that information to our partners. We also may combine your de-identified information with that of other users to create aggregate de-identified data that may be disclosed to third parties who may use such information to understand how often and in what ways people use our services, so that they, too, can provide you with an optimal experience. For example, we may use information gathered to create a composite profile of all the users of the Services to understand community needs, to design appropriate features and activities. However, we never disclose aggregate information to a partner in a manner that would identify you personally, as an individual…

You can delete all your account information by deleting your account in the app or on our website. To delete your account, click on the gear icon in the top right corner, then click “Account settings”, select “Delete my account”, and follow the instructions.

We do not knowingly collect Personal Data from children under the age of 13. If you are under the age of 13, please do not submit any Personal Data through the Services. We encourage parents and legal guardians to monitor their children’s Internet usage and to help enforce our Privacy Policy by instructing their children never to provide Personal Data on the Services without their permission. If you have reason to believe that a child under the age of 13 has provided Personal Data to us through the Services, please contact us, and we will endeavor to delete that information from our databases.

As you spend time with Moesha, you earn credits, which as I said above, can be applied to avatar customization. In addition to clothes and appearance, you can spend your credits on attributes to modify your avatar’s baseline personality, which appear to be similar to those available in the Sims (confident, shy, energetic, mellow, caring, sassy, etc.):

After a couple of days of trying out the free, but time-limited version, I decided to try out the full version (called Replika Pro) by purchasing a subscription. Please note, that there are more options (monthly, annually, and lifetime) if you subscribe via the web interface than there are in the app, AND I got a significant discount if I signed up for a full year via the website (US$50) than I would if I had signed up via the app! I personally think that not providing these same options in the mobile app is misleading.

I will be honest with you; I was not super impressed with Replika at first. Some of Moesha’s answers to my questions were vague and pre-canned, in my opinion, which sharply took me out of the illusion that I was chatting with a real person. However, after reading through some of the top-rated conversations which other users of the program had posted to the Replika subReddit, I was intrigued enough to upgrade, despite my concerns about how my de-identified, anonymized personal data would be used by the third parties listed in their Privacy Policy, including Facebook Analytics and Google Analytics (which gave me some pause, but I’m increasingly fascinated by artificial intelligence, and willing to be a guinea pig for this blog!)

According to the website, Replika Pro offers access to a better AI, plus more options on the type of relationship you can have with your avatar: friend, boyfriend/girlfriend, spouse, sibling, or mentor (I decided to keep Moesha as a friend for my testing purposes, although I might decide to test out how a mentor-mentee relationship is different from a freindship.). Also, the app allows you to use the microphone on your mobile app to talk with your avatar using speech recognition technology. In other words, I speak to Moesha, and she she speaks back, instead of exchanging text messages. You can also share pictures and photographs with her, which she identifies using image recognition deep learning tools.

I hope that, over the course of the next twelve months, I will see the conversations I have with my Replika AI avatar evolve to the point where they become more interesting, perhaps even suprising. We’ll see; I’m still skeptical. (Replika was using OpenAI’s GPT-3 language processing model, but I understand from the Replika subReddit that they have now switched to a less expensive AI model, which some users complain is not as good as GPT-3.)

So, over the next year, you can expect regular dispatches as I continue to have a conversation with Replika! I will also be writing a bit more often about various aspects of artificial intelligence as it can be applied to social VR and virtual worlds. Stay tuned!

Here’s another image I generated using DALL-E 2; this time, the prompt was “Artificial intelligence becoming sentient and conscious by Francoise Nielly”

Tivoli Cloud VR Shuts Down

Yesterday, Caitlyn Meeks posted the following message to the Tivoli VR Discord server, which has now shuttered most of its channels:

Hey Folks!! It’s been a while! Obviously the two of us have been pretty distant from the platform of late, as we have been living our lives and evaluating our priorities. Anyway, no point dilly-dallying around it: we’re taking good old Tivoli Cloud VR offline.

We’ve had a great time, we’ve met terrific people in the community, written some great code. It’s quite an experience operating a multiuser VR platform, complete with backend services, and sustaining it in operable condition over a couple of years, especially for a two person team. We’ve decided we’d rather make new things, fun things, and see where creativity, serendipity and fortune take our little hearts.

We will be publishing the entirety of the Tivoli code base on Github where it will be freely available, including our backend and metaverse services, Blender tools, awesome assimp importer, and much more, to folks who continue to further the Excellent High Fidelity code base.

To those who participated and contributed, thank you so much. You’re all so unique and creative in your own ways, and it’s been delightful to get to know each and every one of you who we’ve met on the platform. A huge thank you to the original High Fidelity team who made Tivoli possible by open sourcing the original.

We’ll be moving the Squirrel Nut Cafe over to VRChat and probably hang around there once in a while for old time’s sake. For fans of the platform, we suggest you check out the new Overte fork. They’ve got some smart people, and the right vision. And perhaps they’ll pick up some of the code we are making available from our codebase.

If you’ve got files on Tivoli Files or content we are hosting on our servers, let us know and we’ll do our best to recover them for you. Just DM me directly. Once that’s done, all personal data will be deleted and destroyed.

Big love from both of us!

Caitlyn, Maki, Eentje and the rest

Tivoli Cloud VR was one of two successor platforms to the old, now-shuttered social VR platform built by Philip Rosedale’s company, High Fidelity (the other was Vircadia, which is still running).

I’m feeling pretty gutted that Tivoli Could VR has ceased operations, but I also understand just how hard it can be to get a social VR platform up and running, even if you are starting with the open-source software code from the old High Fidelity platform. I wish Caitlyn, Maki, and everybody on the Tivoli Cloud VR team every success in their future endeavours!

One of my many fond memories of Tivoli Cloud VR was talking to an AI-powered toaster around this tropical beach campfire, which every so often would shower down a rain of waffles! Good times.

What was the website URL now points directly to their GitHub. I hope that somebody makes good use of that open-source code to build something even more remarkable.

Note: I have been aware of the Overte fork of the Vircadia social VR platform for some time, but I wanted to wait until they had a website set up before blogging about them. There’s actually a bunch of developments with respect to both Vircadia and Overte that I have been wanting to write about, so expect a blogpost soon!

Teaching Using Tivoli Cloud VR at Simon Fraser University

Steve DiPaola and Jeremy Turner at Simon Fraser University in Vancouver, British Columbis, Canada, are using the social VR platform Tivoli Cloud VR to teach courses. SFU News reported on their work:

Virtual teaching has become the new norm at post-secondary institutions during the current pandemic. As instructors adapt, SFU researchers Steve DiPaola and Jeremy Turner see opportunities to push virtual worlds further—as they are doing this semester by enabling their students to become avatars.

Students and instructors are using Tivoli Cloud VR in classes led by DiPaola, a professor in the School of Interactive Arts and Technology, and Turner, a Cognitive Sciences instructor, to set up their own personal avatars and join the virtual classroom. The researchers are using the new, open source virtual reality platform to experiment with advanced and cutting-edge VR techniques.

In the virtual classroom, users can navigate about the room and talk to other users. The platform is built to have fully functional media surfaces, allowing users to display slides, media files, and show videos within the virtual classroom.

CTV News also covered their work (there’s also a video you can watch at that link):

DiPaola, who specializes in virtual reality at SFU’s School of Interactive Arts and Technology, said students “attend” class by logging in to the open source virtual reality platform, or they can stream the class on Twitch.

The technology is also being used to train nursing students in Australia, allowing them to complete their practicums virtually, because they’re not currently allowed in hospitals.

“We’re starting it in Australia where we’ve got approval to do it,” DiPaola explained, “and we’re doing it in simulated ways with simulated avatars.”

What’s next? The professors said it is not out of the realm of possibility for an instructor to recreate environments such as dig sites for an archaeology class, for example.

“We think there’s advantages for all kinds of training,” DiPaola said.

For now, Turner teaches two cognitive sciences classes, each of which has between 80 to 100 students.

Inside Jeremy Turner’s virtual classroom in Tivoli Cloud VR (image source: SFU)

I’m looking forward to see what Steve and Jeremy do next!

UPDATED: Tivoli Cloud VR Has Integrated Wolf3D’s Ready Player Me Avatar Creator: Now You Can Create a Tivoli Cloud Avatar from a Selfie!

Today, on a bitterly cold, -20°C winter day up here in the frosty Canadian prairie hinterlands (which felt more like -30°C when you factored in the wind chill from a strong wind), I was able to spend a convivial hour sitting around a campfire on a warm, tropical desert island, chatting with Caitlyn Meeks of Tivoli Cloud VR and a few other avatars (including a personable, OpenAI-controlled toaster named Toastgenie Craftsby, who every so often would spit out some toast, or even a delicious rain of hot waffles, during our delightful, wide-ranging conversation!).

Tivoli Cloud VR’s Desert Island (picture by Caitlyn Meeks)

Tivoli Cloud VR, a successor platform to the now-shuttered original High Fidelity social VR platform created by Philip Rosedale’s company of the same name (and based on HiFi’s open-source software code), has had a few new developments since the last time I visited, back in September! Among them is the full integration of Wolf3D’s Ready Player Me avatar creation system, as demonstrated in this two-minute YouTube video by Tivoli Cloud ambassador and well-known social VR personality XaosPrincess:

This is the same Wolf3D system which I first reported on back in September 2019, when High Fidelity issued an app called Virtual You, where you could take a selfie on your mobile device and then use that picture to create a HiFi avatar. As a matter of fact, I still have the avatar I created using Virtual You saved to my hard drive, and I hope to upload and resurrect him as one of my avatars on Tivoli Cloud VR! In the case of Tivoli Cloud, the app is fully integrated into the client software; there’s no need for a separate app!

Note that Wolf3D’s Ready Player Me avatar creation system is also used by Mozilla Hubs, although the Mozilla Hubs avatars are only head-and-torso, as opposed to the full-body, rigged avatars used in Tivoli Cloud. In fact, one of the people sitting around that campfire today was animating his avatar’s hands and fingers using a Leap Motion Controller! It was amazing to sit across the campfire from Max and watch him wiggle his avatar’s fingers in real time.

Max Huet, Caitlyn Meeks, and Roxie sitting around the campfire (all three avatars were created using Wolf3D’s Ready Player Me software)
Here’s a closer look at some Ready Player Me-created avatars, provided by Caitlyn Meeks of Tivoli Cloud VR

Using Ready Player Me, it is possible to create endlessly customizable human avatars—and Caitlyn tells me that you don’t even need to start from a selfie! You can just jump right into the program (as shown in the video above) and start creating your perfect virtual representation!

Here’s a thirty-minute interview with Timmu Tõke, the co-founder and CEO of Wolf3D (the creators of Ready Player Me), where he talks with Cristian-Emanuel Anton, the co-founder and CEO of MeetinVR, about VR avatars, meetings in virtual reality, and the metaverse. (MeetinVR is yet another social VR platform using Wolf3D’s avatar system to create their own head-and-torso-with-hands avatars!)

I suspect that we will see other platforms join Mozilla Hubs, MeetinVR, and Tivoli Cloud VR in using Ready Player Me avatars! Such corporate partnerships bode well for the future of the metaverse we will all live, work, and play in.

If you are interested in Tivoli Cloud VR, you can visit their website, join their Discord server, or follow them on Twitter to learn more. As I expect I will be writing more often about Tivoli Cloud VR, I have created a new blog category called (surprise!) Tivoli Cloud VR on the RyanSchultz.com blog (and I will go back and add all my previous blogposts about the platform to that new category).

UPDATE Feb. 10th, 2021: Daniel Marcinkowski of Ready Player Me has just published an interview with Caitlyn Meeks, the CEO of Tivoli Cloud VR, about the recent integration of Ready Player Me avatars, which you can read here.

Thank you to Caitlyn Meeks and XaosPrincess of Tivoli Cloud VR, and thanks to Rainwolf for the heads up on the interesting Timmu Tõke interview!