Niantic, the creator of Pokemon Gorecently invited me to their London offices to see their new mobile pet simulation Peridot. I saw a presentation, tried out the game – which came out on iOS and Android on May 9 – and got to hang out with some of the developers, including the IT guy who heads up the small research team that builds the machine learning technology behind Peridotof potentially revolutionary new augmented reality technology.
But no demo could better illustrate the potential of Peridot over half an hour I spent with the game a few days later when I was given a test version to try at home. I have two young children (ages 4 and 6), who were immediately charmed to see the cute creature from Niantic’s invention, called Peridot, hatch on my phone screen and run around my house. . We petted him, fed him, threw him a ball that seemed to bounce off the walls of my living room, and watched him run around and behind furniture, only slightly erratically.
Image: Niantic
While checking Baby Dot’s wishes, I saw that she wanted to eat a dandelion, which must be picked from grassy areas. Our backyard is paved, but there’s grass across the street, it’s a nice spring evening and there’s a little time to spare before bath time — let’s go -y! We left and the kids screamed as they saw the peridot running past us down the driveway. They were impressed that the game could tell the difference between grass, foliage, and cobblestones, so the creature could seek out different objects of each.
I noticed that my peridot wanted to feed on a habitat, what this game calls local map-based points of interest shared by all of Niantic’s AR games (in Pokemon Go, they are called PokéStops). I saw that it was only a few steps away and that I had never visited before (I am not a Pokemon Go player) — why not? We headed. Turns out the point of interest was a stinky Victorian pipe (I live in London, can you tell?), which looks like a lamp post without a lamp and is actually some kind of giant straw designed to release noxious gases from the sewers below, high above the heads of the queen’s good subjects. I had never heard of these things or noticed this one before. We returned home having had a great time, some unscheduled exercise, and learned a bit about local history (and upgraded our pet). Niantic couldn’t have scripted it better.
Peridot is a fairly typical pet simulation, in the style of something like Nintendogs, crossed with Niantic’s vast map data resource and a new generation of AR technology. You interact with your pet to keep it happy and earn growth points, growing it from baby to teenager to adult. As an adult, the creature will want to be “released” into a habitat where it can breed with other peridots (animals are asexual) and spawn a new baby for you to care for. (Producer Ziah Fogel says the plan was for players to say goodbye to their adult Peridots permanently at this point, but testers got too upset, so you have to hang in there.) Niantic has built enough variables into it. the DNA of the creatures that each one is genetically unique, and each couple will create an equally unique new baby.
Image: Niantic
You can imagine the location-based features that Niantic has incorporated into this kind of game; walk your pet, forage for food in different environments, engage in herding with other local players as a sort of social endgame. You can also imagine the sweet but insistent trickle of goals, progress, currency, and rewards built into a free live service game like this, along with the customization features (yes, your pet can wear hats ).
But Niantic has another mission with Peridot, which is about pushing the boundaries of AR and changing the way people think about it. The idea is that your pet should, on some level, be able to understand and respond to the real world it (or rather, your phone’s camera) is looking at. That’s the work being undertaken by Niantic’s R&D team, which is based on six floors of a cramped, cramped office building in London’s Covent Garden area, and is led by Niantic’s chief researcher Gabe Bostow.
The AI research work begins with precise 3D space mapping and path finding around real-world objects, as well as obstacle occlusion which means your pet will realistically disappear behind objects. chairs and tree trunks. (Bostow seems frustrated that the design team requested that a shadow showing the position of the peridot behind the object be put back into the game, lest players get too stressed out about losing their charges.) It continues with the “semantic segmentation”, as Bostow puts it, of your phone’s video stream of the environment, using machine learning to train the code to tell the difference between grass and water, to recognize a television screen so your peridot can scold you about screen time and to identify classes of objects such as pets, people, and plates of food. The neural networks required for this weren’t possible on mobile phones even five years ago, Niantic says. Peridot certainly exercises your phone’s CPU and GPU, if its effect on battery life is anything to go by.
Image: Niantic
Peridot started life as a tech demo, and while the pet sim genre is a natural fit for Niantic’s endeavors, you can tell. In some ways, it feels even more like the skunkworks project of a technology company hungry for progress than a labor of love from a game developer. The Bostow team is sharing its findings with the academic community, while code will be available to licensed third parties through Niantic’s Lightship suite of developer tools. You can imagine the benefits the technology could have for robotics or other assistive technologies; Bostow likes to imagine a future where he can hold his phone to see the location of plumbing behind walls and under floors when doing renovations. There are probably more sinister uses for this combination of machine learning environments with Niantic’s industrial scale geopositional data farming, but don’t think about it – look how cute these little creatures are!
In practice, the technology is far from perfect: it can’t reliably spot food, can see water better outside than inside, and has trouble with windows. But there’s an undeniable magic to watching your peridot frolicking around your house, jumping on tables, and noticing your cat. It feels a bit alive, and it will only feel more alive as the game releases and players start training Niantic’s neural networks en masse. As Fogel says, a game about growing small creatures and introducing them to the world is a good thematic match for technology that will itself improve its understanding of the world and the realism of its behaviors over time.
I think that’s impressive. But it’s not my reaction that counts, or who tells you if Peridot will be a success. My kids were thrilled with it within seconds, and my 4 year old was walking around the house calling “pennydot” within minutes. Plus, we all know about Victorian stinky pipes by now.
