Social News

Mark Zuckerberg told about the creation of a personal home assistant – “Jarvis”

Founder and CEO of the social networking site Facebook, Mark Zuckerberg, posted on his page a note in which he spoke about developing his own home assistant following the example of Jarvis from Iron Man. Editorial chose the most interesting moments.

In early 2016, Mark Zuckerberg spoke about his goal for the year: to create a home system of artificial intelligence “like Jarvis from Iron Man”, which would help him solve personal and work tasks. On December 19, 2016, Zuckerberg posted a note on his Facebook page that shared his successes.

“On the one hand, it was easier to achieve this goal than the one I set last year. In total, I spent more time on the run. But in some ways the creation of such a system turned out to be much more complicated than I expected, ”admits Zuckerberg.

The assistant-designed assistant controls light, temperature, doors, cameras, and smart devices in the house. Jarvis can be controlled using three different interfaces: a voice application for the iPhone, a bot for Facebook Messenger, or a surveillance camera. Algorithms of speech recognition and analysis and face recognition technology are built into the system.

The first and one of the most laborious steps, according to Mark Zuckerberg, was to combine all the “smart” devices in the house into a single system that could be controlled using one interface. The entrepreneur said that he even had to use reverse engineering (reverse engineering) to build the simplest control system for all devices using a PC.

“All devices in my house use different interfaces and data transfer protocols. Worse, some of them do not connect to the Internet at all. Some of the devices can pass a simple command — for example, turn the power on or off. But that’s not enough, ”says the creator of Facebook.

For example, I found out that almost no modern toaster allows you to lower the bread in advance before the power is on, so that as soon as the power comes, it starts working. I had to use a 1995 toaster paired with a smart switch.

According to Zuckerberg, the same failures befell him when working with the “gun” for T-shirts and with the dog food dispenser. “So that everyone can create an assistant like Jarvis, the industry needs to take care to create uniform APIs and performance standards for all smart devices.”

Speech recognition

Then the entrepreneur added to the system management using text messages in natural language, and later – voice control. At first, the system simply searched for the key words in the received text – for example, “bedroom”, “light” and “turn on”. Then Zuckerberg taught her to recognize synonyms.

“Understanding the context is a very important feature for artificial intelligence. For example, when I ask to turn on the light in my office, it means not at all the same as the same phrase from the mouth of Priscilla. Or when I ask the system to turn on the music without naming the room, it needs to independently determine where I am and turn on the music in this room. ”

One of the most interesting areas for research, according to Zuckerberg, was music. “There are so many albums, artists and various compositions. The light can only be made brighter or dimmed, but how many different music requests start at play: for example, “play someone like you”, “play someone like adele”, and “play some adele” are just requests related to Adele’s work, and each of them has a completely different meaning, although outwardly they are very similar. “

The entrepreneur notes that the more accurate the context of the system, the better it understands what is required of it. “Usually I ask Jarvis to” turn on the music, “he studies what I’ve listened to lately and offers a playlist. If he didn’t guess his mood, I say: “No, this is not an easy composition. Turn on something light,” and then he switches the song and learns at the same time. In addition, he distinguishes who speaks with him – me or Priscilla “.

We use such “open” requests much more often than rigidly defined ones. It seems to me that this is what modern home helpers lack – the ability to work with such requests. And here lies a huge opportunity.

Face and image recognition

Zuckerberg also introduced image and video recognition algorithms into the system. “There are many problems that a home assistant has to face, for example, motion detection on video. Is Max sleeping in his crib or is he already awake? Who is lying on the floor in the room – our dog or carpet? Who is behind the door? ”

The task of recognizing faces, says Zuckerberg, is more difficult than the task of recognizing objects – because mostly people have similar facial features. But the Facebook team has achieved significant success in solving this problem – the social network algorithms are able to identify the user’s friends and himself in the photos. These algorithms and used by the entrepreneur.

Mark Zuckerberg installed several cameras at different angles at the front door of his house – to make sure that no matter how the guest stood, the system would be able to see his face completely.

I created a simple server that constantly checks the images on the cameras. He runs face search algorithms to see if someone is standing by the door. If the system detects a guest, it launches a face recognition algorithm to understand exactly who came.


Realizing what kind of person is standing in front of the door, “Jarvis” checks whether there is such a person in the list of guests who are waiting for the owners of the house. If there is, the system opens the door and lets the guest inside, and also notifies the hosts about who came.

Face recognition algorithms not only help the system to identify guests and control the doors, but also, for example, determine which room Mark, Max and Priscilla are in to correctly respond to requests – or, for example, enable Mandarin for one year old Max when she wakes up , writes Fast Company. “As I said, the more context the system has, the smarter it becomes.”


Bot for Facebook Messenger

At first, Zuckerberg ran the system from his computer. “But for the assistant to be really useful, the owner must have access to the system, no matter where he is located – even if he is at work or on a walk. I decided to create a bot for the messenger – because it is much easier than developing a separate application. ”

Using a bot, Zuckerberg can send both text and voice requests to the Jarvis server. Audio recordings are processed directly on the server and converted to text with which the system can work in the future. The assistant also sends messages or camera images to the entrepreneur, if someone has come to his home, and reminds of business.

Voice and Speech Recognition

In order to give commands to the assistant with the help of voice, Zuckerberg created a separate application that constantly listens to what is happening around and determines when an entrepreneur calls Jarvis. The entrepreneur has developed an application for iOS, and plans to implement an Android program soon.

I have not been developing iOS applications since 2012, and I can see that the tools that Facebook has introduced since then for building applications and voice recognition are impressive.

According to Zuckerberg, he found that voice systems still don’t understand conversational speech very well and work more efficiently with clearly structured queries. At the same time, speech analysis and machine learning systems as a whole are often focused on one area and cannot equally accurately recognize requests from different areas.

Voice systems seem more human to computers than computers, Zuckerberg says: “As soon as I built voice algorithms into Jarvis, I wanted to add a bit of humor to them. Partly due to the fact that the assistant needs to communicate with Max and entertain her, and partly because I myself would be more comfortable. ”

Using internal Facebook tools

As Zuckerberg notes, he did not have to write a lot of code – as the CEO of Facebook, he was able to use the company’s internal libraries and tools. “I do not stop programming, but more and more often I create some personal projects – like Jarvis. And creating a home assistant not only helped me understand the state of artificial intelligence, but also allowed me to feel what it was like to be an engineer on Facebook. And this is impressive. ”

All the code is perfectly organized, and finding what you need is very easy – whatever area it belongs to. One of our values ​​is speed. This means that any developer, having come to the company, can create something faster than anywhere else. And faster than he would have done it on his own.

What’s next

Zuckerberg notes that he will continue to develop the system – he has many more ideas that one could add to it. In the near future – the creation of an Android application and the connection of new devices in the house. Then the entrepreneur plans to work on learning algorithms, and in the long term, to understand how the system could be applied in other homes.

In total, the creation of an assistant at Zuckerberg took about 100 hours. According to him, he is still far from understanding how machine learning should work, but even if he spent more than 1000 hours at work, he would not have come to a clue.

According to the Fast Company correspondent, who managed to visit Zuckerberg at home and look at the system, the assistant still often makes mistakes. For example, the entrepreneur had to ask Jarvis four times to turn off the lights in the room until the system understood the command.

The entrepreneur plans to talk about his goal for 2017 until the end of 2016 or in the first few weeks of 2017.


Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button