The concept of character-extraction can re-purpose the uploading of videos on social media. The research team of Facebook has come out with an AI with which the users can appear games or a different virtual environment with their own image and moves.
What Facebooks Vid2Play does is turning real people into playable characters. Basically, the AI analyses videos, extracts persons and their movement, then puts them into a different environment in controllable way.
The video still has to contain relevant moves. For example, the user has to upload a dancing video to appear in a dancing app, or has to perform his/her own fencing or tennis moves to be playable in a sports game, and so on.
The AI can filter out the redundant objects, identify the relevant moves, replace the background and environment. And it can add certain controls – on the keyboard or a controller – to each identified move for the users to execute with their own image.
Of course, this is an oversimplification. If this video looks too technical at first sight, it shows the process in plain English from 3:30. The user’s video have to go through two neural networks called Pose2Pose and Pose2Frame. The first network is designed to separate the person on the video and the identify the moves of a certain category (e.g. tennis). Pose2Frame then extracts the person’s image – along with the shadow and the items – and puts them into a new scene.
The AI is said to be capable of ’learning’ a person from just a few short videos. The system filters out other people and can compensate for the different came angles.
The result is a little bit blurry and the transfered images of people do not always seem to have a proper physical contact with the replaced environment. It is nevertheless impressive. It is perfectly conceivable that Facebook will come up with a series of online browser games (tennis, dancing, etc) that utilize Vid2Play.
Facebook states that Vid2Play addresses a computational problem not previously fully met. Potentially, this technology can extract and re-insert controllable characters not just from personal videos, but from any video content of social media and/or video games.