ByteDance Develops OmniHuman, an AI Framework That Can Generate Realistic Videos of Humans

ByteDance, the corporate behind TikTok, just lately shared its analysis on a brand new synthetic intelligence (AI) framework. Dubbed OmniHuman, it’s a video-generation framework that may create reasonable human movies with full-body motion and lip-syncing. The researchers acknowledged that it requires a human picture together with movement indicators resembling video or audio to generate output. A number of demonstration movies generated utilizing the AI mannequin have additionally been shared, showcasing the realism of the ultimate output. Notably, the corporate acknowledged that the AI mannequin is on the market within the public area.

OmniHuman Can Generate Life like Human Movies

The researchers shared a number of demonstrations and detailed the framework on its website. It’s an end-to-end system that was constructed utilizing a novel multimodality movement conditioning blended coaching technique, the publish claimed. Whereas the researchers didn’t share any benchmark metrics, they claimed that the AI mannequin “considerably outperforms current strategies.”

OmniHuman can generate movies utilizing a picture of the particular person and a movement sign. Movement indicators will be audio solely, video solely or a mixture of audio and video. The AI mannequin can generate reasonable movies based mostly on textual content prompts. These movies will be full-body the place the limbs, facial expressions, and lip motion will be synced with the audio or music taking part in within the background. OmniHuman can generate movies in several facet ratios, permitting flexibility to customers.

OmniHuman output instance
Picture Credit score: OmniHuman

The usage of movement indicators is a novel method, which the corporate is asking omni-conditions coaching. With this, the AI mannequin is educated on totally different modalities, together with textual content, picture, audio, and video. Researchers stated this allowed the mannequin to study blended conditioning which overcame the shortage of high-quality knowledge.

Notably, the mannequin was educated on 18,700 hours of human video knowledge. The small print concerning the coaching course of have been documented in a paper printed within the on-line pre-print journal arXiv.

The corporate additionally shared a number of demonstrations of movies generated utilizing the mannequin, and the outcomes look like extremely reasonable with pure physique actions, hand gestures, and lip actions. Such realism has additionally raised issues about deepfakes. Nonetheless, the corporate has specified that the AI mannequin is at the moment not out there to be downloaded, and there’s no service folks can use to entry its capabilities.