AR-LOGO-NEW2@3x.png

INDIVIDUAL PROJECT

AI Composer for Live streaming

WHAT'S

MUSIGO?

Musigo is an AI music composer that generates real time background music for live streaming videos based on the user’s preferences.

EXPLORATION

How is computer vision trained in order to interpret the world around us?

Before starting this project. I was curious about how AI is trained to interpret the world. To understand it, I explored many Runway models and existing AI tools. 

One of the runway model is DenseCap. It’s a model that can generate sentence descriptions of regions in an image. I ran the model with live streaming videos. It is able to detect the objects in the video, and describe those objects with adjective words in real-time.

I also learnt that AI can compose music in one click. This is an example of AI composer called AIVA.

By training the machine with great samples of music database, it is able to compose music based on different genres and styles. User can also adjust the AI composing by altering some elements of it. The more training of music samples that machine took, the more natural result it can get.  

AI is able to be a live producer

From the AI study,I learnt that AI is able to analyze mages, and create music in real time. With the power of agentive technology, we may make music for videos while it’s playing. When it comes to video in real-time, the first thing comes into my mind is live streaming videos.

Many people don’t have music while taking live streaming videos.

live streaming services has been growing fast in recent years. I looked into some live streaming apps like periscope, youtube live, instagram live, and twitch. I see that many live videos don’t have background music or using music that doesn’t match.

Main reasons of not using background music:

  • being stressful while on live

  • Difficult to find proper music in short time

  • It’s too complicated to operate for non-professional live streamers

  • Copyright of music

A good background music should interact with the scene

This video is a great example of how music interact with the video scenes. The landscape scenes is using cinematic music with slower tempo. When the vloger starts talking, the volume gets lower, and the tracks are simpler.

OPPORTUNITY

AI can generate real-time music for live streaming videos by analyzing the video contents in real time, based on the training of the video database.

PROTOTYPING AI ALGORITHM

How to train the machine to understand user’s expectations?

To explore how to train the machine to understand user’s expectations, I prototyped the algorithm by playing different genres of music, and asked user about their feelings and imagination. Then the participants checked the adjective words and context music they felt connected with the music.

Insights

People have shared opinions and feelings about music.

They also have their own preferences of using the music.

Design choices

People have shared opinions and feelings about music.

They also have their own preferences of using the music.

SYSTEM DIAGRAM

USER SCENARIO

Michael loves traveling, he also likes sharing his experience on live streaming apps. He sees a new plug-in called Musigo on instagram live. It says that he can use AI to make music for his live. He wants to try it.

Key user points

Having royalty free music along with the live video that matches well.

Michael Plays with Musigo and finds out he can adjust the music with keywords or humming. He hums the melody he likes. The music is tuned based on his melody.

Key user points

Being able to adjust the music based their personal preferences.

Michael likes using Musigo to add tone to his live streaming video. After few months of use. He doesn’t need to make adjustment to get the music he like. He can just click play and start his live immediately. 

 

Key user points

Quickly play the music they like without adjustment after a long period of use.

ITERATION

I want the design to be generative, integrated, and personalized. After many rounds of iteration. I decided to design a minimal style UI that can be easily embedded into an existing live streaming program.

MAIN FEATURES

PLEASE TURN ON THE AUDIO FOR THE BEST RESULT 

GENERATING AI MUSIC

Musigo can generate AI music from what it’s seeing through the camera. Computer vision and real time data detects objects and motion, and monitors weather and time.

DEEP UNDERSTANDING OF USER'S MUSIC PREFERENCE

Musigo can generate different styles of songs until the user finds one they like. The system will learn from the user's behavior, so that it can anticipate the user's preferences.

ADJUST THE MUSIC BY HUMMING A TUNE

User can adjust the melody of the music by humming a tune.

EMPHASIZING HUMAN VOICE

When user starts talking during live streaming, the music will be lighter, and the sound volume will be lower, so that the audience can hear the voice better.

PANNING

When the user is panning the camera, the audio levels automatically adjust based on the direction the camera is facing.

NEXT PROJECT

MUSEUM OF MOTION