INDIVIDUAL PROJECT
AI Composer for Live streaming
WHAT'S
MUSIGO?
Musigo is an AI music composer that generates real time background music for live streaming videos based on the user’s preferences.
EXPLORATION
How is computer vision trained in order to interpret the world around us?
Before starting this project. I was curious about how AI is trained to interpret the world. To understand it, I explored many Runway models and existing AI tools.
One of the runway model is DenseCap. It’s a model that can generate sentence descriptions of regions in an image. I ran the model with live streaming videos. It is able to detect the objects in the video, and describe those objects with adjective words in real-time.
I also learnt that AI can compose music in one click. This is an example of AI composer called AIVA.
By training the machine with great samples of music database, it is able to compose music based on different genres and styles. User can also adjust the AI composing by altering some elements of it. The more training of music samples that machine took, the more natural result it can get.
AI is able to be a live producer
From the AI study,I learnt that AI is able to analyze mages, and create music in real time. With the power of agentive technology, we may make music for videos while it’s playing. When it comes to video in real-time, the first thing comes into my mind is live streaming videos.
Many people don’t have music while taking live streaming videos.
live streaming services has been growing fast in recent years. I looked into some live streaming apps like periscope, youtube live, instagram live, and twitch. I see that many live videos don’t have background music or using music that doesn’t match.
Main reasons of not using background music:
-
being stressful while on live
-
Difficult to find proper music in short time
-
It’s too complicated to operate for non-professional live streamers
-
Copyright of music
A good background music should interact with the scene
This video is a great example of how music interact with the video scenes. The landscape scenes is using cinematic music with slower tempo. When the vloger starts talking, the volume gets lower, and the tracks are simpler.
OPPORTUNITY
AI can generate real-time music for live streaming videos by analyzing the video contents in real time, based on the training of the video database.
PROTOTYPING AI ALGORITHM
How to train the machine to understand user’s expectations?
To explore how to train the machine to understand user’s expectations, I prototyped the algorithm by playing different genres of music, and asked user about their feelings and imagination. Then the participants checked the adjective words and context music they felt connected with the music.
Insights
People have shared opinions and feelings about music.
They also have their own preferences of using the music.
Design choices
People have shared opinions and feelings about music.
They also have their own preferences of using the music.
SYSTEM DIAGRAM
USER SCENARIO
Michael loves traveling, he also likes sharing his experience on live streaming apps. He sees a new plug-in called Musigo on instagram live. It says that he can use AI to make music for his live. He wants to try it.
Key user points
Having royalty free music along with the live video that matches well.
Michael Plays with Musigo and finds out he can adjust the music with keywords or humming. He hums the melody he likes. The music is tuned based on his melody.
Key user points
Being able to adjust the music based their personal preferences.
Michael likes using Musigo to add tone to his live streaming video. After few months of use. He doesn’t need to make adjustment to get the music he like. He can just click play and start his live immediately.
Key user points
Quickly play the music they like without adjustment after a long period of use.
ITERATION
I want the design to be generative, integrated, and personalized. After many rounds of iteration. I decided to design a minimal style UI that can be easily embedded into an existing live streaming program.
