View source for Abhishek/metabrainz GSoC2016Proposal

<h3> Personal Details </h3>

<ul>
<li> <p style="white-space: pre;">Name                         : Abhishek Kumar Singh </p></li>
<li> <p style="white-space: pre;">IRC                              : kanha@freenode.net</p></li>
<li> <p style="white-space: pre;">Location(Time Zone) : India, IST(UTC + 5:30)</p></li>
<li> <p style="white-space: pre;">Education                  : MS in Computer Science and Engineering, IIIT Hyderabad</p></li>
<li> <p style="white-space: pre;">Email                          : abhishekkumarsingh.cse@gmail.com, abhishek.singh@research.iiit.ac.in </p></li>
<li> <p style="white-space: pre;">Blog URL                    : http://abhisheksingh01.wordpress.com/</p></li>
</ul>

<h2> AcousticBrainz </h2>

<h3> Abstract </h3>
<p> Implementation of an audio search system for AcousticBrainz. An Audio Information Retrieval(AIR) system that would provide client API(s) for audio search and information retrieval. Search will be based on annotated text/descriptions or tags/labels associated with an audio as well as on low and high level features. Focus is to implement an AIR system that provides text based and content based search. Users would be able to perform query through the provided client API(s) and will get list of relevant audios in return.</p>

<h3> Benefit to AcousticBrainz community </h3>
<p> “Search, and you will find” </p>
<ul>
<li> Provides client API(s) for text and content based search. Query can be done by providing some text or by submitting a piece of audio. </li>
<li> Support for advanced queries combining content analysis features and other metadata (tags, etc) including filters and group-by options.</li>
<li> Support for duplication detection. It helps in finding duplicate entries in the collection. </li>
<li> Helps in finding relevant audios quickly and efficiently. </li>
</ul>

<h3> Project Details </h3>
<br>
<u> '''INTRODUCTION''' </u>
<p> The idea is to implement a search functionality for AcousticBrainz. This would include implementing an Audio Information Retrieval System(AIRS) which would provide audio content search. This would also simplifies the task of data exploration and investigating content based similarity.</p>

<p>Audio Information Retrieval(AIR) can be: </p>
<ul>
<li> Text based </li>
<li> Content based </li>
</ul>

; Text based AIR :
<p> Query can be any text say 'rock' or 'beethoven' etc and the system will search through the text(tags, artist name, description) associated with the audios and will return list of relevant audios corresponding to matched text. It's simple to implement but doesn't help much in audio retrieval as most of the times the audio doesn't contains enough annotations.</p>

; Content based AIR :
<p> It bridges the semantic gap when text annotations are nonexistent or incomplete. For example, if one comes to a record store and one only know a tune from a song or an album one want to buy, but not title of music, album, composer, or artist. It is difficult problem to solve for one. On the other hand, sales expertise with vast knowledge of the music can identify tunes hummed by one. Content based audio information retrieval can substitute the vast experience of sales expertise with a vast knowledge of the music.</p>

<u>''Benefit''</u> :
<ul>
<li> Given a piece of audio as a query, Content based AIRS will search through the indexed database of audios return list of audios that are similar to a piece of audio given as query.</li>
<li> Can detect (duplicates) copy right issues with little tweak.</li>
</ul>

<u>'''REQUIREMENTS DURING DEVELOPMENT'''</u>
; Hardware Requirements :
<ul>
<li> A modern PC/Laptop</li>
<li> Graphical Processing Unit(GPU) </li>
</ul>
; Software Requirements : 
<ul>
<li> Gaia </li>
<li> Elastic search (storage database)</li>
<li> elastic-py library </li>
<li> Kibana (data visualization)</li>
<li> Python 2.7.x/C </li>
<li> Scikit, Keras </li>
<li> Coverage </li>
<li> Pep8 </li>
<li> Pyflakes </li>
<li> Vim (IDE)</li>
<li> Sphinx (documentation) </li>
<li> Python Unit-testing framework (Unittest/Nosetest) </li>
<li> AcousticBrainz tools
</ul> 

<u>'''THE OUTLINE OF WORK PLAN'''</u>
<p> Audio information retrieval framework consists of some basic processing steps: </p>
<ul>
<li> Feature extraction</li>
<li> Segmentation</li>
<li> Storing feature vectors(indexing)</li>
<li> Similarity matching</li>
<li> Returning similar results </li>
</ul>

<p> [[File:Air.png|center|600px]] </p>
<center> '''Basic flow in audio information retrieval system''' </center><br>
<p> Some proposed models for Content based audio retrieval </p>
<ul>
<li> Vector Space Model(VSM) </li>
<li> Spectrum analysis(Fingerprint) model</li>
<li> Deep Neural network model (Deep learning approach)</li>
</ul>

<u>'''Division of Work'''</u>
<p>
Basically, The whole work is divided into four phases :
<ul>
<li>'''Phase I'''   : Getting Features and index creation </li>
<li>'''Phase II'''  : Vector Space Model </li>
<li>'''Phase III''' : Spectrum analysis(Fingerprint) model </li>
<li>'''Phase IV'''  : Deep Neural Network Model (Deep learning approach) </li>
</ul>
</p>

<p>'''Phase I : Getting Features and index creation''' <br>
Getting features via AcousticBrainz and index creation for extracted features will be done in this phase. In AcousticBrainz have a lot of annotation data, including all of the information in MusicBrainz, which we can be accessed via the MusicBrainz IDs which are used in both projects, and also the generated annotation data. These features could be annotated texts, tags, descriptions, labels etc. It would also include low level and high level metadata. </p>

<p> Some specific list of descriptors that would be a good idea to include in a search index:
* bpm - could be used for finding songs with specific bpm. Also, can be used in making groups.
* artist - user can search for a song/music based on artist name
* tempo - used for searching. some user likes songs with hight tempo while other with low.
* instrumentation - searching songs based on instrument type. Say 'guitar'
* mood - searching songs based on mood. Get songs which have 'sad' or happy 'mood'
* release - get songs based on release info, date etc.
* track name - get music based on name. say 'scuse me'
* key - get songs or music based on key. Can be useful for some musician users.
* gender - get songs sung by male/female singer
* speech/non speech - can be useful for grouping.
* type of file - search by file type. get mp3 format of some track. Helpful in serving field queries
* length - search by total length or duration of a music. Can be useful in identifying duplicates (will be discussed later)
* language - get all 'english' songs
* genre - search songs based on genre. 'rock', 'classic' etc.
<br>
If we have these features then it would simplify many searching and grouping tasks. It would also help in data exploration task. It would be able to support search for a track by its name, or even
search for a release or artist to add all matching recordings to a class. search could also match recordings which match certain tags. For example, there are tags on recordings in MusicBrainz which look like genres. Would we be able to say "find all recordings tagged with 'rock'
and add them to the rock class in our genre dataset". </p>

<p>[[freesound.org|Freesound]] seems to be good platform for downloading and creating datasets. They also provides annotated data and low level and high level descriptors along with videos. These features wil be used in building and testing our search system. </p>

<p> A demo code for previewing audio from freesound. </p>
<source lang="python">
"""
Demo code to show structure of freesound
client api(s).
"""
import freesound, sys,os


c = freesound.FreesoundClient()
c.set_token("<your_api_key>","token")

results = c.text_search(query="dubstep",fields="id,name,previews")

for sound in results:
    sound.retrieve_preview(".",sound.name+".mp3")
    print(sound.name)
</source>

<p> Features retrieved form AcousticBrainz server will be stored. These features will be indexed in the elastic search database.</p>

<p> A sample code to store indexes in elastic search </p>

<source lang="python">
"""
A Demo code to show indexing in elastic search

creating index for some song collections.
It indexes details about a song; singer, title,
tags, release, genre, length, awesomeness
"""

import elasticsearch
es = elasticsearch.Elasticsearch()  # use default of localhost, port 9200

es.index(index='song_details', doc_type='song', id=1, body={
    'singer': 'Ellie Goulding',
    'title': 'love me like you do',
    'tags': ['cherrytree', 'interscope', 'republic'],
    'release': '7 January 2015',
    'genre': 'electropop',
    'length': 4.12,
    'awesomeness': 0.8
})

es.index(index='song_details', doc_type='song', id=2, body={
    'singer': 'Eminem',
    'title': 'love the way you lie',
    'tags': ['Aftermath', 'Interscope', 'shady'],
    'release': '9 August 2010',
    'genre': 'hip hop',
    'length': 4.22,
    'awesomeness': 0.6
})

es.index(index='song_details', doc_type='song', id=3, body={
    'singer': 'Arijith Singh',
    'title': 'Tum hi ho',
    'tags': ['romantic', 'movie song track'],
    'release': '2 April 2013',
    'genre': 'romantic',
    'length': 4.15,
    'awesomeness': 0.95
})

# searching for song with text as 'tum hi'
es.search(index='song_details', q='tum hi')
</source>

<p>Once we have created indexes and stored in elastic search database, search based on texts can easily be done as elastic search support Lucene based search.</p>
<p> Some more functionalities which elastic search provides: </p>
<ul>
<li> Distributed, scalable, and highly available </li>
<li> Real-time search and analytics capabilities </li>
<li> Sophisticated RESTful API </li>
</ul>

<p> Hence, using elastic search for storing index is a good choice. Data stored in the elastic search will further be used to create different visualization using Kibana. It comes very handy to understand the behavior of data once we visualize it.</p>

<u>''Reason for using Kibana''</u>
<p> Kibana provides the following functionality: </P>
<ul>
<li> Flexible analytics and visualization platform </li>
<li> Real-time summary and charting of streaming data </li>
<li> Intuitive interface for a variety of users </li>
<li> Instant sharing and embedding of dashboards </li>
</ul>
<br>

<u> '''Vector Space Model(VSM)''' </u>
<p> This is a basic yet effective model. The idea is to represent every audio entity in vector form in the feature space.</p>

<p> [[File:Vsm.png|center|400px]] </p>
<center>'''Vector space representation in 3D'''</center><br>

<p> Given a query audio, learn the vector representation of it and then find the similarity between the query vector and indexed audio vectors. More the similarity more is its relevance. Cosine similarity measure can be used to calculate relevance score. Features for audio could be tempo, bpm, average volume, genre, mood, fingerprint etc. </p>

<p> This could be done by using Gaia software already present. </p>

<u> '''Spectrum analysis(Fingerprint) model''' </u>
<p> This relies on fingerprinting music based on spectrogram.</p>
<p> Basic steps of processing </p>
<ul>
<li> Preprocessing step: Includes fingerprinting comprehensive collections of music and storing fingerprint data in database (indexing)</li>
<li> Extracting fingerprint of music(m) piece used as query </li>
<li> Matching fingerprint of m against the indexed fingerprints </li>
<li> Returning matched musics in relevance order </li>
</ul>

<u> ''Advantage''</u>
<p>It works on very obscure songs and will do so even with extraneous background noise</p>

<u>''Implementation Details''</u>
<p> Idea here is to think of a piece of music as time frequency graph also called spectrogram. This graph has three axis. Time on x-axis, frequency on y-axis and intensity on z-axis. A line horizontal line represents a continuous pure tone, while vertical line represents sudden rise (burst) white noise.</p>

<p> [[File:Spectrogram.png|center|600px]] </p>
<center><p> sample wave file: [[Media:Scuse_me.ogg|scuse me]] </p></center>
<source lang="python">
"""Spectrogram image generator for a given audio wav sample.
It is a visual representation of spectral frequencies in sound.
horizontal line(x axis) is time, vertical line(y axis) is frequency
and color represents intensity.
"""


import os
import wave
import pylab


def graph_spectrogram(wav_file):
    sound_info, frame_rate = get_wav_info(wav_file)
    pylab.figure(num=None, figsize=(19, 12))
    pylab.subplot(111)
    pylab.title('spectrogram of %r' % wav_file)
    pylab.specgram(sound_info, Fs=frame_rate)
    pylab.savefig('spectrogram.png')


def get_wav_info(wav_file):
    wav = wave.open(wav_file, 'r')
    frames = wav.readframes(-1)
    sound_info = pylab.fromstring(frames, 'Int16')
    frame_rate = wav.getframerate()
    wav.close()
    return sound_info, frame_rate


if __name__ == '__main__':
    wav_file = 'scuse_me.wav'
    graph_spectrogram(wav_file)
</source>
<p> Points of interest are the points with 'peak intensity'. The algorithm keeps track of the frequency and the amount of time from the beginning of the track for each of these peak point. Fingerprint of the song is generated using these information. </p>

<u>''Details of algorithm''</u>
<p> Read the audio/music data as a normal input stream. This data is time-domain data. As we need to use spectrum analysis instead of direct time-domain data. Using Discrete Fourier Transform (DFT) we will convert the time-domain data to frequency domain so that it can be of use. The problem here is that, in frequency domain data we loose track of time. Hence, to overcome this problem we divide whole data into chunks of data and will transform just this bit of information. Basically, we will be using a small window size. For each of these small window we already know the time.</p>

<p> Using Fast Fourier Transform(FFT) on all data chunks we get a list of all the data about frequencies. Now, our goal would be to find interest points also called key music points to save these points on hash and try to match on them against the indexed database. It would be efficient since average lookup time for a hash table is O(1), i.e. constant. Each line of spectrum represents data for a particular window. We would be taking certain ranges say 40-80, 80-120, 120-180, 180-300. Our goal is to take points with highest magnitude from these ranges for each windowed data. These are the interest points or key points.</p>

<p>[[File:Invertedindex.png|center|600px]] </p>

<p> Next step is to use these key points for hashing and generate index database. Key points obtained form each window will be used to generate a hash key. In the dictionary, a hash key is used to obtain the bucketlist stored as value. Index of the bucketlist is songID and at each index value would be details stored in KeyPoint object.</p>

<source lang="python">
class KeyPoint:
    def __init__(self, time, songID):
        self.time = time
        self.songID = songID
</source>

<p> Using the above mentioned technique we can create index for the audio/music collections. Given a query audio we find fingerprint(contains Keypoints and time details) of that audio and will match against the indexed database to find similar audios. Important thing to keep in mind is that we are not just matching key points but time too. We must overlap the timing. We can subtract the current time in our recording (from query fingerprint) with the time of the hash-match (from indexed KeyPoint objects). This difference is stored together with the song ID. Because this offset, this difference, tells us where we possibly could be in the song. When we have gone through all the hashes from our recording we are left with a lot of song id’s and offsets. More the number of hashes with matching offsets more relevant is the song.</p>

''Reference Paper'': http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf <br>

<u>'''Deep Neural Network Model (Deep learning Approach)''' </u>
<p> This method helps in learning features of a audio automatically. It's an unsupervised approach to learn important features. In deep neural network each layer extracts some abstract representation of the audio. The idea is through each computation layer the system is able to learn some representation of the audio. Initial sets of inputs is passed through the input layer and computations are done at the hidden layers. After passing some hidden layer we learn new feature representation of initial input features. These new learned features vectors will be the representations of audios. Now, we could use machine learning algorithm on these features to do various classification and clustering job. </p>

<p>[[File:Deepnets.png|center|600px]] </p>

<p>In feature space, the audio will be close to each other have some kind of similarity notion between them. </p>

<u>''Advantage''</u>
<ul>
<li> Saves the effort of learning features of audio (meta data extraction and manual labelling)  </li>
<li> It can be used to find similar audios/music. Which can be useful in content based search and by using user information it can be used to recommend songs/audios.</li>
<li> It can be used to detect duplicate songs. It generally happens that two songs ends up having different Ids but are indeed same songs.</li>
</ul>

<h3> Time </h3>
<p> I will be spending 45 hours per week for this project. </p>
<u>'''Summer Plans'''</u><br>
I am a first year masters student and my final exams will be over by the end of April, 2016. Then we have summer vacation till end of July. As, I am free till then so, I will be spending much of my time towards my project.<br>

<u>'''Motivation'''</u><br>
I am a kind of person who always aspires to learn more and explore different areas. I am also an open source enthusiast who likes exploring new technologies. GSoC(Google Summer of Code) provides a very good platform for students like me to learn and show case their talents by coming up with some cool application at the end of summer. 

<h3> Bio </h3>
<p> I am a first year MS in Computer Science and Engineering student at IIIT Hyderabad, India. I am passionate about machine learning, deep learning, information retrieval and text processing and have keen interest in Open Source Software. I am also a research assistant at Search and Information Extraction Lab (SIEL) at my university.</p>

<h3> Experiences </h3>
<p>  My work includes information retrieval for text based systems. During my course work I created a search engine on whole Wikipedia corpus from scratch. I have also been using machine learning and have dived into deep learning concepts for representation learning. Currently, I am working on learning efficient representations of nodes in social network graph.</p>

<P>I have sound knowledge of programming language namely Python, Cython, C, C++ etc. I have good understanding of python's advanced concepts like descriptors, decorators, meta-classes, generators and iterators along with other OOPs concepts. Also, I have contributed to some of the open source organizations like Mozilla, Fedora, Tor, Ubuntu before. As a part of GSoc 2014 project I worked for the Mars exploration project of Italian Mars Society. It was a virtual reality project where I implemented a system for full body and hand gesture tracking of astronauts. This allows astronauts in real world to control their avatar in virtual world through their body gestures. Details of this project can be found at:
</p>
<u>'''concept idea'''</u> :  https://wiki.mozilla.org/Abhishek/IMS_Gsoc2014Proposal <br>
<u>'''IMS repo'''</u> : https://bitbucket.org/italianmarssociety/ <br>
<p> Details of some of my contributions can be found [https://wiki.mozilla.org/Abhishek/contributions here]</p>
<u>'''Github'''</u> :  https://github.com/AbhishekKumarSingh <br>
<u>'''Bitbuket'''</u> : https://bitbucket.org/abhisheksingh <br>
<u>'''My CV'''</u> : https://github.com/AbhishekKumarSingh/CV/blob/master/abhishek.pdf<br>