View source for Abhishek/metabrainz GSoC2016Proposal

<!--
<h3> Personal Details </h3>

<ul>
<li> <p style="white-space: pre;">Name                         : Abhishek Kumar Singh </p></li>
<li> <p style="white-space: pre;">IRC                              : kanha@freenode.net</p></li>
<li> <p style="white-space: pre;">Location(Time Zone) : India, IST(UTC + 5:30)</p></li>
<li> <p style="white-space: pre;">Education                  : MS in Computer Science and Engineering, IIIT Hyderabad</p></li>
<li> <p style="white-space: pre;">Email                          : abhishekkumarsingh.cse@gmail.com, abhishek.singh@research.iiit.ac.in </p></li>
<li> <p style="white-space: pre;">Blog URL                    : http://abhisheksingh01.wordpress.com/</p></li>
</ul>

<h2> AcousticBrainz </h2>

<h3> Abstract </h3>
<p> Implementation of an audio search system for AcousticBrainz. An Audio Information Retrieval(AIR) system that would provide client API(s) for audio search and information retrieval. Search will be based on annotated text/descriptions or tags/labels associated with an audio as well as on low and high level features. Focus is to implement an AIR system that provides text based and content based search. Users would be able to perform query through the provided client API(s) and will get list of relevant audios in return.</p>

<h3> Benefit to AcousticBrainz community </h3>
<p> “Search, and you will find” </p>
<ul>
<li> Provides client API(s) for text and content based search. Query can be done by providing some text or by submitting a piece of audio. </li>
<li> Support for advanced queries combining content analysis features and other metadata (tags, etc) including filters and group-by options.</li>
<li> Support for duplication detection. It helps in finding duplicate entries in the collection. </li>
<li> Helps in finding relevant audios quickly and efficiently. </li>
</ul>

<h3> Project Details </h3>
<br>
<u> '''INTRODUCTION''' </u>
<p> The idea is to implement a search functionality for AcousticBrainz. This would include implementing an Audio Information Retrieval System(AIRS) which would provide audio content search. This would also simplifies the task of data exploration and investigating content based similarity.</p>

<p>Audio Information Retrieval(AIR) can be: </p>
<ul>
<li> Text based </li>
<li> Content based </li>
</ul>

; Text based AIR :
<p> Query can be any text say 'rock' or 'beethoven' etc and the system will search through the text(tags, artist name, description) associated with the audios and will return list of relevant audios corresponding to matched text. It's simple to implement but doesn't help much in audio retrieval as most of the times the audio doesn't contains enough annotations.</p>

; Content based AIR :
<p> It bridges the semantic gap when text annotations are nonexistent or incomplete. For example, if one comes to a record store and one only know a tune from a song or an album one want to buy, but not title of music, album, composer, or artist. It is difficult problem to solve for one. On the other hand, sales expertise with vast knowledge of the music can identify tunes hummed by one. Content based audio information retrieval can substitute the vast experience of sales expertise with a vast knowledge of the music.</p>

<u>''Benefit''</u> :
<ul>
<li> Given a piece of audio as a query, Content based AIRS will search through the indexed database of audios return list of audios that are similar to a piece of audio given as query.</li>
<li> Can detect (duplicates) copy right issues with little tweak.</li>
</ul>

<u>'''REQUIREMENTS DURING DEVELOPMENT'''</u>
; Hardware Requirements :
<ul>
<li> A modern PC/Laptop</li>
<li> Graphical Processing Unit(GPU) </li>
</ul>
; Software Requirements : 
<ul>
<li> Gaia </li>
<li> Elastic search (storage database)</li>
<li> elastic-py library </li>
<li> Kibana (data visualization)</li>
<li> Python 2.7.x/C </li>
<li> Scikit, Keras </li>
<li> Coverage </li>
<li> Pep8 </li>
<li> Pyflakes </li>
<li> Vim (IDE)</li>
<li> Sphinx (documentation) </li>
<li> Python Unit-testing framework (Unittest/Nosetest) </li>
<li> AcousticBrainz tools
</ul> 

<u>'''THE OUTLINE OF WORK PLAN'''</u>
<p> Audio information retrieval framework consists of some basic processing steps: </p>
<ul>
<li> Feature extraction</li>
<li> Segmentation</li>
<li> Storing feature vectors(indexing)</li>
<li> Similarity matching</li>
<li> Returning similar results </li>
</ul>

<p> [[File:Air.png|center|600px]] </p>
<center> '''Basic flow in audio information retrieval system''' </center><br>
<p> Some proposed models for Content based audio retrieval </p>
<ul>
<li> Vector Space Model(VSM) </li>
<li> Spectrum analysis(Fingerprint) model</li>
<li> Deep Neural network model (Deep learning approach)</li>
</ul>

<u>'''Division of Work'''</u>
<p>
Basically, The whole work is divided into four phases :
<ul>
<li>'''Phase I'''   : Getting Features and index creation </li>
<li>'''Phase II'''  : Explore and Search Data</li>
<li>'''Phase III''' : Creating Visualization model </li>
<li>'''Phase IV''' : Vector Space Model (VSM) </li>
</ul>
</p>

<p>'''Phase I : Getting Features and index creation''' <br>
Getting features via AcousticBrainz and index creation for extracted features will be done in this phase. AcousticBrainz has a lot of annotated data, including all of the information in MusicBrainz, which we can be accessed via the MusicBrainz IDs which are used in both projects, and also the generated annotation data. These features could be annotated texts, tags, descriptions, labels etc. It would also include low level and high level metadata. </p>

<p> These data comes from the following sources: </p>

* Data from the lowlevel submission files
* Data from the highlevel files
* Data from MusicBrainz

<p> Some specific list of descriptors that would be a good idea to include in a search index: </p>
<u>''Data from the lowlevel submission files'' </u>

* bpm - could be used for finding songs with specific bpm. Also, can be used in making groups.
* key - get songs or music based on key. Can be useful for some musician users.
* pitch - users(researchers) may search for sounds with specific pitch.
* tempo - used for searching. some user likes songs with hight tempo while other with low.
* length - search by total length or duration of a music. Can be useful in identifying duplicates (will be discussed later)
* type of file - search by file type. get mp3 format of some track. Helpful in serving field queries

<u>''Data from the highlevel files'' </u>
* mood - searching songs based on mood. Get songs which have 'sad' or happy 'mood'
* genre - search songs based on genre. 'rock', 'classic' etc.

<u>''Data from MusicBrainz''</u>
* language - get all 'english' songs
* release - get songs based on release info, date etc.
* track name - get music based on name. say 'scuse me'
* gender - get songs sung by male/female singer
* speech/non speech - can be useful for grouping.
* artist - user can search for a song/music based on artist name
* instrumentation - searching songs based on instrument type. Say 'guitar'

<p>We do have metadata in tags in the submissions, many of the tags are of bad quality. However, we have musicbrainz recording id for each submission. Which will be used to perform a lookup on MusicBrainz to obtain more accurate data, and add this data to the index.</p>

<p>If we have these features then it would simplify many searching and grouping tasks. It would also help in data exploration task. It would be able to support search for a track by its name, or even
search for a release or artist to add all matching recordings to a class. search could also match recordings which match certain tags. For example, there are tags on recordings in MusicBrainz which look like genres. Would we be able to say "find all recordings tagged with 'rock'
and add them to the rock class in our genre dataset". </p>

<p>[[freesound.org|Freesound]] seems to be good platform for downloading and creating datasets. They also provides annotated data and low level and high level descriptors along with videos. These features could be used in building and testing our search system. </p>

<p> A demo code for previewing audio from freesound. </p>
<source lang="python">
"""
Demo code to show structure of freesound
client api(s).
"""
import freesound, sys,os


c = freesound.FreesoundClient()
c.set_token("<your_api_key>","token")

results = c.text_search(query="dubstep",fields="id,name,previews")

for sound in results:
    sound.retrieve_preview(".",sound.name+".mp3")
    print(sound.name)
</source>

<p> Features retrieved form AcousticBrainz server will be stored. These features will be indexed in the elastic search database.</p>

<p> A sample code to store indexes in elastic search </p>

<source lang="python">
"""
A Demo code to show indexing in elastic search

creating index for some song collections.
It indexes details about a song; singer, title,
tags, release, genre, length, awesomeness
"""

import elasticsearch
es = elasticsearch.Elasticsearch()  # use default of localhost, port 9200

es.index(index='song_details', doc_type='song', id=1, body={
    'singer': 'Ellie Goulding',
    'title': 'love me like you do',
    'tags': ['cherrytree', 'interscope', 'republic'],
    'release': '7 January 2015',
    'genre': 'electropop',
    'length': 4.12,
    'awesomeness': 0.8
})

es.index(index='song_details', doc_type='song', id=2, body={
    'singer': 'Eminem',
    'title': 'love the way you lie',
    'tags': ['Aftermath', 'Interscope', 'shady'],
    'release': '9 August 2010',
    'genre': 'hip hop',
    'length': 4.22,
    'awesomeness': 0.6
})

es.index(index='song_details', doc_type='song', id=3, body={
    'singer': 'Arijith Singh',
    'title': 'Tum hi ho',
    'tags': ['romantic', 'movie song track'],
    'release': '2 April 2013',
    'genre': 'romantic',
    'length': 4.15,
    'awesomeness': 0.95
})

# searching for song with text as 'tum hi'
es.search(index='song_details', q='tum hi')
</source>

<p>Once we have created indexes and stored in elastic search database, search based on texts can easily be done as elastic search support Lucene based search.</p>
<p> Some more functionalities which elastic search provides: </p>
<ul>
<li> Distributed, scalable, and highly available </li>
<li> Real-time search and analytics capabilities </li>
<li> Sophisticated RESTful API </li>
</ul>

<p> Hence, using elastic search for storing index is a good choice. Data stored in the elastic search will further be used to create different visualization using Kibana in the later phase. </p>
<br>

<p> '''Phase II : Explore and Search Data''' <br>
After the index creation, this phase focus will be on on exploring and searching the data. This way we can evaluate and get to know the behavior of infrastructure components by analyzing the result at each step of component building. </p>

<p>An interface to let people choose a descriptor and choose a value. Searching for this value can show a list of matching recordings. They could also do text based search through the interface. Search like get all queries matching tag 'rock' etc could be done.</p>

<p> A sample code showing an estimate of the the API(s) to be built. </p>

<source lang="python">
"""
A sample to show some of the API(s) provided by
AB search system and its usage.
"""

from AB import search

# create a search client 
c = search.searchClient()

# setting api key for the client
# this may not be a part, depends on
# AcousticBrainz community 
c.set_token("<your_api_key>","token")


# get sound based on id
sound = c.get_sound(108)

# retrieve relevant results for a query
results = c.text_search(query="hip hop")

# fields parameter allows to specify the information
# you want in the results lis
results = c.text_search(query="hip hop",fields="id,name,previews")

# applying filter
sounds = c.text_search(query="hip hop", filter="tag:loop", fields="id,name,url")

# search sounds using a content-based descriptor target and/or filter
sounds = c.content_based_search(target="lowlevel.pitch.mean:220",
                                descriptors_filter="lowlevel.pitch_instantaneous_confidence.mean:[0.8 TO 1]",
                                fields="id,name,url")

# Combine both text and content-based queries.
sounds = c.combined_search(target="lowlevel.pitch.mean:220", filter="single-note")
</source>

<p>After the end of this phase search system would be able to provide some simple search results through the API(s) as mentioned above.</p>

<p> '''Phase III : Creating Visualization Model''' <br>
Under this phase focus is on exploring more about the audio representations by visualization. Once, the search model is implemented. The features which lies in the vector space model will allow us to explore questions like which kind of genres do we have in the collection, what is the distribution on mood, finding link between low level data and high level data. 
</p>

<p> It comes very handy to understand the behavior of data once we visualize it. For the purpose of visualization Kibana will be used. </p>

<u>''Reason for using Kibana''</u>
<p> Kibana provides the following functionality: </P>
<ul>
<li> Flexible analytics and visualization platform </li>
<li> Real-time summary and charting of streaming data </li>
<li> Intuitive interface for a variety of users </li>
<li> Instant sharing and embedding of dashboards </li>
</ul>
<p>An interface showing graphs,  While kibana can be of good use, we should also do some investigation to see if this is in fact necessary. If we can do the same by running some simple ES search queries then Kibana wouldn't be necessary. </p>
<br>
<p>'''Phase IV : Vector Space Model(VSM)'''<br>
This is a basic yet effective model. The idea is to represent every audio entity in vector form in the feature space.</p>

<p> [[File:Vsm.png|center|400px]] </p>
<center>'''Vector space representation in 3D'''</center><br>

<p> Given a query audio, learn the vector representation of it and then find the similarity between the query vector and indexed audio vectors. More the similarity more is its relevance. Cosine similarity measure can be used to calculate relevance score. Features for audio could be tempo, bpm, average volume, genre, mood, fingerprint etc. Hybrid features that is features from high level, low level and meta data will be combined to represent vector space. Each audio will have their representation in this space. Since, it's a high level space we should use PCA to reduce the dimensionality and project it in low dimensional space. Now, in this low dimensional space, similarity measures will be calculated.</p>

<p> This could be done by using Gaia software already present. </p>

<p>'''Nice to have : Deep Neural Network Model (Deep learning Approach)''' <br>
While this will not be in the scope of summer of code but, would be nice to explore afterwards. This method helps in learning features of a audio automatically. It's an unsupervised approach to learn important features. In deep neural network each layer extracts some abstract representation of the audio. The idea is through each computation layer the system is able to learn some representation of the audio. Initial sets of inputs is passed through the input layer and computations are done at the hidden layers. After passing some hidden layer we learn new feature representation of initial input features. These new learned features vectors will be the representations of audios. Now, we could use machine learning algorithm on these features to do various classification and clustering job. This would help in dataset building and model set creation.</p>

<p>[[File:Deepnets.png|center|600px]] </p>

<p>In feature space, the audio will be close to each other have some kind of similarity notion between them. </p>

<u>''Advantage''</u>
<ul>
<li> Saves the effort of learning features of audio (meta data extraction and manual labeling)  </li>
<li> It can be used to find similar audios/music. Which can be useful in content based search and by using user information it can be used to recommend songs/audios.</li>
<li> It can be used to detect duplicate songs. It generally happens that two songs ends up having different Ids but are indeed same songs.</li>
</ul>

<h3> Deliverables </h3>
<p> An Audio retrieval system that supports following features</p>
<ul>
<li> Client API(s) for searching.</li>
<li> Support for text based search. User can search relevant audios by providing tag/label or some keywords.</li>
<li> Support for content based search. It will show similar audios based on content provided by user. Query can be done by providing a piece of audio also.</li>
<li> Fingerprinting and similarity features will help in duplication detection.</li>
<li> Support for advanced queries. User would be able to filter and group results.</li>
<li> Visualization support for the data powered by Kibana.</li>
<li> Proper documentation on the work for users and developers.</li>
</ul>
 
<p> A sample code showing an estimate of the the API(s) provided at the end.</p>

<source lang="python">
"""
A sample to show some of the API(s) provided by
AB search system and its usage.
"""

from AB import search

# create a search client 
c = search.searchClient()

# setting api key for the client
# this may not be a part, depends on
# AcousticBrainz community 
c.set_token("<your_api_key>","token")


# get sound based on id
sound = c.get_sound(108)

# retrieve relevant results for a query
results = c.text_search(query="hip hop")

# fields parameter allows to specify the information
# you want in the results lis
results = c.text_search(query="hip hop",fields="id,name,previews")

# applying filter
sounds = c.text_search(query="hip hop", filter="tag:loop", fields="id,name,url")

# search sounds using a content-based descriptor target and/or filter
sounds = c.content_based_search(target="lowlevel.pitch.mean:220",
                                descriptors_filter="lowlevel.pitch_instantaneous_confidence.mean:[0.8 TO 1]",
                                fields="id,name,url")

# content based search using given audio piece of music
sounds = c.content_based_search(audiofile="sample.mp3")

# Combine both text and content-based queries.
sounds = c.combined_search(target="lowlevel.pitch.mean:220", filter="single-note")

# returns a confidence score for the possibility of two
# audios being exactly same
dup_score = c.get_duplicate_score(audiofile1, audiofile2)

# returns True if duplicate else False
check = c.is_duplicate(audiofile1, audiofile2)
</source>
<br>

<h3> Project Schedule </h3>
<p>
1. Requirements gathering................................................................................................1 week
:1. Writing down requirements of the user.
:2. Setting up test standards.
:3. Writing test specification to test deployed product.
:4. Start writing developer documentation.
:5. Learn more about elastic search and kibana, Gaia

2. Getting features and Indexing......................................................................................3 weeks
:1. Getting features to work on.
:2. Writing code to extract/retrieve features from the server.
:3. Writing automated unit test to test written code.
:4. Adding to developer documentation.

3. Exploring and Searching Data..............................................................................3 weeks
:1. Writing code to explore and visualize data.
:2. Implementation of basic search API(s).
:3. Writing automated unit test to test written code.
:4. Adding to developer documentation.

4. Creating visualization Model.........................................................................................2 week
:1. Writing code for data visualization.
:2. Writing automated unit test to test written code.
:3. Adding to findings to documentation.

5. Implement Vector Space model.....................................................................2 weeks
:1. Implement similarity measure
:2. Comparing VSM model with the Gaia model.
:3. Adding details to documentation

6. Buffer Time......................................................................................................................1 week
:1. Prepare user documentation.
:2. Implement nice-to-have features.
:3. Improve code readability and documentation.
</p>
<br>
<p>
<u>''Task to be done before midterm [23rd May, 2016 - 23th June, 2016]''</u><br>
1. Requirements gathering<br>
2. Getting features and indexing<br>
3. starting work exploring and searching data <br>
<br>
<u>''Task to be done after midterm [25th June, 2016 - 20th August, 2016]''</u><br>
4. Exploring and Searching data<br>
5. Creation of Visualization Model<br>
6. Buffer Time<br>
</p>
<br>

<h3> Time </h3>
<p> I will be spending 45 hours per week for this project. </p>
<u>'''Summer Plans'''</u><br>
I am a first year masters student and my final exams will be over by the end of April, 2016. Then we have summer vacation till end of July. As, I am free till then so, I will be spending much of my time towards my project.<br>

<u>'''Motivation'''</u><br>
I am a kind of person who always aspires to learn more and explore different areas. I am also an open source enthusiast who likes exploring new technologies. GSoC(Google Summer of Code) provides a very good platform for students like me to learn and show case their talents by coming up with some cool application at the end of summer. I am research scholar with interest in information retrieval. This project would help me in getting my hands dirty in AIR field and would provide me with the base to do research in this field with global community interaction and support.

<h3> Bio </h3>
<p> I am a first year MS in Computer Science and Engineering student at IIIT Hyderabad, India. I am passionate about machine learning, deep learning, information retrieval and text processing and have keen interest in Open Source Software. I am also a research assistant at Search and Information Extraction Lab (SIEL) at my university.</p>

<h3> Experiences </h3>
<p>  My work includes information retrieval for text based systems. During my course work I created a search engine on whole Wikipedia corpus from scratch. I have also been using machine learning and have dived into deep learning concepts for representation learning. Currently, I am working on learning efficient representations of nodes in social network graph.</p>

<P>I have sound knowledge of programming language namely Python, Cython, C, C++ etc. I have good understanding of python's advanced concepts like descriptors, decorators, meta-classes, generators and iterators along with other OOPs concepts. Also, I have contributed to some of the open source organizations like Mozilla, Fedora, Tor, Ubuntu before. As a part of GSoc 2014 project I worked for the Mars exploration project of Italian Mars Society. It was a virtual reality project where I implemented a system for full body and hand gesture tracking of astronauts. This allows astronauts in real world to control their avatar in virtual world through their body gestures. Details of this project can be found at:
</p>
<u>'''concept idea'''</u> :  https://wiki.mozilla.org/Abhishek/IMS_Gsoc2014Proposal <br>
<u>'''IMS repo'''</u> : https://bitbucket.org/italianmarssociety/ <br>
<p> Details of some of my contributions can be found [https://wiki.mozilla.org/Abhishek/contributions here]</p>
<u>'''Github'''</u> :  https://github.com/AbhishekKumarSingh <br>
<u>'''Bitbuket'''</u> : https://bitbucket.org/abhisheksingh <br>
<u>'''My CV'''</u> : https://github.com/AbhishekKumarSingh/CV/blob/master/abhishek.pdf<br>
-->