Socorro/Middleware

Socorro Middleware

The Middleware API in Socorro aims at separating the front-end from the back-end by providing an interface to access the data. By doing so, the front-end will not have to care about the storage system, and will retrieve data from Hbase, PostgreSQL or ElasticSearch in a consistent and simple way, through a consistent REST API.

This API is documented in our documentation centre at socorro.readthedocs.org. This page is a place to work on evolution of that API and it's implementation in Socorro.

Implementation

Old implementation organisation

socorro/
        ...
        services/
                search.py
                ...
        search/
                searchapi.py
                elasticsearch.py
                postgresql.py
        ...

New-style implementation organisation

socorro/
        external/
                elasticsearch/
                        base.py
                        report.py
                        search.py
                        ...
                postgresql/
                        base.py
                        products.py
                        report.py
                        search.py
                        util.py
                        ...
        middleware/
                search_service.py
                report_list_service.py
                products_builds_service.py
                ...

The idea is to have all the uri services (services accessible through the REST API) in a same module called middleware. Those services don't implement anything but instead call other classes or functions depending on the storage system to use.

The external module contains ways of communicating with external resources like HBase, PostgreSQL or ElasticSearch. It will contain implementations of the middleware API as well as other useful functions related to the concerned resource.

For example, when calling the search service via the REST API, and with a configuration pointing to ElasticSearch, here is what will be called:

• 1. socorro.middleware.search_service
• 2. socorro.external.elasticsearch.es_middlewareapi
• 3. socorro.external.elasticsearch.search

middlewareapi classes are not supposed to implement anything, but instead direct requests to the good class and function, and send back the result. This way it's easier to add new classes or functions, and it's also easier to share code between components.

Next-Middleware API

The next level of middleware reorganisation will imply less code by using more conventions. We intend to couple more the API urls and the external module's organisation. On the service side, adding a new one will require one line of code and one line of configuration. A DependencySolver will handle the implementation discovery given the configuration and an eventual argument to force the implementation module. (See gist: https://gist.github.com/1508027 ). The base class for services will receive HTTP requests on each method (GET, POST, PUT, DELETE) and redirect those queries to the right implementation class and method.

The implementation class and method will be determined as follow:

• split the base url on "/" (a base url is composed of one or two parts, i.e. crashes/builds, products/versions or crash)
• use the capitalised first part as the class
• if the second part exists, method is action_secondpart where action depends on the HTTP method (GET -> get, POST -> create, PUT -> update, DELETE -> delete)
• if the second part is missing, use the action only

So, a service created with this URL: crashes/signatures, when called by an HTTP GET, will find its implementation in external._resource_.crashes.Crashes.get_signatures().

There is a demo implementation of the generic service here: https://gist.github.com/1529952