Marketplace/ReplaceMonolith

From MozillaWiki
Jump to: navigation, search
Stop (medium size).png
The Marketplace has been placed into maintenance mode. It is no longer under active development. You can read complete details here.

Replacing Monolith

As part of KTLO and reducing technical debt, this page is proposing that we phase out Monolith in favor of other technology that is better maintained and has more contributors. The two options are (1) use Logstash or (2) use Heka. Both of these projects have an ability to load data into Elasticsearch where Marketplace can then run its aggregations that drive Marketplace stats.

Reasons to phase out Monolith:

  • Reduce technical debt in Marketplace.
  • The original developers have moved on to other projects and it is no longer maintained.
  • There are only a couple people who really know how it works.
  • There are other tools that can accomplish similar goals and have a community behind them.

1. Logstash

Benefits:

  • It's a popular open-source project among people who know Elasticsearch.
  • It has commercial backing and is being improved rapidly.

Cons (unless you are a Ruby fan):

  • It runs on Java (JRuby).
  • Plugins (how we would get the Marketplace and Google Analytics data into logstash) are written in JRuby.

2. Heka

Benefits:

Pseudo-Cons (not really cons):

  • Heka is written in Go so Marketplace stack would need to set up a Go stack.
  • Heka plugins are written in Go or Lua.

What Marketplace Stats needs

  • Ability to load data from Marketplace (via API)
  • Ability to load data from Google Analytics
  • Ability to load data from Solitude (via S3)
  • Ability to load data from App Stats (future requirement)
  • Push data into Elasticsearch for aggregation queries