Marketplace/ReplaceMonolith
From MozillaWiki
.png){kind=small}
The Marketplace has been placed into maintenance mode. It is no longer under active development. You can read complete details here.
Replacing Monolith
As part of KTLO and reducing technical debt, this page is proposing that we phase out Monolith in favor of other technology that is better maintained and has more contributors. The two options are (1) use Logstash or (2) use Heka. Both of these projects have an ability to load data into Elasticsearch where Marketplace can then run its aggregations that drive Marketplace stats.
Reasons to phase out Monolith:
- Reduce technical debt in Marketplace.
- The original developers have moved on to other projects and it is no longer maintained.
- There are only a couple people who really know how it works.
- There are other tools that can accomplish similar goals and have a community behind them.
1. Logstash
Benefits:
- It's a popular open-source project among people who know Elasticsearch.
- It has commercial backing and is being improved rapidly.
Cons (unless you are a Ruby fan):
- It runs on Java (JRuby).
- Plugins (how we would get the Marketplace and Google Analytics data into logstash) are written in JRuby.
2. Heka
Benefits:
- It is a Mozilla project.
- Moving to Heka could be part of the Date Pipeline effort.
- We may be able to piggy back off of existing Heka deployments?
- We already use Heka to a degree for handling our stats: https://github.com/mozilla/zamboni/search?utf8=%E2%9C%93&q=heka
Pseudo-Cons (not really cons):
- Heka is written in Go so Marketplace stack would need to set up a Go stack.
- Heka plugins are written in Go or Lua.
What Marketplace Stats needs
- Ability to load data from Marketplace (via API)
- Ability to load data from Google Analytics
- Ability to load data from Solitude (via S3)
- Ability to load data from App Stats (future requirement)
- Push data into Elasticsearch for aggregation queries
