TestEngineering/Services/HekaInfo

Heka Info/Intro

In Progress - brain dump of all notes related to Heka usage, testing, deployment, etc...
Focus 1: Functional testing of the Heka releases before they go out to the public
Focus 2: Specific functional and compatibility testing with internal Services projects before they get deployed to Production.

Heka Development Team

Rob Miller (RaFromBRC)
Ben Bangert (benbangert)
Mike Trinkala (trink)
Victor Ng (vng)

Heka Documentation

Heka Project Main: https://heka-docs.readthedocs.org/en/latest/
HekaArchitecture: https://heka-docs.readthedocs.org/en/latest/architecture/index.html
Developing with Heka: http://heka-docs.readthedocs.org/en/latest/developing.html

Hekad Main: https://hekad.readthedocs.org/en/latest/
Installation: https://hekad.readthedocs.org/en/latest/installing.html
Configuration: https://hekad.readthedocs.org/en/latest/configuration.html
Monitoring: https://hekad.readthedocs.org/en/latest/monitoring/index.html
Testing with Flood: https://hekad.readthedocs.org/en/latest/developing/testing.html

GitHub Repos

https://github.com/mozilla-services/heka-py
- This will be used by Marketplace and Sync 1.1 in the near future
https://github.com/mozilla-services/heka-node
- Was being evaluated by the PiCL/Sync team, but currently has no internal customers

https://github.com/mozilla-services/heka-build - DEPRECATED

Issues and Bugs

https://github.com/mozilla-services/heka/issues

Travis/CI

https://travis-ci.org/mozilla-services/heka

Deploying Heka to AWS

Standard methods can be used here - AWS Console wizard, AWSBox, AWSBoxen - to deploy Heka to one instances or stacks within AWS. Services QA has credentials for deploying to both common regions: East, West. There are two PEM files that can be used - one for East (Virginia), one for West (Oregon). There is separate documentation for installing a PEM file locally for use with specific instances in AWS.
NOTE: Services QA has some documentation for using the AWS Classic Wizard to create EC2 instances of the standard Linux flavors.
NOTE: Heka Development team has further documentation for creating Windows Server instances.

General Installation Notes

The following is the (fairly) complete list of prerequisites for installing Hekad on Mac, Windows, and Linux.
NOTE: There are specific additions for EC2 instances in AWS, since both the Ubuntu and RHEL flavors deploy with minimal package/library support (see below):

Prerequisites (all systems):
- CMake 2.8 or greater http://www.cmake.org/cmake/resources/software.html
- Git http://git-scm.com/download
- Go 1.1 or greater (1.1.1 recommended) http://code.google.com/p/go/downloads/list
- Mercurial http://mercurial.selenic.com/downloads/
- Protobuf 2.3 or greater (optional - only needed if message.proto is modified) http://code.google.com/p/protobuf/downloads/list
- Sphinx (optional - used to generate the documentation) http://sphinx-doc.org/
Others
- g++
- git
- python 2.6 or greater
- python-devel (optional)
- perl
- hg (mercurial)
Prerequisites (Mac/Unix):
- make
- gcc
- patch
- dpkg (optional unless you want to build your own RPMs)
- rpmbuild (optional unless you want to build your own RPMs)
- packagemaker (optional unless you want to build your own RPMs)
Prerequisites (Windows):
- MinGW http://sourceforge.net/projects/tdm-gcc/

Source: https://hekad.readthedocs.org/en/latest/installing.html

Specific additions for Ubuntu EC2 instances:
- Easy (Wizard) installs may only come with make, perl, and python. So, you may need to install cmake, gcc, g++, git, python-devel.

Specific additions for RHEL EC2 instances:
- Easy (Wizard) installs may only come with make, perl, and python. So, you may need to install cmake, gcc, g++, git, python-devel.

Specific additions for stripped-down/minimal Linux installations:
- Similar to the above. Most base Linux installations (not development installations), may be missing compilers and other basic development packages that will be needed.

Using hekad.toml Files To Configure Hekad

The default/sample hekad.toml file that comes with Heka is usually sufficient to being some general testing of the product, especially with Flood (see below).
The default/sample is called "sample-config.toml" and can be found here (in the repo), or in a similar location after installation:
- https://github.com/mozilla-services/heka/blob/dev/testsupport/sample-config.toml

Note: The following, older, sample config file is out-of-date and not suitable for testing, so should be skipped:
- https://github.com/mozilla-services/heka/blob/dev/examples/example.toml

Beyond the sample config file, many additions can be made to the options, input, output, and filters section for specific testing.

REF: https://hekad.readthedocs.org/en/latest/configuration.html

Installing The Heka RPM

TBD

NOTE: Ready-to-install packages (TAR, GZ, RPM) can always be found here:
- https://github.com/mozilla-services/heka/releases/

Installing Heka From GitHub

TBD

NOTE: Ready-to-install packages (TAR, GZ, RPM) can always be found here:
- https://github.com/mozilla-services/heka/releases/

Installing the Heka Package on Mac

The released Mac package can always be found here:
- https://github.com/mozilla-services/heka/releases
This installs similarly to other Mac packages.
Hekad will be installed here: /usr/local/bin/hekad
The hekad.toml file does not get installed by default. You can always get it from here:
- https://github.com/mozilla-services/heka/blob/dev/testsupport/sample-config.toml

NOTE: There is no uninstaller for this package, so it has to be done manually using Mac's pkgutil:

$ pkgutil --pkgs | grep heka
com.mozilla.hekad.pkg

$ pkgutil --files com.mozilla.hekad.pkg
bin
bin/hekad
share
share/man
share/man/man1
share/man/man1/hekad.1
share/man/man5
share/man/man5/hekad.config.5
share/man/man5/hekad.plugin.5

$ pkgutil --pkg-info com.mozilla.hekad.pkg (example)
package-id: com.mozilla.hekad.pkg
version: 0.3
volume: /
location: usr/local
install-time: 1374618711

$ cd /
$ sudo find . -name "*heka*" -print
./usr/local/bin/hekad
./usr/local/share/man/man1/hekad.1
./usr/local/share/man/man5/hekad.config.5
./usr/local/share/man/man5/hekad.plugin.5
Manually remove the 4 files
$ sudo rm -i /usr/local/bin/hekad /usr/local/share/man/man1/hekad.1 /usr/local/share/man/man5/hekad.config.5 /usr/local/share/man/man5/hekad.plugin.5
Unregister the package
$ sudo pkgutil --forget com.mozilla.hekad.pkg

Installing the Heka Package on Windows Server

TBD

Creating an RPM for Heka

TBD

The Hekad Command Line Options

Most of the older command line options are now given in the hekad.toml file. Only the two basic options remain:
- hekad -version
- hekad -config
REFs:
- https://hekad.readthedocs.org/en/latest/configuration.html#global-configuration-options
- https://hekad.readthedocs.org/en/latest/configuration.html#command-line-options

Testing Hekad

TBD

Using/Reconfiguring the Hekad TOML (Configuration) File

TBD

Using Flood and the flood.toml File

Monitoring Hekad with Dashboard

Testing Ideas for Services QA

Functional testing of each Heka release before it gets posted to the public.
QA/Dev environment deployments in AWS using GitHub or the RPM build.
"Load" testing of hekad using Flood and a more complete TOML file.
Testing a more complete (or various types of) hekad.toml file.
Testing with sandboxed filters.
Deployment and testing in the Persona Stage environment.
Testing in the new Sync Stage environment (TBD).

Advanced Configurations for Heka

Getting more out of the hekad.toml file for running the Heka daemon
- See the following: https://hekad.readthedocs.org/en/latest/configuration.html
- and here: https://hekad.readthedocs.org/en/latest/developing/plugin.html

Using Sandboxed Filters

For full details, look here: https://hekad.readthedocs.org/en/latest/sandbox/index.html
Then, talk to Mike Trinkala (trink) for specific uses for our projects

Heka for "New" Sync

The following "front-end" services - all part of Firefox Accounts - are either already using or will be using the latest release of Heka for various types of stats and log collection:
- Auth API service (picl-idp)
- Firefox Account Bridge (FAB)
- Scrypt Helper (scrypt-helper)
Heka will also be used for stats and log collection at the "back-end" data storage service.

Heka for Marketplace

Marketplace is using Heka for stats collection.
An interesting v0.3.0 sample hekad.toml for Marketplace can be seen here:
- https://bugzilla.mozilla.org/show_bug.cgi?id=889161
Note that is supports only minimal stats and json logging output at this time.

Heka for Sync 1.1

Sync 1.1 is using Heka for stats collection.
An interesting v0.3.0 sample hekad.toml for Sync 1.1 can be seen here:
- https://bugzilla.mozilla.org/show_bug.cgi?id=889159
Note that is supports only minimal stats and logging output at this time.