TestEngineering/Services/HekaInfo

From MozillaWiki
Jump to navigation Jump to search

Heka Info/Intro

  • In Progress - brain dump of all notes related to Heka usage, testing, deployment, etc...
  • Focus 1: Functional testing of the Heka releases before they go out to the public
  • Focus 2: Specific functional and compatibility testing with internal Services projects before they get deployed to Production.

Heka Development Team

  • Rob Miller (RaFromBRC)
  • Ben Bangert (benbangert)
  • Mike Trinkala (trink)
  • Victor Ng (vng)

Heka Documentation

GitHub Repos

Issues and Bugs

Travis/CI

Deploying Heka to AWS

  • Standard methods can be used here - AWS Console wizard, AWSBox, AWSBoxen - to deploy Heka to one instances or stacks within AWS. Services QA has credentials for deploying to both common regions: East, West. There are two PEM files that can be used - one for East (Virginia), one for West (Oregon). There is separate documentation for installing a PEM file locally for use with specific instances in AWS.
  • NOTE: Services QA has some documentation for using the AWS Classic Wizard to create EC2 instances of the standard Linux flavors.
  • NOTE: Heka Development team has further documentation for creating Windows Server instances.

General Installation Notes

  • The following is the (fairly) complete list of prerequisites for installing Hekad on Mac, Windows, and Linux.
  • NOTE: There are specific additions for EC2 instances in AWS, since both the Ubuntu and RHEL flavors deploy with minimal package/library support (see below):
  • Specific additions for Ubuntu EC2 instances:
    • Easy (Wizard) installs may only come with make, perl, and python. So, you may need to install cmake, gcc, g++, git, python-devel.
  • Specific additions for RHEL EC2 instances:
    • Easy (Wizard) installs may only come with make, perl, and python. So, you may need to install cmake, gcc, g++, git, python-devel.
  • Specific additions for stripped-down/minimal Linux installations:
    • Similar to the above. Most base Linux installations (not development installations), may be missing compilers and other basic development packages that will be needed.

Installing The Heka RPM

  • Follow the instructions here for installing Heka from an RPM
Get the RPM from the releases folder:
for example
$ wget https://github.com/mozilla-services/heka/releases/download/v0.3.0/hekad-0.3.0-1.x86_64.rpm
or 
$ curl -O https://github.com/mozilla-services/heka/releases/download/v0.3.0/hekad-0.3.0-1.x86_64.rpm

Use the RPM command to install it
$ sudo rpm -ivv hekad-0.3.0-1.x86_64.rpm

Verify that it is installed to /usr/bin or /usr/local/bin
$ which hekad

Verify the version
$ hekad -version
  • File locations
    • ./usr/bin/hekad (or /usr/local/bin/hekad)
    • ./usr/share/man/man5/hekad.config.5.gz
    • ./usr/share/man/man5/hekad.plugin.5.gz
    • ./usr/share/man/man1/hekad.1.g
    • ./etc/hekad.toml.sample
    • Note here that this is probably the out-of-date sample hekad.toml file
  • To uninstall hekad, again, use the RPM command
$ sudo rpm -evv hekad

Installing Heka From GitHub

  • It is sometimes useful to install/build Heka from the GitHub repo, especially if you are testing Dev changes or are interested in building your own RPMs for deployment to AWS or a QA test environment.
  • Follow these steps:
$ git clone https://github.com/mozilla-services/heka
$ cd heka
$ . build.sh # Unix (note the dot: this file must be sourced to properly setup the environment)

After running ". build.sh", you should see the following new ENV:
    GOBIN=/home/ec2-user/heka/build/heka/bin
    GOPATH=/home/ec2-user/heka/build/heka

Note: They are only active/defined until you log out, so best to put them in a bash/login profile
Alternatively, run ". build.sh" after each log in.

Verify hekad in the following directory:
/heka/build/heka/bin

Run a quick verification test:
$ cd heka/build
$ make test

Installing the Heka Package on Mac

  • NOTE: There is no uninstaller for this package, so it has to be done manually using Mac's pkgutil:
$ pkgutil --pkgs | grep heka
com.mozilla.hekad.pkg

$ pkgutil --files com.mozilla.hekad.pkg
bin
bin/hekad
share
share/man
share/man/man1
share/man/man1/hekad.1
share/man/man5
share/man/man5/hekad.config.5
share/man/man5/hekad.plugin.5

$ pkgutil --pkg-info com.mozilla.hekad.pkg (example)
package-id: com.mozilla.hekad.pkg
version: 0.3
volume: /
location: usr/local
install-time: 1374618711

$ cd /
$ sudo find . -name "*heka*" -print
./usr/local/bin/hekad
./usr/local/share/man/man1/hekad.1
./usr/local/share/man/man5/hekad.config.5
./usr/local/share/man/man5/hekad.plugin.5
Manually remove the 4 files
$ sudo rm -i /usr/local/bin/hekad /usr/local/share/man/man1/hekad.1
/usr/local/share/man/man5/hekad.config.5 /usr/local/share/man/man5/hekad.plugin.5
(make sure this sudo command is all on one line)
Unregister the package
$ sudo pkgutil --forget com.mozilla.hekad.pkg

Installing the Heka Package on Windows Server

Creating an RPM for Heka

  • RPM building is probably best done on a Linux AWS instance with the latest heka build, or alternatively, on a Linux VM.
  • Verify that the following packages have been installed:
    • dpkg
    • rpmbuild
    • packagemaker
  • Follow these steps:
$ cd heka/build 
$ cpack
or
$ make package
  • That should build an RPM and a tarball
  • You should then see files like this in the current directory:
    • heka-0_4_0-linux-amd64.rpm
    • heka-0_4_0-linux-amd64.tar.gz

The Hekad Command Line Options

Testing Hekad

  • Unit Testing
    • If Heka was installed through GitHub, then a best practice would be to run through the unit tests before moving on to using Flood:
$ cd heka/build
$ make test
Note: check the Makefile to find out what this is actually doing
  • Starting Hekad
If you installed Heka through GitHub, then run the following
$ cd heka/build/heka
$ ./bin/hekad -config=<Path to hekad.toml file>
Example:
$ ./bin/hekad -config=hekad-sample-config.toml

If you installed Heka with the RPM, then run the following
$ hekad -config=<Path to hekad.toml file>
Example:
$ hekad -config=hekad-sample-config.toml
Note: this assumes heka is in your path - /usr/bin or /usr/local/bin

In either case, you can run this as a background process or in screen or in both to 
detach/log out and have hekad run as with any other daemon on the system.
  • Log, DB, and Dashboard directories and files
    • These directories and files are created on first instantiation of hekad. By default, they will be created in the current directory (the directory where you started hekad). Make sure that you have write access to this directory so that the Log, DB, and Dashboard directories can be created.
    • Verify that these directories are now being created automatically.
  • Testing with Flood
    • See the section below for more specific information
  • Other Testing
    • TBD
  • Log directory info
    • TBD
  • DB directory info
    • TBD
  • Dashboard directory info
    • See separate section

Hello World Tutorial

http://thraxil.org/users/anders/posts/2013/07/27/A-Gentle-Introduction-to-Heka/

Using hekad.toml Files To Configure Hekad

  • Beyond the sample config file, many additions can be made to the options, input, output, and filters section for specific testing.

Using Flood and the flood.toml File

  • Flood is a basic testing tool for exercising various protocols, message types, and errors. It is not really a load test tool, although several can be run in parallel on the same server to "stress" heka a bit.
  • Heka comes with the flood tool and a basic flood.toml file as part of the github and RPM install.
  • Very basic usage
Put flood into your path, then
$ flood -help
to see usage information

Run flood by specifying its toml file and a test name

Note: All the predefined tests are listed in the toml file.
Note: Also, multiple instances of flood can be run
     with the same toml file and test
     with the same toml file and a different test
     with different toml files and tests

Verify the method for stopping tests - do they have a set run time or do you need to kill the process?

Monitoring Hekad with Dashboard

Testing Ideas for Services QA

  • Functional testing of each Heka release before it gets posted to the public.
  • QA/Dev environment deployments in AWS using GitHub or the RPM build.
  • "Load" testing of hekad using Flood and a more complete TOML file.
  • Testing a more complete (or various types of) hekad.toml file.
  • Testing with sandboxed filters.
  • Deployment and testing in the Persona Stage environment.
  • Testing in the new Sync Stage environment (TBD).

Advanced Configurations for Heka

Using Sandboxed Filters

Heka for "New" Sync

  • The following "front-end" services - all part of Firefox Accounts - are either already using or will be using the latest release of Heka for various types of stats and log collection:
    • Auth API service (picl-idp)
    • Firefox Account Bridge (FAB)
    • Scrypt Helper (scrypt-helper)
  • Heka will also be used for stats and log collection at the "back-end" data storage service.

Heka for Marketplace

Heka for Sync 1.1

Retrieved from "https://wiki.mozilla.org/index.php?title=TestEngineering/Services/HekaInfo&oldid=1145570"