A delicious hypertable build with aws emr and mapr. Configurable.
#INSTRUCTIONS:
- Install AWS CLI
- Configure AWS CLI with a profile
- Install Ansible: http://docs.ansible.com/intro_installation.html
- Set ansible env: shell source ansible env
- Turn host key checking off: shell export ANSIBLE_HOST_KEY_CHECKING=False
- Ansible needs Jinja2: apt-get install python-jinja2
- If old ssh like 5.5: shell export ANSIBLE_SSH_ARGS=""
- profile: usp
- name: panda
- type: m1.large
- masters: 1
- cores: 1
- tasks: 3
- log_uri: s3://pandahard/
- mapr_edition: m3
- mapr_version: 3.1.1
- ami_version: 2.4.2
- hypertable_version: 0.9.8.1
- data: vol-e1141fe5
- newrelic: 9910108157933df624056a0f5c26f19df6090a28
Commands are run from app root.
- Command: ansible-playbook -v -i build build.yml
- Notes: Build can take 20+ mins depending upon cluster size.
- Command: ansible-playbook -v -i builds/{{id}}/{{name}}.hosts test.yml
- Notes: Creates namespace, loads schema, loads table and then backs it up.
- Command: ansible-playbook -v -i builds/{{id}}/{{name}}.hosts load.yml
- Notes: Uses an ebs attached mount where import scripts and data live. Mount point: /data Path to data: /data/panda/source/$table_name Path to scripts: /data/panda/source/$table_name/load.hql
- Command: ansible-playbook -v -i builds/{{id}}/{{name}}.hosts backup.yml
- Notes: Uses an ebs attached mount where import scripts and data live. Path: /data/panda/backups
- Command: ansible-playbook -v -i builds/{{id}}/{{name}}.hosts clean.yml
- Hypertable Monitoring @ panda_masters[0].public_ip:15860
- NewRelic available if you entered a licence or used the one in this repo which is totally ok.
#FAQ
-
How do I increase debug output when running playbooks? For maximum ansible debug verbosity use "-vvvv". The more v's, the more debug output.
-
I don't want to use Newrelic? If you don't have a newrelic key or don't want to use newrelic, then comment out the new relic roles in /build.yml on lines 21, 35 & 51.
#TODO
- Support Ganglia Monitoring
- Support Hadoop EMR Builds
- Needs more tests