This nifty util aggregates all the events from the firehose feature in CloudFoundry.

./firehose-to-logstash \
	--domain=cf.installation.domain.com \
	--user=admin-username \
	--password=admin-password \
	--debug

{"cf_app_id":"e626413b-f1f8-436d-8963-c46f7cb345eb","cf_app_name":"php-diego-one","cf_org_id":"ebd95a83-5b6a-43ff-af67-a234ece3fb78","cf_org_name":"GWENN","cf_space_id":"a2e4c75b-fe02-4078-abfb-87539352aeac","cf_space_name":"GWENN-SPACE","cpu_percentage":1.4523587130944957,"disk_bytes":0,"event_type":"ContainerMetric","instance_index":0,"level":"info","memory_bytes":14110720,"msg":"","origin":"executor","time":"2015-04-17T13:59:52-07:00"}

Options

usage: firehose-to-syslog [<flags>]

Flags:
  --help              Show help.
  --debug             Enable debug mode. This disables forwarding to syslog
  --domain="10.244.0.34.xip.io" Domain of your CF installation.
  --syslog-server=SYSLOG-SERVER
                      Syslog server.
  --subscription-id=firehose  Id for the subscription.
  --user=admin      Admin user.
  --password=admin  Admin password.
  --skip-ssl-validation Please don\'t
  --events=LogMessage Comma seperated list of events you would like. Valid options are HttpStart, HttpStop, Heartbeat, HttpStartStop, LogMessage, ValueMetric, CounterEvent, Error, ContainerMetric
  --boltdb-path='my.db' Bolt Database path
  --cc-pull-time=60s  CloudController Pooling time in sec
  --version           Show application version.

Event documentation

See the dropsonde protocol documentation for details on what data is sent as part of each event.

Caching

We use boltdb for caching application name, org and space name.

We have 3 caching strategies:

• Pull all application data on start.
• Pull application data if not cached yet.
• Pull all application data every "cc-pull-time".

To test and build

# Setup repo
go get github.com/cloudfoundry-community/firehose-to-syslog
cd $GOPATH/src/github.com/cloudfoundry-community/firehose-to-syslog

# Test
ginkgo -r .

# Build binary
godep go build

Deploy with Bosh

logsearch-for-cloudfoundry

Run agains a bosh-lite CF deployment

godep go run main.go \
	--debug \
	--skip-ssl-validation

Parsing the logs with Logstash

logsearch-for-cloudfoundry

Devel

This is a Git Flow project. Please fork and branch your features from develop.

Contributors

• Ed King - Added support to skip ssl validation.
• Mark Alston - Added support for more events and general code cleaup.
• Etourneau Gwenn - Added validation of selected events and general code cleanup, caching system..