terraform-sumologic-gcp-soruce

This repo contains terraform code to help you route log messages from a GCP project to a sumologic GCP collector. The parent module creates both a GCP source on a pre-existing collector in sumologic and a topic, pubsub log router, and pubsub push subscription in GCP.

Contents:

• Getting Started
• Basic Implementation
• Requirements
• Providers
• Inputs
• Testing
• Development

Getting Started

Both an existing sumologic hosted collector and GCP project must exist prior to execution, this module does not create those objects. modules/sumologic-gcp-source can be used by itself as a basis for other logging functionality.

In GCP, you will also need to disable domain-restricted sharing for your project directly or by inheritence, as this module needs to add roles/pubsub.publisher to a Google-owned service account.

Basic Implementation

module "gcp-log-export" {
  source                 = "../.."
  project_id             = "my-project-sr2f"
  sumologic_collector_id = "1242652142"
  name                   = "my-gcp-log-export"
  gcp_filters = {
    gke     = "resource.type=\"gke_cluster\" OR resource.type=\"k8s_cluster\" OR resource.type=\"k8s_node\" OR resource.type=\"k8s_pod\"",
    project = "resource.type=\"project\""
  }
}

This will create both a sumologic GCP source as well as all the plumbing in your GCP project to export logs to sumologic. The gcp_filters parameter is important, as you need to set up at least one filter to be able to send any logs at all. These can be built using the Stackdriver log explorer in the GCP Cloud Console. gcp_filters is a map, so you can build an unlimited number of filters based on the logs you need. The example here will get you all GKE logs and project logs.

Requirements

Name	Version
terraform	~> 1.0
google	3.74
sumologic	>= 2.9, < 3.0

Providers

Name	Version
google	3.74

Modules

Name	Source	Version
sumologic-gcp-source	./modules/sumologic-gcp-source	n/a

Resources

Name	Type
google_logging_project_sink.logged_messages	resource
google_pubsub_subscription.push	resource
google_pubsub_topic.topic	resource
google_pubsub_topic_iam_member.member	resource

Inputs

Name	Description	Type	Default	Required
category	Single-word category that logs for this search will go into. Will be concated with parent_categories	string	""	no
gcp_filters	List of map of filters to create and be routed into the pubsub topic and push	map(string)	{}	no
name	Name to use uniformally for the log source, pubsub topic, and pubsub subscription	string	n/a	yes
parent_categories	A hierarchy of terms that make up the parent categories. Important if using search partitioning	list(string)	[]	no
project_id	GCP Project ID where the GCP resources should be created	string	n/a	yes
pubsub_sa_publisher_account	GCP Service Account to assign roles/pubsub.publisher to.	string	"serviceAccount:[email protected]"	no
push_deadline_seconds	Maximum amount of time for the subscription to wait for acknowledgement of reciept of message	number	20	no
source_description	Description to use for the source	string	""	no
sumologic_collector_id	ID of the hosted collector at sumologic that will recieve messages for the new source	string	n/a	yes

Outputs

Name	Description
google_pubsub_subscription_id	n/a
google_pubsub_subscription_name	n/a
google_pubsub_subscription_shortid	n/a
google_pubsub_topic_id	n/a
google_pubsub_topic_name	n/a
google_topic_iam_publisher	n/a
sumologic_endpoint	n/a

Testing

This repository uses Kitchen-Terraform to test the terraform modules. In the examplesdirectory you can find examples of how each module can be used. Those examples are fed to [Test Kitchen][https://kitchen.ci/]. To install test kitchen, first make sure you have Ruby and bundler installed.

brew install ruby
gem install bundler

Then install the prerequisites for test kitchen.

bundle install

You'll need to add some common credentials and secret variables

And now you're ready to run test kitchen. Test kitchen has a couple main commands:

• bundle exec kitchen create initializes terraform.
• bundle exec kitchen converge runs our terraform examples.
• bundle exec kitchen verify runs our inspec scripts against a converged kitchen.
• bundle exec kitchen test does all the above.

Development

Merging Policy

Use GitLab Flow.

• Create feature branches for features and fixes from default branch
• Merge only from PR with review
• After merging to default branch a release is drafted using a github action. Check the draft and publish if you and tests are happy

Pre-commit hooks

Install and configure terraform pre-commit hooks To run them: pre-commit run -a

CI

This project has three workflows enabled:

1. PR labeler: When openning a PR to default branch, a label is given automatically according to the name of your feature branch. The labeler follows thenrules in pr-labeler.yml

2. Release Drafter: When merging to master, a release is drafted using the Release-Drafter Action

3. Kitchen test is run on every commit unless [skip ci] is added to commit message.