Ssong sumo 76740 metadata #7

samjsong · 2017-08-25T17:15:19Z

Added log-opt tag support. https://docs.docker.com/engine/admin/logging/log_tags/
Added log-opt sumo-source-category. Currently supports explicit strings, but can specify {{.Tag}} in the string to include the tag in the _sourceCategory. For example: test_category/{{.Tag}}/hello
Added HTTP headers for X-Sumo-Name and X-Sumo-Category.
Updated README to reflect changes

rvmiller89 · 2017-08-25T21:05:26Z

README.md

 | sumo-url                  | Yes       |               | HTTP Source URL
+| sumo-source-category      | No        | `dockerlog` | Source category to appear when searching on Sumo Logic by `_sourceCategory`. To include the log tag in the source category, use `{{.Tag}}`; see `tag`.


we should add the same support for _sourceHost and _sourceName

should we allow the tag to be in _sourceHost and _sourceName as well?

yep, basically 3 ways for customer to specify the metadata

rvmiller89 · 2017-08-25T22:14:58Z

I'm confused on the usage of --log-opt tag. I would think we can actually skip specifying this, and have the user specify the 3 metadata directly, such as

--log-opt sumo-source-category="Category_{{.ImageName}}" \
--log-opt sumo-source-name="{{.Name}}/{{.ID}}" \

etc.

samjsong · 2017-08-25T22:22:36Z

Yeah, I agree that would probably be simpler. One issue is that the tag option is a Docker supported thing: https://docs.docker.com/engine/admin/logging/log_tags/ so it makes sense to follow this convention of using the "tag" log-opt.

A smaller issue is that Docker provides in their code a way to easily parse the tag in their loggerutils package: tag, err := loggerutils.ParseLogTag(info, loggerutils.DefaultTemplate). To support the functionality you mention, I would have to ignore this package (they hard-coded the key "tag", so it wouldn't work with "sumo-source-category"), but if you feel that this would be a much better design, I can make that change

rvmiller89 · 2017-08-25T22:25:20Z

Yea I think we'll want to look into this more, since we have 3 metadata fields, but only 1 tag field capable of template markup, which limits the usefulness of metadata. Thoughts @bin3377 ?

samjsong · 2017-08-25T22:27:38Z

Definitely. One thing we could do is to support the tag in the backend, in which case one possible approach would be to support templating in the 3 metadata fields, and also to support a fourth tag metadata field.

rvmiller89 · 2017-08-25T23:23:43Z

I wonder if we could:

examine all 3 metadata fields to find set of template keys
create an artificial tag string containing each key separated by separator, e.g. {{.ID}}_{{. ImageName}}_{{.ImageName}}
use ParseLogTag to resolve the template, split again, and create a lookup map of each key to resolved value
use lookup map to string-replace the original metadata fields from (1)

bin3377 · 2017-08-25T23:59:51Z

The problem here is: the native docker log/stats blades don't have ability to resolve the template in -tag (which is a go-lang feature). So for sending metadata in _sourceCategory (and other 2), I proposed to use a Sumo owned tokenized template as {{.MARKUP}} {{#LABEL}} and {{$ENV}}

So if we want make the docker logging driver has best compatible with native docker blades, Ideally sumo-source-XXXX should also support all 3 types of token and also add {{.Tag}} as one extra {{.MARKUP}} so customer can get benefit by both features (from docker -tag or from Sumo {{}})

And we can even give sumo-source-category a default value of {{.Tag}} so if customer just using -tag as in other logging driver, it will be sent as _sourceCategory

bin3377 · 2017-08-26T00:06:00Z

@yuting-liu since we may want the native blades and logging driver has identical syntax on the metadata template

bin3377 · 2017-09-06T23:10:41Z

add few changes around the metadata part:

if sumo-source-category is not given, use the value from HTTP source (e.g. not sending in header)
change token {{.Tag}} to {{tag}}; so later on, it can be extended to {{tag.Id}}, {{tag.name}}, etc.
any unrecognized token(s) will be replaced to empty
token name is case insensitive
extend the code to easy adopt any interpret by just modify dictionary
unit test update for these features

small fixes:

trim leading "/" in the container's name when using default value of _sourceName (consist with docker sources)
validate sumo-url existing and be valid URL when initial

bin3377 · 2017-09-06T23:17:11Z

@latkin @rvmiller89 apply the change we discussed today. Please take a look before I merge it

bin3377 · 2017-09-07T18:53:09Z

@latkin @rvmiller89 can you guys make it higher priority since @bgoleno want it out next week and we still have some validating process on the docker store side after we finalize the code

latkin · 2017-09-07T22:05:11Z

DEVELOPER.md

@@ -0,0 +1,45 @@
+# Guide for developers
+This project is a plugin working in docker engine for delivering log to Sumo Logic cloud service through HTTP source.


Fix some typos:

This project is a plugin for the docker engine, which delivers logs to Sumo Logic by pushing log messages through an HTTP source. This is the guide for developers who want build and extend the plugin. If you just want use this plugin in your docker environment, please refer to the [readme file](README.md).

latkin · 2017-09-07T22:05:54Z

DEVELOPER.md

+```bash
+$ sudo bash ./plugin_install.sh
+```
+If everything goes fine, you can verify the plugin correctly installed with:


If everything goes fine, you can verify that the plugin is correctly installed with:

latkin · 2017-09-07T22:06:42Z

DEVELOPER.md

+```bash
+$ docker run --log-driver=sumologic --log-opt sumo-url=<url> -i -t ubuntu bash
+```
+This will create a bash session in docker container and send all console contents to Sumo HTTP source as log lines


This will create a bash session in a docker container and send all console contents to a Sumo Logic HTTP source as log lines

latkin · 2017-09-07T22:06:55Z

DEVELOPER.md

+This will create a bash session in docker container and send all console contents to Sumo HTTP source as log lines
+
+## Run unit test
+The unit test is written in `XXX_test.go` which `XXX` is the module to be tested. You can launch all unite tests with:


unite = unit

latkin · 2017-09-07T22:07:55Z

README.md

@@ -2,7 +2,7 @@

 A Docker logging driver plugin to send logs to Sumo Logic.

-**Disclaimer:** This plugin is still being developed.  We recommend using this plugin in non-production environments.
+**Disclaimer:** This plugin is still being developed. We recommend using this plugin in non-production environments.


If we are announcing this at Illuminate we should remove this warning. (can be separate PR)

latkin · 2017-09-07T22:10:28Z

README.md

 | sumo-url                  | Yes       |               | HTTP Source URL
+| sumo-source-category      | No        |               | Source category to appear when searching on Sumo Logic by `_sourceCategory`. Use `{{Tag}}` as the placeholder of `tag` option. If not specified, the source category of HTTP source will be used


For all of these, "searching on" should be "searching in"

latkin · 2017-09-07T22:11:27Z

README.md

 | sumo-url                  | Yes       |               | HTTP Source URL
+| sumo-source-category      | No        |               | Source category to appear when searching on Sumo Logic by `_sourceCategory`. Use `{{Tag}}` as the placeholder of `tag` option. If not specified, the source category of HTTP source will be used


If not specified, the default source category configured for the HTTP source will be used.

latkin · 2017-09-07T22:12:17Z

README.md

 | sumo-url                  | Yes       |               | HTTP Source URL
+| sumo-source-category      | No        |               | Source category to appear when searching on Sumo Logic by `_sourceCategory`. Use `{{Tag}}` as the placeholder of `tag` option. If not specified, the source category of HTTP source will be used
+| sumo-source-name          | No        |               | Source name to appear when searching on Sumo Logic by `_sourceName`. Use `{{Tag}}` as the placeholder of `tag` option. If not specified, it will be the container's name


"If not specified, it will be the container's name" This should be in the "Default Value" column it sounds like

it's a little bit different, since default value is always a fixed value but these 3 options are actually a behavior (since you cannot specify a container's name no matter whatever string you assigned here)

I woudl suggest using code style for the fixed value ones, and regular text description for others.

From casual user's perspective, if the "Default Value" column is empty, then they would expect no default value.

ok. make sense

latkin · 2017-09-07T22:14:56Z

README.md

 | sumo-url                  | Yes       |               | HTTP Source URL
+| sumo-source-category      | No        |               | Source category to appear when searching on Sumo Logic by `_sourceCategory`. Use `{{Tag}}` as the placeholder of `tag` option. If not specified, the source category of HTTP source will be used
+| sumo-source-name          | No        |               | Source name to appear when searching on Sumo Logic by `_sourceName`. Use `{{Tag}}` as the placeholder of `tag` option. If not specified, it will be the container's name


"Use {{Tag}} as the placeholder of tag option." I think wording could be improved -- is it correct to say "Within the source name, the token {{Tag}} will be replaced with the value of the Docker tag option."

Similar for category/host.

latkin · 2017-09-07T22:19:39Z

README.md

@@ -49,6 +52,9 @@ To specify additional logging driver options, you can use the `--log-opt NAME=VA
 | sumo-root-ca-path         | No        |               | Set the path to a custom root certificate.
 | sumo-server-name          | No        |               | Name used to validate the server certificate. By default, uses hostname of the `sumo-url`.
 | sumo-queue-size           | No        | 100           | The maximum number of log batches of size `sumo-batch-size` we can store in memory in the event of network failure, before we begin dropping batches. Thus in the worst case, the plugin will use `sumo-batch-size` * `sumo-queue-size` bytes of memory per container (default 100 MB).
+| tag                       | No        | {{.ID}}       | Specify a tag for messages, which interprets some markup. Default value is {{.ID}} (first 12 characters of the container ID). Refer to the [tag log-opt documentation] for customizing the log tag format.


Would update wording to "Specifies a tag for messages, which can be used in the 'source category', 'source name', and source host' fields. Certain tokens of the form {{X}} are supported. Default value is {{.ID}}, the first 12 characters of the container ID. Refer to the [tag log-opt documentation] for more information and a list of supported tokens."

latkin · 2017-09-07T22:28:54Z

driver.go

+
+  for _, v := range re.FindAllSubmatchIndex([]byte(input), -1) {
+      groups := []string{}
+      for i := 0; i < len(v); i += 2 {


Why is it incrementing by 2?

it's because the return of the FindAllSubmatchIndex() actually return a pattern as begin/end index of the matching sections

latkin · 2017-09-07T22:38:40Z

logger.go

@@ -162,6 +162,11 @@ func (sumoLogger *sumoLogger) sendLogs(logs []*sumoLog) error {
  if sumoLogger.gzipCompression {
    request.Header.Add("Content-Encoding", "gzip")
  }
+  if sumoLogger.sourceCategory != "" {


Does != "" also handle nil/null?

That's because in golang the string cannot be nil (but *string can)

latkin · 2017-09-07T22:39:31Z

README.md

 | sumo-url                  | Yes       |               | HTTP Source URL
+| sumo-source-category      | No        |               | Source category to appear when searching on Sumo Logic by `_sourceCategory`. Use `{{Tag}}` as the placeholder of `tag` option. If not specified, the source category of HTTP source will be used
+| sumo-source-name          | No        |               | Source name to appear when searching on Sumo Logic by `_sourceName`. Use `{{Tag}}` as the placeholder of `tag` option. If not specified, it will be the container's name
+| sumo-source-host          | No        |               | Source host to appear when searching on Sumo Logic by `_sourceHost`. Use `{{Tag}}` as the placeholder of `tag` option. If not specified, it will be the machine host name


"If not specified, it will be the machine host name" should be in "default value" column

latkin · 2017-09-07T22:40:32Z

logger.go

+  if sumoLogger.sourceCategory != "" {
+    request.Header.Add("X-Sumo-Category", sumoLogger.sourceCategory)
+  }
+  request.Header.Add("X-Sumo-Name", sumoLogger.sourceName)


I know these should never be empty, but would be good to do != "" check on these too.

ok. I can add the checking

latkin · 2017-09-07T22:48:29Z

Looking at https://docs.docker.com/engine/admin/logging/log_tags/ it seems that if users specify a tag logopt, then the value of the tag will be injected into every log message? Is this the case?

I thought we were using it simply as an easy way to do token evaluation, then only expected the result to be included in sourceCategory, sourceHost, etc.

Besides this, overall LGTM, just a bunch of small doc suggestions and a couple small code suggestions.

bin3377 · 2017-09-07T23:02:49Z

The behavior of tag is decided by the logging driver. For example in Splunk plugin The tag can become a json field or a prefix depends on the format of message

So yes, in sumo plugin, we decide the tag only be used in 3 metadata fields depends on the parameter given (not as prefix because the multiline processing)

latkin · 2017-09-07T23:33:46Z

Ok, so when a Docker + Sumo user specifies a tag, the tag value will NOT be injected into log lines? Because it is up to our driver to decide what to do with it?

latkin · 2017-09-08T00:35:25Z

LGTM

bin3377 · 2017-09-08T01:11:46Z

Merge to make a tag. feel free to comment

aidansteele · 2018-06-07T22:46:40Z

Did you end up implementing the {{.MARKUP}}, {{#LABEL}} and {{$ENV}} functionality @bin3377? I would like to be able to use Docker container labels in both the source host and source category, but right now this appears to not be possible with the current tag workaround.

bin3377 · 2018-06-07T22:57:46Z

Not yet. File an issue #21 for tracking it. But since we will be consistent with the tag using in other SumoLogic source types, the tags will probably be changed to the formats described on this page

samjsong added 5 commits August 23, 2017 12:45

Set default HTTP headers for category and name

4b770b4

Add source category and tag options

38f874e

Update README

13e937f

Fix tab/space issue

c7e0db7

Remove unneeded info logs

08e10a3

rvmiller89 reviewed Aug 25, 2017

View reviewed changes

samjsong requested a review from bin3377 August 25, 2017 21:49

samjsong added 2 commits August 25, 2017 15:02

Add log-opts for sourceName, sourceHost

8a3666f

Update README

ccdf32e

samjsong requested a review from rvmiller89 August 25, 2017 22:06

latest change on the metadata part

5dd7b81

bin3377 force-pushed the ssong-SUMO-76740-metadata branch from 896afad to 5dd7b81 Compare September 6, 2017 23:04

bin3377 requested a review from latkin September 6, 2017 23:16

latkin self-assigned this Sep 6, 2017

developer guide

2dd63c0

latkin reviewed Sep 7, 2017

View reviewed changes

Bin Yi added 3 commits September 7, 2017 16:04

PR - word repahsing

b9c0d55

add space checking for all metadata fields

8c2fd36

Remove the in developing disclaimer

7ddcdf4

default value of meta fields

7951649

fix unit test

6c8863d

bin3377 merged commit 53e8e5e into master Sep 8, 2017

bin3377 deleted the ssong-SUMO-76740-metadata branch September 20, 2017 21:45

		\| sumo-url \| Yes \| \| HTTP Source URL
		\| sumo-source-category \| No \| `dockerlog` \| Source category to appear when searching on Sumo Logic by `_sourceCategory`. To include the log tag in the source category, use `{{.Tag}}`; see `tag`.

		@@ -0,0 +1,45 @@
		# Guide for developers
		This project is a plugin working in docker engine for delivering log to Sumo Logic cloud service through HTTP source.

		\| sumo-url \| Yes \| \| HTTP Source URL
		\| sumo-source-category \| No \| \| Source category to appear when searching on Sumo Logic by `_sourceCategory`. Use `{{Tag}}` as the placeholder of `tag` option. If not specified, the source category of HTTP source will be used

Ssong sumo 76740 metadata #7

Ssong sumo 76740 metadata #7

Conversation

samjsong commented Aug 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rvmiller89 commented Aug 25, 2017 • edited Loading

samjsong commented Aug 25, 2017

rvmiller89 commented Aug 25, 2017

samjsong commented Aug 25, 2017

rvmiller89 commented Aug 25, 2017

bin3377 commented Aug 25, 2017

bin3377 commented Aug 26, 2017

bin3377 commented Sep 6, 2017 • edited Loading

bin3377 commented Sep 6, 2017

bin3377 commented Sep 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

latkin commented Sep 7, 2017 • edited Loading

bin3377 commented Sep 7, 2017 • edited Loading

latkin commented Sep 7, 2017

latkin commented Sep 8, 2017

bin3377 commented Sep 8, 2017

aidansteele commented Jun 7, 2018

bin3377 commented Jun 7, 2018

rvmiller89 commented Aug 25, 2017 •

edited

Loading

bin3377 commented Sep 6, 2017 •

edited

Loading

latkin commented Sep 7, 2017 •

edited

Loading

bin3377 commented Sep 7, 2017 •

edited

Loading