Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grpcclient: Make keepalive time and timeout into config parameters #56

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

aknuds1
Copy link
Collaborator

@aknuds1 aknuds1 commented Oct 7, 2021

What this PR does:
In grpcclient package, make ping time and timeout into configuration parameters so each app can control them instead of having to use hardcoded values. Also add a test.

Which issue(s) this PR fixes:

Checklist

  • Tests updated
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@aknuds1 aknuds1 added the enhancement New feature or request label Oct 7, 2021
@aknuds1 aknuds1 changed the title grpcclient: Make ping time and timeout into config parameters WIP: grpcclient: Make ping time and timeout into config parameters Oct 7, 2021
@aknuds1 aknuds1 changed the title WIP: grpcclient: Make ping time and timeout into config parameters grpcclient: Make ping time and timeout into config parameters Oct 7, 2021
@aknuds1 aknuds1 marked this pull request as ready for review October 7, 2021 16:23
@aknuds1 aknuds1 requested a review from a team October 7, 2021 16:24
@pracucci
Copy link
Contributor

pracucci commented Oct 7, 2021

My concerns about exposing this config haven't changed, but I'm open to discuss it.

@aknuds1
Copy link
Collaborator Author

aknuds1 commented Oct 7, 2021

My concerns about exposing this config haven't changed, but I'm open to discuss it.

@pracucci could you link me to the concerns you expressed? I couldn't recall you had expressed other concerns than embedding it within the other PR.

@aknuds1
Copy link
Collaborator Author

aknuds1 commented Oct 7, 2021

@pracucci never mind, I found your comment. I must've overlooked it before.

@aknuds1
Copy link
Collaborator Author

aknuds1 commented Oct 7, 2021

@pracucci now, about whether to make this parameterizable or not, I think it makes sense to not force choices made by Cortex on other applications using dskit. I think this corresponds with @pstibrany's concerns that these very low ping times have proven problematic. What do you say @pstibrany?

If nothing else, parameterizing these values should document them towards dskit clients and make them aware of the implications. If the currently hardcoded values can cause problems, that certainly should be documented towards dskit clients as @pstibrany raised.

// KeepaliveTime is the number of seconds after which the client will ping the server in case of inactivity.
//
// See `google.golang.org/grpc/keepalive.ClientParameters.Time` for reference.
KeepaliveTime int64 `yaml:"ping_time"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is default value? Looks like it’s 0 now, which is change worth documenting or fixing.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pstibrany thanks for noticing this. I documented in the changelog and for Config.DialOption that the defaults are the same as for the underlying google.golang.org/grpc/keepalive package. Good enough?

PermitWithoutStream: true,
}),
), nil
}

var withDefaultCallOptions = func(cos ...grpc.CallOption) grpc.DialOption {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using global variables for these, can they be passed to dialOption as parameters? (Default publicly-exported DialOption would just use grpc-functions). This avoids needing setup and cleanup, and can be hidden into “fakeDialOption” method or similar.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pstibrany I can try.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this feedback hasn't been addressed, am I wrong?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pracucci you're right! I was deferring addressing this until consensus on whether the PR should be closed or not.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pracucci @pstibrany the drawback I see with stubbing these dependencies via arguments is that it adds complexity. The tests will no longer call Config.DialOption itself, and introduces the risk of having a bug in Config.DialOption where it passes the wrong dependencies. The test coverage will also be lower with this technique.

If you still prefer the injection via method arguments technique, I can change to that. Just want to clarify the problems I see with it.

grpcclient/grpcclient_test.go Show resolved Hide resolved
@aknuds1 aknuds1 changed the title grpcclient: Make ping time and timeout into config parameters grpcclient: Make keepalive time and timeout into config parameters Oct 11, 2021
grpcclient/grpcclient.go Outdated Show resolved Hide resolved
grpcclient/grpcclient.go Outdated Show resolved Hide resolved
grpcclient/grpcclient.go Outdated Show resolved Hide resolved
@pracucci
Copy link
Contributor

I'm still a bit dubious this extra config is going to give a real value. Are we solving a real issue? If not, I've the feeling we're just making our config bigger.

@aknuds1
Copy link
Collaborator Author

aknuds1 commented Dec 22, 2021

I'm still a bit dubious this extra config is going to give a real value. Are we solving a real issue? If not, I've the feeling we're just making our config bigger.

@pracucci IIRC @pstibrany thought this config would make sense. Peter, do you still think we should parameterize these values?

@pstibrany
Copy link
Member

I think default client timeout values used by dskit are too aggressive, and incompatible with default values on gRPC server. We have seen many issues with them in multiple projects (Cortex, Loki). I think dskit should not use such a low defaults. Exposing config (not necessarily to user, but to projects using dskit) would give us option to change defaults in dskit to be more reasonable, while giving projects that have adjusted their gRPC server values an option to use lower values still.

@aknuds1
Copy link
Collaborator Author

aknuds1 commented Dec 22, 2021

Peter's feedback sounds reasonable to me, although I lack the experience to tell whether the defaults are too aggressive or not. Also agree the configuration doesn't have to be user visible.

@pstibrany
Copy link
Member

Change in Cortex: cortexproject/cortex#4168, example user-reported issue: cortexproject/cortex#3606
Change in Loki: grafana/loki#4435, example user-reported issue: grafana/loki#4375

All projects using dskit will need to adjust server-side settings (or modify dskit defaults, but right now they can't).

Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think default client timeout values used by dskit are too aggressive, and incompatible with default values on gRPC server. We have seen many issues with them in multiple projects (Cortex, Loki). I think dskit should not use such a low defaults. Exposing config (not necessarily to user, but to projects using dskit) would give us option to change defaults in dskit to be more reasonable, while giving projects that have adjusted their gRPC server values an option to use lower values still.

I see your point. I agree exposing such config to projects (but not users, at least not here). However, given the breaking change in the config (and the risk downstream projects miss it), I think it's safe if we break the DialOption() signature (and take it explicitly) so that it's a clear signal to downstream projects.

I also don't think such config should be in the Config struct.

PermitWithoutStream: true,
}),
), nil
}

var withDefaultCallOptions = func(cos ...grpc.CallOption) grpc.DialOption {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this feedback hasn't been addressed, am I wrong?

@pstibrany
Copy link
Member

I see your point. I agree exposing such config to projects (but not users, at least not here). However, given the breaking change in the config (and the risk downstream projects miss it), I think it's safe if we break the DialOption() signature (and take it explicitly) so that it's a clear signal to downstream projects.

I also don't think such config should be in the Config struct.

I agree that breaking API is safer to make sure clients notice the change and there is no need to expose it via Config struct.

@aknuds1
Copy link
Collaborator Author

aknuds1 commented Jan 4, 2022

@pstibrany @pracucci As per your feedback, I moved the KeepAliveTime* parameters into Config.DialOption arguments. Also documented in the changelog that their defaults are the same as in the wrapped google.golang.org/grpc/keepalive package. Hope that looks good?

@aknuds1 aknuds1 requested a review from pstibrany January 4, 2022 15:08
// keepaliveTimeout is the number of seconds the client waits after pinging the server, and if no activity is
// seen after that, the connection is closed. See `google.golang.org/grpc/keepalive.ClientParameters.Timeout`
// for reference.
func (cfg *Config) DialOption(unaryClientInterceptors []grpc.UnaryClientInterceptor, streamClientInterceptors []grpc.StreamClientInterceptor, keepaliveTime, keepaliveTimeout int64) ([]grpc.DialOption, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think keepaliveTime and keepaliveTimeout should be time.Duration.

// WithInsecure wraps grpc.WithInsecure.
//
// Stubbable for tests.
var WithInsecure = func() grpc.DialOption {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we wanna to expose it? Looks weird to me.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's technically necessary for grpcclient tests to be able to stub calling out to the grpc package. I'll look again into whether there's another way to do this (dependency injection?).

return grpc.WithInsecure()
}

// GetGRPCDialOptions creates GRPC DialOptions for TLS.
func (cfg *ClientConfig) GetGRPCDialOptions(enabled bool) ([]grpc.DialOption, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Can we rename enabled to tlsEnabled to make it more clear?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pracucci done.

@CLAassistant
Copy link

CLAassistant commented Jun 15, 2022

CLA assistant check
All committers have signed the CLA.

@liguozhong
Copy link

great 👍

charleskorn added a commit that referenced this pull request Aug 11, 2023
* Use ps.Map for Counters and Sets, remove Metadata in favour of Latest.

Also
- Add more complicated report.json for benchmark
- Break up report/topology.go
- Implement our own DeepEqual for ps.Map

* Memoise & cache the result of renderers, so we don't recalculate views multiple times.

* Fix tests

* refactoring deepequal to satisfy linter

* add test for render/detailed/parents and fixed bug

* Decouple Scope lifecycle from Weave lifecycle

- Run the Weave integrations regardless of if weave is detected.
- Make everything backoff and not spam the logs.
- Add miekg dns to vendor.
- Have the app periodically register with weaveDNS, and the probe do lookups there.
- Decide what the local networks are at runtime, not once at startup.
- Correctly resolve app ids, fixes #825

* adding direction to connections from conntrack

* Remove report.EdgeMetadata.MaxConnCountTCP, as we don't display it anywhere
* Remove hostname metadata from local end of connection. We should be using the hostnodeid

* Remove address topology

* Fix spelling mistakes in the codebase.

* Refactoring rendering to remove RenderableNode

Squash of:
- use detailed.Summaries to render topology nodes
- ban merging nodes of different topologies (they should be mapped)
- need to prune parents when mapping node types
- render container images by id if they have no name
- remove separate render ids and prune parents in NewDerived*
- don't render metrics/metadata for groups of nodes
- fixing up tests
- removing pending unit tests (for mapping.go, for now)
- updating experimental dir for RenderableNode removal

* Review Feedback

Squash of:
- including children in topologies_test.go
- report.Node.Prune should prune children also
- rewrote ShortLivedInternetConnections test to express its intent
- adding tests for detail Summary rendering

* Update docker client, to get better state strings in the UI

* Remove load5 and load15

* not really useful
* take up lots of real estate

Fixes #1267

* Adding support for plugins, with basic example of iowait, and ebpf

Squash of:
* Include plugins in the report
* show plugin list in the UI
* moving metric and metadata templates into the probe reports
* update js for prime -> priority
* added retry to plugin handshake
* added iowait plugin
* review feedback
* plugin documentation

* adding a test for pod node rendering

* Add local_networks to weave Overlay nodes, so we can track weave without an exposed weave ip (#1313)

* Removing report.Node.WithID (#1315)

* removing usage of report.Node.WithID

* report.Topology.AddNode can use the node's ID field

* move counting sublabel definition to the topologies

* Reorganise the render/ package

* Fixing grouped node count for filtered children nodes

Squash of:

* We have to keep all the container hostnames until the end so we can
  count how many we've filtered

* Adding tests for ContainerHostnameRenderer and PodServiceRenderer with
  filters

* Because we filter on image name we need the image name before
  filtering

* Alternative approach to passing decorators.

* Refactor out some of the decorator capture

* Don't memoise decorated calls to Render

* Fixing filtered counts on containers topology

  Tricky, because we need the filters to be silent sometimes (when they're
  in the middle), but not when they're at the top, so we take the "top"
  filter's stats. However, this means we have to compose all
  user-specified filters into a single Filter layer, so we can get all
  stats.

  There are no more Silent filters, as all filters are silent (unless they
  are at the top).

  Additionally, I clarified some of the filters as their usage/terminology
  was inconsistent and confused. Now Filter(IsFoo, ...) *keeps* only nodes
  where IsFoo is true.

* Index Pods by UID and join with containers based on this.

* fixing up tests

* Index services by UID, and refactor out common k8s metadata handling

* Deployment and ReplicaSet views for k8s

* Review Feedback

* Use image name without version as id. (#1531)

* Use image name without version as id.

* Review feedback

* Fix lint in all the build-tools scripts

* Squashed 'tools/' changes from 7a66090..8c6170d

8c6170d Fix lint in all the build-tools scripts
239935c shell-lint tool
d9ab133 Script for finding files with a given type
1b64e46 Add Weave Cloud client
f2e40b4 Time out commands after three minutes
2da55ce Don't spell-check compressed files
e9749a5 Make scheduler aware of test parallelisation

git-subtree-dir: tools
git-subtree-split: 8c6170d292d34923d23d5eef6b6ac5e4af6b7188

* Speed up test by using git ls-files

* Remove spurious debugging code from test

* Test directories need ./ prefixes, obviously.

* Squashed 'tools/' changes from 0620e58..e9e7e6b

e9e7e6b Merge pull request #26 from weaveworks/this-time-for-sure
df494d6 Remove dependencies
c045d16 Properly exclude vendor from lint
2cfcf08 Add blacklist to wcloud client
ca6ebfb Merge pull request #25 from weaveworks/fix-brokenness
bfb1747 Test directories need ./ prefixes, obviously.
5b9b314 Merge pull request #24 from weaveworks/find-files
8786427 Remove spurious debugging code from test
8b7ec6e Speed up test by using git ls-files
cf53dc1 Exclude vendor from shell linting
b2ab380 Fix field name
c86fd3d Add notification config for wcloud
f643920 Merge pull request #23 from weaveworks/only-lint-git-files
47a0152 Only lint git files
50d47f9 Merge pull request #22 from weaveworks/shell-lint

git-subtree-dir: tools
git-subtree-split: e9e7e6b0f042eb38bd2d45591f3f5c8364bdc7ce

* Remove Metric Add() method

* Helps reduce garbage (MakeMetric() now takes a slice and there's a shorter version MakeSingletonMetric())
* Fixes bug computing Max (Min) in samples since using MakeMetric()
  was causing a default Max/Min of zero.
* Simplifies code a bit

* Remove Metric WithFirst() method

It was only used in tests and wasn't really necessary

* Append namespace to endpoint scope for loopback connections

* Remove redundant kubernetes ID

* Make dumper a bit more verbose

So it displays differences behind interface that would otherwise go
unnoticed (like string vs []byte).

* Sync changes done directly in scope/tools

Applies tools changes from:

commit 7f46b90e272f00275ce47a29e0cef0c62bbc4e38
Author: Krzesimir Nowak <[email protected]>
Date:   Thu Jul 21 11:55:08 2016 +0200

    Make LatestMap "generic"

    This commit makes the LatestMap type a sort of a base class, that
    should not be used directly. This also adds a generator for LatestMap
    "concrete" types with a specific data type in the LatestEntry.

commit 0f1cb82084435622b2b1c78bd36884cf90f1ab25
Author: Krzesimir Nowak <[email protected]>
Date:   Thu Jul 28 14:26:08 2016 +0200

    Allow testing only a subset of directories

    This can be done by calling TESTDIRS="./report ./probe" make tests

commit 97eb8d033dc07dee338a43c170e809731ca3c9e4
Author: Krzesimir Nowak <[email protected]>
Date:   Thu Aug 4 11:44:21 2016 +0200

    Do not spell check Makefiles and JSON files

    Spell check does not handle JSON files very well. It finds misspelled
    words in hexadecimal hashes (like misspelled "dead" in "123daed456").

    Ignore Makefiles too - there is not so much free text to spell check
    and the spell check complains about words it was told by Makefile to

* Squashed 'tools/' changes from e9e7e6b..b990f48

b990f48 Merge pull request #42 from kinvolk/lorenzo/fix-git-diff
224a145 Check if SHA1 exists before calling `git diff`
1c3000d Add auto_apply config for wcloud
0ebf5c0 Fix wcloud -serivice
4fe078a Merge pull request #39 from weaveworks/fix-wrong-subtree-use
3f4934d Remove generate_latest_map
48beb60 Sync changes done directly in scope/tools
45dcdd5 Merge pull request #37 from weaveworks/fix-mflag-missing
b895344 Use mflag package from weaveworks fork until we find a better solution
e030008 Merge pull request #36 from weaveworks/wcloud-service-flags
9cbab40 Add wcloud Makefile
ef55901 Review feedback, and build wcloud in circle.
3fe92f5 Add wcloud deploy --service flag
3527b56 Merge pull request #34 from weaveworks/repo-branch
92cd0b8 [wcloud] Add support for repo_branch option
9f760ab Allow wcloud users to override username
38037f8 Merge pull request #33 from weaveworks/wcloud-templates
7acfbd7 Propagate the local username
e6876d1 Add template fields to wcloud config.
f1bb537 Merge pull request #30 from weaveworks/mike/shell-lint/dont-error-if-empty
e60f5df Merge pull request #31 from weaveworks/mike/fix-shell-lint-errors
e8e2b69 integrations: Fix a shellcheck linter error
a781575 shell-lint: Don't fail if no shell scripts found
db5efc0 Merge pull request #28 from weaveworks/mike/add-image-tag
5312c40 Import image-tag script into build tools so it can be shared
7e850f8 Fix logs path
dda9785 Update deploy api
f2f4e5b Fix the wcloud client
3925eb6 Merge pull request #27 from weaveworks/wcloud-events
77355b9 Lint
d9a1c6c Add wcloud events, update flags and error nicely when there is no config

git-subtree-dir: tools
git-subtree-split: b990f488bdc7909b62d9245bc4b4caf58f5ae7ea

* Add Weave peers view

* Extend metadata in details panel for Weave Net nodes

* Added container filters as CLI arguments

gofmt load_container_filters.go

removed the environment variable for container label filters

Added the --app.container-label-filter command line argument, and load_container_filters.go now uses the results from that

Changed init() to InitializeTopologies()

Changed init() to InitializeTopologies() so that it can be called after the container filters are loaded from the command line argument. init() executes before main() in prog/main.go, so the flag parsing isn't finished before init() is called

Applied lint fixes

fixed lint issues

brought back the init function for api_topologies.go

Addressed many of the PR comments, except escaping colons

Renamed IsDesired to HasLabel in render/filters.go

Allows for the user to escape colons

added registry function for modifying the container filters

created a separate function that parses the container filter flags

simplified registry.addContainerFilters()

addressed review comments

switched API Topology Description IDs to constants

addressed review comments

joined constants

added test functions

addressed most of the review comments

Changed containerLabelFilters to an array of APItopologyOptions, placing the parsing in the Set() function. Removed parsing from HasLabel in render/filters.go

refactored code

added test that applies to the container filtering by labels

applied golint

made Registry items private and added a MakeRegistry() function

fixed usage of topologyRegistry.RendererForTopology

Added container label filters by exclusion

minor update to report_fixture

Modified container labels test to use existing report

I added labels to the existing containers in the fixed report for testing.

refactored code

refactored code

further code refactoring

addressed @ijsnellf's review comments

unexported Registry, and reduced duplicate code

addressed @ijsnellf's review comments

Addressed review comments

Addressed final review comments

* linter: fix punctuation and capitalization

Symptoms:
> app/control_router.go:44:38: error strings should not be capitalized or end with punctuation or a newline

This is blocking the build on:
https://circleci.com/gh/kinvolk/scope/363

* Squashed 'tools/' content from commit fd875e2

git-subtree-dir: tools
git-subtree-split: fd875e27c5379d443574bcf20f24a52a594871ca

* Remove uncommon things

* Move packages out of common directory

* Don't use scope's reflect

* poll is secretly scope specific

* Squashed 'tools/' changes from fd875e2..03cc598

03cc598 Don't lint generated protobuf code.
2b55c2d Merge pull request #66 from weaveworks/reduce-test-timeout
d4e163c Make timeout a flag
49a8609 Reduce test timeout
8fa15cb Merge pull request #63 from weaveworks/test-defaults
b783528 Tweak test script so it can be run on a mca
a3b18bf Merge pull request #65 from weaveworks/fix-integration-tests
ecb5602 Fix integration tests
f9dcbf6 ... without tab (clearly not my day)
a6215c3 Add break I forgot
0e6832d Remove incorrectly added tab
eb26c68 Merge pull request #64 from weaveworks/remove-test-package-linting
f088e83 Review feedback
2c6e83e Remove test package linting
2b3a1bb Merge pull request #62 from weaveworks/revert-61-test-defaults
8c3883a Revert "Make no-go-get the default, and don't assume -tags netgo"
e75c226 Fix bug in GC of firewall rules.
e49754e Merge pull request #51 from weaveworks/gc-firewall-rules
191f487 Add flag to enale/disable firewall rules' GC.
567905c Add GC of firewall rules for weave-net-tests to scheduler.
03119e1 Fix typo in GC of firewall rules.
bbe3844 Fix regular expression for firewall rules.
c5c23ce Pre-change refactoring: splitted gc_project function into smaller methods for better readability.
ed5529f GC firewall rules
ed8e757 Merge pull request #61 from weaveworks/test-defaults
57856e6 Merge pull request #56 from weaveworks/remove-wcloud
dd5f3e6 Add -p flag to test, run test in parallel
62f6f94 Make no-go-get the default, and don't assume -tags netgo
8946588 Merge pull request #60 from weaveworks/2647-gc-weave-net-tests
4085df9 Scheduler now also garbage-collects VMs from weave-net-tests.
4b7d5c6 Merge pull request #59 from weaveworks/57-fix-lint-properly
b7f0e69 Merge pull request #58 from weaveworks/fix-lint
794702c Pin version of shfmt
ab1b11d Fix lint
d1a5e46 Remove wcloud cli tool
81d80f3 Merge pull request #55 from weaveworks/lint-tf
05ad5f2 Review feedback
4c0d046 Use hclfmt to lint terraform.

git-subtree-dir: tools
git-subtree-split: 03cc5989769d93aa03f8aed3784ddfdb1fffc1c6

* Squashed 'tools/' changes from 03cc598..9857568

9857568 Use mflag and mflagext package from weaveworks/common.
9799112 Quote bash variable.
10a36b3 Merge pull request #67 from weaveworks/shfmt-ignore
a59884f Add support for .lintignore.

git-subtree-dir: tools
git-subtree-split: 98575686df9c946c076bbfd7bc50c62825f1f030

* Add SetEnv() to Cmd mocking interface

* test: Do not block when reading stderr pipe

* Squashed 'tools/' changes from 9857568..41c5622

41c5622 Merge pull request #90 from weaveworks/build-golang-service-conf
e8ebdd5 broaden imagetag regex to fix haskell build image
ba3fbfa Merge pull request #89 from weaveworks/build-golang-service-conf
e506f1b Fix up test script for updated shfmt
9216db8 Add stuff for service-conf build to build-goland image
66a9a93 Merge pull request #88 from weaveworks/haskell-image
cb3e3a2 shfmt
74a5239 Haskell build image
4ccd42b Trying circle quay login
b2c295f Merge branch 'common-build'
0ac746f Trim quay prefix in circle script
c405b31 Merge pull request #87 from weaveworks/common-build
9672d7c Push build images to quay as they have sane robot accounts
a2bf112 Review feedback
fef9b7d Add protobuf tools
10a77ea Update readme
254f266 Don't need the image name in
ffb59fc Adding a weaveworks/build-golang image with tags
b817368 Update min Weave Net docker version
cf87ca3 Merge pull request #86 from weaveworks/lock-kubeadm-version
3ae6919 Add example of custom SSH private key to tf_ssh's usage.
cf8bd8a Add example of custom SSH private key to tf_ansi's usage.
c7d3370 Lock kubeadm's Kubernetes version.
faaaa6f Merge pull request #84 from weaveworks/centos-rhel
ef552e7 Select weave-kube YAML URL based on K8S version.
b4c1198 Upgrade default kubernetes_version to 1.6.1.
b82805e Use a fixed version of kubeadm.
f33888b Factorise and make kubeconfig option optional.
f7b8b89 Install EPEL repo for CentOS.
615917a Fix error in decrypting AWS access key and secret.
86f97b4 Add CentOS 7 AMI and username for AWS via Terraform.
eafd810 Add tf_ansi example with Ansible variables.
2b05787 Skip setup of Docker over TCP for CentOS/RHEL.
84c420b Add docker-ce role for CentOS/RHEL.
00a820c Add setup_weave-net_debug.yml playbook for user issues' debugging.
3eae480 Upgrade default kubernetes_version to 1.5.4.
753921c Allow injection of Docker installation role.
e1ff90d Fix kubectl taint command for 1.5.
b989e97 Fix typo in kubectl taint for single node K8S cluster.
541f58d Remove 'install_recommends: no' for ethtool.
c3f9711 Make Ansible role docker-from-get.docker.com work on RHEL/CentOS.
038c0ae Add frequently used OS images, for convenience.
d30649f Add --insecure-registry to docker.conf
1dd9218 shfmt -i 4 -w push-images
6de96ac Add option to not push docker hub images
310f53d Add push-images script from cortex
8641381 Add port 6443 to kubeadm join commands for K8S 1.6+.
50bf0bc Force type of K8S token to string.
08ab1c0 Remove trailing whitespaces.
ae9efb8 Enable testing against K8S release candidates.
9e32194 Secure GCP servers for Scope: open port 80.
a22536a Secure GCP servers for Scope.
89c3a29 Merge pull request #78 from weaveworks/lint-merge-rebase-issue-in-docs
73ad56d Add linter function to avoid bad merge/rebase artefact
52d695c Merge pull request #77 from kinvolk/schu/fix-relative-weave-path
77aed01 Merge pull request #73 from weaveworks/mike/sched/fix-unicode-issue
7c080f4 integration/sanity_check: disable SC1090
d6d360a integration/gce.sh: update gcloud command
e8def2c provisioning/setup: fix shellcheck SC2140
cc02224 integration/config: fix weave path
9c0d6a5 Fix config_management/README.md
334708c Merge pull request #75 from kinvolk/alban/external-build-1
da2505d gce.sh: template: print creation date
e676854 integration tests: fix user account
8530836 host nameing: add repo name
b556c0a gce.sh: fix deletion of gce instances
2ecd1c2 integration: fix GCE --zones/--zone parameter
3e863df sched: Fix unicode encoding issues
51785b5 Use rm -f and set current dir using BASH_SOURCE.
f5c6d68 Merge pull request #71 from kinvolk/schu/fix-linter-warnings
0269628 Document requirement for `lint_sh`
9a3f09e Fix linter warnings
efcf9d2 Merge pull request #53 from weaveworks/2647-testing-mvp
d31ea57 Weave Kube playbook now works with multiple nodes.
27868dd Add GCP firewall rule for FastDP crypto.
edc8bb3 Differentiated name of dev and test playbooks, to avoid confusion.
efa3df7 Moved utility Ansible Yaml to library directory.
fcd2769 Add shorthands to run Ansible playbooks against Terraform-provisioned virtual machines.
f7946fb Add shorthands to SSH into Terraform-provisioned virtual machines.
aad5c6f Mention Terraform and Ansible in README.md.
dddabf0 Add Terraform output required for templates' creation.
dcc7d02 Add Ansible configuration playbooks for development environments.
f86481c Add Ansible configuration playbooks for Docker, K8S and Weave-Net.
efedd25 Git-ignore Ansible retry files.
765c4ca Add helper functions to setup Terraform programmatically.
801dd1d Add Terraform cloud provisioning scripts.
b8017e1 Install hclfmt on CircleCI.
4815e19 Git-ignore Terraform state files.
0aaebc7 Add script to generate cartesian product of dependencies of cross-version testing.
007d90a Add script to list OS images from GCP, AWS and DO.
ca65cc0 Add script to list relevant versions of Go, Docker and Kubernetes.
aa66f44 Scripts now source dependencies using absolute path (previously breaking make depending on current directory).
7865e86 Add -p option to parallelise lint.
36c1835 Merge pull request #69 from weaveworks/mflag

git-subtree-dir: tools
git-subtree-split: 41c562219dcc03a70dfdd9b1353cd4cd4f1cab46

* fs: add ReadDirCount (#30)

* fs: add ReadDirCount

ReadDirNames is expensive and better avoided when we don't care about
the names.

* fs: add tests and benchmarks for ReadDirNames and ReadDirCount

$ go test -bench Benchmark*
BenchmarkReadDirNames-4   	   10000	    155566 ns/op
BenchmarkReadDirCount-4   	   10000	    124618 ns/op

* Fix lint error

* Add ConfigFromURL helper function

extracted from weaveworks/cortex

* Allow the AWS backend to get credentials from its environment. (#111)

Signed-off-by: Tom Wilkie <[email protected]>

* Default diff-printer to hide details (#103)

This new setting will make it print the difference between String()
output only.  This is more usable for developers in 99% of cases where
the difference can be seen.

* Squashed 'tools/' changes from 41c5622..d6cc704

d6cc704 Fix comment
7139116 Revert "Push comments to the left so they don't appear in scripts"
e47e58f Push comments to the left so they don't appear in scripts
3945fce Remove nonexistent env var GIT_TAG
cd62992 Merge pull request #156 from weaveworks/drop-quay
af0eb51 Merge pull request #157 from weaveworks/fix-image-tag-prefix-length
0b9aee4 Fix image-tag object name prefix length to 8 chars.
813c28f Move from CircleCI 1.0 to 2.0
425cf4e Move from quay.io to Dockerhub
87ccf4f Merge pull request #155 from weaveworks/go-1-12
c31bc28 Update lint script to work with Go 1.12
ed8e380 Update to Go 1.12.1
ec369f5 Merge pull request #153 from dholbach/drop-email
ef7418d weave-users mailing list is closed: https://groups.google.com/a/weave.works/forum/#!topic/weave-users/0QXWGOPdBfY
6954a57 Merge pull request #144 from weaveworks/golang-1.11.1
9649eed Upgrade build image from golang:1.10.0-strech to 1.11.1-strech
59263a7 Merge pull request #141 from weaveworks/update-context
e235c9b Merge pull request #143 from weaveworks/gc-wks-test-vms
c865b4c scheduler: please lint/flake8
da61568 scheduler: please lint/yapf
ce9d78e scheduler: do not cache discovery doc
e4b7873 scheduler: add comment about GCP projects' IAM roles needed to list/delete instances and firewall rules
ff7ec8e scheduler: add comment about CircleCI projects' access via the API
2477d98 scheduler: deploy command now sets the current datetime as the version
5fcd880 scheduler: pass CircleCI API token in for private projects
6b8c323 scheduler: more details in case of failure to get running builds from CircleCI
0871aff scheduler: downgrade google-api-python-client from 1.7.4 to 1.6.7
b631e7f scheduler: add GC of WKS test VMs and firewall rules
a923a32 scheduler: document setup and deployment
013f508 scheduler: lock dependencies' versions
6965a4a Merge pull request #142 from weaveworks/fix-build
23298c6 Fix golint expects import golang.org/x/lint/golint
482f4cd Context is now part of the Go standard library
2bbc9a0 Merge pull request #140 from weaveworks/sched-http-retry
c3726de Add retries to sched util http calls
2cc7b5a Merge pull request #139 from meghalidhoble/master
fd9b0a7 Change : Modified the lint tools to skip the shfmt check if not installed. Why the change : For ppc64le the specific version of shfmt is not available, hence skipped completely the installation of shfmt tool. Thus this change made.
bc645c7 Merge pull request #138 from dholbach/add-license-file
a642e02 license: add Apache 2.0 license text
9bf5956 Merge pull request #109 from hallum/master
d971d82 Merge pull request #134 from weaveworks/2018-07-03-gcloud-regepx
32e7aa2 Merge pull request #137 from weaveworks/gcp-fw-allow-kube-apiserver
bbb6735 Allow CI to access k8s API server on GCP instances
764d46c Merge pull request #135 from weaveworks/2018-07-04-docker-ansible-playbook
ecc2a4e Merge pull request #136 from weaveworks/2018-07-05-gcp-private-ips
209b7fb tools: Add private_ips to the terraform output
369a655 tools: Add an ansible playbook that just installs docker
a643e27 tools: Use --filter instead of --regexp with gcloud
b8eca88 Merge pull request #128 from weaveworks/actually-say-whats-wrong
379ce2b Merge pull request #133 from weaveworks/fix-decrypt
3b906b5 Fix incompatibility with recent versions of OpenSSL
f091ab4 Merge pull request #132 from weaveworks/add-opencontainers-labels-to-dockerfiles
248def1 Inject git revision in Dockerfiles
64f2c28 Add org.opencontainers.image.* labels to Dockerfiles
ea96d8e add information about how to get help (#129)
f066ccd Make yapf diff failure look like an error
34d81d7 Merge pull request #127 from weaveworks/golang-1.10.0-stretch
89a0b4f Use golang:1.10.0-stretch image.
ca69607 Merge pull request #126 from weaveworks/disable-apt-daily-test
f5dc5d5 Create "setup-apt" role
7fab441 Rename bazel to bazel-rules (#125)
ccc8316 Revert "Gocyclo should return error code if issues detected" (#124)
1fe184f Bazel rules for building gogo protobufs (#123)
b917bb8 Merge pull request #122 from weaveworks/fix-scope-gc
c029ce0 Add regex to match scope VMs
0d4824b Merge pull request #121 from weaveworks/provisioning-readme-terraform
5a82d64 Move terraform instructions to tf section
d285d78 Merge pull request #120 from weaveworks/gocyclo-return-value
76b94a4 Do not spawn subshell when reading cyclo output
93b3c0d Use golang:1.9.2-stretch image
d40728f Gocyclo should return error code if issues detected
c4ac1c3 Merge pull request #114 from weaveworks/tune-spell-check
8980656 Only check files
12ebc73 Don't spell-check pki files
578904a Special-case spell-check the same way we do code checks
e772ed5 Special-case on mime type and extension using just patterns
ae82b50 Merge pull request #117 from weaveworks/test-verbose
8943473 Propagate verbose flag to 'go test'.
7c79b43 Merge pull request #113 from weaveworks/update-shfmt-instructions
258ef01 Merge pull request #115 from weaveworks/extra-linting
e690202 Use tools in built image to lint itself
126eb56 Add shellcheck to bring linting in line with scope
63ad68f Don't run lint on files under .git
51d908a Update shfmt instructions
e91cb0d Merge pull request #112 from weaveworks/add-python-lint-tools
0c87554 Add yapf and flake8 to golang build image
35679ee Merge pull request #110 from weaveworks/parallel-push-errors
3ae41b6 Remove unneeded if block
51ff31a Exit on first error
0faad9f Check for errors when pushing images in parallel
d87cd02 Add arg flag override for destination socks host:port in pacfile.
74dc626 Merge pull request #108 from weaveworks/disable-apt-daily
b4f1d91 Merge pull request #107 from weaveworks/docker-17-update
7436aa1 Override apt daily job to not run immediately on boot
7980f15 Merge pull request #106 from weaveworks/document-docker-install-role
f741e53 Bump to Docker 17.06 from CE repo
61796a1 Update Docker CE Debian repo details
0d86f5e Allow for Docker package to be named docker-ce
065c68d Document selection of Docker installation role.
3809053 Just --porcelain; it defaults to v1
11400ea Merge pull request #105 from weaveworks/remove-weaveplugin-remnants
b8b4d64 remove weaveplugin remnants
35099c9 Merge pull request #104 from weaveworks/pull-docker-py
cdd48fc Pull docker-py to speed tests/builds up.
e1c6c24 Merge pull request #103 from weaveworks/test-build-tags
d5d71e0 Add -tags option so callers can pass in build tags
8949b2b Merge pull request #98 from weaveworks/git-status-tag
ac30687 Merge pull request #100 from weaveworks/python_linting
4b125b5 Pin yapf & flake8 versions
7efb485 Lint python linting function
444755b Swap diff direction to reflect changes required
c5b2434 Install flake8 & yapf
5600eac Lint python in build-tools repo
0b02ca9 Add python linting
c011c0d Merge pull request #79 from kinvolk/schu/python-shebang
6577d07 Merge pull request #99 from weaveworks/shfmt-version
00ce0dc Use git status instead of diff to add 'WIP' tag
411fd13 Use shfmt v1.3.0 instead of latest from master.
0d6d4da Run shfmt 1.3 on the code.
5cdba32 Add sudo
c322ca8 circle.yml: Install shfmt binary.
e59c225 Install shfmt 1.3 binary.
30706e6 Install pyhcl in the build container.
960d222 Merge pull request #97 from kinvolk/alban/update-shfmt-3
1d535c7 shellcheck: fix escaping issue
5542498 Merge pull request #96 from kinvolk/alban/update-shfmt-2
32f7cc5 shfmt: fix coding style
09f72af lint: print the diff in case of error
571c7d7 Merge pull request #95 from kinvolk/alban/update-shfmt
bead6ed Update for latest shfmt
b08dc4d Update for latest shfmt (#94)
2ed8aaa Add no-race argument to test script (#92)
80dd78e Merge pull request #91 from weaveworks/upgrade-go-1.8.1
08dcd0d Please ./lint as shfmt changed its rules between 1.0.0 and 1.3.0.
a8bc9ab Upgrade default Go version to 1.8.1.
31d069d Change Python shebang to `#!/usr/bin/env python`

git-subtree-dir: tools
git-subtree-split: d6cc704a2892e8d85aa8fa4d201c1a404f02dfa4

* use https if provided as scheme

* Allow region to be read from environment variable

* Fixed aws.ConfigFromURL() when URL contains @ but no user/pass

Signed-off-by: Marco Pracucci <[email protected]>

* Add provenance and license comments to files migrated from weaveworks/common, update package names

* Update imports to match new package paths.

* Remove exec, fs, test/fs and test/exec packages not used by Loki.

* Add changelog entry.

---------

Signed-off-by: Tom Wilkie <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Co-authored-by: Tom Wilkie <[email protected]>
Co-authored-by: Tom Wilkie <[email protected]>
Co-authored-by: Paul Bellamy <[email protected]>
Co-authored-by: Simon <[email protected]>
Co-authored-by: David <[email protected]>
Co-authored-by: Alfonso Acosta <[email protected]>
Co-authored-by: Ilya Dmitrichenko <[email protected]>
Co-authored-by: Adam Harrison <[email protected]>
Co-authored-by: Jonathan Lange <[email protected]>
Co-authored-by: Alfonso Acosta <[email protected]>
Co-authored-by: Mike Lang <[email protected]>
Co-authored-by: Matthias Radestock <[email protected]>
Co-authored-by: Krzesimir Nowak <[email protected]>
Co-authored-by: lukemarsden <[email protected]>
Co-authored-by: Lorenzo Manacorda <[email protected]>
Co-authored-by: Bowen Li <[email protected]>
Co-authored-by: Jordan Pellizzari <[email protected]>
Co-authored-by: CarltonSemple <[email protected]>
Co-authored-by: Alban Crequy <[email protected]>
Co-authored-by: Filip Barl <[email protected]>
Co-authored-by: Bryan Boreham <[email protected]>
Co-authored-by: Bryan Boreham <[email protected]>
Co-authored-by: Thor <[email protected]>
Co-authored-by: Aditya C S <[email protected]>
Co-authored-by: Marco Pracucci <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants