Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eliminate deadlocks in Subscribe and AddListener #4

Conversation

ajwerner
Copy link

@ajwerner ajwerner commented Jan 8, 2019

The existing code provides two mechanisms for clients to receive events from
a circuit breaker. Both contained logic which could potentially lead to
deadlocks. The logic assumed that clients prefer to receive more recent
events in the case where their channel is full and thus attempts to pull
events off the front of the client channel in the case where sends would block.
This however can lead to deadlocking when another goroutine is trying to send
an event on the channel (which can totally happen). This commit fixes the
possible deadlocks, adds testing, and clarifies in the comments that the event
mechanisms are best effort. This course of action was taken as opposed to
removing the functionality so as to not break existing external clients who may
be relying on it.

Addresses #47 from rubyist/circuitbreaker.


This change is Reviewable

The existing code provides two mechanisms for clients to receive events from
a circuit breaker. Both contained logic which could potentially lead to
deadlocks. The logic assumed that clients prefer to receive more recent
events in the case where their channel is full and thus attempts to pull
events off the front of the client channel in the case where sends would block.
This however can lead to deadlocking when another goroutine is trying to send
an event on the channel (which can totally happen). This commit fixes the
possible deadlocks, adds testing, and clarifies in the comments that the event
mechanisms are best effort. This course of action was taken as opposed to
removing the functionality so as to not break existing external clients who may
be relying on it.
@ajwerner ajwerner changed the title eliminate deadlocks from Subscribe and AddListener eliminate deadlocks in Subscribe and AddListener Jan 8, 2019
Copy link

@a-robinson a-robinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: assuming you've vetted our uses of subscriptions/listeners to confirm that we're ok with dropped events

Reviewed 2 of 2 files at r1.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @ajwerner)


circuitbreaker_test.go, line 562 at r1 (raw file):

		}
	}
	tripAndReset := func() {

I'm assuming this sometimes cause the test to timeout before your fix, right?

@ajwerner
Copy link
Author

ajwerner commented Jan 8, 2019

TFTR!

I'm assuming this sometimes cause the test to timeout before your fix, right?
Correct, it was pretty reliable.

assuming you've vetted our uses of subscriptions/listeners to confirm that we're ok with dropped events
We never actually use either one and they were dropping events before. Just saw this and thought I'd be nice to the next passerby who blindly assumes code on the internet is going to work.

@ajwerner ajwerner merged commit 0e362d8 into cockroachdb:master Jan 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants