Do not unregister the callback in the zk watcher

[jolynch]

Addresses incompatibility of per_callback threading in ZK, the way that
synapse calls unsubscribe within the callback, and ruby 2.X really not
liking joining on the current running thread.

Should fix #137 #99 and possibly #94

I've tested this manually and events seem to get propagated ... I'd like to roll this out to a small portion of our infrastructure and see how it behaves.

[jolynch]

also #134

[jolynch]

For some more context, we do have to call get/stat/exists with :watch => true to reset the watch, but we don't have to unsubscribe and re-subscribe. The zk-ruby documentation is a bit vague.

[jolynch]

I did some testing where an outside actor removes the path that is registered, and we actually didn't properly exit because the callback lives in a thread (which swallows the exception). A simple zk_cleanup suffices.

D, [2015-09-03T21:48:12.580705 #25732] DEBUG -- Synapse::ZookeeperWatcher: synapse: setting watch at /services/service1_backup
E, [2015-09-03T21:48:12.581436 #25732] ERROR -- Synapse::ZookeeperWatcher: synapse: zookeeper watcher path /services/service1_backup does not exist!
I, [2015-09-03T21:48:12.581468 #25732]  INFO -- Synapse::ZookeeperWatcher: synapse: zookeeper watcher cleaning up
E, [2015-09-03T21:48:13.497742 #25732] ERROR -- Synapse::Synapse: synapse: encountered unexpected exception #<RuntimeError: synapse: service watcher service1_backup failed ping!> in main thread
W, [2015-09-03T21:48:13.497799 #25732]  WARN -- Synapse::Synapse: synapse: exiting; sending stop signal to all watchers
W, [2015-09-03T21:48:13.497816 #25732]  WARN -- Synapse::ZookeeperWatcher: synapse: zookeeper watcher exiting
I, [2015-09-03T21:48:13.497826 #25732]  INFO -- Synapse::ZookeeperWatcher: synapse: zookeeper watcher cleaning up
I, [2015-09-03T21:48:13.497916 #25732]  INFO -- Synapse::ZookeeperWatcher: synapse: zookeeper watcher cleaned up successfully
W, [2015-09-03T21:48:13.497941 #25732]  WARN -- Synapse::ZookeeperWatcher: synapse: zookeeper watcher exiting
I, [2015-09-03T21:48:13.497952 #25732]  INFO -- Synapse::ZookeeperWatcher: synapse: zookeeper watcher cleaning up
I, [2015-09-03T21:48:13.497967 #25732]  INFO -- Synapse::ZookeeperWatcher: synapse: closing zk connection to localhost:2181
I, [2015-09-03T21:48:13.497977 #25732]  INFO -- Synapse::ZookeeperWatcher: synapse: zookeeper watcher cleaned up successfully
W, [2015-09-03T21:48:13.497985 #25732]  WARN -- Synapse::ZookeeperWatcher: synapse: zookeeper watcher exiting
I, [2015-09-03T21:48:13.497992 #25732]  INFO -- Synapse::ZookeeperWatcher: synapse: zookeeper watcher cleaning up
I, [2015-09-03T21:48:13.498060 #25732]  INFO -- Synapse::ZookeeperWatcher: synapse: zookeeper watcher cleaned up successfully
/home/ereinion/Hackery/ruby/synapse-airbnb/lib/synapse.rb:54:in `block (2 levels) in run': synapse: service watcher service1_backup failed ping! (RuntimeError)
        from /home/ereinion/Hackery/ruby/synapse-airbnb/lib/synapse.rb:53:in `each'
        from /home/ereinion/Hackery/ruby/synapse-airbnb/lib/synapse.rb:53:in `block in run'
        from /home/ereinion/Hackery/ruby/synapse-airbnb/lib/synapse.rb:52:in `loop'
        from /home/ereinion/Hackery/ruby/synapse-airbnb/lib/synapse.rb:52:in `run'
        from bin/synapse:60:in `<main>'

[jolynch]

@igor47 I'm going to start testing master in our testing environments this week, should be on production by the end of the week.

I'll ask you to cut a new version once I figure out if it all works?

[igor47]

sgtm, i’m going to start testing this version in our infra as well

On Mon, Sep 7, 2015 at 1:10 PM, Joseph Lynch [email protected]
wrote:

@igor47 https://github.com/igor47 I'm going to start testing master in
our testing environments this week, should be on production by the end of
the week.

I'll ask you to cut a new version once I figure out if it all works?

—
Reply to this email directly or view it on GitHub
#138 (comment).

[jolynch]

@igor47 Things seem to be working well on production for us. Mind cutting a release?