Adjust to new redlock behaviour

[nduitz]

There have been some changes to the behaviour of redis-client and redlock-rb which lead to a different behaviour in this gem.

Before leandromoreira/redlock-rb#110 was merged, redlock would return false if there was an error with the connection, now it raises the error.
Before this behaviour could have been handled by configuring on_conflict. Now the behaviour should be more fine grained.

This adds a on_error option, which can be either nil (leading to a reraise) or a proc leading to the evaluation of said proc.

This allows users to handle connections errors to reduce the amount of jobs that get lost because of said connection errors.

[sharshenov]

Hey @nduitz
Thank you for the PR. I'm happy to accept it. But at the same time it would be nice to preserve compatibility with older versions of Ruby.

[sharshenov]

This syntax is not available in Rubies pre-3.1

[nduitz]

thanks for the fast feedback :) will change.

You've been fast, in the meantime i've changed the naming a bit and added support for a global config

[nduitz]

I've been running this PR in production for some days now.
I added some logging around that callback, but it appears to be silent. So this might not be an issue at all. Even though this PR might still be a feature worth adding.

Most of the errors that we encounter come from delete_lock. So I will try and investigate that if I find the time, maybe there is something we can do about that.
redlock's strategy is to simply fire and forget the unlock attempt, rescuing every error that might occur. But I don't know about that, since redlock is more about process based locking. Trying the same might lead to a deadlock situation.

[louiswdavis]

Think we've also been encountering this issue after doing an upgrade with a job with the unique config of; unique :until_executed, lock_ttl: 10.minutes, on_conflict: :log.

Is the gem going to include these changes in the near future?

[rmaspero]

I think we are experiencing a similar issue related to the redis-client and redlock-rb changes. @nduitz are you still running this branch?

@sharshenov are we able to merge or get an official fix?

[nduitz]

I think we are experiencing a similar issue related to the redis-client and redlock-rb changes. @nduitz are you still running this branch?

@sharshenov are we able to merge or get an official fix?

Hey, yes I am still running the branch.
But as I said before, this PR doesn't change anything on the situation. We still have loads of RedisClient::ConnectionError which are in fact triggered by delete_lock.

Since we never experienced any real issues I suspect this arrises through a raise condition and the lock already being deleted but I did not investigate it further