Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Free redis key after completion instead of using a TTL #8

Open
WhyNotHugo opened this issue May 8, 2017 · 2 comments
Open

Free redis key after completion instead of using a TTL #8

WhyNotHugo opened this issue May 8, 2017 · 2 comments

Comments

@WhyNotHugo
Copy link
Contributor

Using an approximate TTL can have issues; for example, a task may have been delayed, so it's still in queue but expired from redis. This would allow enqueueing it twice.

I'd like to propose adding keys to the store, and have the after_return and on_failure remove them from the store.

This makes sure we never have queued tasks that are missing from the redis backend uniqueness store. It also simplifies code quite a bit.

If having stale entries is a strong concern, acks_late can be enforced for unique tasks. However, stale entries should not be an issue, because we only cancel the old entry, so this flow would mean that we just cancel-or-noop the old entry.

@TylerHendrickson
Copy link
Contributor

@hobarrera I like this proposal! Thinking about it, I actually don't think stale entries in redis is a concern; if it ever happened, the stale entry would eventually be replaced by a new task.

@WhyNotHugo
Copy link
Contributor Author

the stale entry would eventually be replaced by a new task.

Yup, we both completely missed that before. 😛

I'll move onto this as soon as I finish the Mixin/Backend refactor (which makes room for non-redis backends).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants