Task Queue Webhook configured at task level

[tacman]

We have several projects sharing a single meilisearch instance. I'd like the option to create the webhook url when the task is created, so that I can assign different tasks to https://project-1.example.com/handle-meili-task, etc.

Otherwise, I have to write a middle layer that distributes the webhook events to their individual projects.

Thanks.

[curquiza]

Hello @tacman
We are working on the v1.6.0 release, and the third release candidate (i.e. rc3) include the webhook URL option. You can test it.
The official release of v1.6.0 is on 15/01/2024

[tacman]

It looks like the webhook URL option is at the server level, not at the task level. My issue is that I need to customize the batch processing callback, since there are different projects using the same server, so having a callback url at the server level is too general.

…

On Tue, Jan 2, 2024 at 2:13 AM Clémentine U. - curqui < ***@***.***> wrote: Hello @tacman <https://github.com/tacman> We are working on the v1.6.0 release, and the third release candidate (i.e. rc3) include the webhook URL option. You can test it <https://github.com/meilisearch/meilisearch/releases>. The official release of v1.6.0 is on 15/01/2024 — Reply to this email directly, view it on GitHub <#715 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEXIQL3LQCMC3PDXHZQ733YMO6TNAVCNFSM6AAAAABBI7735CVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TSOJRGIYDS> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[curquiza]

Ok! Got it! Thank you for your feedback

[tacman]

I'm pretty excited about this new feature. Currently I keep retrying a queued message if isIndexing is set in stats, and when it's not, I run the final step, which is to create a summary and store it in the database. It feels fragile and hackish, when of course what I want is a callback event, as you've proposed.

The other thing to consider is that by making it an option at the server level, it requires the devops team to get involved. I don't want to say "install 1.6 and configure it with the webhook url as xxx", and then a week later say "oh, can you restart the server with this new webhook url"?

I just want to say "install 1.6" and then everything else I can do within PHP. Like other small companies, we outsource the devops and server management, so I'm hoping you'll simply include the webhook and make the entire configuration possible at the API level.

Thanks.

[macraig]

Hi @tacman , thank you for all your feedback. For context, we initially implemented this feature for our own internal needs and hence the setting at the server level. Your use case, however, makes a lot of sense, and we'll be discussing the implementation of the webhook at the task level in our next planning meeting. We'll make sure to keep you in the loop, thanks again for your valuable feedback!

[macraig]

Adding feedback from the notion page below to preserve one source of truth:

Please add a way to configure the webhook url when the task is created. Several projects share the meilisearch server (indexes are prefixed with a project code), and each project needs to have its own listener for when the task is finished.

[tacman]

Hi, it's been a while, but wanted to ping you about a configurable webhook.

I would LOVE to know when a task is finished, without polling. Indeed, the webhook does exactly that.

All of my projects share a single meilisearch instance. I don't want to create a separate project simply to listen for the webhook and distribute it by dispatching another message (via a webhook or queue or whatever).

Sometimes I don't even need the webhook. But when I do, usually large indexes, it's be enormously helpful.

Basically, I'd like to send a webhook URL to any operation that returns a task id. That would eliminate the polling on that task, and provide a way to get errors more easily, since the errors don't happen when queuing.

Thanks for your consideration. It will be MUCH more widely used when it's configurable at the task level. And programmers are used to this pattern, as almost every email service now dispatches a webhook. Plus github, etc.

[macraig]

Hi @tacman , thanks once again for your feedback. We haven't forgotten about this feature request, it's still on our radar and we want to implement it, but we've been occupied with other priorities. We will let you know as soon as we begin active development.

[tacman]

Thanks! I've been thinking about a meili admin server to marshall and distribute the task hooks on a server basis. It's a neat idea if I had extra time, but better will be when task-level web hooks are implemented. Thanks for keeping it on the radar.