Is it recommended to split Jobs onto its own server as recommended in the Meteor Guide?

[jamauro]

Curious if you recommend splitting Jobs on to its own server as mentioned in the Meteor Guide here: https://cloud-guide.meteor.com/background-jobs.html#split

If so, do you have any additional tips?

In my use case, I have several async jobs that are run per userId. I don't think any of the jobs would be considered "long-running" so I'm guessing it should be fine. This is mainly curiosity on my part and having an idea of what the options are should I need to reduce the impact of background jobs.

Thanks!

[wildhart]

Hi @jamauro,

Thanks for your continued interest in this package.

The guide uses long-running jobs as an example something which should ideally be run on a separate server. In my experience - at least for all the 30-odd different jobs I have in my applications - most jobs are fairly quick and each run several async operations (either using Meteor fibers or await), looking something up from a database, sending a notification or email, saving something in the database, etc. This gives plenty of opportunity for each job to defer back to the main thread so that user latency isn't affected. The actual time the job code is hogging the CPU is almost negligible.

Unless you have a very heavy synchronous number-crunching job, or thousands of small jobs which are running continuously, then the effect on the user will be minimal.

In this package currently there isn't the ability to force jobs to run only on a particular server.

If you don't have resource-intensive jobs, then I would NOT recommend making them run on only one specific server. One of the main features of most job schedulers is that if one of your servers goes down, one of the other servers takes control of the job queue. If only one server was configured to run jobs then you wouldn't have that redundancy.

However, if any user of this package does have resource intensive jobs, and would like to run jobs only on a specific server then that is a feature I could add relatively easily...

[jamauro]

Thanks for the great reply! Makes sense.

thousands of small jobs which are running continuously

I wouldn’t be near this for some time but if I get lucky and have enough users, it’s possible. Curious, if you have any ideas on how to approach in this scenario? Maybe a separate Meteor app entirely dedicated for Jobs or is there a better way?

resource-intensive jobs

Also, out of curiosity, what would you consider to be resource-intensive? Is there a certain time threshold or some other measurement to use as a guide?

Appreciate your thoughts.

[wildhart]

'resource-intensive' is a matter of opinion I guess. My criteria is anything which would hog the CPU for long enough for users to notice increased latency. With Meteor methods, if you have the method on the server AND the client then you have Optimistic UI, so latency is even less of a problem anyway.

Like I said, most normal jobs are very quick and perform mostly async operations which let the thread handle incoming requests while they wait for a database or API response.

Things which hog the CPU could be heavy number crunching, zipping files, maybe generating large PDFs. Although I regularly generate single-page invoice PDFs in one of my jobs.

Another way of looking at it: if the jobs are performing the same actions as your users, then their load on the system will be just the same as another user. E.g. my server can generate invoices and receipt PDFs in response to user requests, and this is a quick and mostly async operation, so doing the same thing in a job when it's time to email them a reminder invoice or automatic payment receipt is no different.

Curious, if you have any ideas on how to approach in this scenario?

Like the article said, if you did have CPU-heavy jobs then it would be best to run them on a separate server and run the jobs only on that server. Another solution would be to create these jobs as a Lamda function (or similar), so you can run them via an API, but the point of the article you reference in your OP was that by keeping these jobs in your meteor app code you don't have to re-engineer your code, db access, etc, for a separate system.

To run them in your meteor code but on a specific server, I would have to add that feature to this package.

[jamauro]

Thanks for the helpful reply! That all makes sense. Sounds like I'll just need to monitor how things go.