-
Notifications
You must be signed in to change notification settings - Fork 328
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
router: Make the number of router threads configurable #1385
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, about the usage in the commit example.
- Use "listeners" instead in the example.
{
"listen": {
...
},
"settings": {
"router": {
"threads": 2
}
},
...
}
- I feel developers who are familiar with unit code can understand the "router" object in the "settings" well since threads run in the router process. But I'm not sure if it's good enough to introduce the new "router" option in the configuration. For users, can they understand it well? Thus we have two usages to choose:
a. the current way.
b. just use "listen_threads"
{
"settings": {
"listen_threads": 3
}
}
I'd like to ask the team to suggest.
src/nxt_conf_validation.c
Outdated
static nxt_conf_vldt_object_t nxt_conf_vldt_router_members[] = { | ||
{ | ||
.name = nxt_string("threads"), | ||
.type = NXT_CONF_VLDT_INTEGER, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems we need to add more checks to its value. It's similar to "process" with less check.
static nxt_int_t
nxt_conf_vldt_processes(nxt_conf_validation_t *vldt, nxt_conf_value_t *value,
void *data)
{
int64_t int_value;
nxt_int_t ret;
nxt_conf_vldt_processes_conf_t proc;
if (nxt_conf_type(value) == NXT_CONF_INTEGER) {
int_value = nxt_conf_get_number(value);
if (int_value < 1) {
return nxt_conf_vldt_error(vldt, "The \"processes\" number must be "
"equal to or greater than 1.");
}
if (int_value > NXT_INT32_T_MAX) {
return nxt_conf_vldt_error(vldt, "The \"processes\" number must "
"not exceed %d.", NXT_INT32_T_MAX);
}
return NXT_OK;
}
...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suspect I can just re-use one of the "existing" "threads" check...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, this one should do the trick...
static nxt_int_t
nxt_conf_vldt_threads(nxt_conf_validation_t *vldt, nxt_conf_value_t *value,
void *data)
{
int64_t threads;
threads = nxt_conf_get_number(value);
if (threads < 1) {
return nxt_conf_vldt_error(vldt, "The \"threads\" number must be "
"equal to or greater than 1.");
}
if (threads > NXT_INT32_T_MAX) {
return nxt_conf_vldt_error(vldt, "The \"threads\" number must "
"not exceed %d.", NXT_INT32_T_MAX);
}
return NXT_OK;
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unless of course we change the name... shame to basically duplicate that function...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could leave it until the team are happy with the usage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the team's efforts. I am a developer from China. There are 700M people using nginx products here. if ok I will promote it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, the more the merrier!
It would be great to have tests on it. Is it good to put in |
Well, if it's documented...
I don't mind adding it as a top-level "settings" setting. But maybe there will be other 'router' related things that want configuring. It seems like we will need to allow the listen(2) backlog to be configurable. Would that perhaps go under a 'router' settings? Seeing as it's the router process/threads that do the listening... |
Heh, maybe! |
It depends on if it’s a global option, I feel backlog is bound to specific listener. Let’s discuss it with the team. |
This comment was marked as off-topic.
This comment was marked as off-topic.
1 similar comment
Thank you for the team's efforts. I am a developer from China. There are 700M people using nginx products here. I will promote it. |
Hi @ac000,
I found nginx has a similar feature. |
Yes, the listen backlog is per listen(2) call. However I'm not sure why you'd want it to be different in each of the threads... |
I only mean it belongs to listen and should be the same between threads. |
Exactly! Which is why I suggested to put it under /settings/router... |
Currently we try to do "auto" by default, but there are cases where it's wrong. With #1383 there are more cases where it's correct. But there are still cases where it could be wrong which is when you'd use this setting. IOW
That is essentially how it is now. But can be wrong in certain cases. Or did I misunderstand? |
There might be something confusing between the issues, and let me show my thoughts again.
It's independent of the feature of router threads.
Anyway, it's inside the "settings" object. Let me know if I missed something. |
It seems the default value from the available CPU numbers makes more sense than the default of the total CPU numbers. The other solution is to keep the default value as it is and introduce a new feature that is configurable.
The advantage is users can change available CPU numbers anytime but not compile Unit again. |
OK, yes, that does make sense...
Correct. |
You've lost me. All this patch does is allow you to set the number of router threads that are created.
Which is exactly what this patch does!
Not sure I understand. Are you referring to #1383 ? Sure, with this patch we technically wouldn't need it, but why not try to do better by default? So I still want it... we can do similar with the BSDs I think also...
That's how it is now! We try to determine how many threads to create via
Yes, that's exactly what this patch allows!
Yes, that could be done as future work, but is a separate issue from this patch... |
I’m fine with both the ways. I mean set the default CPU numbers from total CPUs or available CPUs. |
If by "total CPUs" you mean what is returned by It doesn't take into account CPU allowed masks and so can give drastically wrong answers... |
Add missing threads validation check
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The implementation looks good to me, the left thing is about the usage to be made with the team.
So to re-cap, we need to decide two things
Well, that's my understanding anyway... |
"listeners" inside "listeners_threads" means the created threads are working for "listeners". And the configuration is applied to the router process. |
What I meant was that I think "listeners_threads" (which is what it was originally called) is more accurate than "listener_thread", if we went down that naming route.
It is if we do it under
The main concern I have about putting the setting directly under |
I think y'all were talking past each other earlier on. I believe there's agreement that:
The remaining question is what we name the option and where we put it in the hierarchy. And honestly, we shouldn't care too much. If what we choose ends up feeling wrong down the line, we can rename and alias it in a future release. The current patch LGTM. |
(all of the other naming proposals also LGTM. just pick one and run with it) |
@hongzhidao What's your preference for the name and location? |
I prefer the top location but can't find a good name. I'd like just "threads". |
OK, I'll put it under However just "threads" is a little ambiguous there, I'm just gonna call it "listen_threads"... |
|
Unit generally creates an extra number of router threads (to handle client connections, not incl the main thread) to match the number of available CPUs. There are cases when this can go wrong, e.g on a high CPU count machine and Unit is being effectively limited to a few CPUs via the cgroups cpu controller. So Unit may create a large number of router threads when they are only going to effectively run on a couple of CPUs or so. There may be other cases where you would like to tweak the number of router threads, depending on your workload. As it turns out it looks like it was intended to be made configurable but was just never hooked up to the config system. This adds a new '/settings/listen_threads' config option which can be set like { "listen": { ... }, "settings": { "listen_threads": 2, ... }, ... } Before this patch (on a four cpu system) $ ps -efL | grep router andrew 419832 419829 419832 0 5 Aug12 pts/10 00:00:00 unit: router andrew 419832 419829 419833 0 5 Aug12 pts/10 00:00:00 unit: router andrew 419832 419829 419834 0 5 Aug12 pts/10 00:00:00 unit: router andrew 419832 419829 445145 0 5 03:31 pts/10 00:00:00 unit: router andrew 419832 419829 445146 0 5 03:31 pts/10 00:00:00 unit: router After, with a threads setting of 2 $ ps -efL | grep router andrew 419832 419829 419832 0 3 Aug12 pts/10 00:00:00 unit: router andrew 419832 419829 419833 0 3 Aug12 pts/10 00:00:00 unit: router andrew 419832 419829 419834 0 3 Aug12 pts/10 00:00:00 unit: router Closes: nginx#1042 Signed-off-by: Andrew Clayton <[email protected]>
Rebased wity master |
Unit generally creates an extra number of router threads (to handle client connections, not incl the main thread) to match the number of available CPUs.
There are cases when this can go wrong, e.g on a high CPU count machine and Unit is being effectively limited to a few CPUs via the cgroups cpu controller. So Unit may create a large number of router threads when they are only going to effectively run on a couple of CPUs or so.
There may be other cases where you would like to tweak the number of router threads, depending on your workload.
As it turns out it looks like it was intended to be made configurable but was just never hooked up to the config system.
This adds a new '/settings/router/threads' config option which can be set like
Before this patch (on a four cpu system)
After, with a threads setting of 2.