Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "if" option to the "match" object. #1204

Merged
merged 10 commits into from
Aug 20, 2024
Merged

Conversation

hongzhidao
Copy link
Contributor

@hongzhidao hongzhidao commented Mar 28, 2024

Two things:

  1. restrict variables to only support synchronous way.
    Initially, variable query was designed to accomodate both synchronous and asynchronous operations. However, upon consideration of actual requirements, we recognized that asynchronous support was not needed.
    This refactoring is intended to simplify the usage of variable queries and prepare for the subsequent feature of variable updating.

  2. Add if option to match object to make matching scripting. For example:

{
    "listeners": {
        "*:8080": {
            "pass": "routes"
        }
    },
    "routes": [
        {
            "match": {
                "if": "!`${headers['User-Agent'].split('/')[0] == 'curl'}`"
            },
            "action": {
                "return": 204
            }
        }
    ]
}

@ac000
Copy link
Member

ac000 commented Mar 28, 2024

Couple of quick questions.

What variables are we talking about here?

Is the afore mentioned discussion available anywhere?

@hongzhidao
Copy link
Contributor Author

What variables are we talking about here?

Variable-related APIs, correspond to nxt_tstr_{api}.

Is the afore mentioned discussion available anywhere?

I'm afraid no discussion was mentioned.

@ac000
Copy link
Member

ac000 commented Mar 28, 2024

│ What variables are we talking about here?

Variable-related APIs, correspond to nxt_tstr_{api}.

njs variables?

│ Is the afore mentioned discussion available anywhere?

I'm afraid no discussion was mentioned.

I thought the below might have been discussed somewhere...

upon consideration of actual requirements, we recognized that
asynchronous support was not needed.

@hongzhidao
Copy link
Contributor Author

njs variables?

It means Unit variables, for example, $host. This has nothing to do with njs here.

I thought the below might have been discussed somewhere...

We did consider supporting certain variables, such as $db_query_some, so that's why the API has callbacks ready and error handlers that are called after the async operations. But no such requirements are asked.

@ac000
Copy link
Member

ac000 commented Apr 1, 2024

It means Unit variables, for example, $host . This has nothing to do
with njs here.

Perhaps that should be made clear in the commit message then...

@hongzhidao
Copy link
Contributor Author

It means Unit variables, for example, $host . This has nothing to do
with njs here.

Perhaps that should be made clear in the commit message then...

Added, thanks.

@hongzhidao hongzhidao changed the title Make var query only support synchronous operation. Make variable query only support synchronous operation. Apr 17, 2024
@ac000
Copy link
Member

ac000 commented Apr 17, 2024

I wonder if

4d515d54 Var: Removed unused functions and structure fields

and

2cb6b1bf Var: Removed unused functions

should be combined?

@hongzhidao
Copy link
Contributor Author

I wonder if

4d515d54 Var: Removed unused functions and structure fields

and

2cb6b1bf Var: Removed unused functions

should be combined?

Good idea.

Comment on lines -1363 to +1402
nxt_int_t ret;
nxt_router_conf_t *rtcf;
nxt_http_action_t *action;
nxt_http_status_t status;
nxt_http_request_t *r;

r = obj;
action = data;
nxt_int_t ret;
nxt_router_conf_t *rtcf;
nxt_http_status_t status;

Copy link
Member

@ac000 ac000 Apr 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically shouldn't nxt_router_conf_t *rtcf; and nxt_http_status_t status; be swapped around as when types are the same length we order by type alphabetically (yes it's the same order as before), not that I really care about this...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, it makes sense to consider the order. But I'd like to keep it the same as I touched it based on the existing order.

@ac000 ac000 self-requested a review April 18, 2024 21:45
ac000
ac000 previously approved these changes Apr 18, 2024
Copy link
Member

@ac000 ac000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks fairly reasonable to me...

@ac000
Copy link
Member

ac000 commented May 9, 2024

HI @hongzhidao

The only thing you might want to do to be consistent with recent commits is to lowercase the Var's and HTTPs in the commit subjects...

@hongzhidao
Copy link
Contributor Author

HI @hongzhidao

The only thing you might want to do to be consistent with recent commits is to lowercase the Var's and HTTPs in the commit subjects...

Good reminder, thanks.
Btw, I'll wait until variable updating feature or demo is ready before submitting these patches.

@callahad
Copy link
Collaborator

Status: The team needs to discuss if we merge this now or wait until variable updating is done.

As I understand it:

  1. This PR is intended to ease the development of a new feature (updating Unit variables).
  2. That feature is not yet ready.
  3. We can't be certain that this refactor actually achieves its goals with its current design until that feature is developed.

Merging now avoids the risk of merge conflicts and burden of re-review, while waiting ensures that these patches are fully correct, necessary, and sufficient.

@callahad callahad added the X-Needs-Discussion The team needs to discuss this label Jun 18, 2024
@callahad
Copy link
Collaborator

callahad commented Jul 1, 2024

The team's decision is to defer to @hongzhidao :)

If you think we should merge it as-is, please do so.

Otherwise, please convert it back to a draft until it's ready for re-review and merging. There's a button at the top right under Reviewers:

Screenshot of the 'Conver to Draft' button

@callahad callahad removed the X-Needs-Discussion The team needs to discuss this label Jul 1, 2024
@hongzhidao hongzhidao marked this pull request as draft July 2, 2024 06:32
@ac000
Copy link
Member

ac000 commented Jul 10, 2024

Ah, while this is awaiting merging (or not), we should change the commit subjects to be in "imperative mood", e.g

var: Restrict nxt_tstr_query() to only support synchronous operation => no change (already imperative mood)
http: Refactored return action => http: Refactor return action
http: Refactored route pass query => http: Refactor route pass query
http: Refactored static action => http: Refactor static action
http: Refactored access log write => http: Refactor access log write
var: Removed unused functions and structure fields => var: Remove unused functions and structure fields
http: Added "if" option to match object => http: Add "if" option to match object

@hongzhidao
Copy link
Contributor Author

Ah, while this is awaiting merging (or not), we should change the commit subjects to be in "imperative mood", e.g

var: Restrict nxt_tstr_query() to only support synchronous operation => no change (already imperative mood) http: Refactored return action => http: Refactor return action http: Refactored route pass query => http: Refactor route pass query http: Refactored static action => http: Refactor static action http: Refactored access log write => http: Refactor access log write var: Removed unused functions and structure fields => var: Remove unused functions and structure fields http: Added "if" option to match object => http: Add "if" option to match object

Ok. And I'm going to merge the prepared patches after I test the "if" option, then I'll create a new PR for it.

@ac000
Copy link
Member

ac000 commented Jul 10, 2024

No need to make a new PR. You can un-draft this one!

See the Ready for review button? (just below the checks...)

I've had a look at the latest patch, I'm assuming the rest didn't change and it was just rebased against master... You can always re-request a persons review if things changed enough to warrant re-reviewing...

Copy link
Member

@ac000 ac000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This

 http: Refactor out nxt_tstr_cond_t from access log

Still suffers from the problem that nxt_http_request_access_log() was moved and changed in one step making the patch way bigger than it needs to be. If you change it without moving it, that'd be OK...

**EDIT: ** Well, it's not quite the same as before... I just don't understand why it's being done like this.

@ac000
Copy link
Member

ac000 commented Aug 15, 2024

I think the best thing would be to just put it back like before but make

Refactor out nxt_tstr_cond_t from access log.`

Look like I showed here.

I would also change the subject to just Factor out nxt_tstr_cond_t from access log as it's more than just a simple refactor...

@ac000
Copy link
Member

ac000 commented Aug 15, 2024

OK, looking at it again, I don't think it's so bad.

My problem now is the commit subject and message

http: Refactor out nxt_tstr_cond_t from access log

This nxt_tstr_cond_t will be reused for the feature of adding "if"
option to the "match" object. The two "if" options have the same usage.

Which doesn't really describe the change.

Not only does it factor code out of nxt_http_request_access_log(), but it adds a bunch of new code.

I think there should be a commit before this one that adds the nxt_tstr_cond_compile() function and nxt_tstr_cond_t structure.

Then I'm not sure where this change belongs yet...

diff --git ./src/nxt_router_access_log.c ./src/nxt_router_access_log.c
index cc8d5e4f..afecd0b1 100644
--- ./src/nxt_router_access_log.c
+++ ./src/nxt_router_access_log.c
@@ -143,15 +143,8 @@ nxt_router_access_log_create(nxt_task_t *task, nxt_router_conf_t *rtcf,
     if (alcf.expr != NULL) {
         nxt_conf_get_string(alcf.expr, &str);
 
-        if (str.length > 0 && str.start[0] == '!') {
-            rtcf->log_negate = 1;
-
-            str.start++;
-            str.length--;
-        }
-
-        rtcf->log_expr = nxt_tstr_compile(rtcf->tstr_state, &str, 0);
-        if (nxt_slow_path(rtcf->log_expr == NULL)) {
+        ret = nxt_tstr_cond_compile(rtcf->tstr_state, &str, &rtcf->log_cond);
+        if (nxt_slow_path(ret != NXT_OK)) {
             return NXT_ERROR;
         }
     }

Does it directly depend on or is depended on by the change to nxt_http_request_access_log() or could it be it's own commit?

@hongzhidao
Copy link
Contributor Author

hongzhidao commented Aug 15, 2024

http: Refactor out nxt_tstr_cond_t from access log

“access log” in the subject means “access log module” but not “nxt_http_request_access_log()”, the later is one function in the access log module.

In the commit, it touched the access log module but don’t need to mention specific details like which function “nxt_http_request_access_log()”.

Then in the next commit, we get rid of “nxt_http_request_access_log()”.

Does it directly depend on or is depended on by the change to nxt_http_request_access_log() or could it be it's own commit?

Belongs to "the access log module".

Btw, I change "access log" to "the access log module" in the subject. Hope it eliminates the confusion.

@ac000
Copy link
Member

ac000 commented Aug 16, 2024

I'm Looking at this again and by this, I'm talking about

    http: Refactor out nxt_tstr_cond_t from the access log module
    
    This nxt_tstr_cond_t will be reused for the feature of adding "if"
    option to the "match" object. The two "if" options have the same usage.

I'm struggling to reconcile the commit message with the change itself.

It really looks like to me this commit should be split into three

  1. Add the nxt_tstr_cond_t structure and nxt_tstr_cond_compile()
    function.

  2. Do the main change of extracting the code out of
    nxt_http_request_access_log() into nxt_http_cond_value()

    Do NOT remove the log_str & log_negate structure members from
    nxt_router_conf_t

  3. Do the change that updates nxt_router_access_log_create()

    Now you can remove the log_str & log_negate structure members.

Let me show you what I mean.

That commit now becomes the following in cronological order
(i.e git log --reverse)

(1)

    tstr: Add nxt_tstr_cond_t and nxt_tstr_cond_compile()

diff --git ./src/nxt_tstr.c ./src/nxt_tstr.c
index a6d2e7ad..50df4c47 100644
--- ./src/nxt_tstr.c
+++ ./src/nxt_tstr.c
@@ -196,6 +196,26 @@ nxt_tstr_state_release(nxt_tstr_state_t *state)
 }
 
 
+nxt_int_t
+nxt_tstr_cond_compile(nxt_tstr_state_t *state, nxt_str_t *str,
+    nxt_tstr_cond_t *cond)
+{
+    if (str->length > 0 && str->start[0] == '!') {
+        cond->negate = 1;
+
+        str->start++;
+        str->length--;
+    }
+
+    cond->expr = nxt_tstr_compile(state, str, 0);
+    if (nxt_slow_path(cond->expr == NULL)) {
+        return NXT_ERROR;
+    }
+
+    return NXT_OK;
+}
+
+
 nxt_bool_t
 nxt_tstr_is_const(nxt_tstr_t *tstr)
 {
diff --git ./src/nxt_tstr.h ./src/nxt_tstr.h
index 2aa905df..aca74e20 100644
--- ./src/nxt_tstr.h
+++ ./src/nxt_tstr.h
@@ -37,12 +37,20 @@ typedef enum {
 } nxt_tstr_flags_t;
 
 
+typedef struct {
+    nxt_tstr_t          *expr;
+    uint8_t             negate;  /* 1 bit */
+} nxt_tstr_cond_t;
+
+
 nxt_tstr_state_t *nxt_tstr_state_new(nxt_mp_t *mp, nxt_bool_t test);
 nxt_tstr_t *nxt_tstr_compile(nxt_tstr_state_t *state, const nxt_str_t *str,
     nxt_tstr_flags_t flags);
 nxt_int_t nxt_tstr_test(nxt_tstr_state_t *state, nxt_str_t *str, u_char *error);
 nxt_int_t nxt_tstr_state_done(nxt_tstr_state_t *state, u_char *error);
 void nxt_tstr_state_release(nxt_tstr_state_t *state);
+nxt_int_t nxt_tstr_cond_compile(nxt_tstr_state_t *state, nxt_str_t *str,
+    nxt_tstr_cond_t *cond);
 
 nxt_bool_t nxt_tstr_is_const(nxt_tstr_t *tstr);

(2)

    http: Factor out code from nxt_http_request_access_log()

diff --git ./src/nxt_http.h ./src/nxt_http.h
index fe5e72a8..5369c8e1 100644
--- ./src/nxt_http.h
+++ ./src/nxt_http.h
@@ -441,6 +441,9 @@ void nxt_h1p_complete_buffers(nxt_task_t *task, nxt_h1proto_t *h1p,
     nxt_bool_t all);
 nxt_msec_t nxt_h1p_conn_request_timer_value(nxt_conn_t *c, uintptr_t data);
 
+int nxt_http_cond_value(nxt_task_t *task, nxt_http_request_t *r,
+    nxt_tstr_cond_t *cond);
+
 extern const nxt_conn_state_t  nxt_h1p_idle_close_state;
 
 #endif  /* _NXT_HTTP_H_INCLUDED_ */
diff --git ./src/nxt_http_request.c ./src/nxt_http_request.c
index ccd2b141..a65163d0 100644
--- ./src/nxt_http_request.c
+++ ./src/nxt_http_request.c
@@ -915,44 +915,11 @@ static nxt_int_t
 nxt_http_request_access_log(nxt_task_t *task, nxt_http_request_t *r,
     nxt_router_conf_t *rtcf)
 {
-    nxt_int_t                ret;
-    nxt_str_t                str;
-    nxt_bool_t               expr;
     nxt_router_access_log_t  *access_log;
 
     access_log = rtcf->access_log;
 
-    expr = 1;
-
-    if (rtcf->log_expr != NULL) {
-
-        if (nxt_tstr_is_const(rtcf->log_expr)) {
-            nxt_tstr_str(rtcf->log_expr, &str);
-
-        } else {
-            ret = nxt_tstr_query_init(&r->tstr_query, rtcf->tstr_state,
-                                      &r->tstr_cache, r, r->mem_pool);
-            if (nxt_slow_path(ret != NXT_OK)) {
-                return NXT_DECLINED;
-            }
-
-            ret = nxt_tstr_query(task, r->tstr_query, rtcf->log_expr, &str);
-            if (nxt_slow_path(ret != NXT_OK)) {
-                return NXT_DECLINED;
-            }
-        }
-
-        if (str.length == 0
-            || nxt_str_eq(&str, "0", 1)
-            || nxt_str_eq(&str, "false", 5)
-            || nxt_str_eq(&str, "null", 4)
-            || nxt_str_eq(&str, "undefined", 9))
-        {
-            expr = 0;
-        }
-    }
-
-    if (rtcf->log_negate ^ expr) {
+    if (nxt_http_cond_value(task, r, &rtcf->log_cond)) {
         access_log->handler(task, r, access_log, rtcf->log_format);
         return NXT_OK;
     }
@@ -1364,3 +1331,48 @@ nxt_http_cookie_hash(nxt_mp_t *mp, nxt_str_t *name)
 {
     return nxt_http_field_hash(mp, name, 1, NXT_HTTP_URI_ENCODING_NONE);
 }
+
+
+int
+nxt_http_cond_value(nxt_task_t *task, nxt_http_request_t *r,
+    nxt_tstr_cond_t *cond)
+{
+    nxt_int_t          ret;
+    nxt_str_t          str;
+    nxt_bool_t         expr;
+    nxt_router_conf_t  *rtcf;
+
+    rtcf = r->conf->socket_conf->router_conf;
+
+    expr = 1;
+
+    if (cond->expr != NULL) {
+
+        if (nxt_tstr_is_const(cond->expr)) {
+            nxt_tstr_str(cond->expr, &str);
+
+        } else {
+            ret = nxt_tstr_query_init(&r->tstr_query, rtcf->tstr_state,
+                                      &r->tstr_cache, r, r->mem_pool);
+            if (nxt_slow_path(ret != NXT_OK)) {
+                return -1;
+            }
+
+            ret = nxt_tstr_query(task, r->tstr_query, cond->expr, &str);
+            if (nxt_slow_path(ret != NXT_OK)) {
+                return -1;
+            }
+        }
+
+        if (str.length == 0
+            || nxt_str_eq(&str, "0", 1)
+            || nxt_str_eq(&str, "false", 5)
+            || nxt_str_eq(&str, "null", 4)
+            || nxt_str_eq(&str, "undefined", 9))
+        {
+            expr = 0;
+        }
+    }
+
+    return cond->negate ^ expr;
+}
diff --git ./src/nxt_router.h ./src/nxt_router.h
index cfc7258c..60fc9557 100644
--- ./src/nxt_router.h
+++ ./src/nxt_router.h
@@ -56,6 +56,7 @@ typedef struct {
     nxt_tstr_t               *log_format;
     nxt_tstr_t               *log_expr;
     uint8_t                  log_negate;  /* 1 bit */
+    nxt_tstr_cond_t          log_cond;
 } nxt_router_conf_t;


(3)

    Convert nxt_router_access_log_create() to use nxt_tstr_cond_compile()

diff --git ./src/nxt_router.h ./src/nxt_router.h
index 60fc9557..06c6bb32 100644
--- ./src/nxt_router.h
+++ ./src/nxt_router.h
@@ -54,8 +54,6 @@ typedef struct {
 
     nxt_router_access_log_t  *access_log;
     nxt_tstr_t               *log_format;
-    nxt_tstr_t               *log_expr;
-    uint8_t                  log_negate;  /* 1 bit */
     nxt_tstr_cond_t          log_cond;
 } nxt_router_conf_t;
 
diff --git ./src/nxt_router_access_log.c ./src/nxt_router_access_log.c
index cc8d5e4f..afecd0b1 100644
--- ./src/nxt_router_access_log.c
+++ ./src/nxt_router_access_log.c
@@ -143,15 +143,8 @@ nxt_router_access_log_create(nxt_task_t *task, nxt_router_conf_t *rtcf,
     if (alcf.expr != NULL) {
         nxt_conf_get_string(alcf.expr, &str);
 
-        if (str.length > 0 && str.start[0] == '!') {
-            rtcf->log_negate = 1;
-
-            str.start++;
-            str.length--;
-        }
-
-        rtcf->log_expr = nxt_tstr_compile(rtcf->tstr_state, &str, 0);
-        if (nxt_slow_path(rtcf->log_expr == NULL)) {
+        ret = nxt_tstr_cond_compile(rtcf->tstr_state, &str, &rtcf->log_cond);
+        if (nxt_slow_path(ret != NXT_OK)) {
             return NXT_ERROR;
         }
     }

I think that is much clearer what is going on. The commits have become smaller,
more concise and really do just one thing.

I've pushed this up to https://github.com/ac000/unit/commits/var-sync-rework/
so you can see the thing as a whole. Feel free to take them if you want...

@hongzhidao
Copy link
Contributor Author

hongzhidao commented Aug 16, 2024

I've pushed this up to https://github.com/ac000/unit/commits/var-sync-rework/
so you can see the thing as a whole. Feel free to take them if you want...

Thanks a lot. It's a different way to think of it, I'd like to share my thoughts while I was implementing it.

  1. First, a few things around "tstr cond" should be introduced together, for example:
nxt_tstr_cond_t;
nxt_tstr_cond_compile(...);
nxt_tstr_cond_query(...);

They should be created at the same time, like "var_" stuff, "compile" and "query" are paired, one happens in configuration phase, and the other happens in the request runtime phase.
But I agree that it's not perfect that I used "nxt_http_cond_value()" instead of "nxt_tstr_cond_query(...)" since it has to depend on "nxt_http_request_t" object.

  1. Then I replaced related code with "tstr cond" in the access log module as much as possible.

  2. Since "nxt_http_request_access_log()" is confusing, I separated an individual patch to eliminate it.

Is this design idea clear for you?

@ac000
Copy link
Member

ac000 commented Aug 19, 2024

I've pushed this up to https://github.com/ac000/unit/commits/var-sync-rework/
so you can see the thing as a whole. Feel free to take them if you want...

Thanks a lot. It's a different way to think of it, I'd like to share my thoughts while I was implementing it.

I thik the difference is you're talking about committing the feature (i.e this commit) as a whole while, I'm talking about splitting it into logical components.

But I don't think we're going to reach equilibrium on this, so I'll say OK to this and move onto looking at the rest of the commits...

Comment on lines 1633 to 1635
if (ret <= 0) {
action = (nxt_http_action_t *) (intptr_t) ret;
return action;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor thing, feel free to ignore...

Do we need the action = step rather than just returning the thing directly (casted of course).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense.

Copy link
Member

@ac000 ac000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests: "if" option in http route match.

Should of course be written as

tests: "if" option in http route match

(The most important thing is to get rid of that trailing period!)

@hongzhidao hongzhidao force-pushed the var-sync branch 2 times, most recently from 6349386 to 121eba5 Compare August 19, 2024 14:16
@@ -1594,8 +1626,15 @@ nxt_http_route_match(nxt_task_t *task, nxt_http_request_t *r,
nxt_http_route_match_t *match)
{
nxt_int_t ret;
nxt_http_action_t *action;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to remove this now it's not needed!?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@ac000
Copy link
Member

ac000 commented Aug 19, 2024

OK, thanks I think this is good to go now...

Initially, variable query was designed to accomodate both synchronous
and asynchronous operations. However, upon consideration of actual
requirements, we recognized that asynchronous support was not needed.

The refactoring ensures that the success or failure of the variable
query operation is now directly indicated by its return value. This
change streamlines the function's usage and enhances code clarity,
as it facilitates immediate error handling without the need for
asynchronous callbacks or additional error checking functions.

Note the patch only works for Unit native variables but not njs
variables.
This nxt_tstr_cond_t will be reused for the feature of adding "if"
option to the "match" object. The two "if" options have the same usage.
This feature allows users to specify conditions to check if one
route is matched. It is used the same way as the "if" option in
the access log.

Example:

    {
        "match": {
            "if": "`${headers['User-Agent'].split('/')[0] == 'curl'}`"
        },
        "action": {
            "return": 204
        }
    }
@hongzhidao hongzhidao merged commit 43c4bfd into nginx:master Aug 20, 2024
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants