Inconsistent NEWLINE handling

[cathaysia]

I noticed that the handling of NEWLINE when the rule appears in a COMMENT is inconsistent with that elsewhere.
How to allow NEWLINE in rules in comment?

here is a simple grammar:

WHITESPACE = _{ WHITE_SPACE }
identifier = { ASCII_ALPHA+ }
cid = { "//@" ~ identifier }

COMMENT = { cid }
document = { cid | identifier }

here is the test case:

input :

//@Ab
c

parse cid:

• cid > identifier: "Ab\nc"

parse comment:

• COMMENT: "//@ab"

expected:

• COMMENT: "//@ab\nc"

[tomtau]

Is this related to #966 ?

[cathaysia]

here is the full code about above:

WHITESPACE = _{ WHITE_SPACE | NEWLINE }
identifier = { ASCII_ALPHA+ }
two_identifier = { identifier ~ "=" ~ identifier }
COMMENT = { two_identifier }

as a interesting result, two_identifier cannot match:

abc=abc

it need a space between c and =. it seems WHITESPACE rule not work normally.

[tomtau]

One thing to keep in mind here is that each ~ inserts implicit WHITESPACE | COMMENT in-between that the parser may consume.
You can try the debugger: https://github.com/pest-parser/awesome-pest/blob/main/debugger.md and add all rules if you'd like to see what is being tried at which position.

[cathaysia]

~ inserts implicit WHITESPACE | COMMENT*

but if a rule appears in COMMENT, and this rule maybe contains WHITESPACE, how can I do it?

[tomtau]

Perhaps define different explicit rules for those instead of using the implicit ones and use atomic rules: http://pest.rs/book/grammars/syntax.html#atomic ?

[cathaysia]

Explicit definition these whitespace does work, but there are too many places to define. :(

[cathaysia]

Would you consider allowing WHITESPACE appears in COMMENT?

[tomtau]

pest3 will have a bit different grammar style, so hopefully this shouldn't be an issue