Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix rendering of custom tags with more than one parameter #301

Merged
merged 2 commits into from
Apr 24, 2024

Conversation

pmenhart
Copy link
Contributor

Trying to add moderately complex custom tags, I encountered a problem:

  • All tag parameters were concatenated into one string, e.g. {% mu for foo as bar %} was presented as one node ["forfooasbar"] in Tag method render(context, nodes)
  • Caused by combining text of all children of token 'other_tag_parameters' in NodeVisitor.visitSimple_tag()
    Implementation was changed in Optimize the parsing of large templates #227. I reviewed the related tickets and didn't find justification for the concatenation
  • This PR separates each parameter to a distinct node, i.e. ["for", "foo", "as", "bar"]

I am focused on the Liquid flavor, but the change should work well for Jekyll as well.
There was one Jekyll scenario with failing tests, revealing an unrelated problem with parsing of include_relative. I submitted #300 as a prerequisite of this PR.

@msangel I'd appreciate your insight. I'm wondering if there was a reason for such string concatenation in #227 I am not aware of?

For reference, the original pre-227 code:

public BlockNode visitOther_tag_block(Other_tag_blockContext ctx) {
    BlockNode node = new BlockNode(isRootBlock);
    for (AtomContext child : ctx.atom()) {
      node.add(visit(child));
    }
    return node;
  }

cc @bkiers

* In Jekyll style, 'include_relative' was represented as token SimpleTag,
  instead of IncludeRelative, as expected since PR bkiers#288.
* As a consequence, filename components were parsed as separate parameters
  instead of token file_name_or_output. This behaviour got unnoticed only
  because all parameters were incorrectly concatenated into one string
  (to be fixed outside of this commit)
* In Liquid style, tag 'include_relative' is not defined (token InvalidTagId),
  but can be defined as a custom block (token SimpleBlock) or a tag (SimpleTag).
* To facilitate flavor-specific lexical analysis, flag isLiquidStyleInclude was
  was added to LiquidLexer; same code as in the LiquidParser since bkiers#196
* All tag parameters were concatenated into one string,
  e.g. {% mu for foo as bar %} was presented as one node ["forfooasbar"] in Tag method render()
* Caused by combining text of all childen of token 'other_tag_parameters' in NodeVisitor.visitSimple_tag().
  Change to one string in bkiers#227 seems to be not intentional; reverting to a list
* This commit separates each parameter to a distinct node, i.e. ["for", "foo", "as", "bar"]
@msangel
Copy link
Collaborator

msangel commented Apr 24, 2024

I don't know the answer.

I think it was made so for giving end-user to define own way of parsing parameters. Interesting related scenario: #196 (comment)

No clear reasoning for me why it was made so, as well as what is better. In the end, calling str.split('\\s') is not that hard.

Lets see what @bkiers will say on this.

@pmenhart
Copy link
Contributor Author

True, parsing can be delegated to custom tag implementation. Unfortunately, whitespace is not preserved at the moment: for foo as bar is transformed into a single string "forfooasbar". We cannot detect original boundaries.

@msangel
Copy link
Collaborator

msangel commented Apr 24, 2024

It’s true.

@msangel msangel merged commit 416d98a into bkiers:master Apr 24, 2024
4 checks passed
@bkiers
Copy link
Owner

bkiers commented Apr 24, 2024

I don't know the answer.

I think it was made so for giving end-user to define own way of parsing parameters. Interesting related scenario: #196 (comment)

No clear reasoning for me why it was made so, as well as what is better. In the end, calling str.split('\\s') is not that hard.

Lets see what @bkiers will say on this.

Me neither. @pmenhart's changes look sound en well tested! 👍

Thanks both!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants