templates: add raw_text to directly output content

[avamsi]

Templates can be formatted (using labels) and are usually sanitized
(unless for plain text output). raw_text(content) bypasses both.

	'hyperlink(url, text)' = '''
		raw_text("\e]8;;" ++ url ++ "\e\\") ++ text ++ raw_text("\e]8;;\e\\")
	'''

In this example, raw_text not only outputs the intended escape codes,
it also strips away any escape codes that might otherwise be part of the
url (from any labels attached to the url content).

---

Checklist

If applicable:

• I have updated CHANGELOG.md
• I have updated the documentation (README.md, docs/, demos/)
• I have updated the config schema (cli/src/config-schema.json)
• I have added tests to cover my changes

TODO: will update changelog and documentation post alignment

[martinvonz]

templates: add plain_text to strip formatting

This is missing the most important part (IMO): the "why" (the "what" can at least be inferred from the patch). Unless it's obvious why it's a good thing, I think a commit description should explain what the point is, e.g. by providing a use case.

[avamsi]

Re: why, apologies -- I was a bit over eager with sending this out as I wasn't sure if I had a good implementation.
I'll amend the commit shortly, but to answer your question in the meantime, I'm trying to hyperlink change IDs in jj log to Gerrit CLs, with something like this:

	'hyperlink(url, text)' = '''
		"\e]8;;" ++ plain_text(url) ++ "\e\\" ++ text ++ "\e]8;;\e\\"
	'''
	'format_short_change_id(id)' = '''
		hyperlink("https://gerrit/q/Id0000000" ++ id.normal_hex(), id.shortest())
	'''

(From avamsi#1)

id.normal_hex() in this example would have formatting applied (from the change_id label) and the corresponding ANSI escape codes would result in a wrong URL. plain_text proposed in this PR strips them (and so, corrects the URL).

[martinvonz]

I'm trying to hyperlink change IDs in jj log to Gerrit CLs

Nice!

[yuja]

Perhaps, is_raw: bool will have to be passed to the callback. The "indent" template function prints prefix by using the callback formatter argument, and the prefix part should be colorized if any.

BTW, I think it's okay to disable raw_text() within reformatting template functions. We can add the support later if needed.

[avamsi]

I think without this support, raw_text won't work with wrapping :(.

The (/ your) comment in write_indented says prefix inherits current labels, which arguably is nothing while raw_text is in play? Basically, it doesn't sound to me like we're breaking the existing behaviour.

But either way, I think indent() is the only function that needs this? And do you think this should be blocking or is it okay to fix in a follow up?

[yuja]

I think without this support, raw_text won't work with wrapping :(.

Right. However, wrapping wouldn't work correctly if raw_text() contained escaped characters. In your use case, I assume you wouldn't want to count raw_text() content as visible characters, so it might make sense to put them as FormatOp::Raw(Vec<u8>). This suggests that raw_text() might be a bad function name.

The (/ your) comment in write_indented says prefix inherits current labels, which arguably is nothing while raw_text is in play?

With the current implementation, indent text will be printed by PlainTextFormatter if the trailing text is raw. That's the problem I pointed out. Try something like indent(label("error", "foo"), raw_text("bar\n")).

And do you think this should be blocking or is it okay to fix in a follow up?

The problem of indent() isn't a blocker, but I guess we would need a non-trivial rewrite to fix the problem. So it's nice if you can address it within this PR.

[avamsi]

However, wrapping wouldn't work correctly if raw_text() contained escaped characters.

Could you elaborate on this? I was going through the code earlier and there was a comment (again, yours :P) saying textwrap handles escape codes, so I just didn't worry about it. My testing now seems to confirm this as well:

    #[test]
    fn test_wrap() {
        assert_eq!(
            wrap_bytes(
                b"\x1b]8;;http://example.com\x1b\\Example Text\x1b]8;;\x1b\\",
                12
            ),
            vec![b"\x1b]8;;http://example.com\x1b\\Example Text\x1b]8;;\x1b\\".as_ref()]
        );
        assert_eq!(
            wrap_bytes(
                b"\x1b]8;;http://example.com\x1b\\Example Text\x1b]8;;\x1b\\",
                10
            ),
            vec![
                b"\x1b]8;;http://example.com\x1b\\Example".as_ref(),
                b"Text\x1b]8;;\x1b\\".as_ref(),
            ]
        );
    }

(I haven't tested if the 2nd example above is broken by graph nodes in log, but I'm satisfied with this level of support from textwrap.)

---

With the current implementation, indent text will be printed by PlainTextFormatter if the trailing text is raw. That's the problem I pointed out. Try something like indent(label("error", "foo"), raw_text("bar\n")).

I understand that but I'm not sure why you say it's a problem -- behaviour of indent(label("prefix", "foo"), label("rest", "bar")) and indent(label("prefix", "foo"), raw_text(label("rest", "bar"))) seems consistent to me (which is, the formatting (or the lack of) of the actual content affect the prefix as well).

[yuja]

However, wrapping wouldn't work correctly if raw_text() contained escaped characters.

Could you elaborate on this? I was going through the code earlier and there was a comment (again, yours :P) saying textwrap handles escape codes,

Oh, I didn't remember textrap() could parse ANSI escapes (because the existing code doesn't depend on it.) Then, the remaining issue is escape sequence containing \n? If we add some padding function, it won't work, but we don't have such function right now.

Anyway, if the primary use case is to insert (invisible) escape characters, it should be easier to process these characters as metadata rather than content to be reformatted. The hard part is to find a short descriptive name for it. :)

With the current implementation, indent text will be printed by PlainTextFormatter if the trailing text is raw. That's the problem I pointed out. Try something like indent(label("error", "foo"), raw_text("bar\n")).

I understand that but I'm not sure why you say it's a problem -- behaviour of indent(label("prefix", "foo"), label("rest", "bar")) and indent(label("prefix", "foo"), raw_text(label("rest", "bar"))) seems consistent to me (which is, the formatting (or the lack of) of the actual content affect the prefix as well).

The expected out put for indent(label("prefix", "foo"), raw_text(label("rest", "bar"))) is <prefix>foo</prefix>bar. You'll get foobar because foo is printed through the PlainTextFormatter.

[yuja]

nit: can you move the escape syntax test to test_string_literal() in template_parser.rs?

It's also better to add a few non-ASCII characters (e.g. \xff), invalid escapes (e.g. \x, \x0, ..)

[yuja]

Appears that requiring T: Template is better. (I thought String version would be way simpler, but I was wrong.)

impl<T: Template> Template for RawTextTemplate<T> {
    fn format(&self, formatter: &mut TemplateFormatter) -> io::Result<()> {
        let rewrap = formatter.rewrap_fn();
        let mut raw_formatter = PlainTextFormatter::new(formatter.raw());
        self.0.format(&mut rewrap(&mut raw_formatter))
    }
}

[yuja]

reminder: you'll need to update the templates doc.

[yuja]

I think without this support, raw_text won't work with wrapping :(.

Right. However, wrapping wouldn't work correctly if raw_text() contained escaped characters. In your use case, I assume you wouldn't want to count raw_text() content as visible characters, so it might make sense to put them as FormatOp::Raw(Vec<u8>). This suggests that raw_text() might be a bad function name.

The (/ your) comment in write_indented says prefix inherits current labels, which arguably is nothing while raw_text is in play?

With the current implementation, indent text will be printed by PlainTextFormatter if the trailing text is raw. That's the problem I pointed out. Try something like indent(label("error", "foo"), raw_text("bar\n")).

And do you think this should be blocking or is it okay to fix in a follow up?

The problem of indent() isn't a blocker, but I guess we would need a non-trivial rewrite to fix the problem. So it's nice if you can address it within this PR.

[yuja]

nit: for future reference, can you include the "why" in raw_text() patch description?
#4593 (comment)