[Feature discussion] Calling system commands from rgbasm

[Rangi42]

This is a rough draft for a set of features which I would like to cover the most obvious use cases of the proposed #928, without needing potentially years of planning and prep work to make it usable. I also want to make as few changes as necessary, as opposed to inventing new ad-hoc features for slightly different use cases. (I.e. no adding built-in COMPRESS, RANDOM, FILESIZE, READURL, etc functions.)

Macros are already powerful enough to do conditional logic, building up state through multiple macro calls, etc; they just aren't efficient for things like compression algorithms, nor capable of accessing external data from the filesystem or network (unless you INCLUDE a file in rgbasm's own syntax, and have Makefile rules to create the file appropriately).

System commands can take a stream of stdin input, and they give streams of stdout and stderr output, plus a numeric return code. These could all be handled by a single SYSEXEC directive, which sets the values of global __SYSOUT__, __SYSERR__, and __SYSRET__ symbols, and optionally takes a string for stdin:

SYSEXEC "false"
assert __SYSRET__ == 1

DEF greeting EQUS "hello\nworld"
SYSEXEC "cat -n", "{greeting}"
assert !STRCMP("{__SYSOUT__}", "  1  hello\n  2  world")

DEF rawtext EQUS "SHOPKEEP: Here's a long speech.<END>"
SYSEXEC "textcomp", "{rawtext}"
if __SYSRET__
    fail "textcomp failed: {__SYSERET__}\n{__SYSERR__}"
endc
db "{__SYSOUT__}"

(I went with global variables for the out/err/ret values since allowing up to four parameters for specifying custom symbols could get awkward; it's easy to confuse the order if they're all comma-separated, and we don't typically have keywords like SYSEXEC "cat" IN "hello" OUT outstring RET catcode.)

@ISSOtm You compared something like this unfavorable to LaTeX's \write18 in #928. I don't think it would be a big deal in practice, since repos can already put evil commands in obscure input files for commands in their Makefile; but I do understand the concern. Apparently TeX Live and MiKTeX have a workaround for this: "restricted \write18" where only whitelisted commands can execute. So you would have to run rgbasm --allow-sysexec false,cat,textcomp foo.asm for the above to work. (It's a lot easier for the user to audit just the Makefile for any --allow-sysexecs than all the .asm files.)

[Rangi42]

I already know @ISSOtm is not going to accept this. :P

[ISSOtm]

Yeah, in general, I'm more in favour of the multi-step assembling (i.e. progressively processing data towards its final form) than "back-and-forth" between RGBASM and Something Else. Notably because that dangerously veers towards RGBASM becoming a shell, especially #928.

Another concern is that this would strengthen how tied to a platform assembly code would be; I'm already dissatisfied that backslashes can be used on Windows but not Linux, though I've accepted that 1. it's a necessity to accommodate the Real World™ that RGBASM lives in ("why isn't INCBIN "\\\\network_share\\stuff.bin" working?!?"), and 2. we're not going back on that. But I do not like the idea of enabling more avenues for tying source code to a particular platform or machine. (#928 would have it even worse, since each plugin is subject to the same distribution pains that programs like RGBDS itself is.)

Finally, I just don't think I'm ready to shoulder the additional code complexity and edge cases brought by process management...

Also you and ax6 know that if we're adding this I'm definitely making a wrapper macro for INCLUDE/INCBIN that invokes make, to free us from -MG :^)

[Rangi42]

dangerously veers towards RGBASM becoming a shell

Come on, let's have RGBDSH... ;)