Using lib argument causes Haxe to access system shell

[justin-espedal]

I'd like to use the Haxe compiler in Windows environments where access to cmd.exe is restricted (which is not uncommon in, for example, schools).

Two arguments passed to the haxe compiler will cause the compiler to invoke cmd.exe to start a new process. These are cmd and lib.

See:
https://ocaml.org/api/Unix.html#VALopen_process_full
https://ocaml.org/api/Unix.html#VALsystem

The command is interpreted by the shell /bin/sh (or cmd.exe on Windows), cf. Unix.system. The Filename.quote_command function can be used to quote the command and its arguments as appropriate for the shell being used. If the command does not need to be run through the shell, Unix.open_process_args_in can be used as a more robust and more efficient alternative to Unix.open_process_in.

cmd:

haxe/src/compiler/haxe.ml

Line 170 in e70ba87

let pout, pin, perr = Unix.open_process_full cmd (Unix.environment()) in

haxe/src/compiler/haxe.ml

Line 187 in e70ba87

match Unix.system cmd with

lib:

haxe/src/compiler/haxe.ml

Line 111 in e70ba87

let pin, pout, perr = Unix.open_process_full cmd (Unix.environment()) in

I think that cmd has the expectation of being passed to the system shell, so it makes sense to simply avoid using cmd as an argument if shell/interpreter access isn't available. I would expect lib, on the other hand, not to need to use the shell. Is it possible to replace open_process_full with open_process_args_full for the case of lib?

In my case, the Haxe compiler is started from inside an IDE. I can simply have the IDE automatically read the hxml file, find lib arguments, run haxelib path {libs}, and replace the lib arguments with the appropriate classpaths and defines. Then there will be no cmd.exe access and things will work as expected. However, the behavior is a little bit surprising as-is.

[justin-espedal]

I found my workaround a little lacking when the Haxe compiler was run indirectly by other tools. As such, I tried making the needed changes to the compiler myself.

https://github.com/justin-espedal/haxe/compare/9a7bffd05f46f7080124b9144050f198319636a2..f37c14a826b7a947fe0e07271d3dfa3272a00833

The specific function that I was looking at in the ocaml docs is available starting with ocaml 4.08, while Haxe's minimum version is ocaml 4.02. Since I can't use it directly without changing the version dependency, I copied and adapted the new function as a wrapper around Unix.create_process_env instead.

https://github.com/ocaml/ocaml/blob/5158c85e0f67a92176b612cdb67af0ba4efa3d90/otherlibs/unix/unix.ml#L990..L1015

If updating the minimum required ocaml version isn't a big deal, now would be a nice time to update. Otherwise, proper attribution for the source of the backported code may need to be included.

Since this isn't passed through the shell/cmd anymore, the full path to haxelib needs to be specified, for which I wrote a really simple function to pick the executable up from the path.

If this is useful as a starting point, feel free to copy it, or I can create a PR. This change passed all the CI tests on Windows, and I tested the change locally on Linux as well. I grabbed the Windows binary that github actions built and it does what I need it to, so I'm happy.

[justin-espedal]

Looks like I missed a few places.

Many of the generators run their specific haxelib, and that goes through Sys.command which is basically the same as Unix.system.

haxe/src/context/common.ml

Line 674 in fc8633e

run_command = Sys.command;

Unix.open_process_in is also used in a few places. A couple are to run uname on unix systems, but another is used by the cpp target for haxelib path hxcpp.

haxe/src/generators/gencpp.ml

Line 6946 in 6eb36b2

let cmd = Unix.open_process_in "haxelib path hxcpp" in

[justin-espedal]

Took care of all the generators.

https://github.com/justin-espedal/haxe/compare/9a7bffd05f46f7080124b9144050f198319636a2..6b0fb64d51d50d97a239a5327fdc1de28a2cfbd5

Here's the status of all the places where Haxe invokes the shell, as far as I know.

• Haxe.add_libs (results from -lib argument, calls haxelib path {libs})
• Haxe.run_command (results from -cmd argument, this should go through the system shell, right?)
  Is it possible that this was also called from the various code generators, in the case that the compilation server is running?
  (See Haxe.setup_common_context)
  If so, then this really is only for the -cmd argument now, because all the generators call run_command_args instead now.
• Gencpp.write_build_options (calls haxelib path hxcpp)
• Gencpp.generate_source (calls haxelib run hxcpp ...)
• Gencs.generate (calls haxelib run hxcs ...)
• Genhl.generate (calls haxelib run hashlink ...)
• Genjava.generate (calls haxelib run hxjava ...)
• Genneko.generate (calls nekoc ...)
• Hlinterp.load_native (std.sys_string, calls uname on unix, doesn't apply to cmd.exe)
• EvalStdLib.StdSys (system name, calls uname on unix, doesn't apply to cmd.exe)

This does enough for my needs. I don't know if there's any reason to block shell access on Linux as well. If that were the case, the last two would need to be modified as well.

For this to actually be useful to anybody, changes will need to be made on the Haxe code side as well for whatever build tooling you happen to use. Haxelib itself, and anything else like hxp, lime, openfl, hxcpp, etc., will need to avoid Sys.command and use new Process instead for running subprocesses.

[justin-espedal]

I've turned the work I've done here into a PR.

#10393