Implement boot bindings support. #21

jsirois · 2022-11-09T20:46:54Z

Now commands can depend on {boot.bindings.<binding name>} placeholders which are satisfied by boot bindings commands that are run exactly once.

This is used in the Python example to produce a scie-pants Pants binary that prepares a fully pre-compiled Pants venv that runs at the full speed of a native Pants venv. The boot binding command that creates the venv also uses a private PEX_ROOT and cleans this up in addition to the Pants PEX used to build the venv. The result is an ~/.nce with just the CPython interpreter and a venv that symlinks to it. There is no wasted speed or space as compared to a traditional Pants Python venv (save for the size of the scie binary itself which may be addressed by #9 or #19).

Closes #7
Closes #20

Now commands can depend on `{boot.bindings.<binding name>}` placeholders which are satisfied by boot bindings commands that are run exactly once. This is used in the Python example to produce a scie-pants Pants binary that prepares a fully pre-compiled Pants venv that runs at the full speed of a native Pants venv. The boot binding command that creates the venv also uses a private PEX_ROOT and cleans this up in addition to the Pants PEX used to build the venv. The result is an ~/.nce with just the CPython interpreter and a venv that symlinks to it. There is no wasted speed or space as compared to a traditional Pants Python venv (save for the size of the scie binary itself which may be addressed by a-scie#9 or a-scie#19). Closes a-scie#7 Closes a-scie#20

jsirois · 2022-11-09T20:47:49Z

@sureshjoshi sent you an invite to the repo. I can add you as a reviewer if you want to accept that. I just started with the read role.

jsirois · 2022-11-09T21:12:41Z

Aha - OK. I broke the scie-tote. I need to document that! Fix coming.

The node example uses the *scie-tote; so that's why just it failed. The M2 mac also doesn't have vm / post run cleanup; so it worked due to garbage left over in the ~/.nce from prior burns. That's an interesting issue with self hosted runners!

* scie-tote: This is where you package a scie that has no zips - the boot-pack inserts an artifical scie-tote zip for you to hold the files and make sure there is zip EOCD magic bytes at the end of the scie the scie-jump can scan back for to locate the manifest at boot time).

jsirois · 2022-11-09T21:17:10Z

jump/src/context.rs

+
+        let mut file_entries = vec![];
+        for file in &self.lift.files {
+            if self.replacements.contains(&file) {


This is the bug introduced. If there is a replacement for, say {node}, that is added as FileEntry::Install - so far so good. If that file thogh has zero size (which indicates its stored in the scie-tote), the scie-tote needs to be added as a FileEntry::Install as well.

Now the intracies of recognizing a scie tote are self contained in the Context.

…der.

jsirois · 2022-11-09T23:08:03Z

Alright - should be good for review now. The scie-tote bug is fixed and the Circle CI Linux aarch64 build is now properly wired to forked PRs.

sureshjoshi · 2022-11-10T02:01:10Z

Piece of the puzzle I don't quite understand, which I think it's due to my lack of Rust knowledge.

The boot bindings occur exactly once (on first run of the executable, I believe), but I don't quite understand the mechanism that makes this work. Is there a clone made of the fingerprint'd, expanded ~/.nce/... folder - and the bindings are run on that?

Or is the original folder altered in-place?

In either case, how does the system know that the boot binding has successfully run?

jsirois · 2022-11-10T02:26:48Z

jump/src/context.rs

+
+impl Binding {
+    pub(crate) fn execute(self) -> Result<(), String> {
+        atomic_path(&self.target.clone(), Target::File, |lock| {


Right here we pass a closure that only gets executed if the file at the target path does not exist. If you read the implementation of atomic_path, that's done with a file lock.

As a by the by, you may be aware atomic makes skis, boots and bindings.

For more clarity, the current structure of the non-compact executable is:

~/.nce/ <file hash directory>/ <file>.lck <file - expanded if archive or directory, blob if neither> ... <lift manifest hash directory>/ lift.lck lift.json bindings/ locks/ <binding cmd name empty file> <binding cmd name empty file>.lck

It's the last <binding cmd name empty file>{,.lck} pair that is used as the gate for each binding that runs in the bindings sandbox above.

I guess I should have named that directory gates. Some atomics threading the gates right there: https://external-preview.redd.it/VpSdsY8FC78KrqgHfW1eS0aDTgCEoIOwNwtF5oI3zAU.jpg?auto=webp&s=ecd159eb357406f68bcde49a859b61a4c8a851ae

Ah. Awesome - turns out the piece of the puzzle I missed out on was the <lift manifest hash directory>/ - I didn't realize that was also stored in there.

👍🏽

If the binding always fails, the app never runs. You'd need to fix permissions or intervene in some way.

As far as what to rule in / out, you should just be aware the lock is cooperative and so you just have to think about non-cooperative uses. If you try to mess with paths not in your cooperative control, like /usr/local, you may have to contend with a package manager running concurrently to you. I can't help you there!

This install bindings mechanism only guarantees:

Your binding runs (successfully) only once.

You have *complete control of {scie.binding} during that run - it's dedicated to your app alone by dint of your app manifest hash being unique.

* Obviously external commands could be malicious and guess your binding dir or else run blanket and catch you that way (rm -rf ~/.nce).

It should be clear that in a container build step, you're solid. That environment has the guarantees of you running in isolation with nothing else concurrent. On a bare metal host, things are less solid. You probably want to stick to mutating the scie.binding dir. This will be the situation Pants is in installing itself sometimes on end user machines, sometimes in CI, sometimes in production.

I've added #22 to the 0.2.x release and will work on that today. There needs to be a user guide obviously explaining how you build a scie including how you write the lift manifest / what features it supports.

Great, thanks for the info!

The "This install bindings mechanism only guarantees" snippet you added above would be great in the docs, as well as the bare-metal vs container difference. Those two snippets basically cleared up everything for me about binding operation.

Ok, good to know. I'll add a section on boot bindings in a new PR on top of #24 after this and that land and be sure to include those bits.

jsirois · 2022-11-10T02:27:23Z

@sureshjoshi right here: #21 (comment)

And this is the same trick Pex uses FWIW. The Rust is incidental here.

sureshjoshi · 2022-11-11T13:55:12Z

Absolutely trivial comment, just something I noted while running the Pants python example (and now looking at it, this may already be handled by #23):

⏺ examples/python % time ./scie-pants --version                                                                                                                                                                                 ⎇ issues/7
Traceback (most recent call last):
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/pex", line 246, in <module>
    sys.exit(func())
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 112, in main
    PantsLoader.main()
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 108, in main
    cls.run_default_entrypoint()
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 90, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 64, in run
    options_bootstrapper = OptionsBootstrapper.create(
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/option/options_bootstrapper.py", line 135, in create
    bargs = cls._get_bootstrap_args(args)
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/option/options_bootstrapper.py", line 198, in _get_bootstrap_args
    options = GlobalOptions.get_options_flags()  # type: ignore[call-arg]
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/util/memo.py", line 123, in memoize
    result = func(*args, **kwargs)
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/option/global_options.py", line 2008, in get_options_flags
    return GlobalOptionsFlags.create(cast("Type[GlobalOptions]", cls))
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/option/global_options.py", line 2025, in create
    for options_info in collect_options_info(BootstrapOptions):
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/option/option_types.py", line 29, in collect_options_info
    attr = getattr(cls, attrname)
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/option/option_types.py", line 191, in __get__
    return OptionsInfo(self._flag_names, self.get_flag_options(objtype))
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/option/option_types.py", line 174, in get_flag_options
    default=_eval_maybe_dynamic(self._default, subsystem_cls),
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/option/option_types.py", line 59, in _eval_maybe_dynamic
    return val(subsystem_cls) if inspect.isfunction(val) else val  # type: ignore[no-any-return]
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/option/global_options.py", line 787, in <lambda>
    default=lambda _: os.path.join(get_buildroot(), ".pants.d"),
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/base/build_environment.py", line 34, in get_buildroot
    return BuildRoot().path
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/base/build_root.py", line 54, in path
    self._root_dir = os.path.realpath(self.find_buildroot())
  File "/Users/sj/.nce/4a776ed096c69129ecf631f7ccc7221144bcbe803a58b400363bd606da99cd10/bindings/venv/lib/python3.9/site-packages/pants/base/build_root.py", line 31, in find_buildroot
    raise self.NotFoundError(
pants.base.build_root.BuildRoot.NotFoundError: No build root detected. Pants detects the build root by looking for at least one file from ['pants', 'BUILDROOT', 'BUILD_ROOT'] in the cwd and its ancestors. If you have none of these files, you can create an empty file in your build root.
./scie-pants --version  4.17s user 3.08s system 47% cpu 15.361 total

sureshjoshi

I'm about 3-4 more months of Rust learning away before I'd feel remotely comfortable commenting on even a trivial Rust code change.

But all of my comments/questions have been answered - and some of the potential quirks of bindings make sense to me now (covered in the documentation ticket).

jsirois · 2022-11-11T15:17:10Z

Absolutely trivial comment, just something I noted while running the Pants python example (and now looking at it, this may already be handled by #23):

It's actually handled in this PR: https://github.com/a-scie/jump/pull/21/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR64

That said, yeah #23 is the sane approach. You can just run examples/run.sh python.

jsirois · 2022-11-11T19:52:57Z

I'm going to go ahead and submit this in the next hour but @benjyw I'd love your eyes on it at some point. I'll circle back if you spot anything awry. It's not code I'm worried about, just semantics. It would be good to get the concept of one time install actions done right, especially because Pants will likely rely on this. @sureshjoshi asked some good questions that I think the current scheme stood up well to, but more questions are good if they're out there.

benjyw · 2022-11-12T01:37:32Z

Going offline for 24 hours, will review tomorrow night or Sunday.

benjyw · 2022-11-16T23:29:52Z

Finally got a chance to look at this, and yep, I think it makes sense semantically!

Future readers - note the details are a little different from the initial description above, notably the placeholder names are (rightly) {scie.bindings.<binding_name>}, and not as written there.

Just excellent ski gear puns all around.

jsirois requested a review from benjyw November 9, 2022 20:46

jsirois requested a review from sureshjoshi November 9, 2022 21:09

jsirois commented Nov 9, 2022

View reviewed changes

jsirois added 2 commits November 9, 2022 14:26

Elevate the scie tote to a 1st class concept.

fb4da7b

Now the intracies of recognizing a scie tote are self contained in the Context.

Save a little work preparing a binding used in more than one placehol…

9d3bc4d

…der.

jsirois commented Nov 10, 2022

View reviewed changes

sureshjoshi approved these changes Nov 11, 2022

View reviewed changes

Merge remote-tracking branch 'refs/remotes/origin/main' into issues/7

12092ee

jsirois merged commit 8653c40 into a-scie:main Nov 11, 2022

jsirois deleted the issues/7 branch November 11, 2022 23:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement boot bindings support. #21

Implement boot bindings support. #21

jsirois commented Nov 9, 2022

jsirois commented Nov 9, 2022

jsirois commented Nov 9, 2022

jsirois Nov 9, 2022

jsirois commented Nov 9, 2022

sureshjoshi commented Nov 10, 2022

jsirois Nov 10, 2022

jsirois Nov 10, 2022

jsirois Nov 10, 2022 •

edited

Loading

jsirois Nov 10, 2022

sureshjoshi Nov 10, 2022

jsirois Nov 10, 2022

jsirois Nov 10, 2022 •

edited

Loading

jsirois Nov 10, 2022

sureshjoshi Nov 11, 2022

jsirois Nov 11, 2022

jsirois commented Nov 10, 2022 •

edited

Loading

sureshjoshi commented Nov 11, 2022

sureshjoshi left a comment

jsirois commented Nov 11, 2022

jsirois commented Nov 11, 2022

benjyw commented Nov 12, 2022

benjyw commented Nov 16, 2022

Implement boot bindings support. #21

Implement boot bindings support. #21

Conversation

jsirois commented Nov 9, 2022

jsirois commented Nov 9, 2022

jsirois commented Nov 9, 2022

Choose a reason for hiding this comment

jsirois commented Nov 9, 2022

sureshjoshi commented Nov 10, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsirois Nov 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsirois Nov 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsirois commented Nov 10, 2022 • edited Loading

sureshjoshi commented Nov 11, 2022

sureshjoshi left a comment

Choose a reason for hiding this comment

jsirois commented Nov 11, 2022

jsirois commented Nov 11, 2022

benjyw commented Nov 12, 2022

benjyw commented Nov 16, 2022

jsirois Nov 10, 2022 •

edited

Loading

jsirois Nov 10, 2022 •

edited

Loading

jsirois commented Nov 10, 2022 •

edited

Loading