make the @quickactivate macro into its own package with no dependencies

[jesseylin]

As of Julia 1.9, there is now a package precompilation cache. The trade-off is that cache misses result in compilation times even slower than pre-1.9.

The suggested workflow with DrWatson to have DrWatson installed in a global shared environment and then invoking

using DrWatson
@quickactivate

but this frequently triggers such a cache miss if sometimes packages are installed into a project environment the usual way by

] activate .
] add SomePackage

As far as I can tell the @quickactivate macro doesn't depend on any other code and could be extracted into its own package. If this package has no dependencies, then it can be safely installed in the global environment and will never cause such a cache miss as the dependency graph would be identical whether the project is activated the canonical way via Pkg or via @quickactivate

There are probably much nicer solutions to this problem but this is by far the easiest one I can think of.

[Datseris]

Hello,
thanks for opening this. Almost daily I have been suffering from re-compilations of what should have been precompiled already.

Happy to make quick activate a package with no dependencies, but will this solve the issue? Do you mind giving it a try (or I will when I get the chance) by just copying the @quickactivate source and making it a package you dev locally? Because, even quick activate will have Pkg as a dependency.

[Datseris]

whether the project is activated the canonical way via Pkg or via @quickactivate

@quickactivate is a Pkg activation. It uses the Pkg.activate command.

[jesseylin]

Thanks for the comments. I have been meaning to try what you suggested (make some dev package) to see if it works.

I just glanced at the quickactivate source, I did not realize it depends on Pkg. I think one can just escape the reference to Pkg and then the Pkg.activate call can be resolved at the call-site, so then (the macro) has no dependency even on Pkg.

re:

@quickactivate is a Pkg activation. It uses the Pkg.activate command.

You are right. What I meant by the canonical method is to directly use Pkg.activate in the project directory without using DrWatson at all.

[Datseris]

I think one can just escape the reference to Pkg and then the Pkg.activate call can be resolved at the call-site, so then (the macro) has no dependency even on Pkg.

I dont understand how this would be possible.

[jesseylin]

Here is an implementation: https://github.com/jesseylin/QuickActivate.jl/blob/1928b448f34e858f8c4582c0b25375382b067459/src/QuickActivate.jl#L33C1-L52C4

This solved the precompilation cache invalidation issues I was having (i.e., at the REPL I do not get annoying restarts of precompilation). It is currently not a full solution for me because I use Pluto a lot and they actually hardcode a check for the DrWatson package when it computes the evaluation topology of the notebook. I would prefer that a similar solution could be upstreamed into DrWatson. The new recommended workflow would be to install DrWatson per-project and not globally, but install the other small package which only contains @quickactivate globally.

[Datseris]

I would prefer that a similar solution could be upstreamed into DrWatson

I don't understand. What is the similar solution that should be upstreamed?

[jesseylin]

I don't understand. What is the similar solution that should be upstreamed?

I wrote the code in the repo to show that one can implement the @quickactivate macro in such a way that it has no dependencies whatsoever (by escaping the call to Pkg, as described before): the solution is exactly the package in the repo I linked. With this, DrWatson could remove its implementation of @quickactivate and suggest that users install a separate package QuickActivate which is used only to activate a DrWatson project. The project itself has to have DrWatson installed but then a user does not need DrWatson installed in their global environment. Then, this solves the precompilation cache misses.

[Datseris]

https://github.com/jesseylin/QuickActivate.jl/blob/1928b448f34e858f8c4582c0b25375382b067459/src/QuickActivate.jl#L40

How ca this work without Pkg being a dependency? In any case I don't think this hack is the way to go. You are in essence having a dependency without having it in the Project.toml by @eval "hacking" it. It doesn't appear a good idea.

Have you tried it with having Pkg as the only dependency? Why would that trigger pre-compilation problens?

[jesseylin]

How ca this work without Pkg being a dependency?

Because the code is evaluated in the scope of the caller, i.e., the user code, and this will already have Pkg because it's part of the stdlib since Julia 1.0. So strictly speaking, Pkg is a runtime dependency but it's one we want to resolve dynamically in the caller's scope. The essential point is that the package does not bring in any dependencies of it's own because its Manifest.toml and Project.toml are empty.

by @eval "hacking" it. It doesn't appear a good idea.

There is no usage of eval anywhere, I think it's a relatively kosher use of esc for metaprogramming. It is possible that my implementation of the idea could be better but I don't see anything obviously wrong with the idea itself (i.e., the idea of using the caller's Pkg instead of bringing in your own). The failure modes in my eyes for this kind of metaprogramming are

1. the caller scope does not have Pkg
2. our functionality may actually depend on a particular implementation (i.e., a particular version) of Pkg.activate to work and the caller may have a different one.

But I think both of these are rather unrealistic.

Have you tried it with having Pkg as the only dependency? Why would that trigger pre-compilation problens?

No unfortunately, I just whipped this up in the middle of the workday to fix the problem 😅. But if you add Pkg to Project.toml it brings in ~10 dependencies of its own into Manifest.toml. If my understanding of the cache miss is correct (it might not be), then if any of these dependencies overlap with the full dependency graph of the user's DrWatson project environment, it could invalidate the cache and trigger a recompilation.

As an aside, in general I think it's a good idea to use Julia's default package environment tools as much as possible and resolving this problem would let one remove DrWatson from their global environment and install it in each specific environment where it can itself be managed by Pkg.

Let me know what you think. The precompilation thing is very annoying to my workflow and I am happy to help pursue a fix in whatever way moving forward.

[Datseris]

Because the code is evaluated in the scope of the caller, i.e., the user code, and this will already have Pkg because it's part of the stdlib since Julia 1.0. So strictly speaking, Pkg is a runtime dependency but it's one we want to resolve dynamically in the caller's scope. The essential point is that the package does not bring in any dependencies of it's own because its Manifest.toml and Project.toml are empty.

Yes, that is correct. However, notice that this code snippet would work for any package. Pkg isn't special. It is a normal Julia package after all. One could use the code you have now to load any package without having it as a formal dependency.

While it is true that Pkg is stdlib since Julia 1.0, since Julia 1.9 stdlibs must now be listed as formal dependencies of a package for it to be registered. So my problem is that even if the code works, we won't be able to register it anyways...

However. I am really thinking that Quickactivate as a package would work with Pkg a formal dependency. Since the version of Pkg is linked to the Julia version, it could never trigger cache misses, right? I think this is the most promising option if you are willing to test it.

There is no usage of eval anywhere, I think it's a relatively kosher use of esc for metaprogramming.

Right, yes, you are correct, I was wrong and was exaggerating. But still, I would count the current code as a "hack" bcause it is "hacking" in a dependency without listing it as a formal dependency. Doesn't matter if the package loaded is stdlib or not, it is still a dependency.

Have you tried it with having Pkg as the only dependency? Why would that trigger pre-compilation problens?

No unfortunately, I just whipped this up in the middle of the workday to fix the problem 😅. But if you add Pkg to Project.toml it brings in ~10 dependencies of its own into Manifest.toml. If my understanding of the cache miss is correct (it might not be), then if any of these dependencies overlap with the full dependency graph of the user's DrWatson project environment, it could invalidate the cache and trigger a recompilation.

I think this is worth trying out nevertheless though. I have been talking to the developers of Pkg about this. They would like to help, but they need a reproducible MWE. The problem I am raising to them is: cache invalidation occurs even if the two environments have exactly the same package versions. In our case, one environment is the global/default one. If we can show the invalidation happening in a reproducible way with a shareable manifest and the code that loads it, it could be that this problem gets fixed directly in Pkg.

As an aside, in general I think it's a good idea to use Julia's default package environment tools as much as possible and resolving this problem would let one remove DrWatson from their global environment and install it in each specific environment where it can itself be managed by Pkg.

Sure, I agree and I am totally happy to make, test, and register a Quickactivate.jl package, provided that we have confirmation that this works, and also provided that the package can actually be registered (which would mean that Pkg must be listed as a formal dependency).

Let me know what you think. The precompilation thing is very annoying to my workflow and I am happy to help pursue a fix in whatever way moving forward.

I completely feel you. I am suffering from the same problem. All of my science projects have Make+DifferentialEquations as dependencies; the packages that take notoriously the most to precompile. It is an absolute killer to my productivity that these packages get precompiled several times per day.

The most productive way forwards I think is to get the Pkg developers involved somehow, by providing them something "tangible", which is why I am asking you to test the quickactivate with Pkg dependency, and then we share this as a MWE that reproduces the problem.

[jesseylin]

While it is true that Pkg is stdlib since Julia 1.0, since Julia 1.9 stdlibs must now be listed as formal dependencies of a package for it to be registered. So my problem is that even if the code works, we won't be able to register it anyways...

Thanks, I did not know this: it is a very important point.

The problem I am raising to them is: cache invalidation occurs even if the two environments have exactly the same package versions.

I also suspect this happens but I have no real evidence.

The most productive way forwards I think is to get the Pkg developers involved somehow, by providing them something "tangible", which is why I am asking you to test the quickactivate with Pkg dependency, and then we share this as a MWE that reproduces the problem.

I am glad to hear that you have been talking to the Pkg guys. I like this course of action, I will try the test you suggest later in the week when I have more time.

[Datseris]

Perhaps related discussion: https://discourse.julialang.org/t/case-study-method-invalidations-caused-by-pkg-jl-with-julia-1-11/109038