Replicating v1-sandbox functionality during v2-install --lib step for later executables compilation

[okeuday]

Describe the bug
The removal of the v1-sandbox command (#6445) lacks a clear path for replacement and this issue is part of that cabal-install adventure! The first obvious step for obtaining something similar to sandbox functionality is to provide the --store-dir argument to the v2-configure command. However, usage of v2-install with --lib fails to utilize a local directory in a way that works. The v2-install --lib functionality is incomplete or non-functional based on #6481 . This issue was filed to search for the best path forward, based on the v1-sandbox warning issued in the past: Please file a bug if you cannot replicate a working v1- use case with the new-style commands..

To Reproduce
Steps to reproduce the behavior:

$ git clone https://github.com/CloudI/CloudI.git
$ cd CloudI/src
$ ./autogen.sh && ./configure --enable-haskell-support --with-integration-tests-ran && make

Usage with cabal-install >= 3.4 will break in:
https://github.com/CloudI/CloudI/blob/3256f29cf0b749e45b618a0216601ac1f9a9bd78/src/api/haskell/Makefile.am#L24
(past sandbox usage is shown in the same Makefile.am file). The install of the library there is meant to be used for creating integration tests executables in:
https://github.com/CloudI/CloudI/blob/3256f29cf0b749e45b618a0216601ac1f9a9bd78/src/tests/count/haskell/Makefile.am#L10-L13
https://github.com/CloudI/CloudI/blob/3256f29cf0b749e45b618a0216601ac1f9a9bd78/src/tests/http_req/haskell/Makefile.am#L10-L13
https://github.com/CloudI/CloudI/blob/3256f29cf0b749e45b618a0216601ac1f9a9bd78/src/tests/messaging/haskell/Makefile.am#L10-L13
https://github.com/CloudI/CloudI/blob/3256f29cf0b749e45b618a0216601ac1f9a9bd78/src/tests/msg_size/haskell/Makefile.am#L10-L13
https://github.com/CloudI/CloudI/blob/3256f29cf0b749e45b618a0216601ac1f9a9bd78/src/tests/null/haskell/Makefile.am#L10-L13

Expected behavior
I expected cabal v2-install --lib --package-env directory/ to work as a way to install a library for local use (similar to past sandbox use). I am not sure if this use-case is meant to be covered in the env command in the future, as described at #6481 .

System information

• Ubuntu 22.04
• cabal-install 3.4.1.0 and 3.6.2.0
• GHC 9.0.2 and 9.2.4

Additional context
The goal is to ensure the build occurs with local dependencies, without an internet connection. This is basic repeatable build functionality.

[fgaz]

So, if I understand correctly you don't need to expose those libraries to the user. Rather, you just want to have them built in advance for later compilation of executables.

If that's the case, I suggest simply having a cabal.project that references all your packages. If you want to pre-cache the dependencies for later use, you can pass --only-download or --only-dependencies to v2-build.
Other than that, there is no need for preparatory install steps. In particular, avoid using install --lib for this use case, since that flag is for exposing packages to the user and to external tools. Actually you could shorten the *_haskell targets to this one-liner:

count_haskell: Main.hs
	$(CABAL) --store-dir=$(CABAL_SANDBOX) v2-install \
                 --project-file=path/to/cabal.project \
                 --with-compiler=$(GHC) \
                 --installdir=. \
                 --install-method=copy \
                 count_haskell

I left --store-dir there, but removing is almost always better because it allows cabal to cache stuff globally, and with v2- there will not be conflicts. You can also omit --install-method=copy if a symlink is enough, and --project-file=PATH if your project file can be found in the working directory or any parent

[okeuday]

@fgaz I added your suggestion as shown at:
https://github.com/CloudI/CloudI/blob/019293798a0f829fbedda71505c81744cc2a0f48/src/tests/count/haskell/Makefile.am#L11-L17

However, the output is:

make[4]: Entering directory '/home/george/src/CloudI/src/tests/count/haskell'
echo "packages: "./count_haskell.cabal" "../../../api/haskell/cloudi.cabal > cabal.project
/home/george/installed/ghc-9.0.2/bin/cabal --store-dir=/home/george/src/CloudI/src/api/haskell/sandbox v2-install \
                 --project-file=./cabal.project \
                 --with-compiler=/home/george/installed/ghc-9.0.2/bin/ghc \
                 --installdir=. \
                 --install-method=copy \
                 count_haskell
Wrote tarball sdist to
/home/george/src/CloudI/src/tests/count/haskell/dist-newstyle/sdist/Count-2.0.4.tar.gz
Wrote tarball sdist to
/home/george/src/CloudI/src/tests/count/haskell/dist-newstyle/sdist/cloudi-2.0.4.tar.gz
Resolving dependencies...
Build profile: -w ghc-9.0.2 -O1
In order, the following will be built (use -v for more details):
 - hsc2hs-0.68.8 (exe:hsc2hs) (requires build)
 - zlib-0.6.3.0 (lib) (requires build)
 - network-3.1.2.7 (lib:network) (requires build)
 - cloudi-2.0.4 (lib) (requires build)
 - Count-2.0.4 (exe:count_haskell) (requires build)
Starting     hsc2hs-0.68.8 (exe:hsc2hs)
Starting     zlib-0.6.3.0 (lib)
Building     hsc2hs-0.68.8 (exe:hsc2hs)
Building     zlib-0.6.3.0 (lib)
Installing   zlib-0.6.3.0 (lib)
Completed    zlib-0.6.3.0 (lib)
Installing   hsc2hs-0.68.8 (exe:hsc2hs)
Completed    hsc2hs-0.68.8 (exe:hsc2hs)
Starting     network-3.1.2.7 (all, legacy fallback)
Building     network-3.1.2.7 (all, legacy fallback)
Installing   network-3.1.2.7 (all, legacy fallback)
Completed    network-3.1.2.7 (all, legacy fallback)
cabal: Failed to unpack cloudi-2.0.4 (which is required by exe:count_haskell
from Count-2.0.4). The exception was:
Invalid file name in tar archive: "cloudi-2.0.4/../../"

make[4]: *** [Makefile:659: count_haskell] Error 1
make[4]: Leaving directory '/home/george/src/CloudI/src/tests/count/haskell'

It is unclear why it would need to build what was previously built (in the api/haskell directory). It is also unclear why it would need a sdist tarball before the build. However, the error is due to "../.." getting into names in the tarball. Is there a way to avoid this failure?

[okeuday]

Removing the sandbox functionality in cabal-install appears to have opened a portal into the "Cabal hell" that existed in the past. I removed the cabal-install usage I previously had to avoid the problem by doing the build manually (CloudI/CloudI@c8a80c7).

[ivanperez-keera]

I'm having the same issue.

Having to write a separate cabal.project is quite annoying. Being able to just call runhaskell or ghci on a haskell file with all packages in the sandbox exposed was great for quick tests, debugging, DSLs, learning/inexperienced users, etc., as well as to help haskell get adopted (adding a requirement to have a cabal.project just makes the learning curve much steeper and the immediate cost by potential adopters higher).

I know this was closed long ago but I don't think this is solved more generally and it's still important that we find a good solution for these use cases.

[fgaz]

@ivanperez-keera I think you're referring to #9581. If you want to discuss more, let's continue there, where it's more on-topic.

---

@okeuday The original goal was offline builds, right? There is --only-download now, you could use that to fetch the dependencies in advance

[okeuday]

@fgaz Not using cabal-install has been best to ensure consistent behavior is possible. There never was a clear replacement for the sandbox functionality and there weren't dependable releases of cabal-install (that could be compiled from source in a consistent way), so I would prefer to avoid cabal-install in the future whenever possible.

[ivanperez-keera]

@okeuday It's a pity that that was ultimately necessary, and it's what I've been fearing more and more.

Can I trouble you to please go to #9581 and express that sandboxes were useful to you, and also that the reliability, consistency and stability of cabal-install was also important for your use case (if you agree with that; it's what I'm reading from your message).

I've been trying to get the Haskell community to focus more in that direction, but I'm struggling to gain support because the people who care about that don't always come to github to comment.

I've also left a comment here #7548 (comment) that tries to instill the same ideas. If you agree with that message, it would help if you could express it there.

Thanks!

[andreabedini]

I've been trying to get the Haskell community to focus more in that direction, but I'm struggling to gain support because the people who care about that don't always come to github to comment.

@ivanperez-keera Just chiming in to say that your voice is heard, at least by me but also by many others.

[hasufell]

Removal of sandboxes was a mistake.

Given that cabal-install is split into several packages, I think it might make sense to simply write a different interface to Cabal from scratch instead of trying to re-introduce it into cabal-install.

That is also in line with #9063

[andreabedini]

TBH cabal-install is also a library so you could reuse most of cabal-install without having to go down to Cabal. There is a chance the removed code would still mostly work.