SP 001: Common variant names

Proposal by Mark Miller, LLNL

There are a number of aspects of package installation that are frequently duplicated from package to package.

In Spack parlance, these might be best considered variants. However, in some sense they are global variants as they effect many packages in the same way.

I'd like to start a list of these here and keep track of this. Here's a start. Also, would like to keep variant tags short so here we try to use just 3-character variant names.

debug/optimized

• +opt means to build optimized (production) -O3, etc. (default)
• -opt means to build a package with (maximal) debugging features enabled (e.g. -g, -DNDEBUG, etc., maybe special profiling features enabled, etc.). Note that currently Spack automatically sets standard CMake switches to include CMAKE_BUILD_TYPE=RelWithDebInfo. That practice would be replaced with this new way of handling this as a variant (though the proposed variant in fact does not deal with the subtle RelWithDebInfo flavor)

static/dynamic libraries

• +dyn means to build both dynamic and static libraries in the install point
• -dyn means to build only static (default)

32bit/64bit

• +b64 means to build both 32 and 64 bit libraries (default)
• -b64 means to build only 32 bit

Interface Languages (fortran/cxx/c…)

• +fort means to build (all) fortran interfaces (where available) when fc compiler is specified (default)
• +cxx means to build C++ interfaces (where available) when cxx compiler is specified
• +c means to build C interfaces (default)
• `+pyth' means build python interfaces (where available) if python is also available

Note that C/CXX interface variant switches may be less relevant as typically whenever you have a C compiler, you also have a C++ compiler. The same hasn't always tended to be true for Fortran though (OS X for example) and so having that variant switch is important. OTOH, maybe specifying a fortran compiler is sufficient cause to enable fortran wherever its possible and all three of these variant switches can be argued away.

serial/parallel

• +par means to build both serial and parallel variants (default)
• -par means to build only serial variant

In a context where the only parallelism comes from MPI, this works. In a context where there are multiple forms of parallelism (threads, mpi, etc.) it gets more complicated and parallel may not be the appropriate moniker. Nonetheless, there are a lot of libraries out there that are designed to work in serial and (MPI) parallel. More thought is required here.

Libraries, Tools, Tests/Checks and Examples

• +test means to both build and run tests as part of installation (failing the installation if tests don't pass), default=False
• -tools means to NOT build/install support tools (for example, for HDF5, I don't wanna build/install support tools like h5ls on compute nodes, only for login nodes)
• +examples means to build/install examples

Other switches since starting this page

• handling of -PIC, -fPIC, -qpic and variants (already have a post in google groups about this). Its essential if the package you are building will ever be intended to be used in the creation of shared object (.so).

Variants for dependent libraries

In my experience, there are basically two kinds of dependent libraries for a package; required and optional. Required dependencies are already addressed by Spack and do not involve the use of variants. But, what about variants for optional dependencies? One possibility is to have a package support each such dependency as a variant. But, that seems like it could potentially get unwieldly. This is especially true if the dependency has the potential to ripple down through the dependency graph. Furthermore, with different packages possibly having different optional dependencies, a single spack command-line to build a given package could wind up getting quite long due to package-specific optional dependency variants. Is there any Spack shorthand we can develop to simplify this a bit and perhaps come up with a set of global variants that nonetheless help to specify how optional dependencies in any given package be handled?

What are the downsides of simply treating all optional dependencies as required? In other words, what if we opted to maximize optional dependencies? Well, in a nutshell, its less robust; since more things are being built, there is more chance of a failure either to build or to access a given package on the network; an application linking to the desired package must link more packages into the same executable and maybe not all the packages play well together; some packages are optional because they represent experimental capability that is not fully tested and/or works in limited contexts (e.g. compilers, MPI implementations, etc.). So, there are good reasons for Spack to, by default, minimize optional dependencies.

What are the downsides to minimizing optional dependencies? The key downside is that needed functionality isn't available in a given package requiring that package and possibly some portion of its dependencies to be re-built. As different users in a HPC stack install Spack is managing encounter the need for different combinations of optional dependent libraries, eventually that installation could wind up getting populated with the combinatoric expansion of all possible combinations of optional dependencies. Though Spack is designed to handle this situation, is it really an outcome we want to design towards?

Lets propose the following global variants for handling optional dependent packages

• variant('reqdep', default=False, 'Minimize dependencies. Build only with required dependent libraries')
• variant('stddep', defualt=True, 'Standard dependencies. Build with standard/common optional dependent libraries.')
• variant('alldep', default=False, 'Maximal dependencies. Build with all possible dependent libraries')

The meaning of reqdep should be obvious. It means to turn off all optional dependent libraries and build only with required dependent libraries. The meaning of stddep means to build the package in its standard or common configuration with those optional libraries the packages that the package is commonly used with. Furthermore, I am proposing this as the default. What constitutes the set of standard or common optional dependent packages can be determined on a package-specific basis. For example, the Silo library is commonly used with the HDF5 library so activation of the HDF5 dependency in Silo will occur by default but can be turned off by setting the variant '+reqdep'.

These are sort of mutually exclusive options in that setting one has implications for the others. It would be nicer to have a 3-value variant such as

• variant('optdeps', values=['off':'Minimize dependencies. Build only with required dependent libraries', 'std':'Standard dependencies. Build with standard/common optional dependent libraries.', 'all':'Maximal dependencies. Build with all possible dependent libraries']