libctl version conflict in freshly installed conda package?

[wbogaert]

I tried to install Meep as a conda package (Ubuntu 20.04LTS with all updates + miniconda 3), according to the very simple instructions in the docs. Encountered the same problem on Windows Subsystem for Linux with the same Ubuntu and miniconda3.

However, there seems to be a version problem with some libctl packages.

When I perform 'import meep' I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/miniconda3/envs/mp/lib/python3.6/site-packages/meep/__init__.py", line 8, in <module>
    from . import _meep
ImportError: libctlgeom.so.6: cannot open shared object file: No such file or directory

Running locate showed that the conda environment has libctlgeom.so.7, but not 6

I would expect that the dependencies are correctly defined in the conda manifest. Any suggestions which packages to upgrade/downgrade to make this work?

Thanks!

[oskooi]

I just tested the PyMeep Conda package (version 1.22) on a fresh c5.large instance on AWS EC2 running Ubuntu 18.04 (ami-0e472ba40eb589f49) and it seems to be running fine.

Here are the exact set of commands I used:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.6 LTS
Release:	18.04
Codename:	bionic

$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
$ bash miniconda.sh -b
$ export PATH="/home/ubuntu/miniconda3/bin:${PATH}"
$ conda create -n pmp -c conda-forge pymeep=*=mpi_mpich_* pymeep-extras
$ conda init bash
$ . .bashrc
$ conda activate pmp

$ git clone https://github.com/NanoComp/meep.git
$ cd meep/python/examples
$ mpirun -np 2 python3 binary_grating.py
Using MPI version 3.1, 2 processes
-----------
Initializing structure...
Halving computational cell along direction y
Splitting into 2 chunks by voxels
time for choose_chunkdivision = 0.000310183 s
Working in 2D dimensions.
Computational cell is 8.5 x 10 x 0 with resolution 60
time for set_epsilon = 0.164703 s
-----------
on time step 5376 (time=44.8), 0.000744175 s/step
field decay(t = 50.00833333333333): 0.11392641853211995 / 0.11392641853211995 = 1.0
on time step 10937 (time=91.1417), 0.000719375 s/step
field decay(t = 100.01666666666667): 1.011578498395928e-15 / 0.11392641853211995 = 8.879226709920031e-15
run 0 finished at t = 100.01666666666667 (12002 timesteps)
-----------
Initializing structure...
Halving computational cell along direction y
Splitting into 2 chunks by voxels
time for choose_chunkdivision = 0.000403166 s
Working in 2D dimensions.
Computational cell is 8.5 x 10 x 0 with resolution 60
     block, center = (-2.25,0,0)
          size (4,1e+20,1e+20)
          axes (1,0,0), (0,1,0), (0,0,1)
          dielectric constant epsilon diagonal = (2.25,2.25,2.25)
     block, center = (0,0,0)
          size (0.5,5,1e+20)
          axes (1,0,0), (0,1,0), (0,0,1)
          dielectric constant epsilon diagonal = (2.25,2.25,2.25)
time for set_epsilon = 0.298062 s
-----------
on time step 5559 (time=46.325), 0.000719672 s/step

[wbogaert]

Thanks. This exact sequence of steps seems to work. I just followed the steps from the documentation:

$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
$ bash miniconda.sh -b
$ export PATH="/home/ubuntu/miniconda3/bin:${PATH}"
$ conda create -n mp -c conda-forge pymeep
$ conda activate mp
$ python3
>>> import meep

Which then gave me the error.

Your installation gives me quite a different list of package versions.

Packages (conda list) in my installation:

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
blas                      1.1                    openblas    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
certifi                   2021.5.30        py36h5fab9bb_0    conda-forge
fftw                      3.3.10          nompi_h77c792f_102    conda-forge
gsl                       2.2.1           blas_openblashddceaf2_6  [blas_openblas]  conda-forge
h5utils                   1.13                          4    chogan
harminv                   1.4.1                         1    chogan
hdf5                      1.10.1                        2    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libctl                    4.2.0                h14c3975_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 11.2.0              h1d223b6_12    conda-forge
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            11.2.0              h69a702a_12    conda-forge
libgfortran5              11.2.0              h5c6108e_12    conda-forge
libgomp                   11.2.0              h1d223b6_12    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_12    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
mpb                       1.6.1                         0    chogan
ncurses                   6.3                  h9c3ff4c_0    conda-forge
numpy                     1.15.4          py36_blas_openblashb06ca3d_0  [blas_openblas]  conda-forge
openblas                  0.3.3                ha44fe06_1    conda-forge
openssl                   1.1.1l               h7f98852_0    conda-forge
pip                       20.0.2                   py36_1    conda-forge
pymeep                    1.4.3                    py36_1    chogan
python                    3.6.15          hb7a2778_0_cpython    conda-forge
python_abi                3.6                     2_cp36m    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                49.6.0           py36h5fab9bb_3    conda-forge
sqlite                    3.37.0               h9cd32fc_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h36c2ea0_1013    conda-forge

Packages in your installation:

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
aom                       3.2.0                h9c3ff4c_2    conda-forge
autograd                  1.3                        py_0    conda-forge
brotli                    1.0.9                h7f98852_6    conda-forge
brotli-bin                1.0.9                h7f98852_6    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
certifi                   2021.10.8       py310hff52083_1    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
ffmpeg                    4.4.1                h6987444_1    conda-forge
fftw                      3.3.10          mpi_mpich_h8b92f0a_2    conda-forge
fonttools                 4.29.1          py310h6acc77f_0    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
future                    0.18.2          py310hff52083_4    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
gsl                       2.7                  he838d99_0    conda-forge
h5py                      3.6.0           mpi_mpich_py310h83ac6b3_0    conda-forge
h5utils                   1.13.1          mpi_mpich_h88c8ee5_1013    conda-forge
harminv                   1.4.1             hbf3b5f1_1007    conda-forge
hdf5                      1.12.1          mpi_mpich_ha8a74db_3    conda-forge
icu                       69.1                 h9c3ff4c_0    conda-forge
jbig                      2.1               h7f98852_2003    conda-forge
jpeg                      9e                   h7f98852_0    conda-forge
kiwisolver                1.3.2           py310h91b1402_1    conda-forge
krb5                      1.19.2               h48eae69_3    conda-forge
lame                      3.100             h7f98852_1001    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0           13_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h7f98852_6    conda-forge
libbrotlidec              1.0.9                h7f98852_6    conda-forge
libbrotlienc              1.0.9                h7f98852_6    conda-forge
libcblas                  3.9.0           13_linux64_openblas    conda-forge
libctl                    4.5.1                h7f98852_0    conda-forge
libcurl                   7.81.0               h494985f_0    conda-forge
libdeflate                1.10                 h7f98852_0    conda-forge
libdrm                    2.4.109              h7f98852_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_12    conda-forge
libgdsii                  0.21                 h4bd325d_4    conda-forge
libgfortran-ng            11.2.0              h69a702a_12    conda-forge
libgfortran5              11.2.0              h5c6108e_12    conda-forge
libgomp                   11.2.0              h1d223b6_12    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0           13_linux64_openblas    conda-forge
libnghttp2                1.46.0               ha19adfc_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.18          pthreads_h8fe5266_0    conda-forge
libpciaccess              0.16                 h516909a_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libssh2                   1.10.0               ha35d2d1_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_12    conda-forge
libtiff                   4.3.0                h542a066_3    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libva                     2.14.0               h7f98852_0    conda-forge
libvpx                    1.11.0               h9c3ff4c_3    conda-forge
libwebp                   1.2.2                h3452ae3_0    conda-forge
libwebp-base              1.2.2                h7f98852_1    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxml2                   2.9.12               h885dcf4_1    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
matplotlib-base           3.5.1           py310h23f4a51_0    conda-forge
mpb                       1.11.1          mpi_mpich_h6f879dd_4    conda-forge
mpi                       1.0                       mpich    conda-forge
mpi4py                    3.1.3           py310h853ac07_0    conda-forge
mpich                     3.4.3              h846660c_100    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.3                  h9c3ff4c_0    conda-forge
nettle                    3.6                  he412f7d_0    conda-forge
numpy                     1.22.2          py310h454958d_0    conda-forge
openh264                  2.1.1                h780b84a_0    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   3.0.0                h7f98852_2    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pillow                    9.0.1           py310he619898_2    conda-forge
pip                       22.0.3             pyhd8ed1ab_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pymeep                    1.22.0          mpi_mpich_py310h1234567_1    conda-forge
pymeep-extras             1.22.0          mpi_mpich_py310h23d43cc_1    conda-forge
pyparsing                 3.0.7              pyhd8ed1ab_0    conda-forge
python                    3.10.2          hc74c709_3_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    2_cp310    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
scipy                     1.8.0           py310hea5193d_1    conda-forge
setuptools                60.9.3          py310hff52083_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.37.0               h9cd32fc_0    conda-forge
svt-av1                   0.9.0                h27087fc_1    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
unicodedata2              14.0.0          py310h6acc77f_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
x264                      1!161.3030           h7f98852_1    conda-forge
x265                      3.5                  h4bd325d_1    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h36c2ea0_1013    conda-forge
zstd                      1.5.2                ha95c52a_0    conda-forge

A lot more packages (which I assume has to do with the MPI support and pymeep-extras) and higher versions (including a different python version)

Not sure what went wrong with the vanilla installation.

Anyway. It works. Thanks a lot.

[wbogaert]

And it also works under Windows through WSL.

[oskooi]

Note that you were trying to install the serial version of PyMeep whereas my instructions were for the parallel/MPI version. Regardless, when I try to install the serial version via conda create -n mp -c conda-forge pymeep, the list of packages is different from yours, in particular:

  harminv            conda-forge/linux-64::harminv-1.4.1-hbf3b5f1_1007
  libctl             conda-forge/linux-64::libctl-4.5.1-h7f98852_0
  mpb                conda-forge/linux-64::mpb-1.11.1-nompi_h70f4da8_104
  pymeep             conda-forge/linux-64::pymeep-1.22.0-nompi_py310h1234567_101

In your installation, you were using libctl 4.2.0 (January 2019) and the Conda packages for harminv, mpb 1.6.1 (January 2018), and pymeep 1.4.3 (February 2018) were from the chogan channel which has long been obsolete. The key for conda create is to use the -c conda-forge flag to specify the Conda forge channel.

[wbogaert]

Thanks Ardavan.

I realizethe difference between the serial version and the parallel version (I'm trying to set up some simple worker nodes for teaching - single-core operation is just fine for that).

But I did use the conda-forge channel during the installation. Not sure why it pulled in these obsolete packages.