-
Notifications
You must be signed in to change notification settings - Fork 35
PyOP2 on cx1
This page gives a detailed tutorial on how to build PyOP2 on the Imperial College CX1 compute cluster and run the provided example applications.
Follow the detailed instructions in the OP2-Common wiki.
git clone git://github.com/OP2/PyOP2.git
cd PyOP2
Create a file .env with the following content (adjust OP2_DIR to your OP2-Common installation location):
module load cmake/2.8.9
module load cuda/5.0.35
module load intel-suite/11.1
module load git/1.8.1
module load mpi/intel-3.1
module load python/2.7.3
module load opencl/intel-2.0
# load this ONLY if you're not building your own PETSc
module load petsc/3.3-p3-intel-11
export OP2_DIR=/path/to/OP2-Common/op2
and source it with . .env.
The easiest way to install PyOP2 dependencies is by setting up a virtualenv, which also make sure these packages don't interfere with your regular Python environment.
Install virtualenv (if you don't have it already):
git clone git://github.com/pypa/virtualenv.git
cd virtualenv/
python setup.py install --user
Create a virtualenv in your PyOP2 tree:
python -m virtualenv --distribute .venv
and activate it:
. .venv/bin/activate
You can now use pip to install packages in your virtualenv!
To leave the virutalenv, use deactivate.
When not using a virtualenv, use pip install --user
pip install codepy cython decorator instant pyyaml pytest ply jinja2 mako numpy
pip install hg+https://bitbucket.org/eliben/pycparser#egg=pycparser-2.09.1
If you want OpenMP support, don't use the petsc/3.3 module. Build PETSc 3.3 with OpenMP support:
LD_PRELOAD= CC= CXX= FC= F77= F90= \
PETSC_CONFIGURE_OPTIONS="--with-fortran-interfaces=1 --with-c++-support --with-openmp --download-f-blas-lapack=1 CC=mpicc CXX=mpicxx FC=mpifc F77=mpif77 F90=mpif90" \
pip install hg+https://bitbucket.org/ggorman/petsc-3.3-omp
You need to build our petsc4py fork:
pip install hg+https://bitbucket.org/fr710/petsc4py#egg=petsc4py
Note: Make sure the bzr/2.5.0 module is not loaded when you're building PETSc. Otherwise the PETSc build system will pick up the bzrlib commands module instead of the commands module from the Python standard library.
Create a file ~/.aksetup-defaults.py or siteconf.py with the following content:
BOOST_INC_DIR = ['/apps/boost/1.52.0/build/boost_1_52_0']
BOOST_LIB_DIR = ['/apps/boost/1.52.0/build/boost_1_52_0/stage/lib']
BOOST_COMPILER = 'gcc43'
USE_SHIPPED_BOOST = False
BOOST_PYTHON_LIBNAME = ['boost_python']
BOOST_THREAD_LIBNAME = ['boost_thread']
CUDA_TRACE = False
CUDA_ROOT = '/apps/cuda/5.0.35'
CUDA_ENABLE_GL = False
CUDA_ENABLE_CURAND = True
CUDADRV_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CUDADRV_LIBNAME = ['cuda']
CUDART_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CUDART_LIBNAME = ['cudart']
CURAND_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CURAND_LIBNAME = ['curand']
CXXFLAGS = []
LDFLAGS = []
Install PyCUDA from source:
git clone http://git.tiker.net/trees/pycuda.git
cd pycuda
git submodule init
git submodule update
./configure.py
LIBRARY_PATH=${PYTHON_HOME}/lib python setup.py install
The Cusp library is required for using op2.solve with the CUDA backend. You will also need to tell to look for cusp by adding the following after line 457 in pyop2.cuda.py:
nvcc_toolchain.add_library('cusp',['<path to cusp>'],[],[])
Since the OpenCL headers and library are in a non-standard location, PyOpenCL needs to be configured and installed by hand:
git clone http://git.tiker.net/trees/pyopencl.git
cd pyopencl
git submodule init
git submodule update
./configure.py --cl-inc-dir=${OPENCL_HOME}/usr/include --cl-lib-dir=${OPENCL_HOME}/usr/lib64
LIBRARY_PATH=${PYTHON_HOME}/lib python setup.py install
Install FFC and all dependencies via pip:
pip install \
bzr+http://bazaar.launchpad.net/~mapdes/ffc/pyop2#egg=ffc \
bzr+http://bazaar.launchpad.net/~florian-rathgeber/ufc/python-setup#egg=ufc_utils \
bzr+http://bazaar.launchpad.net/~ufl-core/ufl/main#egg=ufl \
bzr+http://bazaar.launchpad.net/~fiat-core/fiat/main#egg=fiat \
https://sourcesup.renater.fr/frs/download.php/2309/ScientificPython-2.8.tar.gz
pycuda-2012.1-py2.7-linux-x86_64.egg/pycuda/_driver.so: undefined symbol: cuIpcGetMemHandle
You get this error when trying import the CUDA backend
tau_exec -T serial,cupti -cupti ./a.out