Get Start

Introduction

The largest user base for Cactus is in the field of numerical relativity where, for example, over 100 components are now shared among over fifteen different groups through the Einstein Toolkit.

Cactus is a component framework. Its components are called thorns whereas the framework itself is called the flesh. The flesh is the core of Cactus, it provides the APIs for thorns to communicate with each other, and performs a number of administrative tasks at build–time and run–time.

Note

The Cactus API supports C/C++ and F77/F90 programming languages for the thorns. This makes it easier for scientists to turn existing codes into thorns which can then make use of the complete Cactus infrastructure, and in turn be used by other thorns within Cactus.

Flesh

At run-time the flesh parses a user provided parameter file which defines which thorns are required and provides key-value pairs of parameter assignments. The flesh then activates only the required thorns, sets the given parameters, using default values for parameters which are not specified in the parameter file, and creates the schedule of which functions provided by the activated thorns to run at which time.

../_images/flesh.png

Note

In order to ensure code portability, the flesh is written in ANSI C. Thorns, however, may be written in C, C++, FORTRAN 77 or Fortran 90. In order to ensure that thorns are platform independent, the configuration script determines various machine-specific details, such as the presence or absence of certain non-ANSI C functions or the sizes of variables, which vary from platform to platform – in order to avoid problems with this, e.g. in cross-language calls or parallel jobs distributed across heterogeneous platforms, Cactus defines a set of types, such as CCTK_REAL, CCTK_INT, which are guaranteed to have the same size on all platforms and in all languages. This also means that runs can produce exactly the same results on different machines, and checkpoint files created on one machine can be restarted on any other.

Driver

Drivers are responsible for memory management for grid variables, all parallel operations, Input/Output(IO) and mesh refinement.

Note

basic parallelisation operations:

  • ghost-zone synchronisation between sub-domains;
  • generalised reduction operators;
  • generalised interpolation operators.

There are two driver thorns: the unigrid PUGH driver and the adaptive mesh refinement (AMR) Carpet driver. Which driver is used is determined by which is activated at run-time.

Modularity

A thorn is the basic working module within Cactus. All user supplied code goes into thorns, which are, by and large, independent of each other. The connection from a thorn to the flesh or to other thorns is specified in configuration files that are parsed at compile time and used to generate glue code that encapsulates the external appearance of a thorn.

A thorn consists of a subdirectory of an arrangement containing four administrative files written in the Cactus Configuration Language (CCL):

  • interface.ccl: Defines the thorn interface and inheritance along with variables and aliased functions.
  • param.ccl: This defines the parameters that are used to control the thorn.
  • schedule.ccl: This defines which functions are called from the thorn and when they are called. It also handles memory and communication assignment for grid variables.

Thorns can also contain

  • configuration.ccl: This file is optional for a thorn. If it exists, it contains extra configuration options of this thorn.
  • a subdirectory called src, which should hold source files and compilation instructions for the thorn.
  • a subdirectory src/include for include files.
  • a README containing a brief description of the thorn.
  • a doc directory for documentation.
  • a par directory for example parameter files.
  • a test subdirectory may also be added, to hold the thorn’s test suite.

Thorns are grouped into arrangements. This is a logical grouping of thorns which is purely for organisational purposes. The arrangements live in the arrangements directory of the main Cactus directory.

There exists a special include source mechanism to generate include files from sections of user code, allowing inlining of code between different thorns. This mechanism arguably breaks the separation between thorns, but can lead to large performance gains if automatic, compiler-dependent cross-file inlining is not available or not reliable. The include source mechanism supports arbitrary languages, including Fortran, and the inlined code is surrounded by guards making it possible to enable or disable the inlined code at run time. One particular disadvantage of this mechanism is namespace pollution, similar to manually inlined code.

Accuracy

we made the following choices:

  • 4th order accurate finite differences,
  • 4th order accurate Runge-Kutta time integrator,
  • 3 timelevels for evolved grid functions,
  • 3 ghostzones for interprocess synchronization,
  • 5th order accurate spatial and 2nd order accurate temporal interpolation at mesh refinement boundaries,
  • 5th order Kreiss-Oliger dissipation terms added to the right hand side of the evolution equations,

Visualization

For visualizing 1-dimensional ASCII output, standard tools like matplotlib are often used; for 2- and 3-dimensional HDF5 output, VisIt is popular (freely available) options.

I written a python library named CactusTool The source code for CactusTool is hosted on GitHub at https://github.com/YuLiumt/CactusTool

You can clone it with

$ git clone https://github.com/YuLiumt/CactusTool.git

and run

$ pip install -e .

Install

Users of the Einstein Toolkit are encouraged to register to become one of its.

Required Software

The main requirements are:

  • Client tools for Source Code Repositories: CVS, SVN and git
  • Compilers: C, C++ and Fortran 90
  • MPI implementation: This is needed for the Carpet driver
  • Standard development tools: Perl, etc.

Linux

# On Debian/Ubuntu/Mint use this command:
$ sudo apt-get install -y subversion gcc git numactl libgsl-dev libpapi-dev python libhwloc-dev make libopenmpi-dev libhdf5-openmpi-dev libfftw3-dev libssl-dev liblapack-dev g++ curl gfortran patch pkg-config libhdf5-dev libjpeg-turbo?-dev
# On Fedora use this command:
$ sudo dnf install -y libjpeg-turbo-devel gcc git lapack-devel make subversion gcc-c++ which papi-devel python hwloc-devel openmpi-devel hdf5-openmpi-devel openssl-devel libtool-ltdl-devel numactl-devel gcc-gfortran findutils hdf5-devel fftw-devel patch gsl-devel pkgconfig
$ module load mpi/openmpi-x86_64 # You will have to repeat the module load command each time you would like to compile or run the code.
# On Centos use this command:
$ sudo yum install -y epel-release
$ sudo yum install -y libjpeg-turbo-devel gcc git lapack-devel make subversion gcc-c++ which papi-devel hwloc-devel openmpi-devel hdf5-openmpi-devel openssl-devel libtool-ltdl-devel numactl-devel gcc-gfortran hdf5-devel fftw-devel patch gsl-devel

Mac

MacPorts

  1. Install Xcode from the Apple App Store. In addition agree to Xcode license in Terminal

    $ sudo xcodebuild -license

  2. install MacPorts for your version of the Mac operating system, if you did not already install it

  3. Next, please install the following packages, using the commands:

    $ sudo port -N install pkgconfig gcc9 openmpi-gcc9 fftw-3 gsl jpeg zlib hdf5 +fortran +gfortran openssl ld64 +ld64_xcode
$ sudo port select mpi openmpi-gcc9-fortran
Homebrew

  1. Install Homebrew

    $ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

  2. Next, please install the following packages, using the commands:

    $ brew install gnuplot pkg-config gcc fftw gsl hdf5 hwloc jpeg openssl pkg-config szip open-mpi

GetComponents Tools

A script called GetComponents is used to fetch the components of the Einstein Toolkit. You may download and make it executable it as follows:

$ cd ~/
$ curl -kLO https://raw.githubusercontent.com/gridaphobe/CRL/ET_2019_03/GetComponents
$ chmod u+x GetComponents

Below checks out Cactus, the Einstein Toolkit thorns, the Simulation Factory and example parameter files into a directory named Cactus.

$ ./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2019_03/einsteintoolkit.th

Error

  • Some versions of svn might show problems with the parallel checkout. If you see errors like (svn: E155037: Previous operation has not finished), try without the –parallel option.

  • svn: E155004: Could not update module <Thorn>

    $ cd <Thorn_Path>
$ svn cleanup

The traditional way of compiling Cactus

The traditional way of compiling Cactus without SimFactory.

$ make ET-config options=<*.cfg> THORNLIST=<thornlist>

This creates a configuration called “ET”, but any other name could be chosen here.

Once the configuration is done, the compilation process is simply

$ make -j <number of processes> ET

If everything is compiled correctly, the executable cactus_ET will be created under ./exe/.

The typical procedure for running is

$ mpirun -np <num procs> ./exe/cactus_ET <parameter file>

SimFactory Tools

SimFactory needs to be configured before it can be used. The recommended way to do this is to use the setup command. SimFactory contains general support for specific operating systems commonly used on workstations or laptops, including Mac OS, Ubuntu, Cent OS and Scientific Linux. To configure SimFactory for one of these, you need to find the suitable files in simfactory/mdb/optionlists and simfactory/mdb/runscripts and specify their names on the sim setup command line.

Note

Newer versions do not have ubuntu.cfg. All flavors of linux should compile with generic.cfg.

# for Debian
$ ./simfactory/bin/sim setup-silent --optionlist=debian.cfg --runscript debian.sh
# for Ubuntu, Mint
$ ./simfactory/bin/sim setup-silent --optionlist=ubuntu.cfg --runscript debian.sh
# for Fedora (you may have to log out and back in if you have just intalled mpich to make the module command work)
$ module load mpi
$ ./simfactory/bin/sim setup-silent --optionlist=fedora.cfg --runscript debian.sh
# OSX+MacPorts
$ ./simfactory/bin/sim setup-silent --optionlist=osx-macports.cfg --runscript osx-macports.run
# OSX+Homebrew
$ ./simfactory/bin/sim setup-silent --optionlist=osx-homebrew.cfg --runscript generic-mpi.run

Note

Generally, configuring SimFactory means providing an optionlist, for specifying library locations and build options, a submit script for using the batch queueing system, and a runscript, for specifying how Cactus should be run, e.g. which mpirun command to use.

Follow the on-screen prompts. This will output your choices in the configuration file simfactory/etc/defs.local.ini.

Note

It is likely that you will have to further customise this file. You can see some possible option settings in simfactory/etc/defs.local.ini.example.

SimFactory needs to have a machine definition for every machine that it is run on. If you are using a machine that SimFactory already has a definition for, such as a well-known supercomputer used by others in the Cactus community, then no additional setup is required. If, however, you are running SimFactory on an individual laptop or on an unsupported supercomputer, the setup command will also create a new machine definition for the local machine in ./repos/simfactory2/mdb/machines/<hostname>.ini. You may also have to add extra information to this file.

Note

A machine consists of a certain number of nodes, each of which consists of a certain number of cores.

The user chooses the total number of threads (–procs). The user can also choose the number of threads per process (–num-threads) and the number of threads per core (–num-smt). Additionally, the user can also specify the number of cores per node (–ppn) and the number of threads per node (–ppn-used). The number of nodes is always chosen automatically.

Note that nodes and cores are requested from the queuing system, while processes and threads are started by SimFactory.

For more details you can see https://simfactory.bitbucket.io/simfactory2/userguide/processterminology.html

Note

The main SimFactory binary is called “sim” and is located in simfactory/bin. You can execute SimFactory explicitly as ./simfactory/bin/sim, but we recommend that you set up a shell alias in your shell startup file so that you can just use the command “sim”. For bash users this file is .bashrc on Linux. Add the following to the shell startup file:

$ alias sim=./simfactory/bin/sim

Error

  • checking whether the C compiler (gcc-8 -g -std=c11 -lgfortran) works… no

Building the Einstein Toolkit

Assuming that SimFactory has been successfully set up on your machine, you should be able to build the Einstein Toolkit with

$ ./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist ../einsteintoolkit.th | cat

The most important argument to this command is the –thornlist option, as this tells Cactus which thorns from your source tree you want to include in the configuration.

Note

Note that, typically, one will not be compiling all the thorns provided with the ET. Compilation is time-consuming, and different configurations also take a significant amount of disk space. One therefore typically builds a thornlist that is as small as possible, including only the required thorns. Care should be taken, though, as there are often non-trivial dependencies between thorns. If one thorn which is required by another thorn is not mentioned in the thornlist, compilation will abort (with the corresponding error message).

Running the Einstein Toolkit

Simulations must always be created before they can be submitted or run. Since it is very common to want to create a simulation and immediately submit or run it, SimFactory provides the create-run and create-submit commands. These commands create the simulation and then either run or submit it immediately.

Note

If you are working on a laptop or workstation, you can run SimFactory simulations directly in your terminal without going via a queuing system. However, if you are running SimFactory on a supercomputer with a queuing system, you cannot run simulations directly using the run command - they must instead be submitted to the queuing system, such as the PBS queuing system.

SimFactory needs to know a name for the simulation as well as what parameter file to use. You can either specify the name on the command line and give the parameter file with the –parfile option.

example

$ ./simfactory/bin/sim create-run static_tov --parfile=par/static_tov_small_short.par --procs=2 --num-threads=1 --ppn-used=2  --walltime=8:0:0 | cat

It is often useful to use a parameter file script, rather than a parameter file, as a basic description of a simulation. For example, when performing a convergence test, many parameters might change between simulations, and changing them all manually is tedious and error-prone.

A parameter file script is a file with a “.rpar” extension which, when executed, generates a file in the same place but with a “.par” extension. You can write a parameter file script in any language.

We provide examples in python.
#!/usr/bin/env python

import sys
import re
from string import Template

dtfac = 0.5

lines = """
Time::dtfac                   = $dtfac
"""

data = open(re.sub(r'(.*)\.rpar$', r'\1.par', sys.argv[0]), 'w')
data.write(Template(lines).substitute(locals()))

These parameter file scripts look like standard Cactus parameter files but with $var variable replacements (in this case for the dtfac variable). You can define new variables and do calculations in the header and use the variables in the main body.

If you want to use the Cactus $parfile syntax, you need to escape the $

IO::out_dir = $$parfile