Thorn Write¶

Naming Conventions:

Thorn names must not start with the word “Cactus” (in any case).
Routine names have to be unique among all thorns.
Names of global variables have to be unique among all thorns.

The following naming conventions are followed by the flesh and the supported Cactus arrangements. They are not compulsory.

Parameters: lower case with words separated by an underscore. Examples: my_first_parameter.
Routine names and names of global variables: Prefixed by thorn name with an underscore, then capitalised words, with no spaces. Examples: MyThorn_StartUpRoutine.

A brief introduction to C¶

C is a programming language originally developed for developing the Unix operating system. It is a low-level and powerful language, but it lacks many modern and useful constructs. C++ is a newer language, based on C, that adds many more modern programming language features that make it easier to program than C.

A C program should be written into one or more text files with extension “.c”.

A C program basically consists of the following parts:

Preprocessor Commands
Functions
Variables
Statements & Expressions
Comments

Variables¶

A variable is nothing but a name given to a storage area that our programs can manipulate. The name of a variable can be composed of letters, digits, and the underscore character. It must begin with either a letter or an underscore. Upper and lowercase letters are distinct because C is case-sensitive.

Each variable in C has a specific type, which determines how much space it occupies in storage and how the bit pattern stored is interpreted.

Type	Description
char	A single character.
int	An integer.
float	A single-precision floating point value.
double	A double-precision floating point value.
void	Represents the absence of type.

To declare a variable you use

<variable type> <name of variable>;

Variables can be initialized (assigned an initial value) in their declaration.

<variable type> <name of variable> = <value of variable>;

Constants¶

Constants refer to fixed values that the program may not alter during its execution. There are two simple ways in C to define constants

Using #define preprocessor.
Using const keyword.

The #define Preprocessor¶

Given below is the form to use #define preprocessor to define a constant

#define <identifier> <value>

The const Keyword¶

The word “const” in front of a type name means that the variable is constant and thus its value cannot be modified by the program. Constant variables are initialised when they are declared:

const <variable type> <name of variable> = <value of variable>;

Arrays¶

Arrays are special variables which can hold more than one value under the same variable name, organised with an index.

To declare an array in C, a programmer specifies the type of the elements and the number of elements required by an array as follows

<type> <arrayName> [ <arraySize> ];

The arraySize must be an integer constant greater than zero and type can be any valid C data type.

Initializing Arrays¶

You can initialize an array in C either one by one or using a single statement as follows

double balance[5] = {1000.0, 2.0, 3.4, 7.0, 50.0};

The number of values between braces { } cannot be larger than the number of elements that we declare for the array between square brackets [ ]. If you omit the size of the array, an array just big enough to hold the initialization is created.

Accessing Array Elements¶

An element is accessed by indexing the array name. For example

double salary = balance[9];

The above statement will take the \(10^{th}\) element from the array and assign the value to salary variable.

Strings¶

Strings are actually one-dimensional array of characters terminated by a null character ‘0’.

The following declaration and initialization create a string consisting of the word “Hello”. To hold the null character at the end of the array, the size of the character array containing the string is one more than the number of characters in the word “Hello”.

char greeting[6] = {'H', 'e', 'l', 'l', 'o', '\0'};

If you follow the rule of array initialization then you can write the above statement as follows

char greeting[] = "Hello";

Structures¶

When programming, it is often convenient to have a single name with which to refer to a group of a related values. Structures provide a way of storing many different values in variables of potentially different types under the same name.

Defining a Structure¶

To define a structure, you must use the struct statement. The struct statement defines a new data type, with more than one member. The format of the struct statement is as follows

struct <tag> {
    <members>;
};

Where Tag is the name of the entire type of structure and Members are the variables within the struct.

Accessing Structure Members¶

To access any member of a structure, we use the member access operator ‘.’.

struct <tag> <structure_variables>; // Declare <structure_variables> of type <tag>
<structure_variables>.<members>

Pointers to Structures¶

You can define pointers to structures in the same way as you define pointer to any other variable

struct <tag> *<struct_pointer>;

Now, you can store the address of a structure variable in the above defined pointer variable. To find the address of a structure variable, place the ‘&’; operator before the structure’s name as follows

<struct_pointer> = &<structure_variables>;

To access the members of a structure using a pointer to that structure, you must use the -> operator as follows

<struct_pointer> -> <members>

typedef¶

The C programming language provides a keyword called typedef, which you can use to give a type a new name.

typedef <existing_name> <alias_name>

Local Variable¶

Variables can be declared at any point in the code, provided that, of course, they are declared before they are used. The declaration is valid only within the local block, i.e. within the region limited by braces “{ }”.

Size¶

To get the exact size of a type or a variable on a particular platform, you can use the sizeof operator.

int a
sizeof(a)

Type Casting¶

Converting one datatype into another is known as type casting or, type-conversion. To typecast something, simply put the type of variable you want the actual variable to act as inside parentheses in front of the actual variable.

(type_name) expression

Functions¶

A function is simply a collection of commands that do “something”. You can either use the built-in library functions or you can create your own functions.

They must all be declared before there is a call to them. A function declaration tells the compiler about a function’s name, return type, and a list of arguments.

<return_type> <function_name>(<arguments>)

A function definition provides the actual body of the function.

<return_type> <function_name>(<arguments>) {
    /* Here goes your code */
}

Here are all the parts of a function:

Return Type - A function may return a value. The return_type is the data type of the value the function returns. Some functions perform the desired operations without returning a value. In this case, the return_type is the keyword void.
Function Name - This is the actual name of the function.
Arguments - When a function is invoked, you pass a value to the parameter. This value is referred to as actual parameter or argument. The parameter list refers to the type, order, and number of the parameters of a function.
Function Body - The function body contains a collection of statements that define what the function does.

Note

In C, arguments are copied by value to functions, which means that we cannot change the arguments to affect their value outside of the function. To do that, we must use pointers.

Pointers¶

As you know, every variable is a memory location and every memory location has its address defined which can be accessed using ampersand (&) operator, which denotes an address in memory.

A pointer is a variable whose value is the address of another variable, i.e., direct address of the memory location. Like any variable or constant, you must declare a pointer before using it to store any variable address.

<type> *<varname>;

Here, type is the pointer’s base type; it must be a valid C data type and varname is the name of the pointer variable.

The cool thing is that once you can talk about the address of a variable, you’ll then be able to go to that address and retrieve the data stored in it, use the *. The technical name for this doing this is dereferencing the pointer.

NULL Pointers¶

It is always a good practice to assign a NULL value to a pointer variable in case you do not have an exact address to be assigned. This is done at the time of variable declaration.

Memory allocation¶

The function malloc, residing in the stdlib.h header file, is used to initialize pointers with memory from free store. The argument to malloc is the amount of memory requested (in bytes), and malloc gets a block of memory of that size and then returns a pointer to the block of memory allocated.

Since different variable types have different memory requirements, we need to get a size for the amount of memory malloc should return. So we need to know how to get the size of different variable types. This can be done using the keyword sizeof, which takes an expression and returns its size. For example,

#include <stdlib.h>

int *ptr = malloc(sizeof(int));

This code set ptr to point to a memory address of size int. The memory that is pointed to becomes unavailable to other programs. This means that the careful coder should free this memory at the end of its usage lest the memory be lost to the operating system for the duration of the program (this is often called a memory leak because the program is not keeping track of all of its memory). The free function returns memory to the operating system.

free(ptr);

Passing pointers to functions in C¶

C programming allows passing a pointer to a function. To do so, simply declare the function parameter as a pointer type.

Return pointer from functions in C¶

Operators¶

An operator is a symbol that tells the compiler to perform specific mathematical or logical functions.

Arithmetic Operators¶

The following table shows all the arithmetic operators supported by the C language.

Operator	Description
+	Adds two operands.
-	Subtracts second operand from the first.
*	Multiplies both operands.
/	Divides numerator by de-numerator.
%	Modulus Operator and remainder of after an integer division.
++	Increment operator increases the integer value by one.
- -	Decrement operator decreases the integer value by one.

Relational Operators¶

The following table shows all the relational operators supported by C.

Operator	Description
==	Checks if the values of two operands are equal or not. If yes, then the condition becomes true.
!=	Checks if the values of two operands are equal or not. If the values are not equal, then the condition becomes true.
>	Checks if the value of left operand is greater than the value of right operand. If yes, then the condition becomes true.
<	Checks if the value of left operand is less than the value of right operand. If yes, then the condition becomes true.
>=	Checks if the value of left operand is greater than or equal to the value of right operand. If yes, then the condition becomes true.
<=	Checks if the value of left operand is less than or equal to the value of right operand. If yes, then the condition becomes true.

Logical Operators¶

Following table shows all the logical operators supported by C language.

Operator	Description
&&	Called Logical AND operator.
\|\|	Called Logical OR Operator.
!	Called Logical NOT Operator.

Decision¶

The if statement allows us to check if an expression is true or false, and execute different code according to the result.

if ( expression ) {
    /* Here goes your code */
}
else {
    /* Here goes your code */
}

Note

C programming language assumes any non-zero and non-null values as true, and if it is either zero or null, then it is assumed as false value.

Conditional Operator ? : can be used to replace if...else statements. It has the following general form

Exp1 ? Exp2 : Exp3;

File I/O¶

A file represents a sequence of bytes, regardless of it being a text file or a binary file. C programming language provides access to handle file on your storage devices.

Opening Files¶

You can use the fopen() function to create a new file or to open an existing file. This call will initialize an object of the type FILE, which contains all the information necessary to control the stream. The prototype of this function call is as follows

FILE *fp;
fp = fopen(filename, mode);

Here, filename is a string literal, which you will use to name your file, and access mode can have one of the following values

Mode	Description
r	Opens an existing text file for reading purpose.
w	Opens a text file for writing. If it does not exist, then a new file is created. Here your program will start writing content from the beginning of the file.
a	Opens a text file for writing in appending mode. If it does not exist, then a new file is created. Here your program will start appending content in the existing file content.
r+	Opens a text file for both reading and writing.
w+	Opens a text file for both reading and writing. It first truncates the file to zero length if it exists, otherwise creates a file if it does not exist.
a+	Opens a text file for both reading and writing. It creates the file if it does not exist. The reading will start from the beginning but writing can only be appended.

Closing a File¶

To close a file, use the fclose() function. The prototype of this function is

fclose(fp);

The fclose() function returns zero on success, or EOF if there is an error in closing the file.

Writing a File¶

Following is the simplest function to write argument to a stream

fprintf(fp, comments);

Reading a File¶

Use fscanf() function to read strings from a file, but it stops reading after encountering the first space character.

fscanf(fp, "%s", buff);

Library¶

The C standard library provides numerous built-in functions that your program can call. To access the standard functions that comes with your compiler, you need to include a header with the #include directive.

assert.h¶

TODO

math.h¶

TODO

stdio.h¶

TODO

stdlib.h¶

TODO

string.h¶

string.h is a header file that contains many functions for manipulating strings.

int strcmp(const char * str1, const char * str2);

Compare two strings.

Parameters:	str1 (char) – C string to be compared. str2 (char) – C string to be compared.
Result:	0 (int) - the contents of both strings are equal <0 (int) - the first character that does not match has a lower value in ptr1 than in ptr2 >0 (int) - the first character that does not match has a greater value in ptr1 than in ptr2

Cactus Configuration Language¶

Cactus Configuration Language (CCL) files are text files which tells the flesh all it needs to know about the thorns.

CCL files may contain comments introduced by the hash # character, which indicates that the rest of the line is a comment. If the last non-blank character of a line in a CCL file is a backslash \, the following line is treated as a continuation of the current line.

The interface.ccl File¶

The interface configuration file consists of:

A header block giving details of the thorn’s relationship with other thorns.
A block detailing which include files are used from other thorns, and which include files are provided by this thorn.
Blocks detailing aliased functions provided or used by this thorn.
A series of blocks listing the thorn’s global variables.

implementation name¶

The implementation name is declared by a single line at the top of the file

implements: <name>

The implementation name must be unique among all thorns.

Inheritance relationships between thorns¶

There are two relationship statements that can be used to get variables (actually groups of variables) from other implementations.

Gets all Public variables from implementation <name>, and all variables that <name> has in turn inherited
Inherits: <name>
A thorn cannot inherit from itself. Inheritance is transitive (if A inherits from B, and B inherits from C, then A also implicitly inherits from C), but not commutative.
Gets all Protected variables from implementation <name>, but, unlike inherits, it defines a transitive relation by pushing its own implementation’s Protected variables onto implementation <name>.
Friend: <name>
A thorn cannot be its own friend. Friendship is associative, commutative and transitive (if A is a friend of B, and B is a friend of C, then A is implicitly a friend of C).

Include Files¶

Cactus provides a mechanism for thorns to add code to include files which can be used by any other thorn. Such include files can contain executable source code, or header/declaration information.

INCLUDE[S] [SOURCE|HEADER]: <file_to_include> in <file_name>

This indicates that this thorn adds the code in <file to include> to the include file <file name>.

Any thorn which uses the include file must declare this in its interface.ccl with the line

USES INCLUDE [SOURCE|HEADER]: <file_name>

If the optional [SOURCE|HEADER] is omitted, HEADER is assumed. Note that this can be dangerous, as included source code, which is incorrectly assumed to be header code, will be executed in another thorn even if the providing thorn is inactive. Thus, it is recommended to always include the optional [SOURCE|HEADER] specification.

Thorn variables¶

Cactus variables are used instead of local variables for a number of reasons:

Cactus variables can be made visible to other thorns, allowing thorns to communicate and share data.
Cactus variables can be distributed and communicated among processors, allowing parallel computation.
A database of Cactus variables, and their attributes, is held by the flesh, and this information is used by thorns, for example, for obtaining a list of variables for checkpointing.

Cactus variables are placed in groups with homogeneous attributes, where the attributes describe properties such as the data type, group type, dimension, ghostsize, number of timelevels, and distribution.

[<access>:]

<data_type> <group_name>[[<number>]] [TYPE=<group_type>] [DIM=<dim>] [TIMELEVELS=<num>] [SIZE=<size in each direction>] [DISTRIB=<distribution_type>] [GHOSTSIZE=<ghostsize>] [TAGS=<string>]
{
    <variable_name>,  <variable_name>,  <variable_name>
} ["<group_description>"]

Currently, the names of groups and variables must be distinct. The options TYPE, DIM, etc., following <group name> must all appear on one line.

Access¶

There are three different access levels available for variables

Public: Can be ‘inherited’ by other implementations.
Protected: Can be shared with other implementations which declare themselves to be friends of this one.
Private: Can only be seen by this thorn.

By default, all groups are private, to change this, an access specification of the form public: or protected:

Data Type¶

Cactus supports integer, real, complex and character variable types, in various different sizes. Normally a thorn should use the default types (CCTK_INT, CCTK_REAL, CCTK_COMPLEX) rather than explicitly setting the size, as this gives maximum portability.

Vector Group¶

If [number] present, indicates that this is a vector group.

Group Types¶

Groups can be either scalars, grid functions (GFs), or grid arrays.

SCALAR: This is just a single number.
GF: This is the most common group type. A GF is an array with a specific size, set at run time in the parameter file, which is distributed across processors. All GFs have the same size, and the same number of ghostzones. Groups of GFs can also specify a dimension, and number of timelevels.
ARRAY: This is a more general form of the GF. Each group of arrays can have a distinct size and number of ghostzones, in addition to dimension and number of timelevels. The drawback of using an array over a GF is that a lot of data about the array can only be determined by function calls, rather than the quicker methods available for GFs.

DIM¶

DIM defines the spatial dimension of the ARRAY or GF. The default value is DIM=3.

Timelevels¶

TIMELEVELS defines the number of timelevels a group has if the group is of type ARRAY or GF, and can take any positive value. The default is one timelevel.

Size and Distrib¶

A Cactus grid function or array has a size set at runtime by parameters. This size can either be the global size of the array across all processors (DISTRIB=DEFAULT), or, if DISTRIB=CONSTANT, the specified size on each processor. If the size is split across processors, the driver thorn is responsible for assigning the size on each processor.

Ghost Zones¶

Cactus is based upon a distributed computing paradigm. That is, the problem domain is split into blocks, each of which is assigned to a processor. For hyperbolic and parabolic problems the blocks only need to communicate at the edges. It defaults to zero.

TAGS¶

TAGS defines an optional string which is used to create a set of key-value pairs associated with the group. The keys are case independent. The string (which must be deliminated by single or double quotes) is interpreted by the function Util_TableSetFromString().

Function¶

Cactus offers a mechanism for calling a function in a different thorn where you don’t need to know which thorn is actually providing the function, nor what language the function is provided in. Function aliasing is also comparatively inefficient, and should not be used in a part of your code where efficiency is important.

If any aliased function is to be used or provided by the thorn, then the prototype must be declared with the form:

<return_type> FUNCTION <alias>(<arg1_type> <intent1>, ...)

The <return_type> must be either void, CCTK_INT, CCTK_REAL, CCTK_COMPLEX, CCTK_POINTER, or CCTK_POINTER_TO_CONST. Standard types such as int are not allowed. The keyword SUBROUTINE is equivalent to void FUNCTION.
The name of the aliased function <alias> must contain at least one uppercase and one lowercase letter and follow the C standard for function names.
The type of an argument, <arg*_type>, must be one of scalar types CCTK_INT, CCTK_REAL, CCTK_COMPLEX, CCTK_POINTER, CCTK_POINTER_TO_CONST, or an array or pointer type CCTK_INT ARRAY, CCTK_REAL ARRAY, CCTK_COMPLEX ARRAY, CCTK_POINTER ARRAY. The scalar types are assumed to be not modifiable. If you wish to modify an argument, then it must have intent OUT or INOUT (and hence must be either a CCTK_INT, a CCTK_REAL, or a CCTK_COMPLEX, or an array of one of these types).
The intent of each argument, <intent*>, must be either IN, OUT, or INOUT. The C prototype will expect an argument with intent IN to be a value and one with intent OUT or INOUT to be a pointer.
CCTK_STRING must appear at the end of the argument list.

Using an Aliased Function¶

To use an aliased function you must first declare it in your interface.ccl file. Declare the prototype as, for example,

/* this function will be either required in your thorn by */
REQUIRES FUNCTION <alias>
/* or optionally used in your thorn by */
USES FUNCTION <alias>

A prototype of this function will be available to any C routine that includes the cctk.h header file.

Providing a Function¶

To provide an aliased function you must again add the prototype to your interface.ccl file. A statement containing the name of the providing function and the language it is provided in, must also be given. For example,

PROVIDES FUNCTION <alias> WITH <provider> LANGUAGE <providing_language>

As with the alias name, <provider> must contain at least one uppercase and one lowercase letter, and follow the C standard for function names. It is necessary to specify the language of the providing function; no default will be assumed. Currently, the only supported values of <providing_language> are C and Fortran.

Testing Aliased Functions¶

The calling thorn does not know if an aliased function is even provided by another thorn. Calling an aliased function that has not been provided, will lead to a level 0 warning message, stopping the code. In order to check if a function has been provided by some thorn, use the CCTK_IsFunctionAliased function described in the function reference section.

The param.ccl File¶

Parameters are the means by which the user specifies the runtime behaviour of the code. Each parameter has a data type and a name, as well as a range of allowed values and a default value. These are declared in the thorn’s param.ccl file.

The full specification for a parameter declaration is

[<access>:]

[EXTENDS|USES] <parameter_type> <parameter name>[[<len>]] "<parameter_description>" [AS <alias>] [STEERABLE=<NEVER|ALWAYS|RECOVER>]
{
    <PARAMETER_RANGES>
} <default_value>

The options AS, STEERABLE, etc., following <parameter description>, must all appear in one line.

Access¶

There are three access levels available for parameters:

Global: These parameters are seen by all thorns.
Restricted: These parameters may be used by other implementations if they so desire.
Private: These are only seen by this thorn.

Inheritance relationships between thorns¶

To access restricted parameters from another implementation, a line containing

shares: <name>

Each of these parameters must be qualified by the initial token USES or EXTENDS, where

USES: indicates that the parameters range remains unchanged.
EXTENDS: indicates that the parameters range is going to be extended.

For example, the following block adds possible values to the keyword <par> originally defined in the implementation <name>, and uses the REAL parameter <par>.

shares: <name>

EXTENDS KEYWORD <par>
{
    "KEYWORD"   :: "A description of the parameter"
}

USES CCTK_REAL <par>

Note that you must compile at least one thorn which implements <name>.

parameter type¶

Parameters can be of these types:

CCTK_INT: Can take any integral value

The range specification is of the form

INT <par> "A description of the parameter"
{
    \\ Each range may have a description associated with it by placing a ``::`` on the line, and putting the description afterwards.
    lower:upper:stride :: "Describing the allowed values of the parameter. lower and upper specify the lower and upper allowed range, and stride allows numbers to be missed out. A missing end of range (or a `*`) indicates negative or positive infinity."
} <default value>

CCTK_REAL: Can take any floating point value

REAL <par> "A description of the parameter"
{
    \\ Each range may have a description associated with it by placing a ``::`` on the line, and putting the description afterwards.
    lower:upper :: "Describing the allowed values of the parameter. lower and upper specify the lower and upper allowed range. A missing end of range (or a `*`) implies negative or positive infinity. "
} <default value>

CCTK_KEYWORD: A distinct string with only a few known allowed values.

KEYWORD <par> "A description of the parameter"
{
    "KEYWORD_1"   :: "A description of the parameter"
    "KEYWORD_2"   :: "A description of the parameter"
    "KEYWORD_3"   :: "A description of the parameter"
} <default value>

CCTK_STRING: An arbitrary string, which must conform to a given regular expression. To allow any string, the regular expression “” should be used. Regular expressions and string values should be enclosed in double quotes.
CCTK_BOOLEAN: A boolean type which can take values 1, t, true, yes or 0, f, false, no.
BOOLEAN <par> "A description of the parameter" { } <default value>

Name¶

The <parameter name> must be unique within the scope of the thorn. <len> indicates that this is an array parameter of len values of the specified type. <alias> allows a parameter to appear under a different name in this thorn, other than its original name in another thorn.

Steerable¶

By default, parameters may not be changed after the parameter file has been read, or on restarting from checkpoint. A parameter can be changed dynamically if it is specified to be steerable. It can then be changed by a call to the flesh function CCTK_ParameterSet.

The value RECOVERY is used in checkpoint/recovery situations, and indicates that the parameter may be altered until the value is read in from a recovery par file, but not after.

The schedule.ccl File¶

A schedule configuration file consists of:

Assignment statements to switch on storage for grid variables for the entire duration of program execution.
Schedule blocks to schedule a subroutine from a thorn to be called at specific times during program execution in a given manner.
Conditional statements for both assignment statements and schedule blocks to allow them to be processed depending on parameter values.

Assignment Statements¶

Assignment statements, currently only assign storage.

These lines have the form:

STORAGE: <group>[timelevels]

If the thorn is active, storage will be allocated, for the given groups, for the duration of program execution (unless storage is explicitly switched off by some call to CCTK_DisableGroupStorage within a thorn).

The storage line includes the number of timelevels to activate storage for, this number can be from 1 up to the maximum number or timelevels for the group, as specified in the defining interface.ccl file. Alternatively timelevels can be the name of a parameter accessible to the thorn. The parameter name is the same as used in C routines of the thorn, fully qualified parameter names of the form thorn::parameter are not allowed.

The behaviour of an assignment statement is independent of its position in the schedule file (so long as it is outside a schedule block).

Schedule Blocks¶

The flesh knows about everything in schedule.ccl files, and handles sorting scheduled routines into an order. Each schedule block in the file schedule.ccl must have the syntax

schedule [GROUP] <function|schedule group name> AT|IN <schedule bin|group name> [AS <alias>] [WHILE <variable>] [IF <variable>] [BEFORE|AFTER <item>|(<item> <item> ...)]
{
    [LANG: <FORTRAN|C>]
    [STORAGE: <group>[timelevels]]
    [TRIGGERS: <group>]
    [SYNC: <group>]
    [OPTIONS: <option>]
    [TAGS: [list of keyword=value definitions]]
    [READS: <group>]
    [WRITES: <group>]
} "A description"

Schedule Bins¶

Each schedule item is scheduled either AT a particular scheduling bin, or IN a schedule group. A list of the most useful schedule bins for application thorns is given here.

In the absence of any ordering, functions in a particular schedule bin will be called in an undetermined order.

Group¶

If the optional GROUP specifier is used, the item is a schedule group rather than a normal function. Schedule groups are effectively new, user-defined, schedule bins. Functions or groups may be scheduled IN these, in the same way as they are scheduled AT the main schedule bins.

Schedule Options¶

The options define various characteristics of the schedule item.

AS: This assigns a new name to a function for scheduling purposes.
WHILE: This specifies a CCTK_INT grid scalar which is used to control the execution of this item. As long as the grid scalar has a nonzero value, the schedule item will be executed repeatedly.
IF: This specifies a CCTK_INT grid scalar which is used to control the execution of this item. If the grid scalar has a nonzero value, the schedule item will be executed, otherwise the item will be ignored. If both an IF and a WHILE clause are present, then the schedule is executed according to the following pseudocode:
IF condition WHILE condition SCHEDULE item END WHILE END IF
BEFORE or AFTER: These specify either a function or group before or after which this item will be scheduled.

Further details¶

The schedule block specifies further details of the scheduled function or group.

LANG: This specifies the language of the routine. Currently this is either C or Fortran.
STORAGE: The STORAGE keyword specifies groups for which memory should be allocated for the duration of the routine or schedule group. The storage status reverts to its previous status after completion of the routine or schedule group.
TRIGGER: List of grid variables or groups to be used as triggers for causing an ANALYSIS function or group to be executed.
SYNC: The keyword SYNC specifies groups of variables which should be synchronised (that is, their ghostzones should be exchanged between processors) on exit from the routine.
OPTIONS: Often used schedule options are local (also the default), level, or global. These options are interpreted by the driver, not by Cactus.
TAGS: Schedule tags. These tags must have the form keyword=value, and must be in a syntax accepted by Util_TableCreateFromString.
READS: READS is used to declare which grid variables are read by the routine.
WRITES: WRITES is used to declare which grid variables are written by the routine.

Modes¶

When looping over grids, Carpet has a standard order to loop over grid attributes.

# meta mode
begin loop over mglevel (convergence level)

    # global mode
    begin loop over reflevel (refinement level)

        # level mode
        begin loop over map

            # singlemap mode
            begin loop over component
                # local mode
            end loop over component

        end loop over map

    end loop over reflevel

end loop over mglevel

When you schedule a routine (by writing a schedule block in a schedule.ccl file), you specify what mode it should run in. If you don’t specify a mode, the default is local. Since convergence levels aren’t used at present, for most practical purposes meta mode is identical to global mode. Similarly, unless you’re doing multipatch simulations, singlemap mode is identical to level mode.

Variables	meta	global	level	singlemap	local
mglevel	No	Yes	Yes	Yes	Yes
reflevel	No	No	Yes	Yes	Yes
map	No	No	No	Yes	Yes
component	No	No	No	No	Yes
CCTK_ORIGIN_SPACE	No	No	Yes	Yes	Yes
CCTK_DELTA_SPACE	No	No	Yes	Yes	Yes
CCTK_DELTA_TIME	No	No	Yes	Yes	Yes
cctk_origin_space	No	Yes	Yes	Yes	Yes
cctk_delta_space	No	Yes	Yes	Yes	Yes
cctk_delta_time	No	Yes	Yes	Yes	Yes
grid scalars	No	Yes	Yes	Yes	Yes
grid arrays	No	Yes	Yes	Yes	Yes
grid functions	No	No	No	No	Yes

Local¶

Grid functions are defined only in local mode. Since most physics code needs to manipulate grid functions, it therefore must run in local mode. However, in general code scheduled in local mode will run multiple times (because it’s nested inside loops over mglevel, reflevel, map, and component). Sometimes you don’t want this. For example, you may want to open or close an output file, or initialize some global property of your simulation. Global modes are good for this kind of thing.

Level¶

Synchronization and turning storage on/off happen in level mode. Boundary conditions must be selected in level mode. Cactus output must be done in level mode.

Reduction/interpolation of grid arrays and/or grid functions may be done in either level mode (applying only to that refinement level), or in global mode (applying to all refinement levels).

Singelmap¶

Singelmap mode is mostly useful only if you’re doing multipatch simulations.

Querying and Changing Modes¶

Carpet has various functions to query what mode you’re in, and functions and macros to change modes. These are all defined in Carpet/Carpet/src/modes.hh, and are only usable from C++ code.

To use any of these facilities, put the line uses include: carpet.hh in your interface.ccl, then include "carpet.hh" in your C++ source code (this must come after include "cctk.h").

To query the current mode, just use any of the Boolean predicates is_meta_mode(), is_global_mode(), … , is_local_mode().

#include <cassert>
#include "cctk.h"
#include "carpet.hh"

void my_function(...)
{
    // make sure we’re in level mode
    assert(Carpet::is_level_mode());
}

Conditional Statements¶

Besides schedule blocks, it’s possible to embed C style if/else statements in the schedule.ccl file. These can be used to schedule things based upon the value of a parameter.

if (<conditional-expression>)
{
    [<assignments>]
    [<schedule blocks>]
}

<conditional-expression> can be any valid C construct evaluating to a truth value. Such conditionals are evaluated only at program startup, and are used to pick between different static schedule options.

The Source File¶

Compile¶

By default, the CCTK looks in the src directory of the thorn for source files.

The Cactus make system looks for a file called make.code.defn in that directory (if there is no file called Makefile in the src directory). At its simplest, this file contains two lines

SRCS = <list of all source files in this directory>
SUBDIRS = <list of all subdirectories, including subdirectories of subdirectories>

Each subdirectory listed should then have a make.code.defn file containing just a SRCS = line, a SUBDIRS = line will be ignored.

Then you need to build the code. The command you need to run is the following:

make <name>

Each configuration has a ThornList which lists the thorns to be compiled in. When this list changes, only those thorns directly affected by the change are recompiled.

Routines¶

Any source file using Cactus infrastructure should include the header file cctk.h using the line

#include "cctk.h"

Any routine using Cactus argument lists (for example, all routines called from the scheduler at time bins between CCTK_STARTUP and CCTK_SHUTDOWN) should include at the top of the file the header

#include "cctk_Arguments.h"

A Cactus macro CCTK_ARGUMENTS is defined for each thorn to contain:

General information about the grid hierarchy.
All the grid variables defined in the thorn’s interface.ccl.
All the grid variables required from other thorns as requested by the inherits and friend lines in the interface.ccl.

These variables must be declared at the start of the routine using the macro DECLARE_CCTK_ARGUMENTS.

Any routine using Cactus parameters should include at the top of the file the header

#include "cctk_Parameters.h"

All parameters defined in a thorn’s param.ccl. Booleans and Integers appear as CCTK_INT types (with nonzero/zero values for boolean yes/no), Reals as CCTK_REAL, and Keywords and String parameters as CCTK_STRING. These variables are read only, and changes should not be made to them. The effect of changing a parameter is undefined (at best). To compare a string valued parameter use the function CCTK_Equals().

The parameters should be declared at the start of the routine using them with the macro DECLARE_CCTK_PARAMETERS.

Example¶

The C routine MyCRoutine is scheduled in the schedule.ccl file, and uses Cactus parameters. The source file should look like

#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"

void MyFunction(CCTK_ARGUMENTS)
{
    DECLARE_CCTK_ARGUMENTS
    DECLARE_CCTK_PARAMETERS

    /* Here goes your code */
};

The C++ routine MyCRoutine is scheduled in the schedule.ccl file, and uses Cactus parameters. The source file should look like

#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"

extern "C" void MyFunction(CCTK_ARGUMENTS)
{
    DECLARE_CCTK_ARGUMENTS
    DECLARE_CCTK_PARAMETERS

    /* Here goes your code */
};

The Fortran routine MyCRoutine is scheduled in the schedule.ccl file, and uses Cactus parameters. The source file should look like

#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"
#include "cctk_Functions.h"

subroutine MyFunction(CCTK_ARGUMENTS)

    implicit none

    DECLARE_CCTK_ARGUMENTS
    DECLARE_CCTK_PARAMETERS
    DECLARE_CCTK_FUNCTIONS

    /* Here goes your code */
end

Specifically for C Programmers¶

Grid functions are held in memory as 1-dimensional C arrays. Cactus provides macros to find the 1-dimensional index which is needed from the multidimensional indices which are usually used. There is a macro for each dimension of grid function. Below is an artificial example to demonstrate this using the 3D macro CCTK_GFINDEX3D:

for (k=0; k<cctk_lsh[2]; ++k) {
    for (j=0; j<cctk_lsh[1]; ++j) {
        for (i=0; i<cctk_lsh[0]; ++i) {
            int const ind3d = CCTK_GFINDEX3D(cctkGH,i,j,k);
            rho[ind3d] = exp(-pow(r[ind3d],2));
        }
    }
}

Here, CCTK_GFINDEX3D(cctkGH,i,j,k) expands to ((i) + cctkGH->cctk_lsh[0]*((j)+cctkGH->cctk_lsh[1]*(k))). In Fortran, grid functions are accessed as Fortran arrays, i.e. simply as rho(i,j,k).

To access vector grid functions, one also needs to specify the vector index. This is best done via the 3D macro CCTK_VECTGFINDEX3D:

for (k=0; k<cctk_lsh[2]; ++k) {
    for (j=0; j<cctk_lsh[1]; ++j) {
        for (i=0; i<cctk_lsh[0]; ++i) {
            /* vector indices are 0, 1, 2 */
            vel[CCTK_VECTGFINDEX3D(cctkGH,i,j,k,0)] = 1.0;
            vel[CCTK_VECTGFINDEX3D(cctkGH,i,j,k,1)] = 0.0;
            vel[CCTK_VECTGFINDEX3D(cctkGH,i,j,k,2)] = 0.0;
        }
    }
}

Cactus Variables¶

The Cactus variables which are passed through the macro CCTK_ARGUMENTS are

Variables	Description
cctkGH	A C pointer identifying the grid hierarchy.
cctk_dim	An integer with the number of dimensions used for this grid hierarchy.
cctk_lsh	An array of cctk_dim integers with the local grid size on this processor.
cctk_ash	An array of cctk_dim integers with the allocated size of the array.
cctk_gsh	An array of cctk_dim integers with the global grid size.
cctk_iteration	The current iteration number.
cctk_delta_time	A CCTK_REAL with the timestep.
cctk_time	A CCTK_REAL with the current time.
cctk_delta_space	An array of cctk_dim CCTK_REALs with the grid spacing in each direction.
cctk_nghostzones	An array of cctk_dim integers with the number of ghostzones used in each direction.
cctk_origin_space	An array of cctk_dim CCTK_REALs with the spatial coordinates of the global origin of the grid.
cctk_lbnd	An array of cctk_dim integers containing the lowest index (in each direction) of the local grid, as seen on the global grid.
cctk_ubnd	An array of cctk_dim integers containing the largest index (in each direction) of the local grid, as seen on the global grid.
cctk_bbox	An array of 2*cctk_dim integers, which indicate whether the boundaries are internal boundaries (e.g. between processors), or physical boundaries. A value of 1 indicates a physical (outer) boundary at the edge of the computational grid, and 0 indicates an internal boundary.
cctk_levfac	An array of cctk_dim integer factors by which the local grid is refined in the corresponding direction with respect to the base grid.
cctk_levoff	Two arrays of cctk_dim integers describing the distance by which the local grid is offset with respect to the base grid, measured in local grid spacings.
cctk_timefac	The integer factor by which the time step size is reduced with respect to the base grid.
cctk_convlevel	The convergence level of this grid hierarchy. The base level is 0, and every level above that is coarsened by a factor of cctk_convfac.
cctk_convfac	The factor between convergence levels. The relation between the resolutions of different convergence levels is \(\Delta x_{L}=\Delta x_{0} \cdot F^{L}\), where L is the convergence level and F is the convergence factor. The convergence factor defaults to 2.

CCTK Routines¶

Providing Runtime Information¶

To write from thorns to standard output (i.e. the screen) at runtime, use the macro CCTK_INFO. For example,

CCTK_INFO("Hellow World")

will write the line:

INFO (MyThorn): Hellow World

Including variables in the info message use CCTK_VINFO. For example,

CCTK_VINFO("The integer is %d", myint);

Here are some commonly used conversion specifiers:

specifiers	type
%d	int
%g	real (the shortest representation)
%s	string

For a multiprocessor run, only runtime information from processor zero will be printed to screen by default.

Error Handling, Warnings and Code Termination¶

The Cactus macros CCTK_ERROR and CCTK_VERROR should be used to output error messages and abort the code. The Cactus macros CCTK_WARN and CCTK_VWARN should be used to issue warning messages during code execution.

Along with the warning message, an integer is given to indicate the severity of the warning.

Macros	Level	Description
CCTK_WARN_ABORT	0	abort the Cactus run
CCTK_WARN_ALERT	1	the results of this run will be wrong,
CCTK_WARN_COMPLAIN	2	the user should know about this, but the problem is not terribly surprising
CCTK_WARN_PICKY	3	this is for small problems that can probably be ignored, but that careful people may want to know about
CCTK_WARN_DEBUG	4	these messages are probably useful only for debugging purposes

A level 0 warning indicates the highest severity (and is guaranteed to abort the Cactus run), while larger numbers indicate less severe warnings. For example,

CCTK_WARN(CCTK_WARN_ALERT, "Your warning message");
CCTK_ERROR("Your error message");

Note that if the flesh parameter cctk_full_warnings is set to true, then CCTK_ERROR and CCTK_WARN automatically include the thorn name, the source code file name and line number in the message. The default is to omit the source file name and line number.

Including variables in the warning message use CCTK_VERROR and CCTK_VWARN. For example,

CCTK_VWARN(CCTK_WARN_ALERT, "Your warning message, including %f and %d", myreal, myint);
CCTK_ERROR("Your warning message, including %f and %d", myreal, myint);

Iterating Over Grid Points¶

A grid function consists of a multi-dimensional array of grid points. These grid points fall into several types:

interior: regular grid point, presumably evolved in time
ghost: inter-process boundary, containing copies of values owned by another process
physical boundary: outer boundary, presumably defined via a boundary condition
symmetry boundary: defined via a symmetry, e.g. a reflection symmetry or periodicity

Note

Grid points in the edges and corners may combine several types. For example, a point in a corner may be a ghost point in the x direction, a physical boundary point in the y direction, and a symmetry point in the z direction.

The size of the physical boundary depends on the application. The number of ghost points is defined by the driver; the number of symmetry points is in principle defined by the thorn implementing the respective symmetry condition, but will in general be the same as the number of ghost points to avoid inconsistencies.

The flesh provides a set of macros to iterate over particular types of grid points.

Loop over all grid points

CCTK_LOOP3_ALL(name, cctkGH, i,j,k)
{
    /* body of the loop */
} CCTK_ENDLOOP3_ALL(name);

Loop over all interior grid points

CCTK_LOOP3_INT(name, cctkGH, i,j,k)
{
    /* body of the loop */
} CCTK_ENDLOOP3_INT(name);

Loop over all physical boundary points
LOOP_BND loops over all points that are physical boundaries (independent of whether they also are symmetry or ghost boundaries).
CCTK_LOOP3_BND(name, cctkGH, i,j,k, ni,nj,nk) { /* body of the loop */ } CCTK_ENDLOOP3_BND(name);
Loop over all “interior” physical boundary point
LOOP_INTBND loops over those points that are only physical boundaries (and excludes any points that belongs to a symmetry or ghost boundary).
CCTK_LOOP3_INTBND(name, cctkGH, i,j,k, ni,nj,nk) { /* body of the loop */ } CCTK_ENDLOOP3_INTBND(name);

In all cases, name should be replaced by a unique name for the loop. i, j, and k are names of variables that will be declared and defined by these macros, containing the index of the current grid point. Similarly ni, nj, and nk are names of variables describing the (outwards pointing) normal direction to the boundary as well as the distance to the boundary.

Interpolation Operators¶

There are two different flesh APIs for interpolation, depending on whether the data arrays are Cactus grid arrays or processor-local, programming language built-in arrays.

CCTK_InterpGridArrays() function interpolates a list of CCTK grid variables (in a multiprocessor run these are generally distributed over processors) on a list of interpolation points. The grid topology and coordinates are implicitly specified via a Cactus coordinate system. The interpolation points may be anywhere in the global Cactus grid. Additional parameters for the interpolation operation can be passed in via a handle to a key-value options table.

#include "cctk.h"
#include "util_Table.h"

/* Pointer to a valid Cactus grid hierarchy. */
const cGH *GH;

/* Number of dimensions in which to interpolate. */
#define N_DIMS 3

/* Handle to the local interpolation operator as returned by CCTK_InterpHandle. */
const int operator_handle = CCTK_InterpHandle(interpolator_name);
if (operator_handle < 0) {
    CCTK_VWARN(CCTK_WARN_ABORT, "Couldn't find interpolator \"%s\"!", interpolator_name);
}

/* Handle to a key-value table containing zero or more additional parameters for the interpolation operation. */
const int param_table_handle = Util_TableCreateFromString(interpolator_pars);
if (param_table_handle < 0) {
    CCTK_VWARN(CCTK_WARN_ABORT, "Bad interpolator parameter(s) \"%s\"!", interpolator_pars);
}

/* Handle to Cactus coordinate system as returned by CCTK_CoordSystemHandle. */
const int coord_system_handle = CCTK_CoordSystemHandle(coord_name);
if (coord_system_handle < 0) {
    CCTK_VWARN(CCTK_WARN_ABORT, "Couldn't get coordinate-system \"%s\"!", coord_name);
}

/* The number of interpolation points requested by this processor. */
#define N_INTERP_POINTS 1000

/* (Pointer to) an array of N dims pointers to 1-D arrays giving the coordinates of the interpolation points requested by this processor. These coordinates are with respect to the coordinate system defined by coord system handle. */
CCTK_REAL interp_x[N_INTERP_POINTS], interp_y[N_INTERP_POINTS], interp_z[N_INTERP_POINTS];
const void *interp_coords[N_DIMS];

interp_coords[0] = (const void *) interp_x;
interp_coords[1] = (const void *) interp_y;
interp_coords[2] = (const void *) interp_z;

/* The number of input variables to be interpolated. */
#define N_INPUT_ARRAYS 2

/* (Pointer to) an array of N_input_arrays CCTK grid variable indices (as returned by CCTK_VarIndex) specifying the input grid variables for the interpolation. */
CCTK_INT input_array_variable_indices[N_INPUT_ARRAYS];
input_array_variable_indices[0] = CCTK_VarIndex("my_thorn::var1");
input_array_variable_indices[1] = CCTK_VarIndex("my_thorn::var2");

/* The number of output arrays to be returned from the interpolation. Note that N_output_arrays may differ from N_input_arrays. */
#define N_OUTPUT_ARRAYS 2

/* Giving the data types of the 1-D output arrays pointed to by output_arrays[]. */
CTK_INT output_array_type_codes[N_OUTPUT_ARRAYS]
output_array_type_codes[0] = CCTK_VARIABLE_REAL
output_array_type_codes[0] = CCTK_VARIABLE_COMPLEX

/* (Pointer to) an array of N_output_arrays pointers to the (user-supplied) 1-D output arrays for the interpolation. */
void *output_arrays[N_OUTPUT_ARRAYS];
CCTK_REAL output_for_real_array [N_INTERP_POINTS];
CCTK_COMPLEX output_for_complex_array[N_INTERP_POINTS];
output_arrays[0] = (void *) output_for_real_array;
output_arrays[1] = (void *) output_for_complex_array;

int status = CCTK_InterpGridArrays(
    GH,
    N_DIMS,
    operator_handle,
    param_table_handle,
    coord_system_handle,
    N_INTERP_POINTS,
    CCTK_VARIABLE_REAL, // Giving the data type of the interpolation-point coordinate arrays pointed to by interp_coords[].
    interp_coords,
    N_INPUT_ARRAYS,
    input_array_variable_indices,
    N_OUTPUT_ARRAYS,
    output_array_type_codes,
    output_arrays
)

if (status < 0) {
    CCTK_WARN(CCTK_WARN_ABORT, "error return from interpolator!");
}

CCTK_InterpLocalUniform() interpolate a list of processor-local arrays which define a uniformly-spaced data grid.

Reduction Operators¶

A reduction operation can be defined as an operation on variables distributed across multiple processor resulting in a single number. Typical reduction operations are: sum, minimum/maximum value, and boolean operations. The different operators are identified by their name and/or a unique number, called a handle.

There are two different flesh APIs for reduction, depending on whether the data arrays are Cactus grid arrays or processor-local, programming language built-in arrays.

CCTK_ReduceGridArrays() reduces a list of CCTK grid arrays (in a multiprocessor run these are generally distributed over processors).

#include "cctk.h"
#include "util_Table.h"

/* Pointer to a valid Cactus grid hierarchy. */
const cGH *GH;

/* Handle to the local reduction operator as returned by CCTK_LocalArrayReductionHandle(). */


const int status = CCTK_ReduceGridArrays(
    GH,
    0, // The destination processor
    param_table_handle,
    N_INPUT_ARRAYS,
    input_array_variable_indices,
    M_OUTPUT_VALUES,
    output_value_type_codes,
    output_values
);

CCTK_ReduceLocalArrays() performs reduction on a list of local grid arrays.

Utility Routines¶

As well as the high-level CCTK_* routines, Cactus also provides a set of lower-level Util_* utility routines which thorns developers may use. Cactus functions may need to pass information through a generic interface. In the past, we often had trouble passing extra information that wasn’t anticipated in the original design. Key-value tables provide a clean solution to these problems. They’re implemented by the Util_Table* functions.

ParamCheck¶

In your schedule.ccl file.

SCHEDULE MyCRoutine_ParamCheck AT CCTK_PARAMCHECK
{
    LANG: C
} "ParamCheck"

In your code.

#include "cctk.h"

#include "cctk_Arguments.h"
#include "cctk_Parameters.h"

void MyCRoutine_ParamCheck(CCTK_ARGUMENTS)
{
    DECLARE_CCTK_ARGUMENTS;
    DECLARE_CCTK_PARAMETERS;

    if(! CCTK_EQUALS(metric_type, "physical") &&
       ! CCTK_EQUALS(metric_type, "static conformal"))
    {
        CCTK_PARAMWARN("Unknown ADMBase::metric_type - known types are \"physical\" and \"static conformal\"");
    }
}

API¶

int CCTK_IsThornActive(const char* thorn)¶

Reports whether a thorn was activated in a parameter file.

Parameters:	thorn (char) – The character-string name of the thorn
Result:	status (int) This function returns a non-zero value if thorn was activated in a parameter file, and zero otherwise.

>>> if (CCTK_IsThornActive ("MoL")) {
>>>     /* Here goes your code */
>>> }

char CCTK_ParameterValString(const char *name, const char *thorn)¶

Get the string representation of a parameter’s value.

Discussion:	The memory must be released with a call to `free()` after it has been used.
Parameters:	name (char) – Parameter name thorn (char) – Thorn name (for private parameters) or implementation name (for restricted parameters)
Result:	valstring (char) - Pointer to parameter value as string
Error:	NULL - No parameter with that name was found.

>>> char *valstring = CCTK_ParameterValString("cctk_run_title", "Cactus")
>>> assert( valstring != NULL );
>>> free(valstring);

CCTK_ParameterGet(const char *name, const char *thorn, int *type)¶

Get the data pointer to and type of a parameter’s value.

Parameters:	name (char) – Parameter name thorn (char) – Thorn name (for private parameters) or implementation name (for restricted parameters) type (int) – If not NULL, a pointer to an integer which will hold the type of the parameter
Result:	valstring (char) - Pointer to the parameter value
Error:	NULL - No parameter with that name was found.

>>> const void *ghost_ptr = CCTK_ParameterGet("ghost_size", "Carpet", NULL);
>>> assert( ghost_ptr != NULL );
>>> int ghost = *(const int *)ghost_ptr;

Coordinate¶

Global coordinates¶

Compute the global Cactus xyz coordinates of the current grid point.

#include "cctk.h"

void MyThorn_MyFunction(CCTK_ARGUMENTS)
{
    int i, j, k;
    for (k = 0; k < cctk_lsh[2]; ++k) {
        for (j = 0; j < cctk_lsh[1]; ++j) {
            for (i = 0; i < cctk_lsh[0]; ++i) {
                // variables declared with 'const' added become constants and cannot be altered by the program.
                const int ind3d = CCTK_GFINDEX3D(cctkGH,i,j,k);

                const CCTK_REAL xL = x[ind3d];
                const CCTK_REAL yL = y[ind3d];
                const CCTK_REAL zL = z[ind3d];
                const CCTK_REAL rL = r[ind3d];
            }
        }
    }
}

#include "cctk.h"

void MyThorn_MyFunction(CCTK_ARGUMENTS)
{
    int i, j, k;
    /* Do not compute in ghost zones */
    for(i=ghost; i<cctk_lsh[0]-ghost; i++) {
        for(j=ghost; j<cctk_lsh[1]-ghost; j++) {
            for(k=ghost; k<cctk_lsh[2]-ghost; k++) {
                const int ind3d = CCTK_GFINDEX3D(cctkGH,i,j,k);

                const CCTK_REAL xL = x[ind3d];
                const CCTK_REAL yL = y[ind3d];
                const CCTK_REAL zL = z[ind3d];
                const CCTK_REAL rL = r[ind3d];
            }
        }
    }
}

CartGrid3D¶

CartGrid3D allows you to enforce even or odd parity for any grid function at (across) each coordinate axis. For a function \(\phi(x, y, z)\), even parity symmetry on the x-axis means

\[\phi(-x, y, z)=\phi(x, y, z)\]

while odd parity symmetry means

\[\phi(-x, y, z)=- \phi(x, y, z)\]

You first need to get access to the include ﬁle by putting the line in your interface.ccl file.

uses include: Symmetry.h

Symmetries should obviously be registered before they are used, but since they can be diﬀerent for diﬀerent grids, they must be registered after the CCTK_STARTUP timebin. The usual place to register symmetries is in the CCTK_BASEGRID timebin.

SCHEDULE MyCRoutine_RegisterSymmetry AT CCTK_BASEGRID
{
    LANG: C
    OPTIONS: global
} "Register symmetry"

#include "cctk.h"
#include "cctk_Arguments.h"
#include "Symmetry.h"

void MyCRoutine_RegisterSymmetry(CCTK_ARGUMENTS)
{
    DECLARE_CCTK_ARGUMENTS;

    int sym[3];
    int ierr;

    sym[0] = 1;
    sym[1] = 1;
    sym[2] = 1;

    /* Applies symmetry boundary conditions from variable name */
    ierr = SetCartSymVN(cctkGH, sym, "ADMAnalysis::Ricci23");
    /* error information */
    if(ierr) {
        CCTK_VWarn(0, __LINE__, __FILE__, "Thorn_Name", "Error returned from function SetCartSymVN");
    }
}

Boundary¶

The implementation Boundary provides a number of aliased functions, which allow application thorns to register routines which provide a particular physical boundary condition, and also to select variables or groups of variables to have boundary conditions applied to whenever the ApplyBCs schedule group is scheduled.

SCHEDULE MyCRoutine_Boundaries
{
    LANG: C
}  "Select boundary conditions"

SCHEDULE GROUP ApplyBCs as MyCRoutine_Boundaries
{
} "Apply boundary conditions"

To select a grid variable to have a boundary condition applied to it, use one of the following aliased functions:

#include "cctk.h"
#include "cctk_Arguments.h"

void MyCRoutine_Boundaries(CCTK_ARGUMENTS)
{
    DECLARE_CCTK_ARGUMENTS;

    int ierr;

    /* Select an entire variable group, using its name. */
    err = Boundary_SelectGroupForBC(
        cctkGH, // Grid hierarchy pointer
        CCTK_ALL_FACES, // CCTK_ALL_FACES corresponds to the set of all faces of the domain.
        1,
        -1, // use default values
        "ADMAnalysis::ricci_scalar", // group name: (<implementation>::<group_name>)
        "Flat" // The boundary conditions available are Scalar, Flat, Radiation, Copy, Robin, Static, and None.
    );
    if (err < 0) {
        CCTK_WARN(2, "Error in applying flat boundary condition");
    }
}

Each of these functions takes a faces speciﬁcation, a boundary width, and a table handle as additional arguments. The faces specification is a single integer which identifies a set of faces to which to apply the boundary condition. The boundary width is the thickness, in grid points, of the boundaries. The table handle identifies a table which holds extra arguments for the particular boundary condition that is requested.

Initial Data¶

Timelevel¶

These are best introduced by an example using finite differencing. Consider the 1-D wave equation

\[\frac{\partial^{2} \phi}{\partial t^{2}}=\frac{\partial^{2} \phi}{\partial x^{2}}\]

To solve this by partial differences, one discretises the derivatives to get an equation relating the solution at different times. There are many ways to do this, one of which produces the following difference equation

\[\phi(t+\Delta t, x)-2 \phi(t, x)+\phi(t-\Delta t, x)=\frac{\Delta t^{2}}{\Delta x^{2}}\{\phi(t, x+\Delta x)-2 \phi(t, x)+\phi(t, x-\Delta x)\}\]

which relates the three timelevels \(t+\Delta t\), \(t\), \(t-\Delta t\).

All timelevels, except the current level, should be considered read-only during evolution, that is, their values should not be changed by thorns. The exception to this rule is for function initialisation, when the values at the previous timelevels do need to be explicitly filled out.

if (timelevels == 1) {
    STORAGE: rho[1]
}
else if (timelevels == 2) {
    STORAGE: rho[2]
}
else if (timelevels == 3) {
    STORAGE: rho[3]
}

#include <cctk.h>
#include <cctk_Arguments.h>
#include <cctk_Parameters.h>

void MyCRoutine_Zero(CCTK_ARGUMENTS)
{
    const int np = cctk_ash[0] * cctk_ash[1] * cctk_ash[2];

    if (CCTK_ActiveTimeLevels(cctkGH, "HydroBase::rho") >= 1) {
#pragma omp parallel for
        for (int i = 0; i < np; ++i) {
            rho[i] = 0.0; \* Set rho to 0 *\
        }
    }

    if (CCTK_ActiveTimeLevels(cctkGH, "HydroBase::rho") >= 2) {
#pragma omp parallel for
        for (int i = 0; i < np; ++i) {
            rho_p[i] = 0.0;
        }
    }

    if (CCTK_ActiveTimeLevels(cctkGH, "HydroBase::rho") >= 3) {
#pragma omp parallel for
        for (int i = 0; i < np; ++i) {
            rho_p_p[i] = 0.0;
        }
    }
}

Numerical¶

Analysis¶

Calculation¶

determinant

#include <stddef.h>
#include "cctk.h"

void MyThorn_MyFunction(CCTK_ARGUMENTS)
{
    int i, j, k;
    for (k = 0; k < cctk_lsh[2]; ++k) {
        for (j = 0; j < cctk_lsh[1]; ++j) {
            for (i = 0; i < cctk_lsh[0]; ++i) {
                int index = CCTK_GFINDEX3D(cctkGH,i,j,k);

                double gxxL = gxx[index];
                double gxyL = gxy[index];
                double gxzL = gxz[index];
                double gyyL = gyy[index];
                double gyzL = gyz[index];
                double gzzL = gzz[index];

                double det = -gxzL*gxzL*gyyL + 2*gxyL*gxzL*gyzL - gxxL*gyzL*gyzL - gxyL*gxyL*gzzL + gxxL*gyyL*gzzL;
                double invdet = 1.0 / det;
                double gupxxL=(-gyzL*gyzL + gyyL*gzzL)*invdet;
                double gupxyL=( gxzL*gyzL - gxyL*gzzL)*invdet;
                double gupyyL=(-gxzL*gxzL + gxxL*gzzL)*invdet;
                double gupxzL=(-gxzL*gyyL + gxyL*gyzL)*invdet;
                double gupyzL=( gxyL*gxzL - gxxL*gyzL)*invdet;
                double gupzzL=(-gxyL*gxyL + gxxL*gyyL)*invdet;
            }
        }
    }
}

Volume integral¶

#include "cctk.h"
#include "cctk_Arguments.h"

void MyCRoutine(CCTK_ARGUMENTS)
{
    DECLARE_CCTK_ARGUMENTS;

    CCTK_INT reduction_handle;

    reduction_handle = CCTK_ReductionHandle("sum");
    if (reduction_handle < 0) {
        CCTK_WARN(0, "Unable to get reduction handle.");
    }
    CCTK_Reduce(
        cctkGH,
        -1,
        reduction_handle,
        1,
        CCTK_VARIABLE_REAL,
        &ADMMass_VolumeMass[*ADMMass_LoopCounter],
        1,
        CCTK_VarIndex("ADMMass::ADMMass_VolumeMass_GF")
    )
    // TODO ADMMass
}

Surface integral¶

#include "cctk.h"
#include "cctk_Arguments.h"

void MyCRoutine(CCTK_ARGUMENTS)
{
    DECLARE_CCTK_ARGUMENTS;

    // TODO ADMMass
}

I/O¶

API¶

int CCTK_VarIndex(const char *varname)¶

Get the index for a variable.

Discussion:	The variable name should be the given in its fully qualified form, that is `<implementation>::<variable>` for PUBLIC or PROTECTED variables and `<thorn>::<variable>` for PRIVATE variables.
Parameters:	varname (char) – The name of the variable.
Error:	-1 - no variable of this name exists -2 - no failed to catch error code from `Util_SplitString` -3 - given full name is in wrong format -4 - memory allocation failed

>>> index = CCTK_VarIndex("evolve::phi");
>>> index = CCTK_VarIndex("evolve::vect[0]");

int CCTK_GroupIndexFromVarI(int varindex)¶

Given a variable index, returns the index of the associated group

Parameters:	varindex (int) – The index of the variable
Result:	groupindex (int) - The index of the group

>>> index = CCTK_VarIndex("evolve::phi");
>>> groupindex = CCTK_GroupIndexFromVarI(index);

char CCTK_FullName(int index)¶

Given a variable index, returns the full name of the variable

Discussion:	The full variable name is in the form `<implementation>::<variable>` for PUBLIC or PROTECTED variables and `<thorn>::<variable>` for PRIVATE variables.
Parameters:	index (int) – The variable index
Result:	implementation (char) - The full variable name

>>> name = CCTK_FullName(index);

int CCTK_VarTypeI(int index)¶

Provides variable type index from the variable index

Discussion:	The variable type index indicates the type of the variable. Either character, int, complex or real. The group type can be checked with the Cactus provided macros for CCTK_VARIABLE_INT, CCTK_VARIABLE_REAL, CCTK_VARIABLE_COMPLEX or CCTK_VARIABLE_CHAR.
Parameters:	index (int) – The variable index
Result:	type (int) - The variable type index

>>> vtype = CCTK_VarTypeI(index);
>>> if (vtype == CCTK_VARIABLE_REAL) {
>>>     /* Here goes your code */
>>> }

int CCTK_GroupTypeI(int group)¶

Provides a group type index given a group index

Discussion:	A group type index indicates the type of variables in the group. The group type can be checked with the Cactus provided macros for CCTK_SCALAR, CCTK_GF, CCTK_ARRAY.
Parameters:	group (int) – Group index.
Error:	-1 - the given group index is invalid.

>>> gtype = CCTK_GroupTypeI(gindex);
>>> if (gtype == CCTK_GF) {
>>>     /* Here goes your code */
>>> }

void CCTK_VarDataPtrI(const cGH * cctkGH, int timelevel, int index)¶

Returns the data pointer for a grid variable from the variable index.

Parameters:	cctkGH – pointer to CCTK grid hierarchy timelevel (int) – The timelevel of the grid variable index (int) – The index of the variable

>>> CCTK_REAL *data = NULL;
>>> vindex = CCTK_VarIndex("evolve::phi");
>>> data = (CCTK_REAL*) CCTK_VarDataPtrI(cctkGH, 0, vindex);

CCTK_GroupData(int group_index, cGroup* group_data_buffer)¶

Given a group index, returns information about the group and its variables.

Discussion:

The cGroup structure contains (at least) the following members:

grouptype (int) - group type
vartype (int) - variable type
disttype (int) - distribution type
dim (int) - dimension (rank) of the group
numvars (int) - number of variables in the group
numtimelevels (int) - declared number of time levels for this group’s variables
vectorgroup (int) - 1 if this is a vector group,0 if it’s not
vectorlength (int) - vector length of group (i.e. number of vector elements)
tagstable (int) - handle to the group’s tags table

Parameters:

group_index (int) – The group index for which the information is desired
group_data_buffer (int) – Pointer to a cGroup structure in which the information should be stored.

Error:

-1 - group index is invalid. -2 - group_data_buffer is NULL.

>>> cGroup group_info;
>>> int group_index = CCTK_GroupIndex("BSSN_MoL::ADM_BSSN_metric");
>>> CCTK_GroupData(group_index, &group_info);
>>> CCTK_VINFO("Dim: %d, numvars: %d", group_info.dim, group_info.numvars);

int CCTK_OutputVarAs(const cGH *cctkGH, const char *variable, const char *alias)¶

Output a single variable as an alias by all I/O methods.

Discussion:	If the appropriate file exists the data is appended, otherwise a new file is created. Uses `alias` as the name of the variable for the purpose of constructing a filename.
Parameters:	cctkGH – pointer to CCTK grid hierarchy variable (char) – full name of variable to output alias (char) – alias name to base the output filename on
Result:	istat (int) - the number of IO methods which did output of variable
Error:	negative - if no IO methods were registered

>>> CCTK_OutputVarAs(cctkGH, "HydroBase::rho", "rho");

int CCTK_OutputVarAsByMethod(const cGH *cctkGH, const char *variable, const char *method, const char *alias)¶

Output a variable variable using the method method if it is registered. Uses alias as the name of the variable for the purpose of constructing a filename. If the appropriate file exists the data is appended, otherwise a new file is created.

Parameters:	cctkGH – pointer to CCTK grid hierarchy variable (char) – full name of variable to output method (char) – method to use for output alias (char) – alias name to base the output filename on
Result:	istat (int) - zero for success
Error:	negative - indicating some error

int CCTK_OutputVarByMethod(const cGH *cctkGH, const char *variable, const char *method)¶

Output a variable variable using the IO method method if it is registered. If the appropriate file exists the data is appended, otherwise a new file is created.

Parameters:	cctkGH – pointer to CCTK grid hierarchy variable (char) – full name of variable to output method (char) – method to use for output
Result:	istat (int) - zero for success
Error:	negative - indicating some error

Utility¶

API¶

int CCTK_isinf(double x)¶

int CCTK_isnan(double x)¶

Functions¶

find_closest¶

#include <cctk.h>
#include <math.h>

int find_closest(const cGH *cctkGH, const int *cctk_lsh,
                const CCTK_REAL *cctk_delta_space, int ghost,
                CCTK_REAL *coord, CCTK_REAL coord_min, int dir)
{
    int i, ijk, min_i = -1;
    CCTK_REAL min = 1.e100;

    for(i=ghost; i<cctk_lsh[dir]-ghost; i++) {
        ijk = CCTK_GFINDEX3D(cctkGH, (dir==0)?i:0, (dir==1)?i:0, (dir==2)?i:0);

        if (fabs(coord[ijk] - coord_min) < min) {
            min = fabs(coord[ijk] - coord_min);
            min_i = i;
        }
    }
    return min_i;
}