Guide to parameters

Although many of the parameters in the params.txt file are self explanatory and commented, this page aims to provide a comprehensive description of all the non-problem-specific parameters available in the existing examples. A useful pdf guide on this topic from the latest GRChombo training day can be found in Useful resources.

Checking parameters are sensible

A useful feature in GRChombo is the ability to run a quick check on the validity of the parameters before a job is submitted to the batch system. Such a check is small enough to run on the login nodes of most systems without creating too much of a load, and ensures the job will not fail immediately due to parameter related errors.

This is done by passing the -check_params flag i.e. by running something like the following command:

mpiexec /path/to/exec.ex /path/to/params.txt -check_params

Note that the checks mostly just ensure that the simulation will run and in particular don't tell you if you are doing something very inefficient with regard to the parallelisation of the run. More detailed information about load balancing using the grid parameters is provided on the GRChombo performance optimisation page. These parameters are essential for using computing time efficiently, so this is highly recommended reading!

Updating parameters during a run

Most parameters can in principle be updated on restart from checkpoint, except those related to the Grid Setup and any initial conditions (which obviously require a restart from t=0).

However, one should take care when updating block_factor and max_grid_size as this can cause issues with regridding at later times (usually seen as the boxes becoming "stuck" at certain positions). Also changing the gauge or dissipation parameters will generally cause unphysical transients and so is not recommended (although we do not prohibit it).

Guide to specific parameters

For the purposes of this page, we will refer to the params.txt in the current BinaryBH example.

Note that a hash # denotes a comment and the rest of the line will not be read, so to activate a parameter this should be removed at the start of the line. Parameters that are unlikely to change are set to defaults, and the corresponding values are commented out in the params.txt file so they are easy to identify.

Filesystem parameters

• verbosity can be set to 0, 1, 2,... . Generally, the higher it is set, the more information is output to the pout file from each MPI rank (or just stdout if running in serial). GRChombo currently does not do anything different above 2 but the underlying Chombo library may output more up to 5.
• chk_prefix and plot_prefix are the directory and filename prefixes for the checkpoint and plot hdf5 files respectively, which will be output during the simulation.
• checkpoint_interval is the number of coarse time steps to wait before outputting a checkpoint file. Since checkpoint files are bigger than plot files, this should be greater than plot_interval.
• plot_interval is the number of coarse time steps to wait before outputting a plot file.
• Checkpoint files contain all grid values for all evolution variables on the grid, hence simulations can be restarted from them. Plot files contain all grid values only for evolution and/or diagnostic variables specified in advance. For the BoostedBHComplexScalar example, this is just num_plot_vars=3 - the real part for the scalar field phi_Re and the linear momentum density and source quantities, as specified by plot_vars. As a result, plot files tend to be significantly smaller than checkpoint files and can be output more frequently. One can choose to write_plot_ghosts but this is usually not necessary unless one plans to use them as checkpoints for another run.
• Each checkpoint/plot filename prefix will be appended with a 6 digit number corresponding to the time step on the coarsest level, the number of spatial dimensions (which should be 3 for these examples) and the hdf5 file extension. For example if the prefix is set to BoostedBH_, then a checkpoint file from the 10th coarse time step will have name BoostedBH_000010.3d.hdf5.
• If a directory is not explicitly specified, then files will be output to the directory in which the example has been executed. The can be overwritten but providing a base directory in output_path, which needs to exist at runtime. Generally, when running on a cluster, you'll want to change it to some local scratch storage or a space with fast I/O. Refer to your cluster documentation/administrator for help with this.
• Note that Unix style relative paths, for example ../filename-prefix, will probably not work and you should use absolute paths.
• If you need to restart a run from a checkpoint file for whatever reason (usually because you ran out of time or the system crashed), uncomment (remove the #) before restart_file and set it to the full checkpoint filename (including the absolute path if different from where the example is executed).
• By default 3 folders will be created in output_path to store the hdf5 files for checkpoints and plot files hdf5, the output from processors pout and any data files generated data. These names can be amended by passing values for hdf5_subpath, pout_subpath and data_subpath. These subfolders do not need to exist at runtime and will be created.
• During the simulation each MPI process outputs data on what it is doing to the pout folder. Setting print_progress_only_to_rank_0 to 1 is usually a sensible thing to do as a lot of the output is duplicated between processors. Note that since each processor acts on a different part of the grid, different processors may encounter different issues - e.g. in detecting Nans - so you should check all pout files if your job stops suddenly (a useful command is grep -r 'Nan' pout/). This command just stops the output of data which indicates the current speed and time step on each level, which in general is redundant between processes.

Grid parameters

• N1, N2 and N3 are the number of cells on the coarsest level along each dimension. Note that this needs to be a multiple of the block_factor (minimum box size). For general 3-dimensional problems, you'll probably want to set them to the same value but for problems that are essentially 2 or 1-dimensional dynamically, you can set different values appropriately.
• L is the coordinate length of the longest side of the grid. The spatial step size dx on the coarsest level is L/Nmax where Nmax is the maximum of N1, N2 and N3. Note that dx is the same along each dimension, even if the N values differ.
• There is an option to specify alternatively N_full and L_full which provide the values ignoring the imposition of symmetries. So for example, for a single black hole in a domain of length L=128 with the black hole at the centre, but where symmetries are used to reduce the actual width of the simulation domain to L=64 with the black hole at the edge (ie, only evolving half the grid), L_full would remain set to 128, but L would need to be set to 64. A similar logic applies to N_full and N. These symmetries are also taken into account by the center parameter, which can usually be left to default.

For more information on GRChombo's implementation of the Berger-Rigoutsos AMR algorithm, see §3.1.2 of Katy Clough's PhD thesis. The following bullet points will assume some familiarity with it.

• max_level is the maximum number of refinement levels above the coarsest level, so there are max_level + 1 levels in total.
• regrid_interval is the number of time steps to wait before tagging/regridding occurs on each level. For fixed background problems, you do not need to regrid at all (set regrid_interval to 0 0 0 ...), since the mesh is always refined initially anyway. Note that max_level values are needed (as no regrinding occurs on the top level).
• regrid_threshold is the overall threshold which is used to decide whether or not to tag the cell, commented out as it is not used on the fixed grid used in this version of the code.
• max_grid_size, alias 'max_box_size' is the maximum number of cells along each dimension of a box (the boxes are the subsets of the grid which are shared between processors). This should definitely be greater than or equal to 8 and probably around 16 or 32 but no larger.
• block_factor, alias 'min_box_size' is the minimum number of cells along each dimension of a box. This should ideally be an integer multiple of the vector width for doubles e.g. 8 for AVX-512 (assuming you've compiled with the appropriate flags) and probably shouldn't be smaller as then the high number of ghost cells can cause memory issues. It must be a power of 2 (for a ref_ratio of 2).
• You may want to set max_grid_size and block_factor to the same value (e.g. 16) so that you know exactly how large each box is going to be. This will make it easier to optimise load balancing.
• tag_buffer_size is the number of additional cells that are added to tagged regions to smooth them out, and defaults to 3. For a fixed grid, this is set to 0.
• center should default to the centre of the grid, taking into account any symmetries, but can be overwritten.

Boundary Conditions parameters

• isPeriodic is a vector of boolean values corresponding to whether the grid boundaries along each dimension are periodic or not. If one wants to have periodic boundary conditions it should be set to 1 1 1. However, GRChombo has also implemented fixed, sommerfeld, reflective and extrapolating boundary conditions in which case isPeriodic should be be 0 0 0 and the precise condition must be chosen by using the additional parameters below.
• hi_boundary and lo_boundary are two 3d vectors that each component carries information about the boundary conditions of the coordinate at the high and low part of the simulation box. They should be set to 0 for static, 1 for sommerfeld, 2 for reflective and 3 for extrapolating. For example, if one wanted to have reflective boundary condition in $z=0$ with sommerfeld at $x=y=0$ and all fixed for $x=y=z=L$, the vectors should be set to lo_boundary = 1 1 2 and hi_boundary = 0 0 0.
  • vars_asymptotic_values: If Sommerfeld or extrapolating boundary conditions are chosen the user must specify the num_nonzero_asymptotic_vars and their names nonzero_asymptotic_vars and values nonzero_asymptotic_values in the corresponding order for those using Sommerfeld conditions.
  • vars_parity: If reflective boundary conditions are chosen the user must specify the parity of each variable at each boundary in vars_parity. 1,2,3 = odd $x$, $y$, $z$ and 4,5,6 = odd $xy$, $yz$, $xz$. If a variable is even under all xyz directions it is set to vars_parity = 0. Each component of the vector corresponds to a variable in UserVariables.hpp. Note that the order should be the same as the one in UserVariables.hpp. The same should be provided for DiagnosticVariables.
  • If extrapolating boundaries are selected, the number of extrapolating variables num_extrapolating_vars and their names extrapolating_vars should be specified along with the order of extrapolation extrapolation_order. The remaining variables will be assumed to be Sommerfeld and so any asymptotic values for these must also be provided as above.

Evolution parameters

• dt_multiplier is the factor you need to multiply the level spatial step size dx to get the time step size dt. This is sometimes known as the CFL (Courant-Friedrichs-Lewy) or just Courant factor. For most GR problems, this will probably need to be less than 0.5 for stability and usually 0.25 is a reasonable choice.
• stop_time is the coordinate time up to which the simulation is run.
• max_steps is a cut-off for the maximum number of coarse time steps so that the program does not run out of control.
• nan_check is a bool, and usually set to 1 to make sure the code checks for instabilities and exits normally. You may wish to turn this off (ie, set it to 0) for debugging or for final (well-tested) production runs, but it is strongly advised during development.

Initial Data parameters

• scalar_mass = $M \mu$ sets the ratio between the scalar field mass, $\mu$, and the mass of the BH $M$.
• scalar_amplitude is the initial the amplitude of the field
• `bh_mass' is the mass of the BH, $M$, usually set to 1.
• bh_velocity in the case of a boosted BH, is the velocity of the boost, in units of c.
• bh_spin is the BH spin in the case of a Kerr metric.

Diagnostic extraction parameters

See the guidance at Extraction and integration.