3.7 Compiler Macros

Some of the modules of the CO5BOLD code (with suffix ``.F90'') employ compiler macros to switch between code versions during compile time. Typically you define at least one of the three switches rhd_r01, rhd_r02, or rhd_r03 to choose a radiation transport module. The others have reasonable default values. To find the combination with the optimal performance, you should look into Sect. 3.8

The macros are sorted into different categories:

Some activate a certain feature (like a radiation transport module or the dust module). They have to be selected by the user (typically via environment variables and the configure script, see Sect. 3.6) each time the code is compiled for a certain purpose.

Other macros are meant to improve the performance by offering the choice between e.g. different loop structures or case distinctions. These macros are set by the configure script to the best knowledge of the author(s). Ideally, they should be checked and modified if necessary each time CO5BOLD is compiled on a new machine. It should be save to modify these settings: the results between runs with different settings should only differ slightly due to round-off errors.

Some macros select between different numerical approximations. A change here should be visible in a (more or less drastic) change of the results of a simulation. Usually, the default values should be accepted. Other settings typically only exist to allow the comparison with older versions of CO5BOLD or because there are new developments going on which have not yet managed to become the default.

A couple of macros only activate timing measurements and result in additional output. Some of them are not thread-save und should only be activated for runs on one thread (as done by the configure script). It is always save to switch any of them off (by removing or undefining them).

The macros in the category test mark parts of code under development. The default values should only be changed with great care (typically by the author of that code segment). The configure script does not touch these settings.

General:

• timing_c_factor:
  in timing_module.F90, (``timing count factor'').
  Category: account for property of machine.
  To produce the timing statistics printed at the end of a simulation run the standard Fortran routine SYSTEM_CLOCK is used. The macro timing_c_factor specifies by how much the count rate of this routine is reduced when storing its count value. This does not prevent all overflows but can make the output much more useful. Values:
  • 1: (default) count rate of SYSTEM_CLOCK is used directly.
  • otherwise: e.g. 1000, count rate of SYSTEM_CLOCK is reduced by this factor.
  By a proper choice of this factor the timing measurements of individual routines can be made meaningful: the reduction of the count rate prevents overflows due to the addition of several measurements. An overflow during an individual measurement can not be prevented. Therefore, the count rate for the entire program still tends to produce overflows.

• gasinter_l01:
  in gasinter_routines.F90, (``gas interpolation l01'').
  Category: performance enhancement.
  This switch determines how temporary arrays are handled to improve performance Values:
  • 0: (default) Temporary coefficient arrays are actually copied.
  • 1: Temporary coefficient arrays just get a pointer link into the big arrays.

• rhd_box_grav01:
  in rhd_box_module.F90, (``rhd box gravitation 01'').
  Category: feature activation.
  Switch to activate the array for the gravitational potential in the box structure. If the switch is set to 1, a 3D array for the potential is created, copied, removed, ... There is no module to compute the gravitational potential, yet. Therefore the entire thing has no practical value, yet. Values:
  • 0: (default) no handling of array.
  • 1: array handling activated.

• rhd_box_quc01:
  in rhd_box_module.F90 and rhd.F90, (``rhd box quantity centered 01'').
  Category: feature activation.
  Now, CO5BOLD is able to handle a number of additional quantities (e.g. density arrays) in addition to the basic hydrodynamics quantities ($\rho$, $e{\rm i}$, ...) if this compiler switch is activated. These additional quantities can be e.g. densities of dust distribution moments or densities of molecules. Values:
  • 0: (default) no handling of additional quantities (density arrays).
  • 1: handling of additional density arrays is activated.
  To actually include dust formation in a simulation, it is necessary to
  1. set the switch -Drhd_box_quc01=1 during compilation (this is done by the configure script if the environment variable F90_DUST is set to 1, see the description of the variable in Sect. 3.6),
  2. put arrays specifying the initial conditions of the additional density into the start model (as real quc001, real quc002, ...),
  3. select a proper model describing dust (or molecule) formation in the parameter file (with character dustscheme).

• rhd_box_bmag01:
  in rhd_box_module.F90 and rhd.F90, (``rhd box b magnetic 01'').
  Category: feature activation.
  CO5BOLD can handle magnetic field arrays if this compiler switch is set. Values:
  • : (default) no handling of magnetic field arrays.
  • : handling of magnetic field arrays is activated.
  To actually account for magnetic fiels in a simulation, it is necessary to
  1. set the switch -Drhd_box_bmag01=1 during compilation (this is done by the configure script if the environment variable F90_MHD is set to 1, see Sect. 3.6),
  2. put arrays specifying the initial conditions of the boundary centered magnetic field arrays into the start model (as real bb1, real bb2, real bb3),
  3. select an hydrodynamics scheme that is able to handle magnetic fields in the parameter file (with character hdscheme).

Hydrodynamics (Roe solver):

• rhd_roe1d_slope_l01:
  in rhd_hyd_module.F90, (``rhd roe 1 dimension slope loop 01'').
  Category: feature activation.
  When this compiler switch is set, a new extra stabilization mechanism can be activated: If one of the reconstruction methods VanLeer, Superbee, or PP (see Sect. 5.3.7) is activated, the slope can be reduced (by averaging with the results from a MinMod reconstruction) by setting c_slopered (see Sect. 5.3.7) to a positive non-zero value. This can improve the stability without significantly reducing the effective numerical resolution. Switch values:
  • 0: (default) no slope reduction.
  • 1: slope reduction in case of expansion wave.
  • 2: slope reduction in case of strong density contrast.

• IDF:
  in rhd_hyd_module.F90, (``Integer Delta Flux'').
  Category: performance enhancement.
  Number of padding cells for flux-like variables. This number was introduced to check whether the increase of the size of vectors for flux-like quantities (defined at cell boundaries) can improve the performance (especially on a CRAY machine). The gain is marginal (if present at all). The parameter is usually set to zero or left undefined. Values:
  • 0: (default) no padding cells
  • 1,2,3,...: extra padding cells

• rhd_hyd_gravcorr_p01:
  in rhd_hyd_module.F90, (``rhd hydrodynamics gravitation correction parameter 01'').
  Category: selection of approximation.
  This parameter controls the way the Roe solver handles the source terms due to gravity. A different choice results in different simulation results and not just in slightly faster (or slower) code. The problem is that the original Roe solver interpretes the pressure gradient in a hydrostatic stratification a fluctuation due to shock waves. In case of strong stratification this can lead to weird effects. With activated correction the Roe solver treats only the deviations from a hydrostatic stratification as due to waves (or shocks). Several correction formulas have been tried. The latest is the recommended default. Values:
  • 0: No pressure correction terms in Roe solver.
  • 1: Simple correction with rhomean, no new average pressure.
  • 2: Simple correction with rhomean, new average pressure.
  • 3: Correction with local rho, limited, new average pressure.
  • 4: Correction with local rho, new (different formula) average pressure.
  • 5: (default) Correction with local rho, new limit, new average pressure.

• rhd_hyd_entropyfix_p01:
  in rhd_hyd_module.F90, (``rhd hydrodynamics entropy fix parameter 01'').
  Category: performance enhancement.
  The entropy fix can be done in one of two ways to get optimum performance (with essentially the same results). Values:
  • 0: (default) ``if...then...else'' construction
  • 1: use a mask and the signum function

• rhd_hyd_upwind_p01:
  in rhd_hyd_module.F90, (``rhd hydrodynamics upwind parameter 01'').
  Category: performance enhancement.
  The determination of the upwind direction can be done in one of two ways to get optimum performance (with essentially the same results). Values:
  • 0: (default) ``if...then...else'' construction
  • 1: use a mask and the signum function

• rhd_hyd_roe1d_l01:
  in rhd_hyd_module.F90, (``rhd hydrodynamics roe 1 dimension loop 01'').
  Category: performance enhancement.
  The computation of the Roe fluxes can be done by either of two sets of routines to find the set which gives optimum performance (with essentially the same results). Values:
  • 0: (default) lots of small routines acting on scalars, inlining needed, cache reuse is optimized
  • 1: routines acting on arrays, more temporary arrays necessary, vectorization is easier

• rhd_roe1d_flux_l01:
  in rhd_hyd_module.F90, (``rhd roe 1 dimension flux loop 01'').
  Category: test.
  By setting this switch an alternative way of computing the upwind centered Roe states is activated (only for 'constant' reconstruction, for performance test purposes only: do not activate!). Values:
  • undefined: (default) Use standard method to compute the Roe states.
  • defined: Use non-standard method to compute the Roe states.

• rhd_bound_t01:
  in rhd_hyd_module.F90, (``rhd bound timing 01'').
  Category: additional output.
  Produce timing information for ``inner boundary'' routine (central potential) or lower and upper boundary routines (constant gravitation). It can be used together with OpenMP. Values:
  • undefined: (default) no timing information.
  • defined: call subroutines to measure elapsed time.

• rhd_roe1d_flux_t01:
  in rhd_hyd_module.F90, (``rhd roe 1 dimension flux timing 01'').
  Category: additional output.
  Produce timing information for the routine which computes the Roe fluxes. It should not be used in conjunction with OpenMP. Values:
  • undefined: (default) no timing information
  • defined: call subroutines to measure elapsed time

• rhd_roe1d_step_t01:
  in rhd_hyd_module.F90, (``rhd roe 1 dimension step timing 01'').
  Category: additional output.
  Produce timing information for the routine which performs the Roe step. It should not be used in conjunction with OpenMP. Values:
  • undefined: (default) no timing information
  • defined: call subroutines to measure elapsed time

Hydrodynamics (tensor viscosity):

• rhd_vis_density_p01:
  in rhd_vis_module.F90, (``rhd viscosity density parameter 01'').
  Category: selection of approximation.
  Choose formula for density average at cell boundary in tensor viscosity routines. Values:
  • 0: rhomean=min(rholeft,rhoright)
  • 1: (default) rhomean=0.5 * (rholeft + rhoright)

• rhd_vis_t01:
  in rhd_vis_module.F90, (``rhd viscosity timing 01'').
  Category: additional output.
  Produce timing information for 2D/3D tensor viscosity routines. It should not be used in conjunction with OpenMP. Values:
  • undefined: (default) no timing information
  • defined: call subroutines to measure elapsed time

Radiation transport:

• rhd_r01:
  in rhd.F90, (``rhd radiation 01'').
  Category: feature activation.
  Switch to include LHDrad radiation transport module. It uses long characteristics and is restricted to an equidistant grid and open boundaries at all surfaces (old ``supergiant module''). Values:
  • undefined: (default) LHDrad routines are deactivated.
  • 1: LHDrad routines are recognized by the compiler.

• rhd_r02:
  in rhd.F90, (``rhd radiation 02'').
  Category: feature activation.
  Switch to include MSrad radiation transport module. It uses long characteristics. The lateral boundaries have to be periodic. Top and bottom can be closed or open (``solar module''). Values:
  • undefined: (default) MSrad routines are deactivated.
  • 1: MSrad routines are recognized by the compiler.

• rhd_r03:
  in rhd.F90, (``rhd radiation 03'').
  Category: feature activation.
  Switch to include SHORTrad radiation transport module. It uses short characteristics and is restricted to an equidistant grid and open boundaries at all surfaces (new ``supergiant module''). Values:
  • undefined: (default) SHORTrad routines are deactivated.
  • 1: SHORTrad routines are recognized by the compiler.

• rhd_rad3d_toray_l01:
  in rhd_lhdrad_module.F90, (``rhd radiation 3 dimensions to ray loop 01'').
  Category: performance enhancement.
  There might be a performance gain by splitting the main loop in routine rhd_rad3d_toray into three separate loops. Typically, one big loop is to be preferred. Values:
  • undefined: (default) One big loop
  • defined: Three smaller loops

• rhd_rad3d_fromray_l01:
  in rhd_lhdrad_module.F90, (``rhd radiation 3 dimensions from ray loop 01'').
  Category: performance enhancement.
  There might be a performance gain by splitting a big loop in routine rhd_rad3d_fromray into two separate loops. Typically, one big loop is to be preferred. Values:
  • undefined: (default) One big loop
  • defined: Two smaller loops

• rhd_rad3d_r02:
  in rhd_lhdrad_module.F90, (``rhd radiation 3 dimensions radiation 02'').
  Category: test.
  Module rhd_lhdrad_module contains a routine for the handling of periodic boundaries. It is in an experimental state and is deactivated by default. Values:
  • undefined: (default) Skip routine rhd_rad3d_dirper during compilation
  • defined: Compile routine rhd_rad3d_dirper

• rhd_rad3d_solve_t01:
  in rhd_lhdrad_module.F90, (``rhd radiation 3 dimensions solve timing 01'').
  Category: additional output.
  Produce timing information for the routines which solves the 1D radiation transport equation along single ray. This routine is called very frequently. The timing measurement might slow it down somewhat. It should not be used in conjunction with OpenMP. Values:
  • undefined: (default) no timing information
  • defined: call subroutines to measure elapsed time

• rhd_rad3d_dir_t01:
  in rhd_lhdrad_module.F90, (``rhd radiation 3 dimensions direction timing 01'').
  Category: additional output.
  Produce timing information for the routines which solves the radiation transport equation for one direction field. The timing measurement are called very frequently and might slow down the code. It should not be used in conjunction with OpenMP. Values:
  • undefined: (default) no timing information
  • defined: call subroutines to measure elapsed time

• rhd_rad3d_step_t01:
  in rhd_lhdrad_module.F90, (``rhd radiation 3 dimensions step timing 01'').
  Category: additional output.
  Produce timing information with main 3D radiation transport routine. It can be used together with OpenMP and should cause no noticeable performance loss. Values:
  • undefined: (default) no timing information
  • defined: call subroutines to measure elapsed time

• rhd_shortrad_operator_l01:
  in rhd_shortrad_module.F90, (``rhd short-characteristics radiation operator loop 01'').
  Category: performance enhancement, selection of approximation.
  Choose type of short characteristics operator. The operators usually come in pairs (1/2, 3/4, 5/6). There is a development from 1/2 over 3/4 to 5/6 towards higher stability. Both members of each pair should do the same operation but use different ways to do a case distinction. The 'even' operator has in most case the better performance. But the 'odd' operator might be saver to use. Values:
  • 0: simple test operator, fast but results are utterly useless!
  • 1: case distinction with ``if..then..else'' construct.
  • 2: case distinction with masks (weights 0.0 or 1.0).
  • 3: case distinction with ``if..then..else'' construct, slope reduction of source function.
  • 4: case distinction with masks (weights 0.0 or 1.0), slope reduction of source function.
  • 5: case distinction with ``if..then..else'' construct, modified slope reduction of source function.
  • 6: (default) case distinction with masks (weights 0.0 or 1.0), modified slope reduction of source function.
  • 8: test version.

• rhd_shortrad_operator_l02:
  in rhd_shortrad_module.F90, (``rhd short-characteristics radiation operator loop 02'').
  Category: performance enhancement.
  Select the way the short characteristics operator is accessed. Values:
  • 0: (default) The routine with the short characteristics operator is called within a loop and should be inlined.
  • 1: The program fragment with the short characteristics operator is included. No inlining necessary.

• rhd_shortrad_dtauop_l01:
  in rhd_shortrad_module.F90, (``rhd short-characteristics radiation delta tau operator loop 01'').
  Category: performance enhancement.
  Choose type of short characteristics tau coupling operator. Values:
  • 1: case distinction with ``if..then..else'' construct, default if rhd_shortrad_operator_l01=1,3,5.
  • 2: case distinction with masks (weights 0.0 or 1.0), default if rhd_shortrad_operator_l01=2,4,6.

• rhd_shortrad_dtauop_l02:
  in rhd_shortrad_module.F90, (``rhd short-characteristics radiation delta tau operator loop 02'').
  Category: performance enhancement.
  Select the way the operator for the tau coupling (short characteristics module) is accessed. Values:
  • 0: (default) The routine with the tau coupling operator is called within a loop and should be inlined.
  • 1: The program fragment with the tau coupling operator is included. No inlining necessary.

• rhd_shortrad_formal_l01:
  in rhd_shortrad_module.F90, (``rhd short-characteristics radiation formal loop 01'').
  Category: performance enhancement.
  Select version of loop splitting for exp(-dtau) computation. Values:
  • 0: (default) dtauhalf, exp_mdtauhalf, expl2t_mdtauhalf are computed in single loop
  • 1: (dtauhalf, exp_mdtauhalf), (expl2t_mdtauhalf) are computed in separate loops. This prevents the SUN1 machine (Sunfire, Solaris, Forte 6.2) from doing some performance degrading optimization

• rhd_shortrad_dir1_l01:
  in rhd_shortrad_module.F90, (``rhd short-characteristics radiation direction 1 loop 01'').
  Category: performance enhancement.
  Choose routine version for rays in x1 direction. Values:
  • 0: (default) Use routine with permuted indices for rays in x1 direction. In this case the innermost loop index is the third array index. The transposition of arrays is not needed but some machines (e.g. SUN1) do not like this index arrangement.
  • 1: Transpose arrays and use routine rhd_shortrad_dir3 for rays in x1 direction. The extra step for the transposition of some arrays (and the reverse procedure) needs some time. But now the routine with the optimum index ordering can be used.

• rhd_shortrad_dir_l02:
  in rhd_shortrad_module.F90, (``rhd short-characteristics radiation direction loop 02'').
  Category: performance enhancement.
  Determine position of PARALLEL statement relative to outer loop in rhd_shortrad_dirX. Both settings give the same results but might show a different performance on a specific machine. Values:
  • 0: (default) PARALLEL statement inside of outer loop
  • 1: PARALLEL statement outside of outer loop

• rhd_shortrad_lambda_l01:
  in rhd_shortrad_module.F90, (``rhd short-characteristics radiation lambda loop 01'').
  Category: feature activation.
  Handling of extra arrays to allow partially implicit Lambda* iteration Values:
  • 0: (default) Only fully implicit Lambda* iteration allowed (or fully explicit treatment).
  • 1: Also partially implicit Lambda* iteration allowed.

• rhd_shortrad_formal_t01:
  in rhd_shortrad_module.F90, (``rhd short-characteristics radiation formal timing 01'').
  Category: additional output.
  Produce timing information for routine which gives the formal solution of the radiation transport equation with the help of short characteristics. It can be used together with OpenMP and should cause no noticeable performance loss. Values:
  • undefined: (default) no timing information
  • defined: call subroutines to measure elapsed time

• rhd_shortrad_step_t01:
  in rhd_shortrad_module.F90, (``rhd short-characteristics radiation step timing 01'').
  Category: additional output.
  Produce timing information for main short characteristics routine. It can be used together with OpenMP and should cause no noticeable performance loss. Values:
  • undefined: (default) no timing information
  • defined: call subroutines to measure elapsed time

• MSrad_raytas:
  in MSrad3D.F90, (``Matthias Steffen radiation ray tau s'').
  Category: performance enhancement.
  Values:
  • 0: (default) Loop with IF..THEN..ELSE
  • 1: Loop with ABS,SIGN
  • 2: Loop with MIN,MAX