Getting Started with AstroGK

Obtain the source code

The source code of AstroGK is managed by the source code management software git. It is served by the version control repository hosting service, bitbucket https://bitbucket.org/gyrokinetics/agk/. There are some branches, such as the master branch, master, or the development branch, next, etc. To obtain the source code of the master branch, type the following command

git clone https://bitbucket.org/gyrokinetics/agk.git work

Note: Here, the working copy (directory) is set to work. In the following, the working copy is denoted by work. If the working copy name is omitted in the clone command, the default is the project name, agk.

AstroGK depends on other projects, utils and Makefiles, that are shared by other projects, Those dependent projects are set as the submodules of AstroGK. To obtain the submodules, git submodule command is used in the working copy directory.

cd work
git submodule update --init

Or, the submodules are cloned when downloading the working copy of AstroGK with the following option.

git clone --recurse-submodules https://bitbucket.org/gyrokinetics/agk.git work

Note: The --recurse-submodules option is available for git version 2.13 or higher. For the older version of git, use --recursive.

Note: The server must be accessed via https. To set the proxy used by git, use git config command. The configuration is stored in ${HOME}/.gitconfig. (Consult the network administrator for the information of the proxy server.)

git config --global https.proxy https://<IP ADDRESS>:<PORT NUMBER>

Build the program

To build the program, a Fortran compiler, and MPI, FFTW (v3), NeteCDF, HDF5 libraries are required. (It can be build without those libraries, but is limited its functions.)

In the environment where the required softwares are available, set the environment variables, MAKEFLAGS and GK_SYSTEM, and execute make. Set -IMakefiles to MAKEFLAGS. Set the "system name" on which the program is build and executed to GK_SYSTEM. The system name is used to choose an appropriate Makefile in the Makefiles directory.

Note: In the computation system in Graduate School of Simulation Studies (abbreviated as SS hereafter), the Intel compiler (the intel module) is recommended as the programming environment. For the required libraries, load the mpt, fftw, hdf5-parallel, netcdf_p modules. Check the version of the softwares as they are updated occasionally.

Note: In the computation system in SS, set GK_SYSTEM=rokko. See Makefiles/Makefile.rokko for details. The explicit steps to build AstroGK on the front-end server rokko of the computation system in SS are the following.

module load intel
module load mpt fftw
module load hdf5-parallel netcdf_p
export MAKEFLAGS=“-IMakefiles"
export GK_SYSTEM=rokko
make

To port the code to a new system, create Makefile.${GK_SYSTEM} corresponding to the system name (GK_SYSTEM) in the Makefiles directory. In the system Makefile, configure the compiler and libraries to be used. The available compilers are the compilers whose corresponding Makefile exists in Makefiles. (eg. Makefile.intel)

Execute a sample run

AstroGK is flat-MPI parallelized. The special command (eg. mpirun, mpiexec) is used to run the MPI-parallelized programs. To run AstroGK with the number of processes $N$ , type the following command in the directory where the executable and input file (an arbitrary name with suffix .in)

mpiexc -n N ./agk input.in

The output are stored in the files having the same basename as the input with different suffixes. The suffixes are determined by the output data type. There are the output files for continuation runs which stores the status of the simulation at the end of the calculation. Some files are text files, which can be easily viewed by text editors, and some files are binary files with NetCDF or HDF formats, which are handled by special programs (See below for visualization of the data using GNU Octave).

There are several sample input files in test_jcp2010/. See the AstroGK paper for the details of the samples. For the simulation of magnetic reconnection (tearing instability) problem, use tearing.in. It is just a linear simulation. For this sample case, the parallel efficiency scales well up to 256 cores, and the simulation ends about one hour (with SGI Altix UV1000, Intel Xeon X7550 2.00GHz). For nonlinear simulations, the number of grids in the $y$ direction, ny, should be increased.

Note: It is recommended to use the computation server in SS for the magnetic reconnection simulation as it takes a bit long time. Refer the users guide and manual for the usage of the computation server. The sample job script to run the simulation on the cluster server is provided at rokko1:/data/n272r011/share/run_tearing.sub.

Visualize the result using GNU Octave

Unfortunately, There is no integrated framework for the analyses of the AstroGK simulation results. Users should develop their own programs/scripts at the moment. For the analyses of the magnetic reconnection simulation, some scripts of GNU Octave are provided by Numata, which are available at https://rnumata.org/redmine/projects/octave_scripts. Git repository is https://rnumata.org/git/octave_scripts.git. To clone the code, type the following command

git clone https://rnumata.org/git/octave_scripts.git

The directory named octave_scripts/ is created in the current directory.

Note: To use the script developed by Numata, Octave must be configured appropriately. Check Tools for Simulation Study to see how to configure Octave. It is necessary to set the search patch of scripts and to load required packages (netcdf, io).

Note: On the front-end server of SS, rokko, netcdf and io packages for Octave are already installed. To use them, load the octave module. It is also required to load the hdf5-parallel and netcdf_p (or hdf5, netcdf) modules to use the netcdf package.

Launch Octave, and run some scripts. For example, plgk_field.m plots 2D electromagnetic fields. The script takes mandatory arguments, basename, by which the basename of the input file for AstroGK is specified (here we assume input is the basename), and varname, by which the variable name to be plott is specied (phi=the electrostatic potential ( $φ$ ), apar=the $z$ component of vector potential $A$ _∥，bpar=the $z$ component of magnetic field $δB$ _∥)

plgk_field (‘basename=input’,'varname=apar')

The script reads several AstroGK output files corresponding to the basename, it is convenient to run Octave at the directory where the executable and output exist. See the head of the script files for the details of other arguments. Note that all the arguments are 'key=value' pairs of a character type, so must be surrounded by single quotes '. For example, if ascii (keyword) is 1 (value), the script outputs the plotted data as a readable ASCII file, "plgk_field.dat".

plgk_field (‘basename=input’,'ascii=1')

How to use AstroGK on your own PC?

To run AstroGK on your PC, the following softwares must be installed. The detail of each tool can be found in Tools for Simulation Study.

git
Fortran compiler (eg. gfortran)
MPI library (eg. mpich)
FFTW (v3), HDF5 (parallelized version recommended), NetCDF (parallelized version recommended)
GNU Octave, and netcdf, io packages for Octave
gnuplot (In the plotting scripts for Octave, it is convenient to use gnuplot as a graphics_toolkit backend because it can handle the Greek symbols and special characters.)

The code can be run on Windows PC, but it is not recommended.