Readme file for GRMHD version

Developed by Yosuke Mizuno
RAISHIN ver. 140127

RAISHIN code is used a conservative finite volume shock-capturing schemes (Gudonov-type schemes) to solving the hyperbolic partial differential equations (PDE). This version of code is specialized for solving 3D ideal general relativistic magnetohydrodynamics (3DGRMHD) using 3D MPI parallelization based on the formulation used in Koide (2003).

Including files: 
README: This file, general documentation
makefile: makefile for main exec, data gathering, and output for data analysis 

# Main computation programs

main.f90: Primary routine
pram.f90: module file for general parameter set
bhmodel.f90: Subroutines for Initial parameter set for BH simulation
bnd.f90: Boundary condition
calcha.f90: Subroutines for calculation of characteristics
calflx.f90: Subroutines for calculation of numerical flux (not Riemann solver)
calsf.f90: Subroutines for calculation of source term
caluu.f90: Subroutines for calculation of conserved variables
cdtcfl.f90: Subroutines for calculation of dt from CFL condition
coord.f90: Metrics
ct.f90: Subroutines for constrained transport schemes
damp.f90: Subroutines for dumping (special treatment)
deriv.f90: Subroutines for higher-order interporation (not used)
fluctu.f90: Subroutines for making fluctuations
func.f90: General functions using RAISHIN code
gravp.f90: Subroutines for calculation of gravitational potential
grid.f90: Set for making grid
hll.f90: Subroutines for Approximate Riemann solver schemes
inpf.f90: Subroutines for input flow from boundary
mdrmhd.f90: Subroutines for Initial setup of each models (for RMHD)
mdgrmhd.f90: Subroutines for Initial setup of each models (for GRMHD)
mpisub.f90: Subroutines for MPI
pminmax.f90: Subroutines for minimum and maxmum density & pressure floor
rec.f90: Subroutines for reconstruction schemes
recov.f90: Subroutines for inversion procedure schemes
restar.f90: Subroutines for restart
rkt.f90: Subroutines for time advance schemes
spdlia.f90: Subroutines for check velocity
trans.f90: Subroutines for calculation of radial accretion flow (Bondi solution)

# Data gathering
remake_data3D.f90: Additional routines for data gathering

# Rearrange of data for analysis
inkt: Input parameter set for data rearranging
kamt.f: Additional routines for data rearranging

START RUNNING:

First copy of makefile:
>cp Makefile_xgrmhd Makefile

Then make a exec file:
>make

Please change using compiler and compiler option for your running computers.

Running exec files: 
>./xgrmhd

Example:
Running Mac OS X using gfortran with MPI. In makefile, using
FC = mpif90
FFLAGS = -O2 -Wall
For running exec files,
> mpiexec -n2 ./xgrmhd (using 2 cpus)

The following files will be generated by RAISHIN code:

--./restart[000-NNN].outdat: output data for restart from each cpu. [000-NNN] is cpu number
---./structr[000-NNN]_[000-MMM].outdat: output data for data analysis from each cpu ([000-NNN]) and each output time ([000-MMM]).

Restart simulations:

The RASHIN code is handling the simulation from using computer time (icpu), maximum number of iteration (itmax), or Maximum simulation time (tmax). If the cpu time reachs maximum before maximum simulation time, the simulation is stopped. Therefore we need to restart the simulations. When the simulation restarts, you set "icres=1" in pram.f90 and make the exec again (need "make clean"). Then restart the simulation.   

Data gathering:

If you use multiple cpus for running the simulation, the output data is discreted by each cpu. Therefore we need to gather all discreted output file into one output file.

>cp Makefile_remake Makefile  
>make
>./xremake

Please check consistency of grid number, number of cpus, and start and end data file number from main exec.

Rearrange data set for data analysis:

After making one data file (structr.outdat), we use additional procedure of making data files for IDL or AVS. 

>cp Makefile_xkamt Makefile
>make
>./xkamt

In kamt.f we comvert from (x1,x2,x3) to (x1,x3,x2). If we make 2D data set using "nq3d=2", IDL data set is made as (x1,x3). The making "ok#" is data set for each time step.

Related parameter is writen in "inkt". These parameter should be consistent with "pram.f90"

nh: total number of output files
metric: metric
nt: read time number from structr.outdat
nq3d: output styple, 
nq3d=2: 2D output for IDL (x1,x3) 
nq3d=f: 3D output for IDL (x1,x2,x3)
nq3d=g: 2D output for IDL (x1,x3) at middle of x2 position
nq3d=h: 2D output for IDL (x2,x3) at middle of x1 position
nq3d=i: 2D output for IDL (x1,x2) at middle of x3 position
nq3d=a: output for AVS

In "kamt.f",

imax,jmax,kmax: grid number for original exec (read data)
lmax,mmax,nmax: grid number of output (for IDL)
llmax,mmmax,nnmax: grid number of output (for AVS)

For IDL, ok* files should be opened.
For AVS, [000-NNN]/* files should be opened.

CHANGING INITIAL CONDITION:

For changing simulation size, time, model etc, you change parameters in pram.f90. The parameter in pram.f90 is following:

 imax=180, jmax=180, kmax=90 !Grid number

 iprocs=2, jprocs=2, kprocs=2 !CPU number in i- j- and k- directions, total number of cpus are npe=iprocs*jprocs*kprocs

 icpu=20000 ! CPU time (second)

 itmax=10000 ! Max number of iteration

 nshot=1 ! Number of Output 

 icres=0 ! Researt parameter
!     icres = 0  : restart off
!     icres = 1  : restart on
 
 tmax=1.0d0 ! Maximum simulation time

 cfl=0.3d0 ! CFL number

 pmin=1.0d-20, pmax=1.0d10, dmin=1.0d-10 ! limiter for minimum and maximum pressure and minimum density

 gam=5.0d0/3.0d0 ! adiabatic index

  rbh=1.d0 ! black hole radius (Shwarzschild radius)
  akm=0.d0 ! black hole spin
  
 xmin, xmax, ymin, ymax, zmin, zmax: maxmim and minmum position for x1-, x2-, and x3-directions (normalized by shwarzschild radius)

 rr1=1.d0, rr2=1.d0, rr3=1.d0 ! grid spacing
!rr1 = 1.0 : uniform grid in radial direction
!rr1 = 888.0 : logarithmic grid spacing in radial direction  
 
 model=0 ! simulation model
! For GRMHD Simulation Model, we need to set "model=0" 

 metric=103 ! metric
! Mertic (coordinates) 
!
!     metric =   103 :  Boyer-Lindquist corrdinates with Schwarzschild BH
!     metric =   203:  Boyer-Lindquist corrdinates with Kerr BH
!

 ieos=0 ! EoS
! Equation of State
!
!     ieos=0 : Gamma-law constant equation of state    
!     ieos=1 : TM variable equation of state (cal enthalpy) 
!     ieos=2 : TM variable equation of state (cal internal energy)
!

 icha=1! calculation method for wave speed

 irec=1 ! reconstruction scheme
! Reconstruction scheme
!
!     irec = 1 : MC slope-limiter reconstruction (2nd order)
!     irec = 2 : Minmod slope-limiter reconstruction (2nd order)
!     irec = 3 : MUSCL method (3rd order), not slope-limiter
!     irec = 4 : Convex ENO reconstruction method (3rd order)
!     irec = 5 : rPPM reconstruction method (4th order)
!     irec = 6 : Monotonicity-Preserving 5th reconstruction (5th order)
!     irec = 7 : Weighted ENO 5th reconstruction (5th order)
!     irec = 8 : MP-WENO reconstruction
!     irec = 9 : WENO-Z 5th reconstruction
!     irec = 10: Mapped WENO 5th reconstruction
!     irec = 11: Limited Reconstruction (3rd order)

 ihll=1 ! Approximate Riemann solver scheme
! Approximate Riemann solver scheme
!
!     ihll = 1 : HLLE single-state approximate Riemann solver
!     ihll = 2 : HLLC two-state approximate Riemann solver
!                (Mignone & Bodo 200?)
!     ihll = 3 : HLLC two-state approximate Riemann solver
!                (Honkkila & Janhunen 2007)
!     ihll = 4 : HLLD multi-state approximate Riemann solver
!                (underconstruction)
!     ihll = 5 : HLLD multi-state approximate Riemann solver
!                (Mignone et al. 2010)

 ict=1  ! Constrained Transport scheme
! Constrained Transport scheme
!
!     ict = 0 : No contrained transport
!     ict = 1 : Flux CT sheme (Toth 2000)
!     ict = 2 : Modified Flux CT scheme (Gardiner & Stone 2005)
!     ict = 3 : Upwind Flux CT scheme (Gardiner & Stone 2005)
!

 irkt=3 ! time-advance
! RK time advance scheme
!
!     irkt = 2 : 2nd order TVD Runge-Kutta time advance method
!     irkt = 3 : 3rd order TVD Runge-Kutta time advance method
!
  
 iwvec=7 ! inversion procedure
! Inversion procedure scheme
!
!     iwvec=6 : new 2 variable newton-raphson (from Noble et al. 2005)
!     iwvec=7 : 1 variable newton-raphson with variable EOS
!               (from Mignone & McKinney 2007) 

 iboux1in(9)=(/7, 7, 7, 7, 7, 7, 7, 7, 7/) ! boundary condition for x1-inner
 iboux1ot(9)=(/5, 5, 5, 5, 5, 5, 5, 5, 5/) ! boundary condition for x1-outer
 iboux2in(9)=(/1, 1, 1, 1, 1, 1, 1, 1, 1/) ! boundary condition for x2-inner
 iboux2ot(9)=(/1, 1, 1, 1, 1, 1, 1, 1, 1/) ! boundary condition for x2-outer
 iboux3in(9)=(/1, 1, 1, 1, 1, 1, 1, 1, 1/) ! boundary condition for x3-inner
 iboux3ot(9)=(/1, 1, 1, 1, 1, 1, 1, 1, 1/) ! boundary condition for x3-outer 
! Boundary Condiiton
!
!               1   :    periodic boundary
!               2   :    fixed boundary
!               3   :    Neumann boundary
!               4   :    free boundary
!               5   :    u_0 = u_1       
!               7   :    radiative boundary without eigenvalue
!              12   :    anti-symmetric boundary condition (u_0=-u_1)
!              13   :    regid wall boundary condition (u_0=0.0)
!              15   :    special reflecting boundary 
!                        for jet propagation (z-direction only)
!              17   :    special radiative boundary without eigenvalue
!                        for jet propagation (z-direction only)
!              19   :    special boundary of u_0=u_1
!                        for jet propagation (z-direction only)
 
For changing simulation model, you change parameters in bhmodel.f90. The parameter in bhmodel.f90 is following:

  dd0=1.d0 ! density normalization unit
  ee0=0.1d0 ! internal energy unit 
  vp=1.0d0 ! radial velocity strength (for Bondi accretion) 
  rd=100.d0 ! disk starting position for thin disk model   
  hrel=1.3d0 ! relativistic enthalpy (for Bondi accretion)
  rc=7.d0 ! position of critical point (for Bondi accretion flow) 
  rshock=-1.d0 ! position of critical point (not used)
!
  tctd=100.d0 ! density ratio (ro_disk/ro_corona)
  vd=1.d0 ! disk rotation speed unit
  b0=0.0d0 ! magnetic field strength unit
  qmag=0.d0 ! charge of BH (not used)
!
  al0=0.1d0 ! strength of angular moumentum
!
  icor=5 ! corona (external medium) profile
!
! icor=0 : Bondi-like accretion flow (Koide et al. 2001)
! icor=1 :
! icor=2 : Quasi-hydrostatic external medium (McKinney )
! icor=3 : Quasi-hydrostatic external medium (Komissarov )
! icor=4 : Quasi-hydrostatic external medium (Koide 2003)
! icor=5 : GR Bondi accretion flow (Hawley, Smarr, Wilson 1984)
! icor=6 : GR Bondi accretion flow + angular momentum (Proga et al. 2003)

  idisk=0 : Black Hole disk model
!
! idisk=0 : Keplerian geomertically thin disk (Koide et al. 2001)
! idisk=1 : Geometrically thick torus (Abramowicz et al. 1978)
! idisk=2 : Geometrically thick torus (De Villiers et al. 2003)
 
  ibhmag=5 : Magnetic field structure
!
! ibhmag=1 : Monopole magnetic field
! ibhmag=2 : Dipole magnetic field
! ibhmag=3 : Potential magnetic field (Ouyed & Pudritz 1997)
! ibhmag=4 : Split monopole magnetic field (Sakurai 1987)
! ibhmag=5 : Uniform vertical magnetic field (Wald solution)
! ibhmag=6 : Magnetic field calculated from vector potential (Poloidal loop field)
! ibhmag=7 : Reversed Current Loop Magnetic Field (Koide et al.)

RUNNING 1D or 2D:

If you want to run 1D or 2D model in this 3D code, you set minimum number of grid in x2 and/or x3 directions. Mimimum number of grid depends on choosing schemes. In general minimum number of grid = 2*nm1+1. "nm1" is found in "main.f90" (depend on choosing each schemes).  And number of cpus set 1 (jpocs and/or kprocs in "pram.f90"). We usually use x1-direction in 1D model and x1- & x3-directions in 2D model.

DATA ANALYSIS

I use IDL for 1D and 2D data analysis.

# analysis program
dread.pro: data read program
vonx1d.pro: 1D plot on x-direction
plfccnve2D.pro: 2D plot (color contour + contour + vector)
video.pro: movie for 2D plot

USING IDL:

#For 1D plot
IDL> .r dread
IDL> .r vonx1d

#For 2D plot
IDL> .r dread
IDL> .r plfccnve2D

#For movie
IDL> .r video

If you want to put the data on ps file, please choose plot_choice='ps' (for 1D plot), or plot_choice='cps' (for 2D plot) in "vonx1d.pro" or "plfccnve2D.pro".

