Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LSODE issues: particles outside wall? #332

Open
cfe316 opened this issue Dec 20, 2024 · 5 comments
Open

LSODE issues: particles outside wall? #332

cfe316 opened this issue Dec 20, 2024 · 5 comments
Assignees
Labels
help wanted Extra attention is needed

Comments

@cfe316
Copy link
Collaborator

cfe316 commented Dec 20, 2024

Having two-or-three issues with Beams3D. I'm not sure how they might all be connected, or not.

  • I'm running on perlmutter using version "4.07".

I'm trying to do deposition and slowing runs on a W7X file. My immediate issue is that during the slowing runs I'm getting LSODE issues and finally crashing.

The LSODE issues are something to do with NaNs, shown here:

----- FOLLOWING GYROCENTER TRAJECTORIES -----
       Method: LSODE
   Particles:      2752
       Steps:    500001   dt_min:   23.461E-09   dt_max:  100.000E-09
      NPOINC:         3
         Tol:   10.000E-09  Type: 10
 ------------------
               269        1538        1538          10
               269           4   3.6338588847276410E-010   8.1317076423414774E-008
               269   5.2021317277503769        1.5628771011341376       0.49176378049793323                            NaN                       NaN
               269           2   1.0000000000000000E-008   1.0000000000000000E-008   1.0000000000000000E-008   1.0000000000000000E-008   1.0000000000000000E-008
               269           1          -3           0
               269   0.0000000000000000        0.0000000000000000        0.0000000000000000
               269           0           0           0
               269   1.6147920165610976E-011   6.1599426901286987E-017   3.6338588847276410E-010                       NaN
               269           0           0           0           2           2           4          84          20
 ------------------
 !!!!! ERROR !!!!!
   BEAMS3D ENCOUNTERED AN LSODE ERROR

There is also

  BEAMS3D ENCOUNTERED AN LSODE ERROR
   CALLING FUNCTION beams3d_follow_gc
   IERR:                -3   

and finally traces like

#2  0x7fd335653dbf in ???
#3  0x57c5f9 in out_beams3d_gc_
      at ../Sources/out_beams3d_gc.f90:69
#4  0x5b3a6b in beams3d_follow_gc_
      at /global/homes/j/jschwart/sw/STELLOPT/BEAMS3D/Debug/beams3d_follow_gc.f90: 286
...

I'm now looking at the .h5 file which is the output of the initial deposition step and seeing some unexpected outputs.

Unexpected result 1

Some markers are at the boundary upper and lower boundary box edges: how did they get there?
See the blue points in the image.

image

In the image, the beam lines are shown at right. I don't have a working wall file so I authored this 'rectangular torus' wall file and cut out three of the triangles so the beams can make it into the plasma.
I'm not sure how some of the markers at center and left got there --- did they need to pass through the walls? Did they start with a large random directional scatter?
(I have NPOINC=3 and I am plotting the second locational index.)

Unexpected results 2

I would have expected that markers had positions (gyropositions?) inside the LCFS, indicated by s < 1. However, 7 particles have a neutral ionization point with 1 < S < 1.02, and many more have gyrocenter S in the same range or even up to 1.5:

> s_lines[np.logical_and(s_lines[:,2] > 1, end_state[:] == 0), 2] 

# particles with gyrocenter position S > 1 and which have their end state == 0 ("orbiting"); I've also excluded the NaN-cases mentioned below.

array([1.06066286, 1.00356287, 1.00689848, 1.02347276, 1.00956314,
       1.03393613, 1.02394802, 1.01666715, 1.03470434, 1.11131292,
       1.02210884, 1.07166826, 1.10615378, 1.09248667, 1.00797324,
       1.02235147, 1.00348526])

image
First pic: markers with s > 0.

image
Second pic: there's light between the points and the LCFS.

Unexpected result 3

I was surprised to find that I couldn't load some Datasets of the HDF5 file in Mathematica, related to particle positions, because they had nan. I haven't done a full investigation, but a set of 62 particles has, for its third positional index (the initial position of the gyrocenter) nan for at least R_lines, U_lines and vll_lines. These particles are located within the plasma itself, within the same region as the rest of the normally ionized particles.
See the images here, where a few of them are shown in red:
image
image

Questions on a way forward

Here's a few things I'm considering:

  • Programmatically edit the .h5 file from step 1 (deposition) to remove particles with nan-positions or which have s > 1. The latter may be important since, at the moment, I'm using the LCFS as the wall in Step 2 (slowing).
    • Also remove the stray particles at the simulation boundary.
  • Author a wall file which is like the VMEC boundary but slightly extended in R(\theta, \phi) - R_axis(\phi) and Z(\theta, \phi) in order to encompass the stray s > 1 particles from Step 1.

Might one of these work? Has either been done before? Is there a better way to cull bad particles before Step2 or to make step2 more resilient?

@cfe316
Copy link
Collaborator Author

cfe316 commented Jan 2, 2025

Investigating a bit further, I've found that the "red" particles have a nan in B_lines for the second index == final neutral particle position == initial ion position.

@cfe316
Copy link
Collaborator Author

cfe316 commented Jan 3, 2025

I've constructed a "cleaned" hdf5 file from the DEPO step that removes the particles with nans and all those with s>0.98. However, this did not fix my LSODE issue.

@lazersos
Copy link
Collaborator

lazersos commented Jan 3, 2025

The deposition part of the code follows neutrals on straight lines till they reach a point where RHO<1 here RHO=SQRT(s). It then takes one step back (0.01 m back currently). If no such condition is found (neutral misses plasma completely), then it'll stop once the particle reaches R>Rmax where Rmax is the maximum background grid domain. This is done with rough stepping which is 0.25 m. This can lead to scatter in particles which don't hit the plasma.

Ionization is calculated on a chord which is ~0.01 m outside the plasma domain. Becasue of the cylindircal gridding this can result in particle ioninzing slightly outside the VMEC S=1 domain. This is because one gridpoint is inside the equilibrium and one is outside. So the density and temperature smoothly vary between these two points. This is usually a very small percentage of particles. And for W7-X is an underprediction since we'd expect a rather large SOL which has been confirmed by beam emission measurements (there are births outside the LCFS).

This effect could be an issue if running with the -plasma flag. If a vacuum field cannot be supplied, it is recommended to run with a -depo run first, then run using the -restart flag for the slowing down, as such particles will be filtered out.

If a particle does not deposit in the plasma, then we follow it to the wall or again a domain which is 2*RMAX or <Rmin again Rmin and Rmax come from the background grid. This hopefully explains what you're seeing in your first and second unexpected result.

Could you please post your full log file up to and including the first error. It's hard to understand how the code is being run. It almost looks like you're running with -plasma on the deposition so your wall is being ignored.

@lazersos lazersos self-assigned this Jan 3, 2025
@lazersos lazersos added the help wanted Extra attention is needed label Jan 3, 2025
@cfe316
Copy link
Collaborator Author

cfe316 commented Jan 3, 2025

Thanks @lazersos.

Could you please post your full log file up to and including the first error. It's hard to understand how the code is being run. It almost looks like you're running with -plasma on the deposition so your wall is being ignored.

I don't think that's the issue. I'm not sure if I mentioned this above, but I added two additional triangles to 'catch' the beam. You can see one of them in the top Figure above, with a cluster of black points to the left of the cluster in the plasma. I don't know why the other wall panels don't seem to stop any particles. Is there some assumption of a normal direction? (Do triangles only stop the beam from one side?)

I will post the DEPO run log, which proceeds without errors, and then the SLOW log.

Depo run log

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Temp Dir is:  /global/cfs/cdirs/m4505/jschwart/workdir/fullwall.34514011
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
srun --mpi=pmix /global/cfs/cdirs/m4505/jschwart/sw/STELLOPT/bin/xbeams3d -vmec w7x_180920017_birth -coil /global/cfs/cdirs/m4505/jschwart/w7x_definition/coils.w7x_mc_sc_tc_17_nfp5 -vessel /global/cfs/cdirs/m4505/jschwart/w7x_definition/w7x_s1.2_wall.dat -depo
BEAMS3D Version  4.07
-----  HDF5 (Parallel) Parameters  -----
   HDF5_version:   1.12 release: 01
-----  MPI Parameters  -----
   MPI_version:   3.01
   Open MPI v5.0.3, package: Open MPI ncicd@login01 Distribution, ident: 5.0.3, repo rev: v5.0.3, Apr 08, 2024
   Nproc_total:       128
   Nproc_shared:      128
-----  GIT Repository  -----
   Repository: [email protected]:PrincetonUniversity/STELLOPT.git
   Branch:     develop
   Version:    v4.50-84-gd286-dirty
   Built-on:   22.11.2024 09:07:13
   Hash:       d286eaacfef1e0b0535de596e00aa77dd5cd33ef
----- Input Parameters -----
   FILE: input.w7x_180920017_birth
   R   = [  4.25000,  8.00000];  NR:    128
   PHI = [ 0.00000, 1.25664];  NPHI:  120
   Z   = [-1.10000, 1.10000];  NZ:    128
   # of Particles to Start:      128
   # of Beams:       24
   VESSEL: /global/cfs/cdirs/m4505/jschwart/w7x_definition/w7x_s1.2_wall.dat
   COIL: /global/cfs/cdirs/m4505/jschwart/w7x_definition/coils.w7x_mc_sc_tc_17_nfp5
   COLLISION OPERATOR ON!
   DEPOSITION ONLY!
   SUZUKI DEPOSITION MODEL!
----- Plasma Parameters -----
   Te   = [  0.09132,  2.97900] keV;  NTE:    101;  S_MAX_TE:  1.00000
   Ti   = [  0.06651,  1.45000] keV;  NTI:    101;  S_MAX_TI:  1.00000
   Ne   = [  0.35050,  0.63110] E20 m^-3;  NNE:    101;  S_MAX_NE:  1.00000
   Ni(1)= [  0.35050,  0.63110] E20 m^-3;  M:   1 amu;  Z:  1;  S_MAX_NI:  1.00000
   Zeff = [  1.00000,  1.00000];  NZEFF:    6;  S_MAX_ZEFF:  1.00000
   PLASMA_MASS =    1.00784 amu
   PLASMA_ZMEAN =    1.00000 [Z]
----- COILS Information -----
   FILE: /global/cfs/cdirs/m4505/jschwart/w7x_definition/coils.w7x_mc_sc_tc_17_nfp5
   Coil Periodicity:   5
   Current Systems:  22
   Current Type:      SCALED
   Num Coils  =   10  EXTCUR =   15.000 [kA]
   Num Coils  =   10  EXTCUR =   15.000 [kA]
   Num Coils  =   10  EXTCUR =   15.000 [kA]
   Num Coils  =   10  EXTCUR =   15.000 [kA]
   Num Coils  =   10  EXTCUR =   15.000 [kA]
   Num Coils  =   10  EXTCUR =    0.000 [A]
   Num Coils  =   10  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]
   Num Coils  =    1  EXTCUR =    0.000 [A]

----- VMEC Information -----
   FILE: w7x_180920017_birth
   R       = [  4.63809,  6.21519]
   Z       = [-0.93259, 0.93259]
   BETA    =    0.01;  I  =    0.00 [A]
   AMINOR  =    0.52 [m]
   PHIEDGE =   -2.22 [Wb]
   VOLUME  =   29.13 [m^3]
   NYQUIST DETECTED IN WOUT FILE!
----- Virtual Casing Information -----
   INTEGRAL TYPE: Surface Current (DCUHRE)
   MIN_GRID_DISTANCE =  8.8301E-02
   NORMAL_AREA =  1.3834E+02
   NR =    1;   NU =  128;  NV =  128;  NFP =   5
   NUVP =  81920
   ABS_TOL =  0.0000E+00;   REL_TOL =  1.0000E-03
   MIN_CLS =      0   (16777216)

----- Loading wall data -----
----- Constructing Splines -----
   R   = [  4.25000,  8.00000];  NR:    128
   PHI = [ 0.00000, 1.25664];  NPHI:  120
   Z   = [-1.10000, 1.10000];  NZ:    128
   HERMITE FORM: 1
----- INITIALIZING BEAMS -----
      nbeams:   24
      nparticles_start:    128
            E_BEAM( 1):   55 [keV] P_BEAM( 1):   0.918 [MW]
            E_BEAM( 2):   27 [keV] P_BEAM( 2):   0.540 [MW]
            E_BEAM( 3):   18 [keV] P_BEAM( 3):   0.342 [MW]
            E_BEAM( 4):   55 [keV] P_BEAM( 4):   0.918 [MW]
            E_BEAM( 5):   27 [keV] P_BEAM( 5):   0.540 [MW]
            E_BEAM( 6):   18 [keV] P_BEAM( 6):   0.342 [MW]
            E_BEAM( 7):   55 [keV] P_BEAM( 7):   0.918 [MW]
            E_BEAM( 8):   27 [keV] P_BEAM( 8):   0.540 [MW]
            E_BEAM( 9):   18 [keV] P_BEAM( 9):   0.342 [MW]
            E_BEAM(10):   55 [keV] P_BEAM(10):   0.918 [MW]
            E_BEAM(11):   27 [keV] P_BEAM(11):   0.540 [MW]
            E_BEAM(12):   18 [keV] P_BEAM(12):   0.342 [MW]
            E_BEAM(13):   55 [keV] P_BEAM(13):   0.918 [MW]
            E_BEAM(14):   27 [keV] P_BEAM(14):   0.540 [MW]
            E_BEAM(15):   18 [keV] P_BEAM(15):   0.342 [MW]
            E_BEAM(16):   55 [keV] P_BEAM(16):   0.918 [MW]
            E_BEAM(17):   27 [keV] P_BEAM(17):   0.540 [MW]
            E_BEAM(18):   18 [keV] P_BEAM(18):   0.342 [MW]
            E_BEAM(19):   55 [keV] P_BEAM(19):   0.918 [MW]
            E_BEAM(20):   27 [keV] P_BEAM(20):   0.540 [MW]
            E_BEAM(21):   18 [keV] P_BEAM(21):   0.342 [MW]
            E_BEAM(22):   55 [keV] P_BEAM(22):   0.918 [MW]
            E_BEAM(23):   27 [keV] P_BEAM(23):   0.540 [MW]
            E_BEAM(24):   18 [keV] P_BEAM(24):   0.342 [MW]
 -----  Vessel Information  -----
   Wall Name : W7X s=1.2 surface
   Date      :  2025-01-02
   Faces     :    5800
   Blocks    :     128
   Mean faces per block:    307.00
   Highest faces per block:     999
   R_WALL   = [  4.54466,  6.27544]
   Z_WALL   = [ -1.01245,  1.01168]
----- FOLLOWING NEUTRAL TRAJECTORIES -----
----- BEAM Density Calc. -----
----- BEAM DIAGNOSTICS -----
 BEAMLINE  E [keV]  Q [e]   M [Mp]   Markers [#]    NDIST [#]    Orbit [%]    Lost [%]   Shine. [%]  Port [%]    T_MIN [s]       T_MAX [s]
    1       55        1        1         128          1.4E+12       87.5        0.0       11.7        0.8         2.2E-06        50.0E-03
    2       27        1        1         128          1.8E+12       97.7        0.0        2.3        0.0         3.2E-06        50.0E-03
    3       18        1        1         128          1.7E+12       97.7        0.0        2.3        0.0         3.9E-06        50.0E-03
    4       55        1        1         128          1.3E+12       83.6        0.0       16.4        0.0         2.2E-06        50.0E-03
    5       27        1        1         128          1.8E+12       94.6        0.0        5.4        0.0         3.2E-06        50.0E-03
    6       18        1        1         128          1.7E+12       98.4        0.0        1.6        0.0         3.9E-06        50.0E-03
    7       55        1        1         128          1.3E+12       80.5        0.0       19.5        0.0         2.2E-06        50.0E-03
    8       27        1        1         128          1.7E+12       89.8        0.0       10.2        0.0         3.2E-06        50.0E-03
    9       18        1        1         128          1.7E+12       93.8        0.0        6.2        0.0         3.9E-06        50.0E-03
   10       55        1        1         128          1.4E+12       85.9        0.0       14.1        0.0         2.3E-06        50.0E-03
   11       27        1        1         128          1.8E+12       97.7        0.0        2.3        0.0         3.2E-06        50.0E-03
   12       18        1        1         128          1.7E+12       95.3        0.0        4.7        0.0         3.9E-06        50.0E-03
   13       55        1        1         128          1.4E+12       86.7        0.0       13.3        0.0         2.2E-06        50.0E-03
   14       27        1        1         128          1.8E+12       96.9        0.0        3.1        0.0         3.2E-06        50.0E-03
   15       18        1        1         128          1.7E+12       98.4        0.0        0.8        0.8         3.9E-06        50.0E-03
   16       55        1        1         128          1.4E+12       87.5        0.0       12.5        0.0         2.2E-06        50.0E-03
   17       27        1        1         128          1.8E+12       94.5        0.0        4.7        0.8         3.1E-06        50.0E-03
   18       18        1        1         128          1.7E+12       94.5        0.0        3.1        2.4         3.8E-06        50.0E-03
   19       55        1        1         128          1.2E+12       78.1        0.0       21.9        0.0         2.2E-06        50.0E-03
   20       27        1        1         128          1.6E+12       87.5        0.0       11.7        0.8         3.1E-06        50.0E-03
   21       18        1        1         128          1.6E+12       92.2        0.0        5.5        2.3         3.8E-06        50.0E-03
   22       55        1        1         128          1.3E+12       80.5        0.0       17.9        1.6         2.2E-06        50.0E-03
   23       27        1        1         128          1.7E+12       93.8        0.0        3.9        2.3         3.2E-06        50.0E-03
   24       18        1        1         128          1.7E+12       97.6        0.0        1.6        0.8         3.9E-06        50.0E-03
----- BEAMS3D DONE -----


  Copying files:
     From:  /global/cfs/cdirs/m4505/jschwart/workdir/fullwall.34514011
     To:    /global/cfs/cdirs/m4505/jschwart/2024-11-22-W7X-profiles

Slowing run log

Note that this is after eliminating (hopefully all!) of the particles that had nan in the DEPO run. That's why it only sees 2975 particles instead of 3072.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Temp Dir is:  /global/cfs/cdirs/m4505/jschwart/workdir/fullwall.34539387
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
srun --mpi=pmix /global/cfs/cdirs/m4505/jschwart/sw/STELLOPT/bin/xbeams3d -vmec w7x_180920017_slow -restart beams3d_w7x_180920017_birth.h5 -plasma -collisions
BEAMS3D Version  4.07
-----  HDF5 (Parallel) Parameters  -----
   HDF5_version:   1.12 release: 01
-----  MPI Parameters  -----
   MPI_version:   3.01
   Open MPI v5.0.3, package: Open MPI ncicd@login01 Distribution, ident: 5.0.3, repo rev: v5.0.3, Apr 08, 2024
   Nproc_total:       128
   Nproc_shared:      128
-----  GIT Repository  -----
   Repository: [email protected]:PrincetonUniversity/STELLOPT.git
   Branch:     develop
   Version:    v4.50-84-gd286-dirty
   Built-on:   22.11.2024 09:07:13
   Hash:       d286eaacfef1e0b0535de596e00aa77dd5cd33ef
----- Input Parameters -----
   FILE: input.w7x_180920017_slow
   R   = [  4.25000,  8.00000];  NR:    128
   PHI = [ 0.00000, 1.25664];  NPHI:  120
   Z   = [-1.10000, 1.10000];  NZ:    128
   # of Particles to Start:        0
   COLLISION OPERATOR ON!
   MAGNETIC FIELD FROM PLASMA ONLY!
   Restarting particles!
----- Plasma Parameters -----
   Te   = [  0.09132,  2.97900] keV;  NTE:    101;  S_MAX_TE:  1.00000
   Ti   = [  0.06651,  1.45000] keV;  NTI:    101;  S_MAX_TI:  1.00000
   Ne   = [  0.35050,  0.63110] E20 m^-3;  NNE:    101;  S_MAX_NE:  1.00000
   Ni(1)= [  0.35050,  0.63110] E20 m^-3;  M:   1 amu;  Z:  1;  S_MAX_NI:  1.00000
   Zeff = [  1.00000,  1.00000];  NZEFF:    6;  S_MAX_ZEFF:  1.00000
   PLASMA_MASS =    1.00784 amu
   PLASMA_ZMEAN =    1.00000 [Z]
----- VMEC Information -----
   FILE: w7x_180920017_slow
   R       = [  4.63809,  6.21519]
   Z       = [-0.93259, 0.93259]
   BETA    =    0.01;  I  =    0.00 [A]
   AMINOR  =    0.52 [m]
   PHIEDGE =   -2.22 [Wb]
   VOLUME  =   29.13 [m^3]
   CREATING WALL FROM HARMONICS

----- Constructing Splines -----
   R   = [  4.25000,  8.00000];  NR:    128
   PHI = [ 0.00000, 1.25664];  NPHI:  120
   Z   = [-1.10000, 1.10000];  NZ:    128
   HERMITE FORM: 1
----- Reading Restart File -----
   FILE: beams3d_w7x_180920017_birth.h5
   NPARTICLES_OLD:     2975
   NPOINC_OLD:        3
   Detected old deposition run!
   Detected old plasma only run!
   NPARTICLES_RESTART:     2735
   # of Beams:     24
 -----  Vessel Information  -----
   Wall Name :  HARMONICS
   Date      :  TODAY
   Faces     :   43200
   Blocks    :    4500
   Mean faces per block:    243.00
   Highest faces per block:    2062
   R_WALL   = [  4.63809,  6.21519]
   Z_WALL   = [ -0.93259,  0.93259]
----- FOLLOWING GYROCENTER TRAJECTORIES -----
       Method: LSODE
   Particles:      2735
       Steps:    500001   dt_min:   23.461E-09   dt_max:  100.000E-09
      NPOINC:         3
         Tol:   10.000E-09  Type: 10
 ------------------
               116        2484        2484          10
               116           4   1.5266159816400957E-008   6.5079634694994537E-008
               116   5.6673223729230466        1.5378420019037313       0.20107777639708105                            NaN                       NaN
               116           2   1.0000000000000000E-008   1.0000000000000000E-008   1.0000000000000000E-008   1.0000000000000000E-008   1.0000000000000000E-008
               116           1          -3           0
               116   0.0000000000000000        0.0000000000000000        0.0000000000000000
               116           0           0           0
               116   1.3479729117957988E-011   5.1421085807640030E-017   1.5266159816400957E-008                       NaN
               116           0           0           0           3           3           4          84          20
 ------------------
 !!!!! ERROR !!!!!
   BEAMS3D ENCOUNTERED AN LSODE ERROR
      CALLING FUNCTION beams3d_follow_gc
      IERR:                -3
      ILLEGAL INPUT DETECTED!
 ------------------

@cfe316
Copy link
Collaborator Author

cfe316 commented Jan 3, 2025

I've now updated to the most recent commit, 988b1d and the issue persists.
I should also post the line in beams3d_follow_gc which has the issue: beams3d_follow_gc.f90:286.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants