============================================================================== Hello, and welcome to S3D! ============================================================================== ============================================================== --------------------S3D BUILD CONFIGURATION----------------- Build directory: /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/fortran/ammonia ------------------------------------------------------ S3D Options: Using mixture averaged transport: ON Using ARK integration: OFF Using chemical mechanism: ammonia Using srcs for timescale analysis: OFF ------------------------------------------------------ Defines: SYSTEMCALLWONTWORK SAVEFILEINSEPDIR ARCH_OPTERON MPIIO BLOCKIO MIXAVG FROZENFEED ------------------------------------------------------ FORTRAN COMPILER FLAGS: -cpp -fdefault-double-8 -fdefault-real-8 -fno-range-check -ffree-line-length-none -fallow-argument-mismatch C COMPILER FLAGS: ------------------------------------------------------ S3D Version: ------------------------------------------------------ Chemistry routine header: c**************************************************************************** c ... DO NOT EDIT. AUTOMATICALLY GENERATED FILE. c**************************************************************************** c ... c ... CKWYP-compatible GETRATES: return molar production rates for all c ... species given pressure (P), temperature (T), and mass fractions (Y). c ... c ... Generated by cgetrates 1.0b c ... c**************************************************************************** subroutine getrates(P,T,Y,ICKWRK,RCKWRK,WSPL) ------------------------------------------------------ Transport routine header: !Standard version of tranlib.f. 18Feb 2009 ! (Changes by Mark Fahey, ORNL) ! (Update by Evatt Hawkes, SNL) SUBROUTINE MCABS ------------------------------------------------------ ============================================================== ------------------------------------------------------------------------------ this run started at: date = 06/30/2025 time = 10:26:03 ------------------------------------------------------------------------------ initializing chemkin module... opening chemkin link file... setting chemkin work array lengths... allocating chemkin work arrays... filling chemkin work arrays based on chemkin link file... CKLIB: CHEMKIN-III GAS-PHASE CHEMICAL KINETICS LIBRARY, DOUBLE PRECISION Vers. 5.36 2000/12/15 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 the following was extracted from the ../input/chem.asc file: element names are as follows: index name 1 H 2 O 3 N species names are as follows: index name 1 H2 2 NH3 3 NO 4 N2O 5 O2 6 H 7 O 8 OH 9 HO2 10 H2O 11 H2O2 12 NO2 13 HNO 14 N 15 NNH 16 NH2 17 NH 18 H2NO 19 N2 species molecular weights are as follows: index name value (kg/mol) 1 H2 2.01594E-03 2 NH3 1.70306E-02 3 NO 3.00061E-02 4 N2O 4.40128E-02 5 O2 3.19988E-02 6 H 1.00797E-03 7 O 1.59994E-02 8 OH 1.70074E-02 9 HO2 3.30068E-02 10 H2O 1.80153E-02 11 H2O2 3.40147E-02 12 NO2 4.60055E-02 13 HNO 3.10141E-02 14 N 1.40067E-02 15 NNH 2.90214E-02 16 NH2 1.60226E-02 17 NH 1.50147E-02 18 H2NO 3.20220E-02 19 N2 2.80134E-02 ------------------------------------------------------------------------------ initializing param module... number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 number of reaction third-body reactions = 11 ------------------------------------------------------------------------------ initializing topology module... Initialized topology module, npx = 2 npy = 1 npz = 4 npes = 8 ------------------------------------------------------------------------------ initializing reference module... the various reference values are as follows: universal gas constant (J/mol-K) = 8.31400E+00 freestream temperature (K) = 8.01900E+02 reference ratio of specifice heats = 1.36400E+00 reference speed of sound (m/s) = 5.28000E+02 reference density (kg/m^3) = 1.22300E+01 reference conductivity (W/m-K) = 5.76900E-02 reference temperature (K) = 2.91892E+02 reference pressure (Pa) = 3.40953E+06 standard atmospheric pressure (Pa) = 1.01325E+05 reference time (s) = 1.25428E-06 reference specific heat (J/kg-K) = 9.55094E+02 reference length (m) = 6.62258E-04 reference viscosity (Pa-s) = 4.27649E+00 acoustic Reynolds number = 1.00000E+05 Mach number = 1.00000E-03 convective Reynolds number = 1.00000E+02 ------------------------------------------------------------------------------ initializing ERK module... ERK module information is as follows: rk_method = -64 cont_switch = 0 tstep_init (seconds) = 5.00E-10 tstep_min (seconds) = 5.00E-10 tstep_max (seconds) = 5.00E-10 rk_rtol = 1.00E-03 rk_atol = 1.00E-15 cont_safety = 9.00E-01 k_I = 2.00E-01 k_P = 2.00E-01 k_D = 5.00E-02 k_D2 = 0.00E+00 cont_factor = 3.00E+00 cfl_switch = 1 i_time_cont = 20 cont_n_reg = 1 controller type = NONE rk_tol = 1.00E-12 using RK4(3)6[2R+]C ERK scheme! ------------------------------------------------------------------------------ initializing transport module... opening mixavg.in ... Lagging is switched on WARNING - with coefficient lagging, theoretical temporal convergence order may not be observed opening transport link file... setting lengths of transport work arrays... allocating transport work arrays... creating transport work arrays from transport link file... TRANLIB: CHEMKIN-III MULTICOMPONENT TRANSPORT LIBRARY, DOUBLE PRECISION Vers. 4.6 1999/06/03 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. ------------------------------------------------------------------------------ initializing filter module... using explicit filter of order 10 filter will be applied every 10 time steps ------------------------------------------------------------------------------ initializing derivative module... ------------------------------------------------------------------------------ initializing thermchem module... ------------------------------------------------------------------------------ initializing grid module... ------------------------------------------------------------------------------ initializing boundary condition module... ------------------------------------------------------------------------------ *** WARNING: ofi-conduit failed to configure FI_CXI_* environment variables due to prior conflicting settings. This may lead to unstable behavior and/or degraded performance. If you did not intentionally set FI_CXI_RX_MATCH_MODE='software', FI_CXI_RDZV_THRESHOLD='16384', or FI_CXI_RDZV_GET_MIN then this condition may have resulted from initializing MPI prior to initialization of GASNet. For more information on that scenario, please see "Limits to MPI interoperability" in the ofi-conduit README. Legion Init: Mechanism = 'ammonia' Local Grid Size = ( 64, 64, 256) Global Grid Size = (128, 64, 1024) Proc Grid Size = ( 2, 1, 4) Proc ID = ( 0, 0, 0) nvar_tot = 23, n_spec = 19, n_reg = 3, n_scalar = 0 iorder = 8, iforder = 10, vary = XYZ fuse fields set to 200 from environment set mapper mode to 'allgpu' from environment RHST_ARGS: -ll:gpu 2 -ll:cpu 1 -ll:util 3 -ll:bgwork 3 -ll:fsize 16384 -ll:csize 4096 -ll:zsize 1024 -ll:stacksize 16 -logfile run_%.log -hip:hostreg 0 -lg:no_transitive_reduction 1 -lg:prof 8 -lg:prof_logfile prof_%.gz REGULAR STENCIL initializing variables... Host: frontier08191 Rank: 0 PID: 186598 Time: 10:26:08 STARTING MAIN TASK! initializing bomb (stand back) Mixture fraction module successfully initialized... storing hard inflow conditions... ------------------------------------------------------------------------------ continuing with initialization... ------------------------------------------------------------------------------ storing hard inflow conditions... maximum speed of sound = 839.70 m/s mimimum speed of sound = 839.70 m/s maximum acoustic time = 2.06E-05 sec minimum acoustic time = 2.06E-05 sec initialization completed for time = 0.00E+00 ------------------------------------------------------------------------------ writing save files for: i_time = 0, time = 0.0000E+00 (sec) precision is d generating active.in file... ------------------------------------------------------------------------------ 10:26:17.671 1 5.0000E-10 5.000E-10 6.140E-04 1600.00 30.00 10:26:18.485 2 1.0000E-09 5.000E-10 2.010E-04 1600.00 30.00 10:26:19.115 3 1.5000E-09 5.000E-10 6.685E-05 1600.00 30.00 10:26:19.730 4 2.0000E-09 5.000E-10 2.712E-05 1600.00 30.00 10:26:20.309 5 2.5000E-09 5.000E-10 1.285E-05 1600.00 30.00 10:26:20.886 6 3.0000E-09 5.000E-10 6.797E-06 1600.00 30.00 10:26:21.460 7 3.5000E-09 5.000E-10 3.893E-06 1600.00 30.00 10:26:22.040 8 4.0000E-09 5.000E-10 2.367E-06 1600.00 30.00 10:26:22.615 9 4.5000E-09 5.000E-10 1.508E-06 1600.00 30.00 10:26:24.921 10 5.0000E-09 5.000E-10 9.961E-07 1600.00 30.00 10:26:48.246 20 1.0000E-08 5.000E-10 4.621E-08 1600.00 30.00 10:27:08.239 30 1.5000E-08 5.000E-10 9.442E-09 1600.00 30.00 10:27:15.367 40 2.0000E-08 5.000E-10 2.475E-09 1600.00 30.00 10:27:22.429 50 2.5000E-08 5.000E-10 6.750E-10 1600.00 30.00 10:27:28.666 60 3.0000E-08 5.000E-10 1.821E-10 1600.00 30.00 10:27:35.366 70 3.5000E-08 5.000E-10 1.370E-10 1600.00 30.00 10:27:42.138 80 4.0000E-08 5.000E-10 1.442E-10 1600.00 30.00 10:27:49.025 90 4.5000E-08 5.000E-10 1.514E-10 1600.00 30.00 10:27:55.864 100 5.0000E-08 5.000E-10 1.556E-10 1600.00 30.00 10:28:02.296 110 5.5000E-08 5.000E-10 1.568E-10 1600.00 30.00 10:28:08.860 120 6.0000E-08 5.000E-10 1.551E-10 1600.00 30.00 10:28:16.071 130 6.5000E-08 5.000E-10 1.512E-10 1600.00 30.00 10:28:23.603 140 7.0000E-08 5.000E-10 1.454E-10 1600.00 30.00 10:28:30.562 150 7.5000E-08 5.000E-10 1.386E-10 1600.00 30.00 10:28:37.490 160 8.0000E-08 5.000E-10 1.312E-10 1600.00 30.00 10:28:44.657 170 8.5000E-08 5.000E-10 1.237E-10 1600.00 30.00 10:28:52.396 180 9.0000E-08 5.000E-10 1.164E-10 1600.00 30.00 10:29:01.048 190 9.5000E-08 5.000E-10 1.093E-10 1600.00 30.00 *** Caught a fatal signal (proc 0): SIGSEGV(11) [7 - 7ffd71df3600] 179.733850 {6}{gex}: CRC MISMATCH: arg0=81 header_size=36 payload_size=9600 exp=5a6ba49e act=6d68a690 *** Caught a fatal signal (proc 7): SIGABRT(6) *** NOTICE (proc 7): We recommend linking the debug version of GASNet to assist you in resolving this application issue. [7] Invoking GDB for backtrace... *** Caught a fatal signal (proc 0): SIGSEGV(11) *** Caught a fatal signal (proc 0): SIGSEGV(11) *** NOTICE (proc 0): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** NOTICE (proc 0): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** NOTICE (proc 0): We recommend linking the debug version of GASNet to assist you in resolving this application issue. [7] /usr/bin/gdb -nx -batch -x /tmp/gasnet_e5mmOt '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_2/s3d.x' 3227349 [7] [New LWP 3227367] [7] [New LWP 3227378] [7] [New LWP 3227382] [7] [New LWP 3227386] [7] [New LWP 3227405] [7] [New LWP 3227408] [7] [New LWP 3227409] [7] [New LWP 3227414] [7] [New LWP 3227415] [7] [New LWP 3227416] [7] [New LWP 3227423] [7] [New LWP 3227424] [7] [New LWP 3227446] [7] [New LWP 3227447] [7] [New LWP 3227448] [7] [New LWP 3227449] [7] [New LWP 3227453] [7] [New LWP 3227457] [7] [New LWP 3227465] [7] [Thread debugging using libthread_db enabled] [7] Using host libthread_db library "/lib64/libthread_db.so.1". [7] 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [7] Id Target Id Frame [7] * 1 Thread 0x7fffd5a20600 (LWP 3227349) "s3d.x" 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [7] 2 Thread 0x7fffd35ff6c0 (LWP 3227367) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [7] 3 Thread 0x7ffd77f05600 (LWP 3227378) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 4 Thread 0x7ffd71df3600 (LWP 3227382) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [7] 5 Thread 0x7ffd62dff600 (LWP 3227386) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 6 Thread 0x7ffd8600f600 (LWP 3227405) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 7 Thread 0x7ffd85e3f600 (LWP 3227408) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 8 Thread 0x7ffd81c3f600 (LWP 3227409) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 9 Thread 0x7ffd7dbbf600 (LWP 3227414) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 10 Thread 0x7fc0687fd600 (LWP 3227415) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 11 Thread 0x7fc0685f9600 (LWP 3227416) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 12 Thread 0x7ffd5a1ff600 (LWP 3227423) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 13 Thread 0x7ffd5e7ff600 (LWP 3227424) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 14 Thread 0x7ffd55bff600 (LWP 3227446) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 15 Thread 0x7ffd515ff600 (LWP 3227447) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 16 Thread 0x7ffd4cfff600 (LWP 3227448) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 17 Thread 0x7ffd489ff600 (LWP 3227449) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 18 Thread 0x7ffd42bff600 (LWP 3227453) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] 19 Thread 0x7ffd7d7ff6c0 (LWP 3227457) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [7] 20 Thread 0x7ffd3e5ff600 (LWP 3227465) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] [7] Thread 20 (Thread 0x7ffd3e5ff600 (LWP 3227465) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 Realm::UnfairCondVar::wait (this=this@entry=0x55e1548) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [7] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x55e1528) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [7] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfe4287970) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 19 (Thread 0x7ffd7d7ff6c0 (LWP 3227457) "s3d.x"): [7] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [7] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #3 0x00007fffd6ca320c in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #4 0x00007fffd6c14e44 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #5 0x00007fffd6c14b5a in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #6 0x00007fffd6c07a19 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #7 0x00007fffec3d104b in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [7] #8 0x00007fffec150186 in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [7] #9 0x00007fffec37e29f in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [7] #10 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #11 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 18 (Thread 0x7ffd42bff600 (LWP 3227453) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 Realm::UnfairCondVar::wait (this=this@entry=0x55e1548) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [7] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x55e1528) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [7] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfe41e4bc0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 17 (Thread 0x7ffd489ff600 (LWP 3227449) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 Realm::UnfairCondVar::wait (this=this@entry=0x55e1548) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [7] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x55e1528) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [7] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfe41e8510) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 16 (Thread 0x7ffd4cfff600 (LWP 3227448) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 Realm::UnfairCondVar::wait (this=this@entry=0x58d8af8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [7] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58d8ad8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [7] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfe00c8db0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 15 (Thread 0x7ffd515ff600 (LWP 3227447) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 Realm::UnfairCondVar::wait (this=this@entry=0x58d8af8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [7] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58d8ad8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [7] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfe00c6ec0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 14 (Thread 0x7ffd55bff600 (LWP 3227446) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 Realm::UnfairCondVar::wait (this=this@entry=0x58d8af8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [7] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58d8ad8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [7] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfe00cab40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 13 (Thread 0x7ffd5e7ff600 (LWP 3227424) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5e7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5e7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 Realm::UnfairCondVar::wait (this=this@entry=0x58d8af8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [7] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58d8ad8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [7] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0440013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 12 (Thread 0x7ffd5a1ff600 (LWP 3227423) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 Realm::UnfairCondVar::wait (this=this@entry=0x55e1548) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [7] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x55e1528) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [7] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc03c0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 11 (Thread 0x7fc0685f9600 (LWP 3227416) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc0685f8ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5a91650) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [7] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5a91650, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [7] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5a91650) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [7] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5a91650) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [7] #7 Realm::Thread::thread_entry_wrapper (obj=0x5a91650) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x802da60) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 10 (Thread 0x7fc0687fd600 (LWP 3227415) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc0687fcba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x58a5cb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [7] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x58a5cb0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [7] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x58a5cb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [7] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x58a5cb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [7] #7 Realm::Thread::thread_entry_wrapper (obj=0x58a5cb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x802d4b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 9 (Thread 0x7ffd7dbbf600 (LWP 3227414) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7dbbeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5bba410) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [7] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5bba410, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [7] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5bba410) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [7] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [7] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [7] #8 0x0000000000000000 in ?? () [7] [7] Thread 8 (Thread 0x7ffd81c3f600 (LWP 3227409) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd81c3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56a6a70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [7] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56a6a70, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [7] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56a6a70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [7] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [7] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [7] #8 0x0000000000000000 in ?? () [7] [7] Thread 7 (Thread 0x7ffd85e3f600 (LWP 3227408) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd85e3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56a5490) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [7] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56a5490, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [7] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56a5490) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [7] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [7] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [7] #8 0x0000000000000000 in ?? () [7] [7] Thread 6 (Thread 0x7ffd8600f600 (LWP 3227405) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8600eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56a51a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [7] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56a51a0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [7] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56a51a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [7] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [7] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [7] #8 0x0000000000000000 in ?? () [7] [7] Thread 5 (Thread 0x7ffd62dff600 (LWP 3227386) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd62dfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 Realm::BackgroundWorkThread::main_loop (this=0x5796b20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [7] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5845320) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 4 (Thread 0x7ffd71df3600 (LWP 3227382) "s3d.x"): [7] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [7] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [7] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [7] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [7] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [7] #5 0x00007fffdf46fa28 in gasneti_defaultSignalHandler () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [7] #6 [7] #7 0x00007fffe65acdfc in __pthread_kill_implementation () from /lib64/libc.so.6 [7] #8 0x00007fffe655b842 in raise () from /lib64/libc.so.6 [7] #9 0x00007fffe65435cf in abort () from /lib64/libc.so.6 [7] #10 0x00007fffdf99737b in Realm::verify_packet_crc (arg0=arg0@entry=81, header_base=header_base@entry=0x7ffd71df0b00, header_size=header_size@entry=36, payload_base=payload_base@entry=0x7ffd697979b0, payload_size=payload_size@entry=9600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:130 [7] #11 0x00007fffdf998d60 in Realm::GASNetEXInternal::handle_medium (this=0x5260390, srcrank=0, arg0=81, hdr=0x7ffd71df0b00, hdr_bytes=36, data=0x7ffd697979b0, data_bytes=9600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:5143 [7] #12 0x00007fffe087662f in Realm::GASNetEXHandlers::handle_request_medium (token=0x7ffd69797980, arg0=81, hdr=hdr@entry=0x7ffd71df0b00, hdr_bytes=hdr_bytes@entry=36, data=0x7ffd697979b0, data_bytes=9600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_handlers.cc:708 [7] #13 0x00007fffe0876a93 in Realm::GASNetEXHandlers::handle_request_medium_10 (token=, buf=, nbytes=, arg0=, arg1=, arg2=, arg3=1054, arg4=754, arg5=0, arg6=1048576, arg7=-1, arg8=0, arg9=1517003934) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_handlers.cc:842 [7] #14 0x00007fffe0886f16 in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [7] #15 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [7] #16 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [7] #17 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [7] #18 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [7] #19 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x5260410, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [7] #20 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd71df15c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [7] #21 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x57966e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [7] #22 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5796c40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #23 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #24 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 3 (Thread 0x7ffd77f05600 (LWP 3227378) "s3d.x"): [7] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd77f04ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 Realm::BackgroundWorkThread::main_loop (this=0x57964d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [7] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5796800) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 3227367) "s3d.x"): [7] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [7] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [7] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 1 (Thread 0x7fffd5a20600 (LWP 3227349) "s3d.x"): [7] #0 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [7] #1 0x00007fffe7879d23 in MPIR_Wait_impl () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [7] #2 0x00007fffe7892e3b in MPIR_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [7] #3 0x00007fffe789385d in PMPI_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [7] #4 0x00007fffe9336ddc in pmpi_wait__ () from /opt/cray/pe/lib64/libmpifort_gnu_123.so.12 [7] #5 0x0000000000526d91 in filter_m::filter (fn=..., io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/modules/filter_m.f90:1242 [7] #6 0x00000000005e279c in integrate (io=6, update_fortran_arrays=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/f90_files/solve/integrate_erk.f90:228 [7] #7 0x00000000005da350 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:301 [7] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [7] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [7] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [7] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [7] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [7] [Inferior 1 (process 3227349) detached] srun: error: frontier08191: task 0: Segmentation fault (core dumped) srun: Terminating StepId=3546544.0 slurmstepd: error: *** STEP 3546544.0 ON frontier08191 CANCELLED AT 2025-06-30T10:30:16 *** *** Caught a signal (proc 1): SIGTERM(15) *** Caught a signal (proc 2): SIGTERM(15) *** Caught a signal (proc 3): SIGTERM(15) *** Caught a signal (proc 4): SIGTERM(15) *** Caught a signal (proc 5): SIGTERM(15) *** Caught a signal (proc 6): SIGTERM(15) srun: error: frontier09085: task 7: Aborted (core dumped) aborting job: GASNet abnormal exit srun: error: frontier08191: task 3: Exited with exit code 255 gasnet_exit(): WARNING: timeout during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit gasnet_exit(): WARNING: timeout during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit gasnet_exit(): WARNING: timeout during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit gasnet_exit(): WARNING: timeout during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit gasnet_exit(): WARNING: timeout during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit srun: error: frontier08191: task 2: Exited with exit code 255 srun: error: frontier09085: tasks 4,6: Exited with exit code 255 srun: error: frontier08191: task 1: Exited with exit code 255 srun: error: frontier09085: task 5: Exited with exit code 255