============================================================================== Hello, and welcome to S3D! ============================================================================== ============================================================== --------------------S3D BUILD CONFIGURATION----------------- Build directory: /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/fortran/ammonia ------------------------------------------------------ S3D Options: Using mixture averaged transport: ON Using ARK integration: OFF Using chemical mechanism: ammonia Using srcs for timescale analysis: OFF ------------------------------------------------------ Defines: SYSTEMCALLWONTWORK SAVEFILEINSEPDIR ARCH_OPTERON MPIIO BLOCKIO MIXAVG FROZENFEED ------------------------------------------------------ FORTRAN COMPILER FLAGS: -cpp -fdefault-double-8 -fdefault-real-8 -fno-range-check -ffree-line-length-none -fallow-argument-mismatch C COMPILER FLAGS: ------------------------------------------------------ S3D Version: ------------------------------------------------------ Chemistry routine header: c**************************************************************************** c ... DO NOT EDIT. AUTOMATICALLY GENERATED FILE. c**************************************************************************** c ... c ... CKWYP-compatible GETRATES: return molar production rates for all c ... species given pressure (P), temperature (T), and mass fractions (Y). c ... c ... Generated by cgetrates 1.0b c ... c**************************************************************************** subroutine getrates(P,T,Y,ICKWRK,RCKWRK,WSPL) ------------------------------------------------------ Transport routine header: !Standard version of tranlib.f. 18Feb 2009 ! (Changes by Mark Fahey, ORNL) ! (Update by Evatt Hawkes, SNL) SUBROUTINE MCABS ------------------------------------------------------ ============================================================== ------------------------------------------------------------------------------ this run started at: date = 06/30/2025 time = 10:26:41 ------------------------------------------------------------------------------ initializing chemkin module... opening chemkin link file... setting chemkin work array lengths... allocating chemkin work arrays... filling chemkin work arrays based on chemkin link file... CKLIB: CHEMKIN-III GAS-PHASE CHEMICAL KINETICS LIBRARY, DOUBLE PRECISION Vers. 5.36 2000/12/15 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 the following was extracted from the ../input/chem.asc file: element names are as follows: index name 1 H 2 O 3 N species names are as follows: index name 1 H2 2 NH3 3 NO 4 N2O 5 O2 6 H 7 O 8 OH 9 HO2 10 H2O 11 H2O2 12 NO2 13 HNO 14 N 15 NNH 16 NH2 17 NH 18 H2NO 19 N2 species molecular weights are as follows: index name value (kg/mol) 1 H2 2.01594E-03 2 NH3 1.70306E-02 3 NO 3.00061E-02 4 N2O 4.40128E-02 5 O2 3.19988E-02 6 H 1.00797E-03 7 O 1.59994E-02 8 OH 1.70074E-02 9 HO2 3.30068E-02 10 H2O 1.80153E-02 11 H2O2 3.40147E-02 12 NO2 4.60055E-02 13 HNO 3.10141E-02 14 N 1.40067E-02 15 NNH 2.90214E-02 16 NH2 1.60226E-02 17 NH 1.50147E-02 18 H2NO 3.20220E-02 19 N2 2.80134E-02 ------------------------------------------------------------------------------ initializing param module... number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 number of reaction third-body reactions = 11 ------------------------------------------------------------------------------ initializing topology module... Initialized topology module, npx = 2 npy = 2 npz = 4 npes = 16 ------------------------------------------------------------------------------ initializing reference module... the various reference values are as follows: universal gas constant (J/mol-K) = 8.31400E+00 freestream temperature (K) = 8.01900E+02 reference ratio of specifice heats = 1.36400E+00 reference speed of sound (m/s) = 5.28000E+02 reference density (kg/m^3) = 1.22300E+01 reference conductivity (W/m-K) = 5.76900E-02 reference temperature (K) = 2.91892E+02 reference pressure (Pa) = 3.40953E+06 standard atmospheric pressure (Pa) = 1.01325E+05 reference time (s) = 1.25428E-06 reference specific heat (J/kg-K) = 9.55094E+02 reference length (m) = 6.62258E-04 reference viscosity (Pa-s) = 4.27649E+00 acoustic Reynolds number = 1.00000E+05 Mach number = 1.00000E-03 convective Reynolds number = 1.00000E+02 ------------------------------------------------------------------------------ initializing ERK module... ERK module information is as follows: rk_method = -64 cont_switch = 0 tstep_init (seconds) = 5.00E-10 tstep_min (seconds) = 5.00E-10 tstep_max (seconds) = 5.00E-10 rk_rtol = 1.00E-03 rk_atol = 1.00E-15 cont_safety = 9.00E-01 k_I = 2.00E-01 k_P = 2.00E-01 k_D = 5.00E-02 k_D2 = 0.00E+00 cont_factor = 3.00E+00 cfl_switch = 1 i_time_cont = 20 cont_n_reg = 1 controller type = NONE rk_tol = 1.00E-12 using RK4(3)6[2R+]C ERK scheme! ------------------------------------------------------------------------------ initializing transport module... opening mixavg.in ... Lagging is switched on WARNING - with coefficient lagging, theoretical temporal convergence order may not be observed opening transport link file... setting lengths of transport work arrays... allocating transport work arrays... creating transport work arrays from transport link file... TRANLIB: CHEMKIN-III MULTICOMPONENT TRANSPORT LIBRARY, DOUBLE PRECISION Vers. 4.6 1999/06/03 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. ------------------------------------------------------------------------------ initializing filter module... using explicit filter of order 10 filter will be applied every 10 time steps ------------------------------------------------------------------------------ initializing derivative module... ------------------------------------------------------------------------------ initializing thermchem module... ------------------------------------------------------------------------------ initializing grid module... ------------------------------------------------------------------------------ initializing boundary condition module... ------------------------------------------------------------------------------ *** WARNING: ofi-conduit failed to configure FI_CXI_* environment variables due to prior conflicting settings. This may lead to unstable behavior and/or degraded performance. If you did not intentionally set FI_CXI_RX_MATCH_MODE='software', FI_CXI_RDZV_THRESHOLD='16384', or FI_CXI_RDZV_GET_MIN then this condition may have resulted from initializing MPI prior to initialization of GASNet. For more information on that scenario, please see "Limits to MPI interoperability" in the ofi-conduit README. Legion Init: Mechanism = 'ammonia' Local Grid Size = ( 64, 64, 256) Global Grid Size = (128, 128, 1024) Proc Grid Size = ( 2, 2, 4) Proc ID = ( 0, 0, 0) nvar_tot = 23, n_spec = 19, n_reg = 3, n_scalar = 0 iorder = 8, iforder = 10, vary = XYZ fuse fields set to 200 from environment set mapper mode to 'allgpu' from environment RHST_ARGS: -ll:gpu 2 -ll:cpu 1 -ll:util 3 -ll:bgwork 3 -ll:fsize 16384 -ll:csize 4096 -ll:zsize 1024 -ll:stacksize 16 -logfile run_%.log -hip:hostreg 0 -lg:no_transitive_reduction 1 -lg:prof 16 -lg:prof_logfile prof_%.gz REGULAR STENCIL initializing variables... Host: frontier05632 Rank: 0 PID: 294631 Time: 10:26:46 STARTING MAIN TASK! initializing bomb (stand back) Mixture fraction module successfully initialized... storing hard inflow conditions... ------------------------------------------------------------------------------ continuing with initialization... ------------------------------------------------------------------------------ storing hard inflow conditions... maximum speed of sound = 839.70 m/s mimimum speed of sound = 839.70 m/s maximum acoustic time = 2.06E-05 sec minimum acoustic time = 2.06E-05 sec initialization completed for time = 0.00E+00 ------------------------------------------------------------------------------ writing save files for: i_time = 0, time = 0.0000E+00 (sec) precision is d generating active.in file... ------------------------------------------------------------------------------ 10:26:56.283 1 5.0000E-10 5.000E-10 6.140E-04 1600.00 30.00 10:26:57.283 2 1.0000E-09 5.000E-10 2.010E-04 1600.00 30.00 10:26:58.103 3 1.5000E-09 5.000E-10 6.685E-05 1600.00 30.00 10:26:58.928 4 2.0000E-09 5.000E-10 2.712E-05 1600.00 30.00 10:26:59.609 5 2.5000E-09 5.000E-10 1.285E-05 1600.00 30.00 10:27:00.285 6 3.0000E-09 5.000E-10 6.797E-06 1600.00 30.00 10:27:00.960 7 3.5000E-09 5.000E-10 3.893E-06 1600.00 30.00 10:27:01.631 8 4.0000E-09 5.000E-10 2.367E-06 1600.00 30.00 10:27:02.308 9 4.5000E-09 5.000E-10 1.508E-06 1600.00 30.00 10:27:04.678 10 5.0000E-09 5.000E-10 9.961E-07 1600.00 30.00 10:27:32.275 20 1.0000E-08 5.000E-10 4.621E-08 1600.00 30.00 10:27:59.029 30 1.5000E-08 5.000E-10 9.442E-09 1600.00 30.00 10:28:08.684 40 2.0000E-08 5.000E-10 2.475E-09 1600.00 30.00 10:28:18.759 50 2.5000E-08 5.000E-10 6.750E-10 1600.00 30.00 10:28:28.544 60 3.0000E-08 5.000E-10 1.821E-10 1600.00 30.00 10:28:38.577 70 3.5000E-08 5.000E-10 1.370E-10 1600.00 30.00 10:28:50.438 80 4.0000E-08 5.000E-10 1.442E-10 1600.00 30.00 10:29:02.122 90 4.5000E-08 5.000E-10 1.514E-10 1600.00 30.00 10:29:13.528 100 5.0000E-08 5.000E-10 1.556E-10 1600.00 30.00 10:29:24.947 110 5.5000E-08 5.000E-10 1.568E-10 1600.00 30.00 10:29:36.935 120 6.0000E-08 5.000E-10 1.551E-10 1600.00 30.00 10:29:48.567 130 6.5000E-08 5.000E-10 1.512E-10 1600.00 30.00 10:30:00.285 140 7.0000E-08 5.000E-10 1.454E-10 1600.00 30.00 10:30:12.107 150 7.5000E-08 5.000E-10 1.386E-10 1600.00 30.00 10:30:23.887 160 8.0000E-08 5.000E-10 1.312E-10 1600.00 30.00 10:30:35.669 170 8.5000E-08 5.000E-10 1.237E-10 1600.00 30.00 10:30:47.698 180 9.0000E-08 5.000E-10 1.164E-10 1600.00 30.00 10:31:01.254 190 9.5000E-08 5.000E-10 1.093E-10 1600.00 30.00 *** Caught a fatal signal (proc 0): SIGSEGV(11) *** NOTICE (proc 0): We recommend linking the debug version of GASNet to assist you in resolving this application issue. [0] Invoking GDB for backtrace... srun: error: frontier05632: task 0: Segmentation fault (core dumped) srun: Terminating StepId=3546545.0 slurmstepd: error: *** STEP 3546545.0 ON frontier05632 CANCELLED AT 2025-06-30T10:32:28 *** *** Caught a signal (proc 1): SIGTERM(15) *** Caught a signal (proc 2): SIGTERM(15) *** Caught a signal (proc 3): SIGTERM(15) *** Caught a signal (proc 4): SIGTERM(15) *** Caught a signal (proc 12): SIGTERM(15) *** Caught a signal (proc 5): SIGTERM(15) *** Caught a signal (proc 13): SIGTERM(15) *** Caught a signal (proc 6): SIGTERM(15) *** Caught a signal (proc 14): SIGTERM(15) *** Caught a signal (proc 7): SIGTERM(15) *** Caught a signal (proc 15): SIGTERM(15) *** FATAL ERROR (proc 15): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 15): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 15): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** FATAL ERROR (proc 14): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** FATAL ERROR (proc 14): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -2012228160(Unspecified error) *** NOTICE (proc 14): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 14): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** NOTICE (proc 14): We recommend linking the debug version of GASNet to assist you in resolving this application issue. [15] Invoking GDB for backtrace... *** FATAL ERROR (proc 12): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 12): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 12): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** Details for bug reporting (proc 14): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** Caught a signal (proc 8): SIGTERM(15) *** Caught a signal (proc 9): SIGTERM(15) *** Caught a signal (proc 10): SIGTERM(15) *** FATAL ERROR (proc 12): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -86471632(Unspecified error) *** Details for bug reporting (proc 12): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu [12] Invoking GDB for backtrace... *** Caught a signal (proc 11): SIGTERM(15) [15] /usr/bin/gdb -nx -batch -x /tmp/gasnet_1HNZGd '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_4/s3d.x' 230587 [15] [New LWP 230602] [15] [New LWP 230614] [15] [New LWP 230618] [15] [New LWP 230620] [15] [New LWP 230621] [15] [New LWP 230625] [15] [New LWP 230629] [15] [New LWP 230632] [15] [New LWP 230636] [15] [New LWP 230640] [15] [New LWP 230651] [15] [New LWP 230652] [15] [New LWP 230655] [15] [New LWP 230660] [15] [New LWP 230661] [15] [New LWP 230677] [15] [New LWP 230682] [15] [New LWP 230686] [15] [New LWP 230687] [15] [Thread debugging using libthread_db enabled] [15] Using host libthread_db library "/lib64/libthread_db.so.1". [15] 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [15] Id Target Id Frame [15] * 1 Thread 0x7fffd5a20600 (LWP 230587) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [15] 2 Thread 0x7fffd35ff6c0 (LWP 230602) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [15] 3 Thread 0x7ffd77f05600 (LWP 230614) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 4 Thread 0x7ffd71df3600 (LWP 230618) "s3d.x" 0x00007fffe65a7ecc in __lll_lock_wait () from /lib64/libc.so.6 [15] 5 Thread 0x7ffd62dff600 (LWP 230620) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 6 Thread 0x7ffd8601f600 (LWP 230621) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 7 Thread 0x7ffd86013600 (LWP 230625) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 8 Thread 0x7ffd85e3f600 (LWP 230629) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 9 Thread 0x7ffd81c3f600 (LWP 230632) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 10 Thread 0x7ff07117e600 (LWP 230636) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 11 Thread 0x7ff070f7a600 (LWP 230640) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 12 Thread 0x7ffd5e7ff600 (LWP 230651) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 13 Thread 0x7ffd5a1ff600 (LWP 230652) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 14 Thread 0x7ffd55bff600 (LWP 230655) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 15 Thread 0x7ffd515ff600 (LWP 230660) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 16 Thread 0x7ffd4cfff600 (LWP 230661) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 17 Thread 0x7ffd7d7bf6c0 (LWP 230677) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [15] 18 Thread 0x7ffd489ff600 (LWP 230682) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 19 Thread 0x7ffd42bff600 (LWP 230686) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] 20 Thread 0x7ffd3e5ff600 (LWP 230687) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] [15] Thread 20 (Thread 0x7ffd3e5ff600 (LWP 230687) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 Realm::UnfairCondVar::wait (this=this@entry=0x59bd558) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [15] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x59bd538) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [15] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc135a30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 19 (Thread 0x7ffd42bff600 (LWP 230686) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5781678) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [15] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5781658) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [15] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd1f63a80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 18 (Thread 0x7ffd489ff600 (LWP 230682) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 Realm::UnfairCondVar::wait (this=this@entry=0x59bd558) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [15] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x59bd538) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [15] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc0d8b90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 17 (Thread 0x7ffd7d7bf6c0 (LWP 230677) "s3d.x"): [15] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [15] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #3 0x00007fffd6ca320c in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #4 0x00007fffd6c14e44 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #5 0x00007fffd6c14b5a in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #6 0x00007fffd6c07a19 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #7 0x00007fffec3d104b in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [15] #8 0x00007fffec150186 in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [15] #9 0x00007fffec37e29f in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [15] #10 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #11 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 16 (Thread 0x7ffd4cfff600 (LWP 230661) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5781678) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [15] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5781658) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [15] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd1eb24e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 15 (Thread 0x7ffd515ff600 (LWP 230660) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5781678) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [15] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5781658) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [15] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd1eb2980) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 14 (Thread 0x7ffd55bff600 (LWP 230655) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 Realm::UnfairCondVar::wait (this=this@entry=0x59bd558) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [15] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x59bd538) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [15] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc048048a20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 13 (Thread 0x7ffd5a1ff600 (LWP 230652) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5781678) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [15] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5781658) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [15] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0400013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 12 (Thread 0x7ffd5e7ff600 (LWP 230651) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5e7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5e7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 Realm::UnfairCondVar::wait (this=this@entry=0x59bd558) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [15] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x59bd538) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [15] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0480013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 11 (Thread 0x7ff070f7a600 (LWP 230640) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff070f79ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5a4a370) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [15] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5a4a370, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [15] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5a4a370) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [15] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5a4a370) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [15] #7 Realm::Thread::thread_entry_wrapper (obj=0x5a4a370) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [15] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x80d74b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 10 (Thread 0x7ff07117e600 (LWP 230636) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff07117dba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5c6aa00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [15] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5c6aa00, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [15] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5c6aa00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [15] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5c6aa00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [15] #7 Realm::Thread::thread_entry_wrapper (obj=0x5c6aa00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [15] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x8055d60) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 9 (Thread 0x7ffd81c3f600 (LWP 230632) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd81c3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56a9c10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [15] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56a9c10, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [15] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56a9c10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [15] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [15] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [15] #8 0x0000000000000000 in ?? () [15] [15] Thread 8 (Thread 0x7ffd85e3f600 (LWP 230629) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd85e3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56a9920) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [15] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56a9920, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [15] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56a9920) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [15] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [15] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [15] #8 0x0000000000000000 in ?? () [15] [15] Thread 7 (Thread 0x7ffd86013600 (LWP 230625) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd86012ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56aa4b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [15] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56aa4b0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [15] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56aa4b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [15] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [15] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [15] #8 0x0000000000000000 in ?? () [15] [15] Thread 6 (Thread 0x7ffd8601f600 (LWP 230621) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8601eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5919c80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [15] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5919c80, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [15] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5919c80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [15] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [15] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [15] #8 0x0000000000000000 in ?? () [15] [15] Thread 5 (Thread 0x7ffd62dff600 (LWP 230620) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd62dfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 Realm::BackgroundWorkThread::main_loop (this=0x5607030) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [15] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5b938b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 4 (Thread 0x7ffd71df3600 (LWP 230618) "s3d.x"): [15] #0 0x00007fffe65a7ecc in __lll_lock_wait () from /lib64/libc.so.6 [15] #1 0x00007fffe65ae615 in pthread_mutex_lock@@GLIBC_2.2.5 () from /lib64/libc.so.6 [15] #2 0x00007fffe087c366 in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #3 0x00007fffe087c92e in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #4 0x00007fffe089665c in gasneti_AMPSHMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #5 0x00007fffe087c04b in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #6 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [15] #7 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [15] #8 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [15] #9 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x5260a00, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [15] #10 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd71df15c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [15] #11 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x5b93640) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [15] #12 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5b934e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #13 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #14 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 3 (Thread 0x7ffd77f05600 (LWP 230614) "s3d.x"): [15] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [15] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd77f04ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [15] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [15] #3 Realm::BackgroundWorkThread::main_loop (this=0x5b933c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [15] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x56ae150) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [15] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 230602) "s3d.x"): [15] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [15] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [15] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [15] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [15] [15] Thread 1 (Thread 0x7fffd5a20600 (LWP 230587) "s3d.x"): [15] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [15] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [15] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #5 0x00007fffdf464ff7 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #6 0x00007fffdf4650e8 in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #7 0x00007fffe08861e2 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #8 0x00007fffe088631c in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #9 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #10 0x00007fffe087c52f in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #11 0x00007fffdf46f9ed in gasneti_defaultSignalHandler () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [15] #12 [15] #13 0x00007fffe65acdfc in __pthread_kill_implementation () from /lib64/libc.so.6 [15] #14 0x00007fffe655b842 in raise () from /lib64/libc.so.6 [15] #15 [15] #16 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [15] #17 0x00007fffe7879d23 in MPIR_Wait_impl () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [15] #18 0x00007fffe7892e3b in MPIR_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [15] #19 0x00007fffe789385d in PMPI_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [15] #20 0x00007fffe9336ddc in pmpi_wait__ () from /opt/cray/pe/lib64/libmpifort_gnu_123.so.12 [15] #21 0x00000000005290b4 in filter_m::filter (fn=..., io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/modules/filter_m.f90:1512 [15] #22 0x00000000005e279c in integrate (io=6, update_fortran_arrays=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/f90_files/solve/integrate_erk.f90:228 [15] #23 0x00000000005da350 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:301 [15] #24 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [15] #25 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [15] #26 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [15] #27 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [15] #28 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [15] [Inferior 1 (process 230587) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit srun: error: frontier07422: task 15: Exited with exit code 255 aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit srun: error: frontier05632: task 3: Exited with exit code 255 srun: error: frontier07422: task 13: Exited with exit code 255 srun: error: frontier07294: tasks 8,10-11: Exited with exit code 255 srun: error: frontier06144: tasks 5-7: Exited with exit code 255 srun: error: frontier07294: task 9: Exited with exit code 255 srun: error: frontier06144: task 4: Exited with exit code 255 gasnet_exit(): WARNING: timeout during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit gasnet_exit(): WARNING: timeout during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit srun: error: frontier05632: task 2: Exited with exit code 255 srun: error: frontier05632: task 1: Exited with exit code 255