============================================================================== Hello, and welcome to S3D! ============================================================================== ============================================================== --------------------S3D BUILD CONFIGURATION----------------- Build directory: /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/fortran/ammonia ------------------------------------------------------ S3D Options: Using mixture averaged transport: ON Using ARK integration: OFF Using chemical mechanism: ammonia Using srcs for timescale analysis: OFF ------------------------------------------------------ Defines: SYSTEMCALLWONTWORK SAVEFILEINSEPDIR ARCH_OPTERON MPIIO BLOCKIO MIXAVG FROZENFEED ------------------------------------------------------ FORTRAN COMPILER FLAGS: -cpp -fdefault-double-8 -fdefault-real-8 -fno-range-check -ffree-line-length-none -fallow-argument-mismatch C COMPILER FLAGS: ------------------------------------------------------ S3D Version: ------------------------------------------------------ Chemistry routine header: c**************************************************************************** c ... DO NOT EDIT. AUTOMATICALLY GENERATED FILE. c**************************************************************************** c ... c ... CKWYP-compatible GETRATES: return molar production rates for all c ... species given pressure (P), temperature (T), and mass fractions (Y). c ... c ... Generated by cgetrates 1.0b c ... c**************************************************************************** subroutine getrates(P,T,Y,ICKWRK,RCKWRK,WSPL) ------------------------------------------------------ Transport routine header: !Standard version of tranlib.f. 18Feb 2009 ! (Changes by Mark Fahey, ORNL) ! (Update by Evatt Hawkes, SNL) SUBROUTINE MCABS ------------------------------------------------------ ============================================================== ------------------------------------------------------------------------------ this run started at: date = 06/30/2025 time = 10:26:03 ------------------------------------------------------------------------------ initializing chemkin module... opening chemkin link file... setting chemkin work array lengths... allocating chemkin work arrays... filling chemkin work arrays based on chemkin link file... CKLIB: CHEMKIN-III GAS-PHASE CHEMICAL KINETICS LIBRARY, DOUBLE PRECISION Vers. 5.36 2000/12/15 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 the following was extracted from the ../input/chem.asc file: element names are as follows: index name 1 H 2 O 3 N species names are as follows: index name 1 H2 2 NH3 3 NO 4 N2O 5 O2 6 H 7 O 8 OH 9 HO2 10 H2O 11 H2O2 12 NO2 13 HNO 14 N 15 NNH 16 NH2 17 NH 18 H2NO 19 N2 species molecular weights are as follows: index name value (kg/mol) 1 H2 2.01594E-03 2 NH3 1.70306E-02 3 NO 3.00061E-02 4 N2O 4.40128E-02 5 O2 3.19988E-02 6 H 1.00797E-03 7 O 1.59994E-02 8 OH 1.70074E-02 9 HO2 3.30068E-02 10 H2O 1.80153E-02 11 H2O2 3.40147E-02 12 NO2 4.60055E-02 13 HNO 3.10141E-02 14 N 1.40067E-02 15 NNH 2.90214E-02 16 NH2 1.60226E-02 17 NH 1.50147E-02 18 H2NO 3.20220E-02 19 N2 2.80134E-02 ------------------------------------------------------------------------------ initializing param module... number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 number of reaction third-body reactions = 11 ------------------------------------------------------------------------------ initializing topology module... Initialized topology module, npx = 4 npy = 2 npz = 4 npes = 32 ------------------------------------------------------------------------------ initializing reference module... the various reference values are as follows: universal gas constant (J/mol-K) = 8.31400E+00 freestream temperature (K) = 8.01900E+02 reference ratio of specifice heats = 1.36400E+00 reference speed of sound (m/s) = 5.28000E+02 reference density (kg/m^3) = 1.22300E+01 reference conductivity (W/m-K) = 5.76900E-02 reference temperature (K) = 2.91892E+02 reference pressure (Pa) = 3.40953E+06 standard atmospheric pressure (Pa) = 1.01325E+05 reference time (s) = 1.25428E-06 reference specific heat (J/kg-K) = 9.55094E+02 reference length (m) = 6.62258E-04 reference viscosity (Pa-s) = 4.27649E+00 acoustic Reynolds number = 1.00000E+05 Mach number = 1.00000E-03 convective Reynolds number = 1.00000E+02 ------------------------------------------------------------------------------ initializing ERK module... ERK module information is as follows: rk_method = -64 cont_switch = 0 tstep_init (seconds) = 5.00E-10 tstep_min (seconds) = 5.00E-10 tstep_max (seconds) = 5.00E-10 rk_rtol = 1.00E-03 rk_atol = 1.00E-15 cont_safety = 9.00E-01 k_I = 2.00E-01 k_P = 2.00E-01 k_D = 5.00E-02 k_D2 = 0.00E+00 cont_factor = 3.00E+00 cfl_switch = 1 i_time_cont = 20 cont_n_reg = 1 controller type = NONE rk_tol = 1.00E-12 using RK4(3)6[2R+]C ERK scheme! ------------------------------------------------------------------------------ initializing transport module... opening mixavg.in ... Lagging is switched on WARNING - with coefficient lagging, theoretical temporal convergence order may not be observed opening transport link file... setting lengths of transport work arrays... allocating transport work arrays... creating transport work arrays from transport link file... TRANLIB: CHEMKIN-III MULTICOMPONENT TRANSPORT LIBRARY, DOUBLE PRECISION Vers. 4.6 1999/06/03 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. ------------------------------------------------------------------------------ initializing filter module... using explicit filter of order 10 filter will be applied every 10 time steps ------------------------------------------------------------------------------ initializing derivative module... ------------------------------------------------------------------------------ initializing thermchem module... ------------------------------------------------------------------------------ initializing grid module... ------------------------------------------------------------------------------ initializing boundary condition module... ------------------------------------------------------------------------------ *** WARNING: ofi-conduit failed to configure FI_CXI_* environment variables due to prior conflicting settings. This may lead to unstable behavior and/or degraded performance. If you did not intentionally set FI_CXI_RX_MATCH_MODE='software', FI_CXI_RDZV_THRESHOLD='16384', or FI_CXI_RDZV_GET_MIN then this condition may have resulted from initializing MPI prior to initialization of GASNet. For more information on that scenario, please see "Limits to MPI interoperability" in the ofi-conduit README. Legion Init: Mechanism = 'ammonia' Local Grid Size = ( 64, 64, 256) Global Grid Size = (256, 128, 1024) Proc Grid Size = ( 4, 2, 4) Proc ID = ( 0, 0, 0) nvar_tot = 23, n_spec = 19, n_reg = 3, n_scalar = 0 iorder = 8, iforder = 10, vary = XYZ fuse fields set to 200 from environment set mapper mode to 'allgpu' from environment RHST_ARGS: -ll:gpu 2 -ll:cpu 1 -ll:util 3 -ll:bgwork 3 -ll:fsize 16384 -ll:csize 4096 -ll:zsize 1024 -ll:stacksize 16 -logfile run_%.log -hip:hostreg 0 -lg:no_transitive_reduction 1 -lg:prof 32 -lg:prof_logfile prof_%.gz REGULAR STENCIL initializing variables... Host: frontier03584 Rank: 0 PID: 347383 Time: 10:26:09 STARTING MAIN TASK! initializing bomb (stand back) Mixture fraction module successfully initialized... storing hard inflow conditions... ------------------------------------------------------------------------------ continuing with initialization... ------------------------------------------------------------------------------ storing hard inflow conditions... maximum speed of sound = 839.70 m/s mimimum speed of sound = 839.70 m/s maximum acoustic time = 2.06E-05 sec minimum acoustic time = 2.06E-05 sec initialization completed for time = 0.00E+00 ------------------------------------------------------------------------------ writing save files for: i_time = 0, time = 0.0000E+00 (sec) precision is d generating active.in file... ------------------------------------------------------------------------------ 10:26:19.059 1 5.0000E-10 5.000E-10 6.140E-04 1600.00 30.00 10:26:20.060 2 1.0000E-09 5.000E-10 2.010E-04 1600.00 30.00 10:26:20.897 3 1.5000E-09 5.000E-10 6.685E-05 1600.00 30.00 10:26:21.678 4 2.0000E-09 5.000E-10 2.712E-05 1600.00 30.00 10:26:22.409 5 2.5000E-09 5.000E-10 1.285E-05 1600.00 30.00 10:26:23.050 6 3.0000E-09 5.000E-10 6.797E-06 1600.00 30.00 10:26:23.691 7 3.5000E-09 5.000E-10 3.893E-06 1600.00 30.00 10:26:24.336 8 4.0000E-09 5.000E-10 2.367E-06 1600.00 30.00 10:26:24.974 9 4.5000E-09 5.000E-10 1.508E-06 1600.00 30.00 10:26:27.308 10 5.0000E-09 5.000E-10 9.961E-07 1600.00 30.00 10:26:55.959 20 1.0000E-08 5.000E-10 4.621E-08 1600.00 30.00 10:27:23.317 30 1.5000E-08 5.000E-10 9.442E-09 1600.00 30.00 10:27:32.639 40 2.0000E-08 5.000E-10 2.475E-09 1600.00 30.00 10:27:42.349 50 2.5000E-08 5.000E-10 6.750E-10 1600.00 30.00 10:27:51.723 60 3.0000E-08 5.000E-10 1.821E-10 1600.00 30.00 10:28:01.287 70 3.5000E-08 5.000E-10 1.370E-10 1600.00 30.00 10:28:12.193 80 4.0000E-08 5.000E-10 1.442E-10 1600.00 30.00 10:28:23.202 90 4.5000E-08 5.000E-10 1.514E-10 1600.00 30.00 10:28:33.932 100 5.0000E-08 5.000E-10 1.556E-10 1600.00 30.00 10:28:45.077 110 5.5000E-08 5.000E-10 1.568E-10 1600.00 30.00 10:28:55.952 120 6.0000E-08 5.000E-10 1.551E-10 1600.00 30.00 10:29:06.567 130 6.5000E-08 5.000E-10 1.512E-10 1600.00 30.00 10:29:17.703 140 7.0000E-08 5.000E-10 1.454E-10 1600.00 30.00 10:29:28.547 150 7.5000E-08 5.000E-10 1.386E-10 1600.00 30.00 10:29:39.691 160 8.0000E-08 5.000E-10 1.312E-10 1600.00 30.00 10:29:50.525 170 8.5000E-08 5.000E-10 1.237E-10 1600.00 30.00 10:30:01.293 180 9.0000E-08 5.000E-10 1.164E-10 1600.00 30.00 10:30:13.877 190 9.5000E-08 5.000E-10 1.093E-10 1600.00 30.00 *** Caught a fatal signal (proc 0): SIGSEGV(11) *** Caught a fatal signal (proc 0): SIGSEGV(11) *** Caught a fatal signal (proc 0): SIGSEGV(11) *** Caught a fatal signal (proc 0): SIGSEGV(11) *** NOTICE (proc 0): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** NOTICE (proc 0): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** NOTICE (proc 0): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** NOTICE (proc 0): We recommend linking the debug version of GASNet to assist you in resolving this application issue. srun: error: frontier03584: task 0: Segmentation fault (core dumped) srun: Terminating StepId=3546541.0 slurmstepd: error: *** STEP 3546541.0 ON frontier03584 CANCELLED AT 2025-06-30T10:31:43 *** *** Caught a signal (proc 1): SIGTERM(15) *** Caught a signal (proc 2): SIGTERM(15) *** Caught a signal (proc 3): SIGTERM(15) *** Caught a signal (proc 4): SIGTERM(15) *** Caught a signal (proc 12): SIGTERM(15) *** Caught a signal (proc 5): SIGTERM(15) *** Caught a signal (proc 13): SIGTERM(15) *** Caught a signal (proc 6): SIGTERM(15) *** Caught a signal (proc 14): SIGTERM(15) *** Caught a signal (proc 7): SIGTERM(15) *** Caught a signal (proc 15): SIGTERM(15) *** Caught a signal (proc 16): SIGTERM(15) *** Caught a signal (proc 17): SIGTERM(15) *** Caught a signal (proc 18): SIGTERM(15) *** Caught a signal (proc 20): SIGTERM(15) *** Caught a signal (proc 19): SIGTERM(15) *** Caught a signal (proc 21): SIGTERM(15) *** Caught a signal (proc 22): SIGTERM(15) *** Caught a signal (proc 23): SIGTERM(15) *** Caught a signal (proc 28): SIGTERM(15) *** Caught a signal (proc 29): SIGTERM(15) *** Caught a signal (proc 30): SIGTERM(15) *** FATAL ERROR (proc 16): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** Caught a signal (proc 31): SIGTERM(15) *** NOTICE (proc 16): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 16): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** Caught a signal (proc 8): SIGTERM(15) *** Caught a signal (proc 9): SIGTERM(15) *** Caught a signal (proc 10): SIGTERM(15) *** Caught a signal (proc 11): SIGTERM(15) [16] Invoking GDB for backtrace... *** Caught a signal (proc 24): SIGTERM(15) *** FATAL ERROR (proc 24): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 24): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 24): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** Caught a signal (proc 25): SIGTERM(15) *** Caught a signal (proc 26): SIGTERM(15) [24] Invoking GDB for backtrace... *** Caught a signal (proc 27): SIGTERM(15) *** FATAL ERROR (proc 28): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** FATAL ERROR (proc 28): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -3(No such process) *** NOTICE (proc 28): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 28): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** Details for bug reporting (proc 28): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** FATAL ERROR (proc 30): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 30): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 30): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** FATAL ERROR (proc 31): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 31): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 31): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu [30] Invoking GDB for backtrace... [31] Invoking GDB for backtrace... [16] /usr/bin/gdb -nx -batch -x /tmp/gasnet_6AhzoP '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_8/s3d.x' 217276 [16] [New LWP 217297] [16] [New LWP 217328] [16] [New LWP 217329] [16] [New LWP 217333] [16] [New LWP 217342] [16] [New LWP 217346] [16] [New LWP 217348] [16] [New LWP 217350] [16] [New LWP 217353] [16] [New LWP 217356] [16] [New LWP 217371] [16] [New LWP 217372] [16] [New LWP 217379] [16] [New LWP 217394] [16] [New LWP 217395] [16] [New LWP 217403] [16] [New LWP 217405] [16] [New LWP 217408] [16] [New LWP 217409] [16] [Thread debugging using libthread_db enabled] [16] Using host libthread_db library "/lib64/libthread_db.so.1". [16] 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [16] Id Target Id Frame [16] * 1 Thread 0x7fffd5a20600 (LWP 217276) "s3d.x" 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [16] 2 Thread 0x7fffcf5ff6c0 (LWP 217297) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [16] 3 Thread 0x7ffd783fc600 (LWP 217328) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [16] 4 Thread 0x7ffd721f3600 (LWP 217329) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 5 Thread 0x7ffd631ff600 (LWP 217333) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 6 Thread 0x7ffd8281f600 (LWP 217342) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 7 Thread 0x7ffd82813600 (LWP 217346) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 8 Thread 0x7ffd8261f600 (LWP 217348) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 9 Thread 0x7ffd82613600 (LWP 217350) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 10 Thread 0x7ff07057d600 (LWP 217353) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 11 Thread 0x7ff070379600 (LWP 217356) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 12 Thread 0x7ffd5ebff600 (LWP 217371) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 13 Thread 0x7ffd5a5ff600 (LWP 217372) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 14 Thread 0x7ffd55fff600 (LWP 217379) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 15 Thread 0x7ffd519ff600 (LWP 217394) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 16 Thread 0x7ffd4d3ff600 (LWP 217395) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 17 Thread 0x7ffd48dff600 (LWP 217403) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 18 Thread 0x7ffd42fff600 (LWP 217405) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] 19 Thread 0x7ffd79eff6c0 (LWP 217408) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [16] 20 Thread 0x7ffd3e9ff600 (LWP 217409) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] [16] Thread 20 (Thread 0x7ffd3e9ff600 (LWP 217409) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3e9feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3e9feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b6f968) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [16] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b6f948) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [16] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc170440) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 19 (Thread 0x7ffd79eff6c0 (LWP 217408) "s3d.x"): [16] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [16] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #3 0x00007fffd6ca320c in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #4 0x00007fffd6c14e44 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #5 0x00007fffd6c14b5a in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #6 0x00007fffd6c07a19 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #7 0x00007fffec3d104b in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [16] #8 0x00007fffec150186 in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [16] #9 0x00007fffec37e29f in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [16] #10 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #11 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 18 (Thread 0x7ffd42fff600 (LWP 217405) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd42ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd42ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5987de8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [16] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5987dc8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [16] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfe5ed6300) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 17 (Thread 0x7ffd48dff600 (LWP 217403) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd48dfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd48dfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5987de8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [16] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5987dc8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [16] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfe5ed78f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 16 (Thread 0x7ffd4d3ff600 (LWP 217395) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd4d3feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd4d3feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b6f968) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [16] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b6f948) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [16] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc0c85d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 15 (Thread 0x7ffd519ff600 (LWP 217394) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd519feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd519feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b6f968) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [16] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b6f948) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [16] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc0ca570) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 14 (Thread 0x7ffd55fff600 (LWP 217379) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd55ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd55ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5987de8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [16] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5987dc8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [16] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc050048f40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 13 (Thread 0x7ffd5a5ff600 (LWP 217372) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b6f968) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [16] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b6f948) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [16] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0480013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 12 (Thread 0x7ffd5ebff600 (LWP 217371) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5ebfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5ebfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5987de8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [16] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5987dc8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [16] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0500013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 11 (Thread 0x7ff070379600 (LWP 217356) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff070378ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x58b91f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [16] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x58b91f0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [16] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x58b91f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [16] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x58b91f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [16] #7 Realm::Thread::thread_entry_wrapper (obj=0x58b91f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [16] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5a62ad0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 10 (Thread 0x7ff07057d600 (LWP 217353) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff07057cba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5c74930) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [16] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5c74930, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [16] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5c74930) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [16] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5c74930) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [16] #7 Realm::Thread::thread_entry_wrapper (obj=0x5c74930) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [16] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x80f3f30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 9 (Thread 0x7ffd82613600 (LWP 217350) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd82612ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5610a80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [16] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5610a80, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [16] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5610a80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [16] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [16] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [16] #8 0x0000000000000000 in ?? () [16] [16] Thread 8 (Thread 0x7ffd8261f600 (LWP 217348) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8261eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56c9bb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [16] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56c9bb0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [16] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56c9bb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [16] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [16] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [16] #8 0x0000000000000000 in ?? () [16] [16] Thread 7 (Thread 0x7ffd82813600 (LWP 217346) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd82812ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56c85d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [16] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56c85d0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [16] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56c85d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [16] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [16] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [16] #8 0x0000000000000000 in ?? () [16] [16] Thread 6 (Thread 0x7ffd8281f600 (LWP 217342) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8281eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56c82e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [16] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56c82e0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [16] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56c82e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [16] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [16] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [16] #8 0x0000000000000000 in ?? () [16] [16] Thread 5 (Thread 0x7ffd631ff600 (LWP 217333) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd631feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 Realm::BackgroundWorkThread::main_loop (this=0x53b2630) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [16] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x53b19e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 4 (Thread 0x7ffd721f3600 (LWP 217329) "s3d.x"): [16] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [16] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd721f2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [16] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [16] #3 Realm::BackgroundWorkThread::main_loop (this=0x53b2440) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [16] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x53b2750) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 3 (Thread 0x7ffd783fc600 (LWP 217328) "s3d.x"): [16] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [16] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [16] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #5 0x00007fffdf464ff7 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #6 0x00007fffdf4650e8 in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #7 0x00007fffe08861e2 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #8 0x00007fffe088631c in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #9 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #10 0x00007fffe087c52f in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #11 0x00007fffe087c92e in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #12 0x00007fffe0886dc0 in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #13 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [16] #14 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [16] #15 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [16] #16 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [16] #17 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x52660a0, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [16] #18 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd783fa5c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [16] #19 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x53b19b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [16] #20 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x53b2520) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [16] #21 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #22 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 2 (Thread 0x7fffcf5ff6c0 (LWP 217297) "s3d.x"): [16] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [16] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [16] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [16] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [16] [16] Thread 1 (Thread 0x7fffd5a20600 (LWP 217276) "s3d.x"): [16] #0 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [16] #1 0x00007fffe7879d23 in MPIR_Wait_impl () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [16] #2 0x00007fffe7892e3b in MPIR_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [16] #3 0x00007fffe789385d in PMPI_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [16] #4 0x00007fffe9336ddc in pmpi_wait__ () from /opt/cray/pe/lib64/libmpifort_gnu_123.so.12 [16] #5 0x0000000000527524 in filter_m::filter (fn=..., io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/modules/filter_m.f90:1495 [16] #6 0x00000000005e279c in integrate (io=6, update_fortran_arrays=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/f90_files/solve/integrate_erk.f90:228 [16] #7 0x00000000005da350 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:301 [16] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [16] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [16] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [16] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [16] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [16] [Inferior 1 (process 217276) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit [24] /usr/bin/gdb -nx -batch -x /tmp/gasnet_zId7T5 '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_8/s3d.x' 391596 [24] [New LWP 391615] [24] [New LWP 391643] [24] [New LWP 391646] [24] [New LWP 391649] [24] [New LWP 391659] [24] [New LWP 391663] [24] [New LWP 391667] [24] [New LWP 391671] [24] [New LWP 391678] [24] [New LWP 391681] [24] [New LWP 391689] [24] [New LWP 391690] [24] [New LWP 391691] [24] [New LWP 391713] [24] [New LWP 391714] [24] [New LWP 391715] [24] [New LWP 391730] [24] [New LWP 391731] [24] [New LWP 391732] [24] [Thread debugging using libthread_db enabled] [24] Using host libthread_db library "/lib64/libthread_db.so.1". [24] 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [24] Id Target Id Frame [24] * 1 Thread 0x7fffd5a20600 (LWP 391596) "s3d.x" 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [24] 2 Thread 0x7fffcf5ff6c0 (LWP 391615) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [24] 3 Thread 0x7ffd723f3600 (LWP 391643) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 4 Thread 0x7ffd633ff600 (LWP 391646) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 5 Thread 0x7ffd5edff600 (LWP 391649) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [24] 6 Thread 0x7ffd8271f600 (LWP 391659) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 7 Thread 0x7ffd82713600 (LWP 391663) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 8 Thread 0x7ffd8243f600 (LWP 391667) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 9 Thread 0x7ffd7e31f600 (LWP 391671) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 10 Thread 0x7ff0783ff600 (LWP 391678) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 11 Thread 0x7ff07057d600 (LWP 391681) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 12 Thread 0x7ffd5a7ff600 (LWP 391689) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 13 Thread 0x7ffd561ff600 (LWP 391690) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 14 Thread 0x7ffd51bff600 (LWP 391691) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 15 Thread 0x7ffd4d5ff600 (LWP 391713) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 16 Thread 0x7ffd48fff600 (LWP 391714) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 17 Thread 0x7ffd431ff600 (LWP 391715) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 18 Thread 0x7ffd79ebf6c0 (LWP 391730) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [24] 19 Thread 0x7ffd3ebff600 (LWP 391731) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 20 Thread 0x7ffd3a5ff600 (LWP 391732) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] [24] Thread 20 (Thread 0x7ffd3a5ff600 (LWP 391732) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3a5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3a5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5965c48) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5965c28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd1f273e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 19 (Thread 0x7ffd3ebff600 (LWP 391731) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3ebfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3ebfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5895be8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5895bc8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc157d60) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 18 (Thread 0x7ffd79ebf6c0 (LWP 391730) "s3d.x"): [24] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [24] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #3 0x00007fffd6ca320c in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #4 0x00007fffd6c14e44 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #5 0x00007fffd6c14b5a in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #6 0x00007fffd6c07a19 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #7 0x00007fffec3d104b in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [24] #8 0x00007fffec150186 in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [24] #9 0x00007fffec37e29f in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [24] #10 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #11 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 17 (Thread 0x7ffd431ff600 (LWP 391715) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd431feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd431feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5965c48) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5965c28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd1ec5340) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 16 (Thread 0x7ffd48fff600 (LWP 391714) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd48ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd48ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5895be8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5895bc8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc0b9540) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 15 (Thread 0x7ffd4d5ff600 (LWP 391713) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd4d5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd4d5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5895be8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5895bc8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc0c4cb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 14 (Thread 0x7ffd51bff600 (LWP 391691) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd51bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd51bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5965c48) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5965c28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c048700) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 13 (Thread 0x7ffd561ff600 (LWP 391690) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd561feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd561feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5895be8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5895bc8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0500013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 12 (Thread 0x7ffd5a7ff600 (LWP 391689) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5965c48) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5965c28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 11 (Thread 0x7ff07057d600 (LWP 391681) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff07057cba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5bed8b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5bed8b0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5bed8b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5bed8b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [24] #7 Realm::Thread::thread_entry_wrapper (obj=0x5bed8b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [24] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5568650) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 10 (Thread 0x7ff0783ff600 (LWP 391678) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff0783feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5c87c10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5c87c10, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5c87c10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5c87c10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [24] #7 Realm::Thread::thread_entry_wrapper (obj=0x5c87c10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [24] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x80f4a70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 9 (Thread 0x7ffd7e31f600 (LWP 391671) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7e31eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x58dafa0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x58dafa0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x58dafa0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [24] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [24] #8 0x0000000000000000 in ?? () [24] [24] Thread 8 (Thread 0x7ffd8243f600 (LWP 391667) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8243eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x41d5700) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x41d5700, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x41d5700) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [24] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [24] #8 0x0000000000000000 in ?? () [24] [24] Thread 7 (Thread 0x7ffd82713600 (LWP 391663) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd82712ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56c85c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56c85c0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56c85c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [24] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [24] #8 0x0000000000000000 in ?? () [24] [24] Thread 6 (Thread 0x7ffd8271f600 (LWP 391659) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8271eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56c82d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56c82d0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56c82d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [24] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [24] #8 0x0000000000000000 in ?? () [24] [24] Thread 5 (Thread 0x7ffd5edff600 (LWP 391649) "s3d.x"): [24] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [24] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [24] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #5 0x00007fffdf464ff7 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #6 0x00007fffdf4650e8 in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #7 0x00007fffe08861e2 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #8 0x00007fffe088631c in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #9 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #10 0x00007fffe087c52f in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #11 0x00007fffe087c92e in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #12 0x00007fffe0886dc0 in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #13 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [24] #14 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [24] #15 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [24] #16 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [24] #17 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x5265d70, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [24] #18 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd5edfd5c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [24] #19 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x583aa00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [24] #20 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x583af80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #21 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #22 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 4 (Thread 0x7ffd633ff600 (LWP 391646) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd633feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::BackgroundWorkThread::main_loop (this=0x583a830) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [24] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x583ab20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 3 (Thread 0x7ffd723f3600 (LWP 391643) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd723f2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::BackgroundWorkThread::main_loop (this=0x5fe66d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [24] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x583a910) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 2 (Thread 0x7fffcf5ff6c0 (LWP 391615) "s3d.x"): [24] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [24] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 1 (Thread 0x7fffd5a20600 (LWP 391596) "s3d.x"): [24] #0 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [24] #1 0x00007fffe7879d23 in MPIR_Wait_impl () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [24] #2 0x00007fffe7892e3b in MPIR_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [24] #3 0x00007fffe789385d in PMPI_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [24] #4 0x00007fffe9336ddc in pmpi_wait__ () from /opt/cray/pe/lib64/libmpifort_gnu_123.so.12 [24] #5 0x00000000005290b4 in filter_m::filter (fn=..., io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/modules/filter_m.f90:1512 [24] #6 0x00000000005e279c in integrate (io=6, update_fortran_arrays=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/f90_files/solve/integrate_erk.f90:228 [24] #7 0x00000000005da350 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:301 [24] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [24] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [24] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [24] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [24] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [24] [Inferior 1 (process 391596) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit [30] /usr/bin/gdb -nx -batch -x /tmp/gasnet_za4lDa '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_8/s3d.x' 2921000 [30] [New LWP 2921016] [30] [New LWP 2921056] [30] [New LWP 2921060] [30] [New LWP 2921064] [30] [New LWP 2921087] [30] [New LWP 2921091] [30] [New LWP 2921095] [30] [New LWP 2921104] [30] [New LWP 2921106] [30] [New LWP 2921107] [30] [New LWP 2921108] [30] [New LWP 2921109] [30] [New LWP 2921116] [30] [New LWP 2921128] [30] [New LWP 2921129] [30] [New LWP 2921136] [30] [New LWP 2921142] [30] [New LWP 2921150] [30] [New LWP 2921151] [30] [Thread debugging using libthread_db enabled] [30] Using host libthread_db library "/lib64/libthread_db.so.1". [30] 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [30] Id Target Id Frame [30] * 1 Thread 0x7fffd5a20600 (LWP 2921000) "s3d.x" 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [30] 2 Thread 0x7fffd35ff6c0 (LWP 2921016) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [30] 3 Thread 0x7ffd77f05600 (LWP 2921056) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 4 Thread 0x7ffd71df3600 (LWP 2921060) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 5 Thread 0x7ffd62dff600 (LWP 2921064) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [30] 6 Thread 0x7ffd8600f600 (LWP 2921087) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 7 Thread 0x7ffd85e3f600 (LWP 2921091) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 8 Thread 0x7ffd81c3f600 (LWP 2921095) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 9 Thread 0x7ffd7db9f600 (LWP 2921104) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 10 Thread 0x7fc0687fd600 (LWP 2921106) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 11 Thread 0x7fc0685f9600 (LWP 2921107) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 12 Thread 0x7ffd5e7ff600 (LWP 2921108) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 13 Thread 0x7ffd5a1ff600 (LWP 2921109) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 14 Thread 0x7ffd55bff600 (LWP 2921116) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 15 Thread 0x7ffd515ff600 (LWP 2921128) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 16 Thread 0x7ffd4cfff600 (LWP 2921129) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 17 Thread 0x7ffd489ff600 (LWP 2921136) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 18 Thread 0x7ffd7d7ff6c0 (LWP 2921142) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [30] 19 Thread 0x7ffd42bff600 (LWP 2921150) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 20 Thread 0x7ffd3e5ff600 (LWP 2921151) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] [30] Thread 20 (Thread 0x7ffd3e5ff600 (LWP 2921151) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x59a01a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x59a0188) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd816bdb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 19 (Thread 0x7ffd42bff600 (LWP 2921150) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5807108) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58070e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfdc243a40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 18 (Thread 0x7ffd7d7ff6c0 (LWP 2921142) "s3d.x"): [30] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [30] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #3 0x00007fffd6ca320c in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #4 0x00007fffd6c14e44 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #5 0x00007fffd6c14b5a in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #6 0x00007fffd6c07a19 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #7 0x00007fffec3d104b in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [30] #8 0x00007fffec150186 in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [30] #9 0x00007fffec37e29f in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [30] #10 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #11 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 17 (Thread 0x7ffd489ff600 (LWP 2921136) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5807108) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58070e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfdc1e9680) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 16 (Thread 0x7ffd4cfff600 (LWP 2921129) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x59a01a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x59a0188) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd80c8180) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 15 (Thread 0x7ffd515ff600 (LWP 2921128) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x59a01a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x59a0188) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd80ca070) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 14 (Thread 0x7ffd55bff600 (LWP 2921116) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5807108) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58070e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c048a20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 13 (Thread 0x7ffd5a1ff600 (LWP 2921109) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x59a01a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x59a0188) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0440013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 12 (Thread 0x7ffd5e7ff600 (LWP 2921108) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5e7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5e7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5807108) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58070e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 11 (Thread 0x7fc0685f9600 (LWP 2921107) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc0685f8ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5a442d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5a442d0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5a442d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5a442d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [30] #7 Realm::Thread::thread_entry_wrapper (obj=0x5a442d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [30] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x80e3e30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 10 (Thread 0x7fc0687fd600 (LWP 2921106) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc0687fcba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5c8bef0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5c8bef0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5c8bef0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5c8bef0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [30] #7 Realm::Thread::thread_entry_wrapper (obj=0x5c8bef0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [30] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x80e3880) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 9 (Thread 0x7ffd7db9f600 (LWP 2921104) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7db9eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5596040) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5596040, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5596040) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #8 0x0000000000000000 in ?? () [30] [30] Thread 8 (Thread 0x7ffd81c3f600 (LWP 2921095) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd81c3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56b6550) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56b6550, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56b6550) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #8 0x0000000000000000 in ?? () [30] [30] Thread 7 (Thread 0x7ffd85e3f600 (LWP 2921091) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd85e3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56b4f70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56b4f70, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56b4f70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #8 0x0000000000000000 in ?? () [30] [30] Thread 6 (Thread 0x7ffd8600f600 (LWP 2921087) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8600eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56b4c80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56b4c80, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56b4c80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #8 0x0000000000000000 in ?? () [30] [30] Thread 5 (Thread 0x7ffd62dff600 (LWP 2921064) "s3d.x"): [30] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [30] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [30] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #5 0x00007fffdf464ff7 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #6 0x00007fffdf4650e8 in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #7 0x00007fffe08861e2 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #8 0x00007fffe088631c in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #9 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #10 0x00007fffe087c52f in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #11 0x00007fffe087c92e in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #12 0x00007fffe0886dc0 in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #13 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [30] #14 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [30] #15 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [30] #16 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [30] #17 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x5261020, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [30] #18 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd62dfd5c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [30] #19 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x5a46400) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [30] #20 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5955960) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #21 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #22 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 4 (Thread 0x7ffd71df3600 (LWP 2921060) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd71df2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::BackgroundWorkThread::main_loop (this=0x5955a70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [30] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5955ff0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 3 (Thread 0x7ffd77f05600 (LWP 2921056) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd77f04ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::BackgroundWorkThread::main_loop (this=0x5a46500) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [30] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5955b90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 2921016) "s3d.x"): [30] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [30] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 1 (Thread 0x7fffd5a20600 (LWP 2921000) "s3d.x"): [30] #0 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [30] #1 0x00007fffe7879d23 in MPIR_Wait_impl () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [30] #2 0x00007fffe7892e3b in MPIR_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [30] #3 0x00007fffe789385d in PMPI_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [30] #4 0x00007fffe9336ddc in pmpi_wait__ () from /opt/cray/pe/lib64/libmpifort_gnu_123.so.12 [30] #5 0x0000000000527078 in filter_m::filter (fn=..., io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/modules/filter_m.f90:1258 [30] #6 0x00000000005e279c in integrate (io=6, update_fortran_arrays=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/f90_files/solve/integrate_erk.f90:228 [30] #7 0x00000000005da350 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:301 [30] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [30] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [30] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [30] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [30] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [30] [Inferior 1 (process 2921000) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit [31] /usr/bin/gdb -nx -batch -x /tmp/gasnet_fxOIil '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_8/s3d.x' 2921001 [31] [New LWP 2921019] [31] [New LWP 2921058] [31] [New LWP 2921062] [31] [New LWP 2921066] [31] [New LWP 2921086] [31] [New LWP 2921090] [31] [New LWP 2921094] [31] [New LWP 2921098] [31] [New LWP 2921101] [31] [New LWP 2921105] [31] [New LWP 2921111] [31] [New LWP 2921110] [31] [New LWP 2921126] [31] [New LWP 2921130] [31] [New LWP 2921131] [31] [New LWP 2921143] [31] [New LWP 2921144] [31] [New LWP 2921146] [31] [New LWP 2921149] [31] [Thread debugging using libthread_db enabled] [31] Using host libthread_db library "/lib64/libthread_db.so.1". [31] 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [31] Id Target Id Frame [31] * 1 Thread 0x7fffd5a20600 (LWP 2921001) "s3d.x" 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [31] 2 Thread 0x7fffd35ff6c0 (LWP 2921019) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [31] 3 Thread 0x7ffd77f05600 (LWP 2921058) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 4 Thread 0x7ffd71df3600 (LWP 2921062) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [31] 5 Thread 0x7ffd62dff600 (LWP 2921066) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 6 Thread 0x7ffd8600f600 (LWP 2921086) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 7 Thread 0x7ffd85e3f600 (LWP 2921090) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 8 Thread 0x7ffd81c3f600 (LWP 2921094) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 9 Thread 0x7ffd7db9f600 (LWP 2921098) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 10 Thread 0x7fc0697fe600 (LWP 2921101) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 11 Thread 0x7fc0685f9600 (LWP 2921105) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 12 Thread 0x7ffd5e7ff600 (LWP 2921111) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 13 Thread 0x7ffd5a1ff600 (LWP 2921110) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 14 Thread 0x7ffd55bff600 (LWP 2921126) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 15 Thread 0x7ffd515ff600 (LWP 2921130) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 16 Thread 0x7ffd4cfff600 (LWP 2921131) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 17 Thread 0x7ffd489ff600 (LWP 2921143) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 18 Thread 0x7ffd42bff600 (LWP 2921144) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 19 Thread 0x7ffd7d7bf6c0 (LWP 2921146) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [31] 20 Thread 0x7ffd3e5ff600 (LWP 2921149) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] [31] Thread 20 (Thread 0x7ffd3e5ff600 (LWP 2921149) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5fafd88) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5fafd68) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc129dc0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 19 (Thread 0x7ffd7d7bf6c0 (LWP 2921146) "s3d.x"): [31] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [31] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #3 0x00007fffd6ca320c in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #4 0x00007fffd6c14e44 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #5 0x00007fffd6c14b5a in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #6 0x00007fffd6c07a19 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #7 0x00007fffec3d104b in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [31] #8 0x00007fffec150186 in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [31] #9 0x00007fffec37e29f in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [31] #10 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #11 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 18 (Thread 0x7ffd42bff600 (LWP 2921144) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57e3ad8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57e3ab8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd1f55380) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 17 (Thread 0x7ffd489ff600 (LWP 2921143) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5fafd88) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5fafd68) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc12bc60) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 16 (Thread 0x7ffd4cfff600 (LWP 2921131) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57e3ad8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57e3ab8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd1eb3020) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 15 (Thread 0x7ffd515ff600 (LWP 2921130) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57e3ad8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57e3ab8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd1eb3690) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 14 (Thread 0x7ffd55bff600 (LWP 2921126) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5fafd88) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5fafd68) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc0ca130) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 13 (Thread 0x7ffd5a1ff600 (LWP 2921110) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57e3ad8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57e3ab8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0480013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 12 (Thread 0x7ffd5e7ff600 (LWP 2921111) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5e7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5e7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5fafd88) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5fafd68) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0400013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 11 (Thread 0x7fc0685f9600 (LWP 2921105) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc0685f8ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5b758d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5b758d0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5b758d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5b758d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [31] #7 Realm::Thread::thread_entry_wrapper (obj=0x5b758d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [31] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x56b2490) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 10 (Thread 0x7fc0697fe600 (LWP 2921101) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc0697fdba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5beaab0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5beaab0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5beaab0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5beaab0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [31] #7 Realm::Thread::thread_entry_wrapper (obj=0x5beaab0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [31] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x56b1ee0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 9 (Thread 0x7ffd7db9f600 (LWP 2921098) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7db9eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5928c80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5928c80, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5928c80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [31] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [31] #8 0x0000000000000000 in ?? () [31] [31] Thread 8 (Thread 0x7ffd81c3f600 (LWP 2921094) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd81c3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x41cf900) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x41cf900, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x41cf900) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [31] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [31] #8 0x0000000000000000 in ?? () [31] [31] Thread 7 (Thread 0x7ffd85e3f600 (LWP 2921090) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd85e3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56b4f10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56b4f10, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56b4f10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [31] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [31] #8 0x0000000000000000 in ?? () [31] [31] Thread 6 (Thread 0x7ffd8600f600 (LWP 2921086) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8600eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56b4c20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56b4c20, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56b4c20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [31] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [31] #8 0x0000000000000000 in ?? () [31] [31] Thread 5 (Thread 0x7ffd62dff600 (LWP 2921066) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd62dfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::BackgroundWorkThread::main_loop (this=0x5b81b80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [31] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5b81240) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 4 (Thread 0x7ffd71df3600 (LWP 2921062) "s3d.x"): [31] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [31] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [31] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #5 0x00007fffdf464ff7 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #6 0x00007fffdf4650e8 in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #7 0x00007fffe08861e2 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #8 0x00007fffe088631c in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #9 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #10 0x00007fffe087c52f in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #11 0x00007fffe087c92e in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #12 0x00007fffe0886dc0 in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #13 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [31] #14 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [31] #15 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [31] #16 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [31] #17 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x5261030, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [31] #18 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd71df15c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [31] #19 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x5626f00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [31] #20 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5b81c60) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #21 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #22 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 3 (Thread 0x7ffd77f05600 (LWP 2921058) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd77f04ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::BackgroundWorkThread::main_loop (this=0x5b81d70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [31] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5b816a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 2921019) "s3d.x"): [31] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [31] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 1 (Thread 0x7fffd5a20600 (LWP 2921001) "s3d.x"): [31] #0 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [31] #1 0x00007fffe7879d23 in MPIR_Wait_impl () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [31] #2 0x00007fffe7892e3b in MPIR_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [31] #3 0x00007fffe789385d in PMPI_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [31] #4 0x00007fffe9336ddc in pmpi_wait__ () from /opt/cray/pe/lib64/libmpifort_gnu_123.so.12 [31] #5 0x00000000005290b4 in filter_m::filter (fn=..., io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/modules/filter_m.f90:1512 [31] #6 0x00000000005e279c in integrate (io=6, update_fortran_arrays=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/f90_files/solve/integrate_erk.f90:228 [31] #7 0x00000000005da350 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:301 [31] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [31] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [31] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [31] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [31] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [31] [Inferior 1 (process 2921001) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit srun: error: frontier07040: task 16: Exited with exit code 255 srun: error: frontier07423: task 24: Exited with exit code 255 srun: error: frontier07551: tasks 30-31: Exited with exit code 255 aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit srun: error: frontier03584: tasks 2-3: Exited with exit code 255 srun: error: frontier07551: task 29: Exited with exit code 255 srun: error: frontier07040: tasks 17,19: Exited with exit code 255 srun: error: frontier07423: tasks 25-26: Exited with exit code 255 srun: error: frontier06527: tasks 8-9,11: Exited with exit code 255 srun: error: frontier06399: tasks 4-6: Exited with exit code 255 srun: error: frontier07295: tasks 21-23: Exited with exit code 255 srun: error: frontier06783: tasks 13-15: Exited with exit code 255 srun: error: frontier03584: task 1: Exited with exit code 255 srun: error: frontier07423: task 27: Exited with exit code 255 srun: error: frontier07040: task 18: Exited with exit code 255 srun: error: frontier06783: task 12: Exited with exit code 255 srun: error: frontier06527: task 10: Exited with exit code 255 srun: error: frontier07295: task 20: Exited with exit code 255 srun: error: frontier06399: task 7: Exited with exit code 255 srun: error: frontier07551: task 28: Aborted (core dumped)