============================================================================== Hello, and welcome to S3D! ============================================================================== ============================================================== --------------------S3D BUILD CONFIGURATION----------------- Build directory: /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/fortran/ammonia ------------------------------------------------------ S3D Options: Using mixture averaged transport: ON Using ARK integration: OFF Using chemical mechanism: ammonia Using srcs for timescale analysis: OFF ------------------------------------------------------ Defines: SYSTEMCALLWONTWORK SAVEFILEINSEPDIR ARCH_OPTERON MPIIO BLOCKIO MIXAVG FROZENFEED ------------------------------------------------------ FORTRAN COMPILER FLAGS: -cpp -fdefault-double-8 -fdefault-real-8 -fno-range-check -ffree-line-length-none -fallow-argument-mismatch C COMPILER FLAGS: ------------------------------------------------------ S3D Version: ------------------------------------------------------ Chemistry routine header: c**************************************************************************** c ... DO NOT EDIT. AUTOMATICALLY GENERATED FILE. c**************************************************************************** c ... c ... CKWYP-compatible GETRATES: return molar production rates for all c ... species given pressure (P), temperature (T), and mass fractions (Y). c ... c ... Generated by cgetrates 1.0b c ... c**************************************************************************** subroutine getrates(P,T,Y,ICKWRK,RCKWRK,WSPL) ------------------------------------------------------ Transport routine header: !Standard version of tranlib.f. 18Feb 2009 ! (Changes by Mark Fahey, ORNL) ! (Update by Evatt Hawkes, SNL) SUBROUTINE MCABS ------------------------------------------------------ ============================================================== ------------------------------------------------------------------------------ this run started at: date = 06/30/2025 time = 12:47:15 ------------------------------------------------------------------------------ initializing chemkin module... opening chemkin link file... setting chemkin work array lengths... allocating chemkin work arrays... filling chemkin work arrays based on chemkin link file... CKLIB: CHEMKIN-III GAS-PHASE CHEMICAL KINETICS LIBRARY, DOUBLE PRECISION Vers. 5.36 2000/12/15 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 the following was extracted from the ../input/chem.asc file: element names are as follows: index name 1 H 2 O 3 N species names are as follows: index name 1 H2 2 NH3 3 NO 4 N2O 5 O2 6 H 7 O 8 OH 9 HO2 10 H2O 11 H2O2 12 NO2 13 HNO 14 N 15 NNH 16 NH2 17 NH 18 H2NO 19 N2 species molecular weights are as follows: index name value (kg/mol) 1 H2 2.01594E-03 2 NH3 1.70306E-02 3 NO 3.00061E-02 4 N2O 4.40128E-02 5 O2 3.19988E-02 6 H 1.00797E-03 7 O 1.59994E-02 8 OH 1.70074E-02 9 HO2 3.30068E-02 10 H2O 1.80153E-02 11 H2O2 3.40147E-02 12 NO2 4.60055E-02 13 HNO 3.10141E-02 14 N 1.40067E-02 15 NNH 2.90214E-02 16 NH2 1.60226E-02 17 NH 1.50147E-02 18 H2NO 3.20220E-02 19 N2 2.80134E-02 ------------------------------------------------------------------------------ Initialized ADIOS; 0 initializing param module... number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 number of reaction third-body reactions = 11 ------------------------------------------------------------------------------ initializing topology module... Initialized topology module, npx = 4 npy = 2 npz = 4 npes = 32 ------------------------------------------------------------------------------ initializing reference module... the various reference values are as follows: universal gas constant (J/mol-K) = 8.31400E+00 freestream temperature (K) = 8.01900E+02 reference ratio of specifice heats = 1.36400E+00 reference speed of sound (m/s) = 5.28000E+02 reference density (kg/m^3) = 1.22300E+01 reference conductivity (W/m-K) = 5.76900E-02 reference temperature (K) = 2.91892E+02 reference pressure (Pa) = 3.40953E+06 standard atmospheric pressure (Pa) = 1.01325E+05 reference time (s) = 1.25428E-06 reference specific heat (J/kg-K) = 9.55094E+02 reference length (m) = 6.62258E-04 reference viscosity (Pa-s) = 4.27649E+00 acoustic Reynolds number = 1.00000E+05 Mach number = 1.00000E-03 convective Reynolds number = 1.00000E+02 ------------------------------------------------------------------------------ initializing ERK module... ERK module information is as follows: rk_method = -64 cont_switch = 0 tstep_init (seconds) = 1.00E-09 tstep_min (seconds) = 1.00E-09 tstep_max (seconds) = 1.00E-09 rk_rtol = 1.00E-03 rk_atol = 1.00E-15 cont_safety = 9.00E-01 k_I = 2.00E-01 k_P = 2.00E-01 k_D = 5.00E-02 k_D2 = 0.00E+00 cont_factor = 3.00E+00 cfl_switch = 1 i_time_cont = 20 cont_n_reg = 1 controller type = NONE rk_tol = 1.00E-12 using RK4(3)6[2R+]C ERK scheme! ------------------------------------------------------------------------------ initializing transport module... opening mixavg.in ... Lagging is switched on WARNING - with coefficient lagging, theoretical temporal convergence order may not be observed opening transport link file... setting lengths of transport work arrays... allocating transport work arrays... creating transport work arrays from transport link file... TRANLIB: CHEMKIN-III MULTICOMPONENT TRANSPORT LIBRARY, DOUBLE PRECISION Vers. 4.6 1999/06/03 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. ------------------------------------------------------------------------------ initializing filter module... using explicit filter of order 10 filter will be applied every 10 time steps ------------------------------------------------------------------------------ initializing derivative module... ------------------------------------------------------------------------------ initializing thermchem module... ------------------------------------------------------------------------------ initializing grid module... ------------------------------------------------------------------------------ initializing boundary condition module... ------------------------------------------------------------------------------ *** WARNING: ofi-conduit failed to configure FI_CXI_* environment variables due to prior conflicting settings. This may lead to unstable behavior and/or degraded performance. If you did not intentionally set FI_CXI_RX_MATCH_MODE='software', FI_CXI_RDZV_THRESHOLD='16384', or FI_CXI_RDZV_GET_MIN then this condition may have resulted from initializing MPI prior to initialization of GASNet. For more information on that scenario, please see "Limits to MPI interoperability" in the ofi-conduit README. Legion Init: Mechanism = 'ammonia' Local Grid Size = ( 64, 64, 256) Global Grid Size = (256, 128, 1024) Proc Grid Size = ( 4, 2, 4) Proc ID = ( 0, 0, 0) nvar_tot = 23, n_spec = 19, n_reg = 3, n_scalar = 0 iorder = 8, iforder = 10, vary = XYZ fuse fields set to 200 from environment set mapper mode to 'allgpu' from environment RHST_ARGS: -ll:gpu 2 -ll:cpu 1 -ll:util 3 -ll:bgwork 3 -ll:fsize 16384 -ll:csize 4096 -ll:zsize 1024 -ll:stacksize 16 -logfile run_%.log -hip:hostreg 0 -lg:no_transitive_reduction 1 -lg:prof 32 -lg:prof_logfile prof_%.gz -level barrier=1 REGULAR STENCIL initializing variables... Host: frontier02431 Rank: 0 PID: 241404 Time: 12:47:20 STARTING MAIN TASK! initializing pressure wave test... initializing species 5 as 0.233 initializing species 19 as 0.767 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ continuing with initialization... ------------------------------------------------------------------------------ storing hard inflow conditions... maximum speed of sound = 347.72 m/s mimimum speed of sound = 347.72 m/s maximum acoustic time = 4.98E-05 sec minimum acoustic time = 4.98E-05 sec initialization completed for time = 0.00E+00 ------------------------------------------------------------------------------ writing save files for: i_time = 0, time = 0.0000E+00 (sec) precision is d writing adios files for: i_time = 0, time = 0.0000E+00 (sec) precision is d Writing data with adios to ../data/pressure_wave_test.0.0000E+00.field.bp Adios ierr_adios: 0 ------------------------------------------------------------------------------ IO Started at 47:21.396 IO Finished at 47:22.086 Per-thread IO = 201.3MB Total IO = 6442.5MB Time taken = 0.690s Total IO Rate = 9336.885MB/s ------------------------------------------------------------------------------ generating active.in file... ------------------------------------------------------------------------------ 12:47:31.476 1 1.0000E-09 1.000E-09 1.881E-24 300.00 1.00 12:47:32.657 2 2.0000E-09 1.000E-09 1.881E-24 300.00 1.00 12:47:33.669 3 3.0000E-09 1.000E-09 1.881E-24 300.00 1.00 12:47:34.591 4 4.0000E-09 1.000E-09 1.881E-24 300.00 1.00 12:47:35.401 5 5.0000E-09 1.000E-09 1.881E-24 300.00 1.00 12:47:36.185 6 6.0000E-09 1.000E-09 1.881E-24 300.00 1.00 12:47:36.945 7 7.0000E-09 1.000E-09 1.881E-24 300.00 1.00 12:47:37.692 8 8.0000E-09 1.000E-09 1.881E-24 300.00 1.00 12:47:38.442 9 9.0000E-09 1.000E-09 1.881E-24 300.00 1.00 12:47:41.106 10 1.0000E-08 1.000E-09 1.881E-24 300.00 1.00 12:48:19.831 20 2.0000E-08 1.000E-09 1.881E-24 300.00 1.00 12:48:56.022 30 3.0000E-08 1.000E-09 1.881E-24 300.00 1.00 12:49:05.254 40 4.0000E-08 1.000E-09 1.881E-24 300.00 1.00 12:49:18.656 50 5.0000E-08 1.000E-09 1.881E-24 300.00 1.00 12:49:28.483 60 6.0000E-08 1.000E-09 1.881E-24 300.00 1.00 12:49:37.967 70 7.0000E-08 1.000E-09 1.881E-24 300.00 1.00 12:49:47.389 80 8.0000E-08 1.000E-09 1.881E-24 300.00 1.00 12:49:56.877 90 9.0000E-08 1.000E-09 1.881E-24 300.00 1.00 12:50:06.277 100 1.0000E-07 1.000E-09 1.881E-24 300.00 1.00 writing save files for: i_time = 100, time = 1.0000E-07 (sec) precision is d writing adios files for: i_time = 100, time = 1.0000E-07 (sec) precision is d Writing data with adios to ../data/pressure_wave_test.1.0000E-07.field.bp Adios ierr_adios: 808464640 ------------------------------------------------------------------------------ IO Started at 50:06.278 IO Finished at 50:06.955 Per-thread IO = 201.3MB Total IO = 6442.5MB Time taken = 0.677s Total IO Rate = 9516.176MB/s ------------------------------------------------------------------------------ 12:50:16.250 110 1.1000E-07 1.000E-09 1.881E-24 300.00 1.00 12:50:25.401 120 1.2000E-07 1.000E-09 1.881E-24 300.00 1.00 12:50:34.838 130 1.3000E-07 1.000E-09 1.881E-24 300.00 1.00 12:50:44.398 140 1.4000E-07 1.000E-09 1.881E-24 300.00 1.00 12:50:53.707 150 1.5000E-07 1.000E-09 1.881E-24 300.00 1.00 12:51:03.308 160 1.6000E-07 1.000E-09 1.881E-24 300.00 1.00 12:51:13.290 170 1.7000E-07 1.000E-09 1.881E-24 300.00 1.00 12:51:22.355 180 1.8000E-07 1.000E-09 1.881E-24 300.00 1.00 12:51:35.010 190 1.9000E-07 1.000E-09 1.881E-24 300.00 1.00 *** Caught a fatal signal (proc 0): SIGSEGV(11) *** NOTICE (proc 0): We recommend linking the debug version of GASNet to assist you in resolving this application issue. [0] Invoking GDB for backtrace... [26 - 7ffd77905600] 265.342014 {6}{gex}: CRC MISMATCH: arg0=81 header_size=36 payload_size=9600 exp=e61d3bec act=52057e03 *** Caught a fatal signal (proc 26): SIGABRT(6) *** FATAL ERROR (proc 24): in _gasneti_malloc() at on_s3d_subrank/gasnet/GASNet-2024.5.0/gasnet_internal.h:116: gasneti_malloc(140736967032608) failed *** NOTICE (proc 24): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 24): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** NOTICE (proc 26): We recommend linking the debug version of GASNet to assist you in resolving this application issue. [26] Invoking GDB for backtrace... [24] Invoking GDB for backtrace... [20 - 7ffd5e7ff600] 265.316830 {6}{gex}: CRC MISMATCH: arg0=81 header_size=36 payload_size=9600 exp=e61d3bec act=c69258a3 *** Caught a fatal signal (proc 20): SIGABRT(6) *** NOTICE (proc 20): We recommend linking the debug version of GASNet to assist you in resolving this application issue. [20] Invoking GDB for backtrace... [24] /usr/bin/gdb -nx -batch -x /tmp/gasnet_pKvdtH '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/pwave_x_8_ammonia/s3d.x' 208760 [24] [New LWP 208781] [24] [New LWP 208789] [24] [New LWP 208790] [24] [New LWP 208791] [24] [New LWP 208801] [24] [New LWP 208805] [24] [New LWP 208809] [24] [New LWP 208813] [24] [New LWP 208831] [24] [New LWP 208832] [24] [New LWP 208845] [24] [New LWP 208847] [24] [New LWP 208855] [24] [New LWP 208856] [24] [New LWP 208857] [24] [New LWP 208858] [24] [New LWP 208859] [24] [New LWP 208876] [24] [New LWP 208882] [24] [Thread debugging using libthread_db enabled] [24] Using host libthread_db library "/lib64/libthread_db.so.1". [24] 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [24] Id Target Id Frame [24] * 1 Thread 0x7fffd5a20600 (LWP 208760) "s3d.x" 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [24] 2 Thread 0x7fffcf5ff6c0 (LWP 208781) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [24] 3 Thread 0x7ffd71df3600 (LWP 208789) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 4 Thread 0x7ffd62dff600 (LWP 208790) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [24] 5 Thread 0x7ffd5e7ff600 (LWP 208791) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 6 Thread 0x7ffd8210f600 (LWP 208801) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 7 Thread 0x7ffd81e3f600 (LWP 208805) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 8 Thread 0x7ffd7dd1f600 (LWP 208809) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 9 Thread 0x7ffd7dd13600 (LWP 208813) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 10 Thread 0x7ffd5a1ff600 (LWP 208831) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 11 Thread 0x7ffd55bff600 (LWP 208832) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 12 Thread 0x7fc056dff600 (LWP 208845) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 13 Thread 0x7fc0567ff600 (LWP 208847) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 14 Thread 0x7ffd515ff600 (LWP 208855) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 15 Thread 0x7ffd4cfff600 (LWP 208856) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 16 Thread 0x7ffd489ff600 (LWP 208857) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 17 Thread 0x7ffd42bff600 (LWP 208858) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 18 Thread 0x7ffd3e5ff600 (LWP 208859) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] 19 Thread 0x7ffd798bf6c0 (LWP 208876) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [24] 20 Thread 0x7ffd39fff600 (LWP 208882) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] [24] Thread 20 (Thread 0x7ffd39fff600 (LWP 208882) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd39ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd39ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b416b8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b41698) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfe800b380) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 19 (Thread 0x7ffd798bf6c0 (LWP 208876) "s3d.x"): [24] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [24] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #3 0x00007fffd6ca320c in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #4 0x00007fffd6c14e44 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #5 0x00007fffd6c14b5a in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #6 0x00007fffd6c07a19 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #7 0x00007fffec3d104b in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [24] #8 0x00007fffec150186 in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [24] #9 0x00007fffec37e29f in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [24] #10 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #11 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 18 (Thread 0x7ffd3e5ff600 (LWP 208859) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57b0908) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57b08e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd5f40de0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 17 (Thread 0x7ffd42bff600 (LWP 208858) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b416b8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b41698) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc1ebf240) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 16 (Thread 0x7ffd489ff600 (LWP 208857) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b416b8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b41698) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c05eca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 15 (Thread 0x7ffd4cfff600 (LWP 208856) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57b0908) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57b08e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd5eafcf0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 14 (Thread 0x7ffd515ff600 (LWP 208855) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57b0908) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57b08e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd5ead820) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 13 (Thread 0x7fc0567ff600 (LWP 208847) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc0567feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56e3280) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56e3280, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56e3280) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x56e3280) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [24] #7 Realm::Thread::thread_entry_wrapper (obj=0x56e3280) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [24] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfd5eadd00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 12 (Thread 0x7fc056dff600 (LWP 208845) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc056dfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x560d470) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x560d470, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x560d470) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x560d470) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [24] #7 Realm::Thread::thread_entry_wrapper (obj=0x560d470) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [24] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c05d470) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 11 (Thread 0x7ffd55bff600 (LWP 208832) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57b0908) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57b08e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0440013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 10 (Thread 0x7ffd5a1ff600 (LWP 208831) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b416b8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [24] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b41698) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [24] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 9 (Thread 0x7ffd7dd13600 (LWP 208813) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7dd12ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5bd5130) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5bd5130, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5bd5130) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [24] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [24] #8 0x0000000000000000 in ?? () [24] [24] Thread 8 (Thread 0x7ffd7dd1f600 (LWP 208809) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7dd1eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56d1450) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56d1450, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56d1450) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [24] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [24] #8 0x0000000000000000 in ?? () [24] [24] Thread 7 (Thread 0x7ffd81e3f600 (LWP 208805) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd81e3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56d0370) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56d0370, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56d0370) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [24] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [24] #8 0x0000000000000000 in ?? () [24] [24] Thread 6 (Thread 0x7ffd8210f600 (LWP 208801) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8210eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56cfe70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [24] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56cfe70, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [24] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56cfe70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [24] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [24] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [24] #8 0x0000000000000000 in ?? () [24] [24] Thread 5 (Thread 0x7ffd5e7ff600 (LWP 208791) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd5e7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::BackgroundWorkThread::main_loop (this=0x5fd5150) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [24] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5fd56d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 4 (Thread 0x7ffd62dff600 (LWP 208790) "s3d.x"): [24] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [24] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [24] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [24] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [24] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [24] #5 0x00007fffdf464ff7 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [24] #6 0x00007fffdf4650e8 in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [24] #7 0x00007fffdf464020 in gasnetc_ofi_am_send_short[cold] () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [24] #8 0x0000000000000402 in ?? () [24] #9 0x0000000000000120 in ?? () [24] #10 0x00007ffd684bcd40 in ?? () [24] #11 0x0000000000000000 in ?? () [24] [24] Thread 3 (Thread 0x7ffd71df3600 (LWP 208789) "s3d.x"): [24] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [24] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd71df2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [24] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [24] #3 Realm::BackgroundWorkThread::main_loop (this=0x5fd4f70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [24] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5fd5020) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [24] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 2 (Thread 0x7fffcf5ff6c0 (LWP 208781) "s3d.x"): [24] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [24] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [24] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [24] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [24] [24] Thread 1 (Thread 0x7fffd5a20600 (LWP 208760) "s3d.x"): [24] #0 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [24] #1 0x00007fffe7879d23 in MPIR_Wait_impl () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [24] #2 0x00007fffe7892e3b in MPIR_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [24] #3 0x00007fffe789385d in PMPI_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [24] #4 0x00007fffe9336ddc in pmpi_wait__ () from /opt/cray/pe/lib64/libmpifort_gnu_123.so.12 [24] #5 0x00000000005290b4 in filter_m::filter (fn=..., io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/modules/filter_m.f90:1512 [24] #6 0x00000000005e279c in integrate (io=6, update_fortran_arrays=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/f90_files/solve/integrate_erk.f90:228 [24] #7 0x00000000005da350 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:301 [24] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [24] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [24] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [24] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [24] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [24] [Inferior 1 (process 208760) detached] *** Caught a fatal signal (proc 24): SIGABRT(6) *** Details for bug reporting (proc 24): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu [20] /usr/bin/gdb -nx -batch -x /tmp/gasnet_C5Z3ht '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/pwave_x_8_ammonia/s3d.x' 208298 [20] [New LWP 208314] [20] [New LWP 208328] [20] [New LWP 208332] [20] [New LWP 208336] [20] [New LWP 208339] [20] [New LWP 208343] [20] [New LWP 208348] [20] [New LWP 208351] [20] [New LWP 208359] [20] [New LWP 208363] [20] [New LWP 208364] [20] [New LWP 208380] [20] [New LWP 208381] [20] [New LWP 208386] [20] [New LWP 208396] [20] [New LWP 208401] [20] [New LWP 208402] [20] [New LWP 208409] [20] [New LWP 208413] [20] [Thread debugging using libthread_db enabled] [20] Using host libthread_db library "/lib64/libthread_db.so.1". [20] 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [20] Id Target Id Frame [20] * 1 Thread 0x7fffd5a20600 (LWP 208298) "s3d.x" 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [20] 2 Thread 0x7fffcf5ff6c0 (LWP 208314) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [20] 3 Thread 0x7ffd71df3600 (LWP 208328) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 4 Thread 0x7ffd62dff600 (LWP 208332) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 5 Thread 0x7ffd5e7ff600 (LWP 208336) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [20] 6 Thread 0x7ffd81e3f600 (LWP 208339) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 7 Thread 0x7ffd7dd3f600 (LWP 208343) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 8 Thread 0x7ffd7dd33600 (LWP 208348) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 9 Thread 0x7ffd7dd27600 (LWP 208351) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 10 Thread 0x7fc0695fa600 (LWP 208359) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 11 Thread 0x7ffd5a1ff600 (LWP 208363) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 12 Thread 0x7ffd55bff600 (LWP 208364) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 13 Thread 0x7ff070bff600 (LWP 208380) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 14 Thread 0x7fc06a7ff600 (LWP 208381) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 15 Thread 0x7ffd515ff600 (LWP 208386) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 16 Thread 0x7ffd4cfff600 (LWP 208396) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 17 Thread 0x7ffd489ff600 (LWP 208401) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 18 Thread 0x7ffd42bff600 (LWP 208402) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 19 Thread 0x7ffd3e5ff600 (LWP 208409) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] 20 Thread 0x7ffd798bf6c0 (LWP 208413) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [20] [20] Thread 20 (Thread 0x7ffd798bf6c0 (LWP 208413) "s3d.x"): [20] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [20] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #3 0x00007fffd6ca320c in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #4 0x00007fffd6c14e44 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #5 0x00007fffd6c14b5a in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #6 0x00007fffd6c07a19 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #7 0x00007fffec3d104b in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [20] #8 0x00007fffec150186 in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [20] #9 0x00007fffec37e29f in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [20] #10 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #11 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 19 (Thread 0x7ffd3e5ff600 (LWP 208409) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3e5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5fe8618) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [20] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5fe85f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [20] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfedf91810) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 18 (Thread 0x7ffd42bff600 (LWP 208402) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd42bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b438f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [20] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b438d8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [20] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc046c1d050) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 17 (Thread 0x7ffd489ff600 (LWP 208401) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd489feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5fe8618) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [20] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5fe85f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [20] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfedf508c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 16 (Thread 0x7ffd4cfff600 (LWP 208396) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd4cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b438f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [20] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b438d8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [20] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc046b6e8a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 15 (Thread 0x7ffd515ff600 (LWP 208386) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd515feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b438f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [20] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b438d8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [20] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc046b6c4f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 14 (Thread 0x7fc06a7ff600 (LWP 208381) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7fc06a7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7fc06a7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5fe8618) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [20] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5fe85f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [20] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfec042810) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 13 (Thread 0x7ff070bff600 (LWP 208380) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff070bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5c8e260) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [20] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5c8e260, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [20] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5c8e260) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [20] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5c8e260) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [20] #7 Realm::Thread::thread_entry_wrapper (obj=0x5c8e260) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [20] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c063790) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 12 (Thread 0x7ffd55bff600 (LWP 208364) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd55bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b438f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [20] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b438d8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [20] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0440013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 11 (Thread 0x7ffd5a1ff600 (LWP 208363) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5fe8618) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [20] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5fe85f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [20] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 10 (Thread 0x7fc0695fa600 (LWP 208359) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc0695f9ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5909430) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [20] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5909430, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [20] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5909430) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [20] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5909430) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [20] #7 Realm::Thread::thread_entry_wrapper (obj=0x5909430) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [20] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5936760) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 9 (Thread 0x7ffd7dd27600 (LWP 208351) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7dd26ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5a34020) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [20] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5a34020, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [20] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5a34020) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [20] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [20] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [20] #8 0x0000000000000000 in ?? () [20] [20] Thread 8 (Thread 0x7ffd7dd33600 (LWP 208348) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7dd32ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56d12e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [20] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56d12e0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [20] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56d12e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [20] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [20] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [20] #8 0x0000000000000000 in ?? () [20] [20] Thread 7 (Thread 0x7ffd7dd3f600 (LWP 208343) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7dd3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56d0200) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [20] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56d0200, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [20] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56d0200) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [20] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [20] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [20] #8 0x0000000000000000 in ?? () [20] [20] Thread 6 (Thread 0x7ffd81e3f600 (LWP 208339) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd81e3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56cfd00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [20] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56cfd00, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [20] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56cfd00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [20] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [20] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [20] #8 0x0000000000000000 in ?? () [20] [20] Thread 5 (Thread 0x7ffd5e7ff600 (LWP 208336) "s3d.x"): [20] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [20] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [20] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [20] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [20] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [20] #5 0x00007fffdf46fa28 in gasneti_defaultSignalHandler () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [20] #6 [20] #7 0x00007fffe65acdfc in __pthread_kill_implementation () from /lib64/libc.so.6 [20] #8 0x00007fffe655b842 in raise () from /lib64/libc.so.6 [20] #9 0x00007fffe65435cf in abort () from /lib64/libc.so.6 [20] #10 0x00007fffdf99737b in Realm::verify_packet_crc (arg0=arg0@entry=81, header_base=header_base@entry=0x7ffd5e7fcb00, header_size=header_size@entry=36, payload_base=payload_base@entry=0x7ffd687cc0f8, payload_size=payload_size@entry=9600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:130 [20] #11 0x00007fffdf998d60 in Realm::GASNetEXInternal::handle_medium (this=0x526d280, srcrank=0, arg0=81, hdr=0x7ffd5e7fcb00, hdr_bytes=36, data=0x7ffd687cc0f8, data_bytes=9600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:5143 [20] #12 0x00007fffe087662f in Realm::GASNetEXHandlers::handle_request_medium (token=0x7ffd687cc0c8, arg0=81, hdr=hdr@entry=0x7ffd5e7fcb00, hdr_bytes=hdr_bytes@entry=36, data=0x7ffd687cc0f8, data_bytes=9600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_handlers.cc:708 [20] #13 0x00007fffe0876a93 in Realm::GASNetEXHandlers::handle_request_medium_10 (token=, buf=, nbytes=, arg0=, arg1=, arg2=, arg3=1054, arg4=754, arg5=0, arg6=1048576, arg7=-1, arg8=0, arg9=-434291732) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_handlers.cc:842 [20] #14 0x00007fffe0886f16 in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [20] #15 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [20] #16 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [20] #17 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [20] #18 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [20] #19 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x526d300, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [20] #20 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd5e7fd5c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [20] #21 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x5925a70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [20] #22 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x56cc710) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #23 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #24 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 4 (Thread 0x7ffd62dff600 (LWP 208332) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd62dfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 Realm::BackgroundWorkThread::main_loop (this=0x5925860) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [20] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5925b90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 3 (Thread 0x7ffd71df3600 (LWP 208328) "s3d.x"): [20] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [20] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd71df2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [20] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [20] #3 Realm::BackgroundWorkThread::main_loop (this=0x59257e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [20] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5925940) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [20] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 2 (Thread 0x7fffcf5ff6c0 (LWP 208314) "s3d.x"): [20] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [20] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [20] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [20] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [20] [20] Thread 1 (Thread 0x7fffd5a20600 (LWP 208298) "s3d.x"): [20] #0 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [20] #1 0x00007fffe7879d23 in MPIR_Wait_impl () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [20] #2 0x00007fffe7892e3b in MPIR_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [20] #3 0x00007fffe789385d in PMPI_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [20] #4 0x00007fffe9336ddc in pmpi_wait__ () from /opt/cray/pe/lib64/libmpifort_gnu_123.so.12 [20] #5 0x0000000000527524 in filter_m::filter (fn=..., io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/modules/filter_m.f90:1495 [20] #6 0x00000000005e279c in integrate (io=6, update_fortran_arrays=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/f90_files/solve/integrate_erk.f90:228 [20] #7 0x00000000005da350 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:301 [20] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [20] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [20] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [20] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [20] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [20] [Inferior 1 (process 208298) detached] [26] /usr/bin/gdb -nx -batch -x /tmp/gasnet_A0kbs3 '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/pwave_x_8_ammonia/s3d.x' 208762 [26] [New LWP 208776] [26] [New LWP 208794] [26] [New LWP 208797] [26] [New LWP 208800] [26] [New LWP 208804] [26] [New LWP 208808] [26] [New LWP 208810] [26] [New LWP 208814] [26] [New LWP 208817] [26] [New LWP 208819] [26] [New LWP 208827] [26] [New LWP 208828] [26] [New LWP 208848] [26] [New LWP 208860] [26] [New LWP 208863] [26] [New LWP 208864] [26] [New LWP 208865] [26] [New LWP 208866] [26] [New LWP 208878] [26] [Thread debugging using libthread_db enabled] [26] Using host libthread_db library "/lib64/libthread_db.so.1". [26] 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [26] Id Target Id Frame [26] * 1 Thread 0x7fffd5a20600 (LWP 208762) "s3d.x" 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [26] 2 Thread 0x7fffd35ff6c0 (LWP 208776) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [26] 3 Thread 0x7ffd77905600 (LWP 208794) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [26] 4 Thread 0x7ffd717f3600 (LWP 208797) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 5 Thread 0x7ffd627ff600 (LWP 208800) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 6 Thread 0x7ffd85a0f600 (LWP 208804) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 7 Thread 0x7ffd8583f600 (LWP 208808) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 8 Thread 0x7ffd8163f600 (LWP 208810) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 9 Thread 0x7ffd7d59f600 (LWP 208814) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 10 Thread 0x7ff070b7e600 (LWP 208817) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 11 Thread 0x7ff07097a600 (LWP 208819) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 12 Thread 0x7ffd59bff600 (LWP 208827) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 13 Thread 0x7ffd5e1ff600 (LWP 208828) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 14 Thread 0x7ffd555ff600 (LWP 208848) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 15 Thread 0x7ffd50fff600 (LWP 208860) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 16 Thread 0x7ffd4c9ff600 (LWP 208863) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 17 Thread 0x7ffd483ff600 (LWP 208864) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 18 Thread 0x7ffd425ff600 (LWP 208865) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 19 Thread 0x7ffd3dfff600 (LWP 208866) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 20 Thread 0x7ffd7d1bf6c0 (LWP 208878) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [26] [26] Thread 20 (Thread 0x7ffd7d1bf6c0 (LWP 208878) "s3d.x"): [26] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [26] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #3 0x00007fffd6ca320c in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #4 0x00007fffd6c14e44 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #5 0x00007fffd6c14b5a in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #6 0x00007fffd6c07a19 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #7 0x00007fffec3d104b in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [26] #8 0x00007fffec150186 in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [26] #9 0x00007fffec37e29f in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [26] #10 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #11 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 19 (Thread 0x7ffd3dfff600 (LWP 208866) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3dffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3dffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::UnfairCondVar::wait (this=this@entry=0x583f0f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [26] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x583f0d8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [26] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc5ec7130) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 18 (Thread 0x7ffd425ff600 (LWP 208865) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd425feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd425feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::UnfairCondVar::wait (this=this@entry=0x55e6ac8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [26] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x55e6aa8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [26] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c08c110) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 17 (Thread 0x7ffd483ff600 (LWP 208864) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd483feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd483feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::UnfairCondVar::wait (this=this@entry=0x583f0f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [26] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x583f0d8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [26] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc5ec90e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 16 (Thread 0x7ffd4c9ff600 (LWP 208863) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd4c9feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd4c9feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::UnfairCondVar::wait (this=this@entry=0x583f0f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [26] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x583f0d8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [26] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc5ec2a30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 15 (Thread 0x7ffd50fff600 (LWP 208860) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd50ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd50ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::UnfairCondVar::wait (this=this@entry=0x55e6ac8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [26] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x55e6aa8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [26] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04e78d430) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 14 (Thread 0x7ffd555ff600 (LWP 208848) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd555feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd555feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::UnfairCondVar::wait (this=this@entry=0x55e6ac8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [26] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x55e6aa8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [26] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfcc0b6140) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 13 (Thread 0x7ffd5e1ff600 (LWP 208828) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5e1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5e1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::UnfairCondVar::wait (this=this@entry=0x583f0f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [26] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x583f0d8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [26] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0440013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 12 (Thread 0x7ffd59bff600 (LWP 208827) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd59bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd59bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::UnfairCondVar::wait (this=this@entry=0x55e6ac8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [26] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x55e6aa8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [26] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 11 (Thread 0x7ff07097a600 (LWP 208819) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff070979ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5a5fd80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5a5fd80, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5a5fd80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5a5fd80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [26] #7 Realm::Thread::thread_entry_wrapper (obj=0x5a5fd80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [26] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x56a8960) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 10 (Thread 0x7ff070b7e600 (LWP 208817) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff070b7dba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x57cc5c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x57cc5c0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x57cc5c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x57cc5c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [26] #7 Realm::Thread::thread_entry_wrapper (obj=0x57cc5c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [26] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x56a83b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 9 (Thread 0x7ffd7d59f600 (LWP 208814) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7d59eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56c2080) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56c2080, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56c2080) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [26] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [26] #8 0x0000000000000000 in ?? () [26] [26] Thread 8 (Thread 0x7ffd8163f600 (LWP 208810) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8163eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bd2f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bd2f0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bd2f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [26] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [26] #8 0x0000000000000000 in ?? () [26] [26] Thread 7 (Thread 0x7ffd8583f600 (LWP 208808) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8583eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bc210) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bc210, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bc210) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [26] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [26] #8 0x0000000000000000 in ?? () [26] [26] Thread 6 (Thread 0x7ffd85a0f600 (LWP 208804) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd85a0eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bbd10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bbd10, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bbd10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [26] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [26] #8 0x0000000000000000 in ?? () [26] [26] Thread 5 (Thread 0x7ffd627ff600 (LWP 208800) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd627feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::BackgroundWorkThread::main_loop (this=0x54fe600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [26] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5654ed0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 4 (Thread 0x7ffd717f3600 (LWP 208797) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd717f2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::BackgroundWorkThread::main_loop (this=0x54fe3f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [26] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x54fe720) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 3 (Thread 0x7ffd77905600 (LWP 208794) "s3d.x"): [26] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [26] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [26] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #5 0x00007fffdf46fa28 in gasneti_defaultSignalHandler () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #6 [26] #7 0x00007fffe65acdfc in __pthread_kill_implementation () from /lib64/libc.so.6 [26] #8 0x00007fffe655b842 in raise () from /lib64/libc.so.6 [26] #9 0x00007fffe65435cf in abort () from /lib64/libc.so.6 [26] #10 0x00007fffdf99737b in Realm::verify_packet_crc (arg0=arg0@entry=81, header_base=header_base@entry=0x7ffd77902b00, header_size=header_size@entry=36, payload_base=payload_base@entry=0x7ffd6802f058, payload_size=payload_size@entry=9600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:130 [26] #11 0x00007fffdf998d60 in Realm::GASNetEXInternal::handle_medium (this=0x5267bf0, srcrank=0, arg0=81, hdr=0x7ffd77902b00, hdr_bytes=36, data=0x7ffd6802f058, data_bytes=9600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:5143 [26] #12 0x00007fffe087662f in Realm::GASNetEXHandlers::handle_request_medium (token=0x7ffd6802f028, arg0=81, hdr=hdr@entry=0x7ffd77902b00, hdr_bytes=hdr_bytes@entry=36, data=0x7ffd6802f058, data_bytes=9600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_handlers.cc:708 [26] #13 0x00007fffe0876a93 in Realm::GASNetEXHandlers::handle_request_medium_10 (token=, buf=, nbytes=, arg0=, arg1=, arg2=, arg3=1054, arg4=754, arg5=0, arg6=1048576, arg7=-1, arg8=0, arg9=-434291732) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_handlers.cc:842 [26] #14 0x00007fffe0886f16 in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #15 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #16 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [26] #17 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [26] #18 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [26] #19 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x5267c70, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [26] #20 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd779035c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [26] #21 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x54fe270) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [26] #22 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x54fe4d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #23 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #24 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 208776) "s3d.x"): [26] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [26] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 1 (Thread 0x7fffd5a20600 (LWP 208762) "s3d.x"): [26] #0 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [26] #1 0x00007fffe7879d23 in MPIR_Wait_impl () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [26] #2 0x00007fffe7892e3b in MPIR_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [26] #3 0x00007fffe789385d in PMPI_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [26] #4 0x00007fffe9336ddc in pmpi_wait__ () from /opt/cray/pe/lib64/libmpifort_gnu_123.so.12 [26] #5 0x0000000000526d91 in filter_m::filter (fn=..., io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/modules/filter_m.f90:1242 [26] #6 0x00000000005e279c in integrate (io=6, update_fortran_arrays=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/f90_files/solve/integrate_erk.f90:228 [26] #7 0x00000000005da350 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:301 [26] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [26] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [26] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [26] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [26] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [26] [Inferior 1 (process 208762) detached] srun: error: frontier02431: task 0: Segmentation fault (core dumped) srun: Terminating StepId=3546971.0 slurmstepd: error: *** STEP 3546971.0 ON frontier02431 CANCELLED AT 2025-06-30T12:52:58 *** *** Caught a signal (proc 1): SIGTERM(15) *** Caught a signal (proc 2): SIGTERM(15) *** Caught a signal (proc 3): SIGTERM(15) *** Caught a signal (proc 21): SIGTERM(15) *** Caught a signal (proc 12): SIGTERM(15) *** Caught a signal (proc 22): SIGTERM(15) *** Caught a signal (proc 13): SIGTERM(15) *** Caught a signal (proc 16): SIGTERM(15) *** Caught a signal (proc 23): SIGTERM(15) *** Caught a signal (proc 14): SIGTERM(15) *** Caught a signal (proc 17): SIGTERM(15) *** Caught a signal (proc 15): SIGTERM(15) *** Caught a signal (proc 18): SIGTERM(15) *** Caught a signal (proc 19): SIGTERM(15) *** Caught a signal (proc 8): SIGTERM(15) *** Caught a signal (proc 9): SIGTERM(15) *** Caught a signal (proc 10): SIGTERM(15) *** Caught a signal (proc 11): SIGTERM(15) *** Caught a signal (proc 25): SIGTERM(15) *** Caught a signal (proc 27): SIGTERM(15) *** Caught a signal (proc 4): SIGTERM(15) *** Caught a signal (proc 5): SIGTERM(15) *** Caught a signal (proc 6): SIGTERM(15) *** Caught a signal (proc 7): SIGTERM(15) *** Caught a signal (proc 28): SIGTERM(15) *** Caught a signal (proc 29): SIGTERM(15) *** FATAL ERROR (proc 31): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 31): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 31): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** Caught a signal (proc 31): SIGTERM(15) [31] Invoking GDB for backtrace... *** Caught a signal (proc 30): SIGTERM(15) *** FATAL ERROR (proc 30): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 30): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 30): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu [30] Invoking GDB for backtrace... [30] /usr/bin/gdb -nx -batch -x /tmp/gasnet_jCz4BU '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/pwave_x_8_ammonia/s3d.x' 2155016 [30] [New LWP 2155035] [30] [New LWP 2155047] [30] [New LWP 2155051] [30] [New LWP 2155055] [30] [New LWP 2155066] [30] [New LWP 2155070] [30] [New LWP 2155074] [30] [New LWP 2155080] [30] [New LWP 2155082] [30] [New LWP 2155085] [30] [New LWP 2155093] [30] [New LWP 2155094] [30] [New LWP 2155103] [30] [New LWP 2155116] [30] [New LWP 2155121] [30] [New LWP 2155122] [30] [New LWP 2155123] [30] [New LWP 2155124] [30] [New LWP 2155135] [30] [Thread debugging using libthread_db enabled] [30] Using host libthread_db library "/lib64/libthread_db.so.1". [30] 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [30] Id Target Id Frame [30] * 1 Thread 0x7fffd5a20600 (LWP 2155016) "s3d.x" 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [30] 2 Thread 0x7fffd35ff6c0 (LWP 2155035) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [30] 3 Thread 0x7ffd77905600 (LWP 2155047) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [30] 4 Thread 0x7ffd717f3600 (LWP 2155051) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 5 Thread 0x7ffd627ff600 (LWP 2155055) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 6 Thread 0x7ffd85a0f600 (LWP 2155066) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 7 Thread 0x7ffd8583f600 (LWP 2155070) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 8 Thread 0x7ffd8163f600 (LWP 2155074) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 9 Thread 0x7ffd7d59f600 (LWP 2155080) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 10 Thread 0x7ff070b7e600 (LWP 2155082) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 11 Thread 0x7ff07097a600 (LWP 2155085) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 12 Thread 0x7ffd5e1ff600 (LWP 2155093) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 13 Thread 0x7ffd59bff600 (LWP 2155094) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 14 Thread 0x7ffd555ff600 (LWP 2155103) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 15 Thread 0x7ffd50fff600 (LWP 2155116) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 16 Thread 0x7ffd4c9ff600 (LWP 2155121) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 17 Thread 0x7ffd483ff600 (LWP 2155122) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 18 Thread 0x7ffd425ff600 (LWP 2155123) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 19 Thread 0x7ffd3dfff600 (LWP 2155124) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 20 Thread 0x7ffd7d1bf6c0 (LWP 2155135) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [30] [30] Thread 20 (Thread 0x7ffd7d1bf6c0 (LWP 2155135) "s3d.x"): [30] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [30] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #3 0x00007fffd6ca320c in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #4 0x00007fffd6c14e44 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #5 0x00007fffd6c14b5a in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #6 0x00007fffd6c07a19 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #7 0x00007fffec3d104b in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [30] #8 0x00007fffec150186 in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [30] #9 0x00007fffec37e29f in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [30] #10 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #11 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 19 (Thread 0x7ffd3dfff600 (LWP 2155124) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3dffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3dffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57c25b8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57c2598) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc40cd850) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 18 (Thread 0x7ffd425ff600 (LWP 2155123) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd425feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd425feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5a4cc28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5a4cc08) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfbc0c4c50) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 17 (Thread 0x7ffd483ff600 (LWP 2155122) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd483feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd483feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57c25b8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57c2598) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc40cc360) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 16 (Thread 0x7ffd4c9ff600 (LWP 2155121) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd4c9feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd4c9feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57c25b8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57c2598) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc40c55d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 15 (Thread 0x7ffd50fff600 (LWP 2155116) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd50ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd50ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5a4cc28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5a4cc08) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04005c550) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 14 (Thread 0x7ffd555ff600 (LWP 2155103) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd555feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd555feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5a4cc28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5a4cc08) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc040048ff0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 13 (Thread 0x7ffd59bff600 (LWP 2155094) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd59bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd59bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5a4cc28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5a4cc08) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0400013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 12 (Thread 0x7ffd5e1ff600 (LWP 2155093) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5e1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5e1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57c25b8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57c2598) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0480013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 11 (Thread 0x7ff07097a600 (LWP 2155085) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff070979ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5543dd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5543dd0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5543dd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5543dd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [30] #7 Realm::Thread::thread_entry_wrapper (obj=0x5543dd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [30] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5b50f90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 10 (Thread 0x7ff070b7e600 (LWP 2155082) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff070b7dba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5c790a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5c790a0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5c790a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5c790a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [30] #7 Realm::Thread::thread_entry_wrapper (obj=0x5c790a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [30] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x80ec9e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 9 (Thread 0x7ffd7d59f600 (LWP 2155080) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7d59eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x59f9720) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x59f9720, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x59f9720) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #8 0x0000000000000000 in ?? () [30] [30] Thread 8 (Thread 0x7ffd8163f600 (LWP 2155074) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8163eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bd2b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bd2b0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bd2b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #8 0x0000000000000000 in ?? () [30] [30] Thread 7 (Thread 0x7ffd8583f600 (LWP 2155070) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8583eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bc1d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bc1d0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bc1d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #8 0x0000000000000000 in ?? () [30] [30] Thread 6 (Thread 0x7ffd85a0f600 (LWP 2155066) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd85a0eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bbcd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bbcd0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bbcd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #8 0x0000000000000000 in ?? () [30] [30] Thread 5 (Thread 0x7ffd627ff600 (LWP 2155055) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd627feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::BackgroundWorkThread::main_loop (this=0x57a8950) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [30] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x57a7d80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 4 (Thread 0x7ffd717f3600 (LWP 2155051) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd717f2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::BackgroundWorkThread::main_loop (this=0x580c030) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [30] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x57a8a70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 3 (Thread 0x7ffd77905600 (LWP 2155047) "s3d.x"): [30] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [30] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [30] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #5 0x00007fffdf464ff7 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #6 0x00007fffdf4650e8 in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #7 0x00007fffe08861e2 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #8 0x00007fffe088631c in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #9 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #10 0x00007fffe087c52f in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #11 0x00007fffe087c92e in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #12 0x00007fffe089665c in gasneti_AMPSHMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #13 0x00007fffe087c04b in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #14 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [30] #15 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [30] #16 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [30] #17 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x5267c50, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [30] #18 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd779035c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [30] #19 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x57a7d10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [30] #20 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x57a8820) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #21 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #22 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 2155035) "s3d.x"): [30] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [30] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 1 (Thread 0x7fffd5a20600 (LWP 2155016) "s3d.x"): [30] #0 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [30] #1 0x00007fffe7879d23 in MPIR_Wait_impl () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [30] #2 0x00007fffe7892e3b in MPIR_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [30] #3 0x00007fffe789385d in PMPI_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [30] #4 0x00007fffe9336ddc in pmpi_wait__ () from /opt/cray/pe/lib64/libmpifort_gnu_123.so.12 [30] #5 0x0000000000526d91 in filter_m::filter (fn=..., io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/modules/filter_m.f90:1242 [30] #6 0x00000000005e279c in integrate (io=6, update_fortran_arrays=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/f90_files/solve/integrate_erk.f90:228 [30] #7 0x00000000005da350 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:301 [30] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [30] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [30] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [30] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [30] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [30] [Inferior 1 (process 2155016) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit [31] /usr/bin/gdb -nx -batch -x /tmp/gasnet_gzI0ee '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/pwave_x_8_ammonia/s3d.x' 2155017 [31] [New LWP 2155031] [31] [New LWP 2155048] [31] [New LWP 2155052] [31] [New LWP 2155056] [31] [New LWP 2155065] [31] [New LWP 2155069] [31] [New LWP 2155073] [31] [New LWP 2155078] [31] [New LWP 2155089] [31] [New LWP 2155090] [31] [New LWP 2155104] [31] [New LWP 2155105] [31] [New LWP 2155108] [31] [New LWP 2155114] [31] [New LWP 2155118] [31] [New LWP 2155119] [31] [New LWP 2155120] [31] [New LWP 2155126] [31] [New LWP 2155132] [31] [Thread debugging using libthread_db enabled] [31] Using host libthread_db library "/lib64/libthread_db.so.1". [31] 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [31] Id Target Id Frame [31] * 1 Thread 0x7fffd5a20600 (LWP 2155017) "s3d.x" 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [31] 2 Thread 0x7fffd35ff6c0 (LWP 2155031) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [31] 3 Thread 0x7ffd77905600 (LWP 2155048) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 4 Thread 0x7ffd717f3600 (LWP 2155052) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [31] 5 Thread 0x7ffd627ff600 (LWP 2155056) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 6 Thread 0x7ffd85a0f600 (LWP 2155065) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 7 Thread 0x7ffd8583f600 (LWP 2155069) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 8 Thread 0x7ffd8163f600 (LWP 2155073) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 9 Thread 0x7ffd7d59f600 (LWP 2155078) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 10 Thread 0x7ffd5e1ff600 (LWP 2155089) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 11 Thread 0x7ffd59bff600 (LWP 2155090) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 12 Thread 0x7fc0689ff600 (LWP 2155104) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 13 Thread 0x7fc0683ff600 (LWP 2155105) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 14 Thread 0x7ffd555ff600 (LWP 2155108) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 15 Thread 0x7ffd50fff600 (LWP 2155114) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 16 Thread 0x7ffd4c9ff600 (LWP 2155118) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 17 Thread 0x7ffd483ff600 (LWP 2155119) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 18 Thread 0x7ffd425ff600 (LWP 2155120) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 19 Thread 0x7ffd3dfff600 (LWP 2155126) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] 20 Thread 0x7ffd7d1bf6c0 (LWP 2155132) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [31] [31] Thread 20 (Thread 0x7ffd7d1bf6c0 (LWP 2155132) "s3d.x"): [31] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [31] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #3 0x00007fffd6ca320c in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #4 0x00007fffd6c14e44 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #5 0x00007fffd6c14b5a in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #6 0x00007fffd6c07a19 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #7 0x00007fffec3d104b in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [31] #8 0x00007fffec150186 in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [31] #9 0x00007fffec37e29f in ?? () from /opt/rocm-6.0.0/lib/libamdhip64.so.6 [31] #10 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #11 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 19 (Thread 0x7ffd3dfff600 (LWP 2155126) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd3dffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd3dffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5881a08) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58819e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc410e010) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 18 (Thread 0x7ffd425ff600 (LWP 2155120) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd425feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd425feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x577c3c8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x577c3a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc9ebff40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 17 (Thread 0x7ffd483ff600 (LWP 2155119) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd483feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd483feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x577c3c8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x577c3a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc9eb0d00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 16 (Thread 0x7ffd4c9ff600 (LWP 2155118) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd4c9feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd4c9feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5881a08) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58819e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc4105d60) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 15 (Thread 0x7ffd50fff600 (LWP 2155114) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd50ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd50ffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x577c3c8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x577c3a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc9eb1310) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 14 (Thread 0x7ffd555ff600 (LWP 2155108) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd555feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd555feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5881a08) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58819e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fbfc40b3a20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 13 (Thread 0x7fc0683ff600 (LWP 2155105) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc0683feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5c7a6a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5c7a6a0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5c7a6a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5c7a6a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [31] #7 Realm::Thread::thread_entry_wrapper (obj=0x5c7a6a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [31] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04805a750) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 12 (Thread 0x7fc0689ff600 (LWP 2155104) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc0689feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5be5fc0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5be5fc0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5be5fc0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5be5fc0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [31] #7 Realm::Thread::thread_entry_wrapper (obj=0x5be5fc0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [31] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0400646c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 11 (Thread 0x7ffd59bff600 (LWP 2155090) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd59bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd59bfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x577c3c8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x577c3a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0480013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 10 (Thread 0x7ffd5e1ff600 (LWP 2155089) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5e1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5e1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5881a08) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [31] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x58819e8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [31] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0400013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 9 (Thread 0x7ffd7d59f600 (LWP 2155078) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7d59eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56111f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56111f0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56111f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [31] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [31] #8 0x0000000000000000 in ?? () [31] [31] Thread 8 (Thread 0x7ffd8163f600 (LWP 2155073) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8163eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bd1a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bd1a0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bd1a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [31] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [31] #8 0x0000000000000000 in ?? () [31] [31] Thread 7 (Thread 0x7ffd8583f600 (LWP 2155069) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8583eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bbbc0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bbbc0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bbbc0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [31] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [31] #8 0x0000000000000000 in ?? () [31] [31] Thread 6 (Thread 0x7ffd85a0f600 (LWP 2155065) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd85a0eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bb8d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [31] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bb8d0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [31] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bb8d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [31] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [31] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [31] #8 0x0000000000000000 in ?? () [31] [31] Thread 5 (Thread 0x7ffd627ff600 (LWP 2155056) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd627feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::BackgroundWorkThread::main_loop (this=0x591c6e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [31] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x591ca50) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 4 (Thread 0x7ffd717f3600 (LWP 2155052) "s3d.x"): [31] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [31] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [31] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #5 0x00007fffdf464ff7 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #6 0x00007fffdf4650e8 in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #7 0x00007fffe08861e2 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #8 0x00007fffe088631c in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #9 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #10 0x00007fffe087c52f in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #11 0x00007fffe087c92e in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #12 0x00007fffe089665c in gasneti_AMPSHMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #13 0x00007fffe087c04b in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [31] #14 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [31] #15 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [31] #16 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [31] #17 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x5267c10, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [31] #18 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd717f15c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [31] #19 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x591c560) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [31] #20 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x591c800) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #21 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #22 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 3 (Thread 0x7ffd77905600 (LWP 2155048) "s3d.x"): [31] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [31] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd77904ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [31] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [31] #3 Realm::BackgroundWorkThread::main_loop (this=0x591c930) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [31] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x591bff0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [31] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 2155031) "s3d.x"): [31] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [31] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [31] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [31] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [31] [31] Thread 1 (Thread 0x7fffd5a20600 (LWP 2155017) "s3d.x"): [31] #0 0x00007fffe6618c97 in sched_yield () from /lib64/libc.so.6 [31] #1 0x00007fffe7879d23 in MPIR_Wait_impl () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [31] #2 0x00007fffe7892e3b in MPIR_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [31] #3 0x00007fffe789385d in PMPI_Wait () from /opt/cray/pe/lib64/libmpi_gnu_123.so.12 [31] #4 0x00007fffe9336ddc in pmpi_wait__ () from /opt/cray/pe/lib64/libmpifort_gnu_123.so.12 [31] #5 0x0000000000526d91 in filter_m::filter (fn=..., io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/modules/filter_m.f90:1242 [31] #6 0x00000000005e279c in integrate (io=6, update_fortran_arrays=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/f90_files/solve/integrate_erk.f90:228 [31] #7 0x00000000005da350 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:301 [31] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [31] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [31] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [31] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [31] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [31] [Inferior 1 (process 2155017) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit srun: error: frontier04860: tasks 30-31: Exited with exit code 255 srun: error: frontier04094: task 20: Aborted (core dumped) aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit srun: error: frontier02431: tasks 2-3: Exited with exit code 255 srun: error: frontier04860: task 28: Exited with exit code 255 srun: error: frontier04094: tasks 21,23: Exited with exit code 255 srun: error: frontier04221: tasks 25,27: Exited with exit code 255 srun: error: frontier03965: tasks 16,19: Exited with exit code 255 srun: error: frontier03837: tasks 12,14-15: Exited with exit code 255 srun: error: frontier03709: tasks 8,10-11: Exited with exit code 255 srun: error: frontier03579: tasks 4-5,7: Exited with exit code 255 srun: error: frontier03965: task 18: Exited with exit code 255 srun: error: frontier04860: task 29: Exited with exit code 255 srun: error: frontier04094: task 22: Exited with exit code 255 srun: error: frontier03965: task 17: Exited with exit code 255 srun: error: frontier03579: task 6: Exited with exit code 255 srun: error: frontier03709: task 9: Exited with exit code 255 srun: error: frontier03837: task 13: Exited with exit code 255 gasnet_exit(): WARNING: timeout during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit srun: error: frontier02431: task 1: Exited with exit code 255 srun: error: frontier04221: task 26: Aborted (core dumped) srun: error: frontier04221: task 24: Aborted (core dumped)