============================================================================== Hello, and welcome to S3D! ============================================================================== ============================================================== --------------------S3D BUILD CONFIGURATION----------------- Build directory: /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/fortran/ammonia ------------------------------------------------------ S3D Options: Using mixture averaged transport: ON Using ARK integration: OFF Using chemical mechanism: ammonia Using srcs for timescale analysis: OFF ------------------------------------------------------ Defines: SYSTEMCALLWONTWORK SAVEFILEINSEPDIR ARCH_OPTERON MPIIO BLOCKIO MIXAVG FROZENFEED ------------------------------------------------------ FORTRAN COMPILER FLAGS: -cpp -fdefault-double-8 -fdefault-real-8 -fno-range-check -ffree-line-length-none -fallow-argument-mismatch C COMPILER FLAGS: ------------------------------------------------------ S3D Version: ------------------------------------------------------ Chemistry routine header: c**************************************************************************** c ... DO NOT EDIT. AUTOMATICALLY GENERATED FILE. c**************************************************************************** c ... c ... CKWYP-compatible GETRATES: return molar production rates for all c ... species given pressure (P), temperature (T), and mass fractions (Y). c ... c ... Generated by cgetrates 1.0b c ... c**************************************************************************** subroutine getrates(P,T,Y,ICKWRK,RCKWRK,WSPL) ------------------------------------------------------ Transport routine header: !Standard version of tranlib.f. 18Feb 2009 ! (Changes by Mark Fahey, ORNL) ! (Update by Evatt Hawkes, SNL) SUBROUTINE MCABS ------------------------------------------------------ ============================================================== ------------------------------------------------------------------------------ this run started at: date = 06/30/2025 time = 14:25:46 ------------------------------------------------------------------------------ initializing chemkin module... opening chemkin link file... setting chemkin work array lengths... allocating chemkin work arrays... filling chemkin work arrays based on chemkin link file... CKLIB: CHEMKIN-III GAS-PHASE CHEMICAL KINETICS LIBRARY, DOUBLE PRECISION Vers. 5.36 2000/12/15 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 the following was extracted from the ../input/chem.asc file: element names are as follows: index name 1 H 2 O 3 N species names are as follows: index name 1 H2 2 NH3 3 NO 4 N2O 5 O2 6 H 7 O 8 OH 9 HO2 10 H2O 11 H2O2 12 NO2 13 HNO 14 N 15 NNH 16 NH2 17 NH 18 H2NO 19 N2 species molecular weights are as follows: index name value (kg/mol) 1 H2 2.01594E-03 2 NH3 1.70306E-02 3 NO 3.00061E-02 4 N2O 4.40128E-02 5 O2 3.19988E-02 6 H 1.00797E-03 7 O 1.59994E-02 8 OH 1.70074E-02 9 HO2 3.30068E-02 10 H2O 1.80153E-02 11 H2O2 3.40147E-02 12 NO2 4.60055E-02 13 HNO 3.10141E-02 14 N 1.40067E-02 15 NNH 2.90214E-02 16 NH2 1.60226E-02 17 NH 1.50147E-02 18 H2NO 3.20220E-02 19 N2 2.80134E-02 ------------------------------------------------------------------------------ Initialized ADIOS; 0 initializing param module... number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 number of reaction third-body reactions = 11 ------------------------------------------------------------------------------ initializing topology module... Initialized topology module, npx = 4 npy = 2 npz = 4 npes = 32 ------------------------------------------------------------------------------ initializing reference module... the various reference values are as follows: universal gas constant (J/mol-K) = 8.31400E+00 freestream temperature (K) = 8.01900E+02 reference ratio of specifice heats = 1.36400E+00 reference speed of sound (m/s) = 5.28000E+02 reference density (kg/m^3) = 1.22300E+01 reference conductivity (W/m-K) = 5.76900E-02 reference temperature (K) = 2.91892E+02 reference pressure (Pa) = 3.40953E+06 standard atmospheric pressure (Pa) = 1.01325E+05 reference time (s) = 1.25428E-06 reference specific heat (J/kg-K) = 9.55094E+02 reference length (m) = 6.62258E-04 reference viscosity (Pa-s) = 4.27649E+00 acoustic Reynolds number = 1.00000E+05 Mach number = 1.00000E-03 convective Reynolds number = 1.00000E+02 ------------------------------------------------------------------------------ initializing ERK module... ERK module information is as follows: rk_method = -64 cont_switch = 0 tstep_init (seconds) = 1.00E-09 tstep_min (seconds) = 1.00E-09 tstep_max (seconds) = 1.00E-09 rk_rtol = 1.00E-03 rk_atol = 1.00E-15 cont_safety = 9.00E-01 k_I = 2.00E-01 k_P = 2.00E-01 k_D = 5.00E-02 k_D2 = 0.00E+00 cont_factor = 3.00E+00 cfl_switch = 1 i_time_cont = 20 cont_n_reg = 1 controller type = NONE rk_tol = 1.00E-12 using RK4(3)6[2R+]C ERK scheme! ------------------------------------------------------------------------------ initializing transport module... opening mixavg.in ... Lagging is switched on WARNING - with coefficient lagging, theoretical temporal convergence order may not be observed opening transport link file... setting lengths of transport work arrays... allocating transport work arrays... creating transport work arrays from transport link file... TRANLIB: CHEMKIN-III MULTICOMPONENT TRANSPORT LIBRARY, DOUBLE PRECISION Vers. 4.6 1999/06/03 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. ------------------------------------------------------------------------------ initializing filter module... using explicit filter of order 10 filter will be applied every 10 time steps ------------------------------------------------------------------------------ initializing derivative module... ------------------------------------------------------------------------------ initializing thermchem module... ------------------------------------------------------------------------------ initializing grid module... ------------------------------------------------------------------------------ initializing boundary condition module... ------------------------------------------------------------------------------ *** WARNING: ofi-conduit failed to configure FI_CXI_* environment variables due to prior conflicting settings. This may lead to unstable behavior and/or degraded performance. If you did not intentionally set FI_CXI_RX_MATCH_MODE='software', FI_CXI_RDZV_THRESHOLD='16384', or FI_CXI_RDZV_GET_MIN then this condition may have resulted from initializing MPI prior to initialization of GASNet. For more information on that scenario, please see "Limits to MPI interoperability" in the ofi-conduit README. Legion Init: Mechanism = 'ammonia' Local Grid Size = ( 64, 64, 256) Global Grid Size = (256, 128, 1024) Proc Grid Size = ( 4, 2, 4) Proc ID = ( 0, 0, 0) nvar_tot = 23, n_spec = 19, n_reg = 3, n_scalar = 0 iorder = 8, iforder = 10, vary = XYZ fuse fields set to 200 from environment set mapper mode to 'allgpu' from environment RHST_ARGS: -ll:gpu 2 -ll:cpu 1 -ll:util 3 -ll:bgwork 3 -ll:fsize 16384 -ll:csize 4096 -ll:zsize 1024 -ll:stacksize 16 -logfile run_%.log -hip:hostreg 0 -lg:no_transitive_reduction 1 -lg:prof 32 -lg:prof_logfile prof_%.gz REGULAR STENCIL initializing variables... Host: frontier05621 Rank: 0 PID: 350391 Time: 14:26:05 STARTING MAIN TASK! initializing pressure wave test... initializing species 5 as 0.233 initializing species 19 as 0.767 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ continuing with initialization... ------------------------------------------------------------------------------ storing hard inflow conditions... maximum speed of sound = 347.72 m/s mimimum speed of sound = 347.72 m/s maximum acoustic time = 4.98E-05 sec minimum acoustic time = 4.98E-05 sec initialization completed for time = 0.00E+00 ------------------------------------------------------------------------------ writing save files for: i_time = 0, time = 0.0000E+00 (sec) precision is d writing adios files for: i_time = 0, time = 0.0000E+00 (sec) precision is d Writing data with adios to ../data/pressure_wave_test.0.0000E+00.field.bp Adios ierr_adios: 0 ------------------------------------------------------------------------------ IO Started at 26:05.786 IO Finished at 26:06.661 Per-thread IO = 201.3MB Total IO = 6442.5MB Time taken = 0.875s Total IO Rate = 7362.801MB/s ------------------------------------------------------------------------------ *** Caught a fatal signal (proc 30): SIGSEGV(11) *** NOTICE (proc 30): We recommend linking the debug version of GASNet to assist you in resolving this application issue. [30] Invoking GDB for backtrace... [30] /usr/bin/gdb -nx -batch -x /tmp/gasnet_hAIh4a '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/pwave_x_8_ammonia/s3d.x' 369494 [30] [New LWP 369504] [30] [New LWP 369523] [30] [New LWP 369524] [30] [New LWP 369525] [30] [New LWP 369529] [30] [New LWP 369533] [30] [New LWP 369536] [30] [New LWP 369541] [30] [New LWP 369547] [30] [New LWP 369549] [30] [New LWP 369550] [30] [New LWP 369551] [30] [Thread debugging using libthread_db enabled] [30] Using host libthread_db library "/lib64/libthread_db.so.1". [30] 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [30] Id Target Id Frame [30] * 1 Thread 0x7fffd5a20600 (LWP 369494) "s3d.x" 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [30] 2 Thread 0x7fffd35ff6c0 (LWP 369504) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [30] 3 Thread 0x7ffd77905600 (LWP 369523) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 4 Thread 0x7ffd717f3600 (LWP 369524) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 5 Thread 0x7ffd615ff600 (LWP 369525) "s3d.x" Realm::GASNetEXPoller::do_work (this=0x5267e40, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:186 [30] 6 Thread 0x7ffd85a0f600 (LWP 369529) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [30] 7 Thread 0x7ffd8583f600 (LWP 369533) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 8 Thread 0x7ffd8163f600 (LWP 369536) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 9 Thread 0x7ffd7d59f600 (LWP 369541) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 10 Thread 0x7ff0745fd600 (LWP 369547) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 11 Thread 0x7ff0743f9600 (LWP 369549) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 12 Thread 0x7ffd5cfff600 (LWP 369550) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] 13 Thread 0x7ffd589ff600 (LWP 369551) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] [30] Thread 13 (Thread 0x7ffd589ff600 (LWP 369551) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd589feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9be398 in Realm::Doorbell::wait (this=0x7ffd589feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b451a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf96068b in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b45188) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x7fc0400013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 12 (Thread 0x7ffd5cfff600 (LWP 369550) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9be398 in Realm::Doorbell::wait (this=0x7ffd5cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5a46358) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [30] #4 0x00007fffdf96068b in Realm::Hip::ContextSynchronizer::thread_main (this=0x5a46338) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [30] #5 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x7fc0480013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 11 (Thread 0x7ff0743f9600 (LWP 369549) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ff0743f8ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5630cd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5630cd0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5630cd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf941ac2 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5630cd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [30] #7 Realm::Thread::thread_entry_wrapper (obj=0x5630cd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [30] #8 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x5b6ada0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 10 (Thread 0x7ff0745fd600 (LWP 369547) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ff0745fcba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x59ffb00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x59ffb00, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x59ffb00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf941ac2 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x59ffb00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [30] #7 Realm::Thread::thread_entry_wrapper (obj=0x59ffb00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [30] #8 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x80a57d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 9 (Thread 0x7ffd7d59f600 (LWP 369541) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd7d59eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5a208f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5a208f0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5a208f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #8 0x0000000000000000 in ?? () [30] [30] Thread 8 (Thread 0x7ffd8163f600 (LWP 369536) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd8163eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bd630) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bd630, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bd630) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #8 0x0000000000000000 in ?? () [30] [30] Thread 7 (Thread 0x7ffd8583f600 (LWP 369533) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd8583eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bc550) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [30] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bc550, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [30] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bc550) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [30] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #8 0x0000000000000000 in ?? () [30] [30] Thread 6 (Thread 0x7ffd85a0f600 (LWP 369529) "s3d.x"): [30] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [30] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [30] #2 0x00007fffe088c9d4 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #3 0x00007fffe088d027 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #4 0x00007fffe0890a2e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #5 0x00007fffdf46f9fa in gasneti_defaultSignalHandler () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [30] #6 [30] #7 0x00007fffe185550b in Legion::Internal::LegionProfInstance::record_mapper_call (this=0x0, mapper=0, mapper_proc=..., kind=Legion::Internal::SLICE_TASK_CALL, uid=222, start=6302787634, stop=6306065815) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_profiling.cc:1457 [30] #8 0x00007fffe19ceab6 in Legion::Internal::SerializingManager::finish_mapper_call (this=0x58d30a0, info=0x7ff06d57c880) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/mapper_manager.cc:1156 [30] #9 0x00007fffe19cbde4 in Legion::Internal::MappingCallInfo::~MappingCallInfo (this=0x7ff06d57c880, __in_chrg=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/mapper_manager.cc:58 [30] #10 0x00007fffe19cc01c in Legion::Internal::MapperManager::invoke_slice_task (this=0x58d30a0, task=task@entry=0x7fc0567c0970, input=..., output=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/mapper_manager.cc:269 [30] #11 0x00007fffe19220e2 in Legion::Internal::MultiTask::slice_index_space (this=0x7fc0567c0970) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_tasks.cc:5837 [30] #12 0x00007fffe1aeb1d0 in Legion::Internal::Runtime::legion_runtime_task (args=0x7fc063069c50, arglen=12, userdata=, userlen=, p=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:36268 [30] #13 0x00007fffdf93fc3b in Realm::Task::execute_on_processor (this=0x7fc063069ad0, p=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:330 [30] #14 0x00007fffdf93fd36 in Realm::UserThreadTaskScheduler::execute_task (this=, task=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1690 [30] #15 0x00007fffdf93e11f in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bc050) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1161 [30] #16 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [30] #17 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [30] #18 0x0000000000000000 in ?? () [30] [30] Thread 5 (Thread 0x7ffd615ff600 (LWP 369525) "s3d.x"): [30] #0 Realm::GASNetEXPoller::do_work (this=0x5267e40, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:186 [30] #1 0x00007fffdf8209d9 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd615fd5c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [30] #2 0x00007fffdf821271 in Realm::BackgroundWorkThread::main_loop (this=0x5b79fe0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [30] #3 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x5fd6380) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #4 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #5 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 4 (Thread 0x7ffd717f3600 (LWP 369524) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd717f2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf821bc5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::BackgroundWorkThread::main_loop (this=0x5b79e80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [30] #4 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x5b79eb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 3 (Thread 0x7ffd77905600 (LWP 369523) "s3d.x"): [30] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [30] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd77904ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [30] #2 0x00007fffdf821bc5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [30] #3 Realm::BackgroundWorkThread::main_loop (this=0x5887550) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [30] #4 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x5b7a100) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [30] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 369504) "s3d.x"): [30] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [30] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [30] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [30] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [30] [30] Thread 1 (Thread 0x7fffd5a20600 (LWP 369494) "s3d.x"): [30] #0 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [30] #1 0x00007fffe65aa548 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 [30] #2 0x00007fffdf83d71d in Realm::BarrierImpl::external_wait (this=0x80a3530, gen_needed=2, poisoned=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/barrier_impl.cc:1281 [30] #3 0x00007fffdf82ce75 in Realm::Event::external_wait_faultaware (this=0x40cd270, poisoned=@0x7fffffff3920: false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/event_impl.cc:282 [30] #4 0x00007fffe174cf75 in Legion::Internal::LgEvent::wait_faultaware (this=0x40cd270, poisoned=@0x7fffffff3920: false, from_app=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3459 [30] #5 0x00007fffe1ad0dfa in Legion::Internal::ApEvent::wait_faultignorant (this=0x40cd270) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:2966 [30] #6 Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion (this=0x40cd260) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:7112 [30] #7 0x00000000005da171 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:229 [30] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [30] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [30] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [30] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [30] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [30] [Inferior 1 (process 369494) detached] srun: error: frontier06515: task 30: Segmentation fault (core dumped) srun: Terminating StepId=3547118.0 slurmstepd: error: *** STEP 3547118.0 ON frontier05621 CANCELLED AT 2025-06-30T14:27:04 *** *** Caught a signal (proc 16): SIGTERM(15) *** Caught a signal (proc 28): SIGTERM(15) *** Caught a signal (proc 17): SIGTERM(15) *** Caught a signal (proc 8): SIGTERM(15) *** Caught a signal (proc 4): SIGTERM(15) *** Caught a signal (proc 12): SIGTERM(15) *** Caught a signal (proc 29): SIGTERM(15) *** Caught a signal (proc 24): SIGTERM(15) *** Caught a signal (proc 18): SIGTERM(15) *** Caught a signal (proc 9): SIGTERM(15) *** Caught a signal (proc 5): SIGTERM(15) *** Caught a signal (proc 13): SIGTERM(15) *** Caught a signal (proc 31): SIGTERM(15) *** Caught a signal (proc 25): SIGTERM(15) *** Caught a signal (proc 19): SIGTERM(15) *** Caught a signal (proc 10): SIGTERM(15) *** Caught a signal (proc 6): SIGTERM(15) *** Caught a signal (proc 14): SIGTERM(15) *** FATAL ERROR (proc 14): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 14): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 14): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu [14] Invoking GDB for backtrace... *** Caught a signal (proc 27): SIGTERM(15) *** Caught a signal (proc 11): SIGTERM(15) *** Caught a signal (proc 7): SIGTERM(15) *** Caught a signal (proc 15): SIGTERM(15) *** Caught a signal (proc 26): SIGTERM(15) *** FATAL ERROR (proc 26): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 26): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 26): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu [26] Invoking GDB for backtrace... *** Caught a signal (proc 0): SIGTERM(15) *** Caught a signal (proc 1): SIGTERM(15) *** Caught a signal (proc 2): SIGTERM(15) *** Caught a signal (proc 20): SIGTERM(15) *** Caught a signal (proc 3): SIGTERM(15) *** Caught a signal (proc 21): SIGTERM(15) *** FATAL ERROR (proc 22): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** Caught a signal (proc 22): SIGTERM(15) *** NOTICE (proc 22): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 22): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu [22] Invoking GDB for backtrace... *** Caught a signal (proc 23): SIGTERM(15) [14] /usr/bin/gdb -nx -batch -x /tmp/gasnet_3JTUC4 '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/pwave_x_8_ammonia/s3d.x' 499944 [14] [New LWP 499958] [14] [New LWP 499972] [14] [New LWP 499974] [14] [New LWP 499976] [14] [New LWP 499978] [14] [New LWP 499982] [14] [New LWP 499986] [14] [New LWP 499990] [14] [New LWP 499995] [14] [New LWP 499999] [14] [New LWP 500001] [14] [New LWP 500002] [14] [Thread debugging using libthread_db enabled] [14] Using host libthread_db library "/lib64/libthread_db.so.1". [14] 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [14] Id Target Id Frame [14] * 1 Thread 0x7fffd5a20600 (LWP 499944) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [14] 2 Thread 0x7fffd35ff6c0 (LWP 499958) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [14] 3 Thread 0x7ffd77905600 (LWP 499972) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] 4 Thread 0x7ffd717f3600 (LWP 499974) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] 5 Thread 0x7ffd615ff600 (LWP 499976) "s3d.x" 0x00007fffe65a7ecc in __lll_lock_wait () from /lib64/libc.so.6 [14] 6 Thread 0x7ffd85a0f600 (LWP 499978) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] 7 Thread 0x7ffd8583f600 (LWP 499982) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] 8 Thread 0x7ffd8163f600 (LWP 499986) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] 9 Thread 0x7ffd7d59f600 (LWP 499990) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] 10 Thread 0x7ff0745fd600 (LWP 499995) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] 11 Thread 0x7ff0743f9600 (LWP 499999) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] 12 Thread 0x7ffd5cfff600 (LWP 500001) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] 13 Thread 0x7ffd589ff600 (LWP 500002) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] [14] Thread 13 (Thread 0x7ffd589ff600 (LWP 500002) "s3d.x"): [14] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd589feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [14] #2 0x00007fffdf9be398 in Realm::Doorbell::wait (this=0x7ffd589feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [14] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5b451a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [14] #4 0x00007fffdf96068b in Realm::Hip::ContextSynchronizer::thread_main (this=0x5b45188) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [14] #5 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x7fc0440013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [14] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [14] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [14] [14] Thread 12 (Thread 0x7ffd5cfff600 (LWP 500001) "s3d.x"): [14] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [14] #2 0x00007fffdf9be398 in Realm::Doorbell::wait (this=0x7ffd5cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [14] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5a46358) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [14] #4 0x00007fffdf96068b in Realm::Hip::ContextSynchronizer::thread_main (this=0x5a46338) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [14] #5 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x7fc04c0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [14] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [14] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [14] [14] Thread 11 (Thread 0x7ff0743f9600 (LWP 499999) "s3d.x"): [14] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ff0743f8ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [14] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [14] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5630cd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [14] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5630cd0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [14] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5630cd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [14] #6 0x00007fffdf941ac2 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5630cd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [14] #7 Realm::Thread::thread_entry_wrapper (obj=0x5630cd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [14] #8 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x80e9db0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [14] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [14] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [14] [14] Thread 10 (Thread 0x7ff0745fd600 (LWP 499995) "s3d.x"): [14] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ff0745fcba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [14] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [14] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x59ffb00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [14] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x59ffb00, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [14] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x59ffb00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [14] #6 0x00007fffdf941ac2 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x59ffb00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [14] #7 Realm::Thread::thread_entry_wrapper (obj=0x59ffb00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [14] #8 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x80e9800) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [14] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [14] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [14] [14] Thread 9 (Thread 0x7ffd7d59f600 (LWP 499990) "s3d.x"): [14] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd7d59eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [14] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [14] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5a208f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [14] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5a208f0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [14] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5a208f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [14] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [14] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [14] #8 0x0000000000000000 in ?? () [14] [14] Thread 8 (Thread 0x7ffd8163f600 (LWP 499986) "s3d.x"): [14] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd8163eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [14] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [14] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bd630) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [14] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bd630, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [14] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bd630) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [14] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [14] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [14] #8 0x0000000000000000 in ?? () [14] [14] Thread 7 (Thread 0x7ffd8583f600 (LWP 499982) "s3d.x"): [14] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd8583eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [14] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [14] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bc550) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [14] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bc550, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [14] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bc550) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [14] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [14] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [14] #8 0x0000000000000000 in ?? () [14] [14] Thread 6 (Thread 0x7ffd85a0f600 (LWP 499978) "s3d.x"): [14] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd85a0eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [14] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [14] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bc050) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [14] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bc050, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [14] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bc050) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [14] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [14] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [14] #8 0x0000000000000000 in ?? () [14] [14] Thread 5 (Thread 0x7ffd615ff600 (LWP 499976) "s3d.x"): [14] #0 0x00007fffe65a7ecc in __lll_lock_wait () from /lib64/libc.so.6 [14] #1 0x00007fffe65ae615 in pthread_mutex_lock@@GLIBC_2.2.5 () from /lib64/libc.so.6 [14] #2 0x00007fffe087c7f6 in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #3 0x00007fffe087cdbe in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #4 0x00007fffe0896aec in gasneti_AMPSHMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #5 0x00007fffe087c4db in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #6 0x00007fffe087b066 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [14] #7 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [14] #8 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [14] #9 0x00007fffdf9a34cb in Realm::GASNetEXPoller::do_work (this=0x5267e40, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [14] #10 0x00007fffdf8209d9 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd615fd5c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [14] #11 0x00007fffdf821271 in Realm::BackgroundWorkThread::main_loop (this=0x5b79fe0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [14] #12 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x5fd6380) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [14] #13 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [14] #14 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [14] [14] Thread 4 (Thread 0x7ffd717f3600 (LWP 499974) "s3d.x"): [14] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd717f2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [14] #2 0x00007fffdf821bc5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [14] #3 Realm::BackgroundWorkThread::main_loop (this=0x5b79e80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [14] #4 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x5b79eb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [14] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [14] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [14] [14] Thread 3 (Thread 0x7ffd77905600 (LWP 499972) "s3d.x"): [14] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [14] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd77904ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [14] #2 0x00007fffdf821bc5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [14] #3 Realm::BackgroundWorkThread::main_loop (this=0x5887550) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [14] #4 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x5b7a100) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [14] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [14] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [14] [14] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 499958) "s3d.x"): [14] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [14] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [14] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [14] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [14] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [14] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [14] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [14] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [14] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [14] [14] Thread 1 (Thread 0x7fffd5a20600 (LWP 499944) "s3d.x"): [14] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [14] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [14] #2 0x00007fffe088c9d4 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #3 0x00007fffe088d027 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #4 0x00007fffe0890a2e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #5 0x00007fffdf464fc9 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #6 0x00007fffdf4650ba in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #7 0x00007fffe0886672 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #8 0x00007fffe08867ac in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #9 0x00007fffe087c4e0 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #10 0x00007fffe087c9bf in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #11 0x00007fffdf46f9bf in gasneti_defaultSignalHandler () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [14] #12 [14] #13 0x00007fffe65acdfc in __pthread_kill_implementation () from /lib64/libc.so.6 [14] #14 0x00007fffe655b842 in raise () from /lib64/libc.so.6 [14] #15 [14] #16 0x00007fffe65a7c1c in __futex_abstimed_wait_common () from /lib64/libc.so.6 [14] #17 0x00007fffe65aa548 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 [14] #18 0x00007fffdf83d71d in Realm::BarrierImpl::external_wait (this=0x80a0ed0, gen_needed=2, poisoned=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/barrier_impl.cc:1281 [14] #19 0x00007fffdf82ce75 in Realm::Event::external_wait_faultaware (this=0x40cd270, poisoned=@0x7fffffff3920: .FALSE.) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/event_impl.cc:282 [14] #20 0x00007fffe174cf75 in Legion::Internal::LgEvent::wait_faultaware (this=0x40cd270, poisoned=@0x7fffffff3920: .FALSE., from_app=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3459 [14] #21 0x00007fffe1ad0dfa in Legion::Internal::ApEvent::wait_faultignorant (this=0x40cd270) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:2966 [14] #22 Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion (this=0x40cd260) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:7112 [14] #23 0x00000000005da171 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:229 [14] #24 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [14] #25 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [14] #26 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [14] #27 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [14] #28 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [14] [Inferior 1 (process 499944) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit [26] /usr/bin/gdb -nx -batch -x /tmp/gasnet_C9nzRx '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/pwave_x_8_ammonia/s3d.x' 387844 [26] [New LWP 387855] [26] [New LWP 387868] [26] [New LWP 387871] [26] [New LWP 387874] [26] [New LWP 387879] [26] [New LWP 387883] [26] [New LWP 387887] [26] [New LWP 387893] [26] [New LWP 387896] [26] [New LWP 387899] [26] [New LWP 387904] [26] [New LWP 387905] [26] [Thread debugging using libthread_db enabled] [26] Using host libthread_db library "/lib64/libthread_db.so.1". [26] 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [26] Id Target Id Frame [26] * 1 Thread 0x7fffd5a20600 (LWP 387844) "s3d.x" 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [26] 2 Thread 0x7fffd35ff6c0 (LWP 387855) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [26] 3 Thread 0x7ffd77905600 (LWP 387868) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [26] 4 Thread 0x7ffd717f3600 (LWP 387871) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 5 Thread 0x7ffd615ff600 (LWP 387874) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 6 Thread 0x7ffd85a0f600 (LWP 387879) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 7 Thread 0x7ffd8583f600 (LWP 387883) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 8 Thread 0x7ffd8163f600 (LWP 387887) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 9 Thread 0x7ffd7d59f600 (LWP 387893) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 10 Thread 0x7ff0745fd600 (LWP 387896) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 11 Thread 0x7ff0743f9600 (LWP 387899) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 12 Thread 0x7ffd5cfff600 (LWP 387904) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] 13 Thread 0x7ffd589ff600 (LWP 387905) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] [26] Thread 13 (Thread 0x7ffd589ff600 (LWP 387905) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd589feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9be398 in Realm::Doorbell::wait (this=0x7ffd589feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5bca998) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [26] #4 0x00007fffdf96068b in Realm::Hip::ContextSynchronizer::thread_main (this=0x5bca978) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [26] #5 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x7fc0480013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 12 (Thread 0x7ffd5cfff600 (LWP 387904) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9be398 in Realm::Doorbell::wait (this=0x7ffd5cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::UnfairCondVar::wait (this=this@entry=0x59ca2c8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [26] #4 0x00007fffdf96068b in Realm::Hip::ContextSynchronizer::thread_main (this=0x59ca2a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [26] #5 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x7fc0500013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 11 (Thread 0x7ff0743f9600 (LWP 387899) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ff0743f8ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5fd56c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5fd56c0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5fd56c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf941ac2 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5fd56c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [26] #7 Realm::Thread::thread_entry_wrapper (obj=0x5fd56c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [26] #8 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x80ece80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 10 (Thread 0x7ff0745fd600 (LWP 387896) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ff0745fcba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5c795d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5c795d0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5c795d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf941ac2 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5c795d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [26] #7 Realm::Thread::thread_entry_wrapper (obj=0x5c795d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [26] #8 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x80ec8d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 9 (Thread 0x7ffd7d59f600 (LWP 387893) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd7d59eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bc5c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bc5c0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bc5c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [26] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [26] #8 0x0000000000000000 in ?? () [26] [26] Thread 8 (Thread 0x7ffd8163f600 (LWP 387887) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd8163eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bc0c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bc0c0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bc0c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [26] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [26] #8 0x0000000000000000 in ?? () [26] [26] Thread 7 (Thread 0x7ffd8583f600 (LWP 387883) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd8583eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bc960) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bc960, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bc960) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [26] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [26] #8 0x0000000000000000 in ?? () [26] [26] Thread 6 (Thread 0x7ffd85a0f600 (LWP 387879) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd85a0eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x54e0da0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [26] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x54e0da0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [26] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x54e0da0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [26] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [26] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [26] #8 0x0000000000000000 in ?? () [26] [26] Thread 5 (Thread 0x7ffd615ff600 (LWP 387874) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd615feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf821bc5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::BackgroundWorkThread::main_loop (this=0x54ed560) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [26] #4 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x54edae0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 4 (Thread 0x7ffd717f3600 (LWP 387871) "s3d.x"): [26] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [26] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd717f2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [26] #2 0x00007fffdf821bc5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [26] #3 Realm::BackgroundWorkThread::main_loop (this=0x54ed350) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [26] #4 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x54ed680) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 3 (Thread 0x7ffd77905600 (LWP 387868) "s3d.x"): [26] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [26] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [26] #2 0x00007fffe088c9d4 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #3 0x00007fffe088d027 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #4 0x00007fffe0890a2e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #5 0x00007fffdf464fc9 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #6 0x00007fffdf4650ba in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #7 0x00007fffe0886672 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #8 0x00007fffe08867ac in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #9 0x00007fffe087c4e0 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #10 0x00007fffe087c9bf in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #11 0x00007fffe087cdbe in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #12 0x00007fffe0896aec in gasneti_AMPSHMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #13 0x00007fffe087c4db in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [26] #14 0x00007fffe087b066 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [26] #15 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [26] #16 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [26] #17 0x00007fffdf9a34cb in Realm::GASNetEXPoller::do_work (this=0x5267e60, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [26] #18 0x00007fffdf8209d9 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd779035c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [26] #19 0x00007fffdf821271 in Realm::BackgroundWorkThread::main_loop (this=0x54ed320) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [26] #20 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x54ed430) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [26] #21 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #22 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 387855) "s3d.x"): [26] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [26] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [26] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [26] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [26] [26] Thread 1 (Thread 0x7fffd5a20600 (LWP 387844) "s3d.x"): [26] #0 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [26] #1 0x00007fffe65aa548 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 [26] #2 0x00007fffdf83d71d in Realm::BarrierImpl::external_wait (this=0x80a4020, gen_needed=2, poisoned=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/barrier_impl.cc:1281 [26] #3 0x00007fffdf82ce75 in Realm::Event::external_wait_faultaware (this=0x40cd270, poisoned=@0x7fffffff3920: .FALSE.) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/event_impl.cc:282 [26] #4 0x00007fffe174cf75 in Legion::Internal::LgEvent::wait_faultaware (this=0x40cd270, poisoned=@0x7fffffff3920: .FALSE., from_app=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3459 [26] #5 0x00007fffe1ad0dfa in Legion::Internal::ApEvent::wait_faultignorant (this=0x40cd270) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:2966 [26] #6 Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion (this=0x40cd260) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:7112 [26] #7 0x00000000005da171 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:229 [26] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [26] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [26] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [26] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [26] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [26] [Inferior 1 (process 387844) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit [22] /usr/bin/gdb -nx -batch -x /tmp/gasnet_0mdyjo '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/pwave_x_8_ammonia/s3d.x' 565316 [22] [New LWP 565335] [22] [New LWP 565346] [22] [New LWP 565349] [22] [New LWP 565351] [22] [New LWP 565355] [22] [New LWP 565359] [22] [New LWP 565363] [22] [New LWP 565367] [22] [New LWP 565370] [22] [New LWP 565373] [22] [New LWP 565380] [22] [New LWP 565381] [22] [Thread debugging using libthread_db enabled] [22] Using host libthread_db library "/lib64/libthread_db.so.1". [22] 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [22] Id Target Id Frame [22] * 1 Thread 0x7fffd5a20600 (LWP 565316) "s3d.x" 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [22] 2 Thread 0x7fffd35ff6c0 (LWP 565335) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [22] 3 Thread 0x7ffd77905600 (LWP 565346) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] 4 Thread 0x7ffd717f3600 (LWP 565349) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] 5 Thread 0x7ffd615ff600 (LWP 565351) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [22] 6 Thread 0x7ffd85a0f600 (LWP 565355) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] 7 Thread 0x7ffd8583f600 (LWP 565359) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] 8 Thread 0x7ffd8163f600 (LWP 565363) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] 9 Thread 0x7ffd7d59f600 (LWP 565367) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] 10 Thread 0x7ff0745fd600 (LWP 565370) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] 11 Thread 0x7ff0743f9600 (LWP 565373) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] 12 Thread 0x7ffd5cfff600 (LWP 565380) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] 13 Thread 0x7ffd589ff600 (LWP 565381) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] [22] Thread 13 (Thread 0x7ffd589ff600 (LWP 565381) "s3d.x"): [22] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd589feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [22] #2 0x00007fffdf9be398 in Realm::Doorbell::wait (this=0x7ffd589feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [22] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5894a98) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [22] #4 0x00007fffdf96068b in Realm::Hip::ContextSynchronizer::thread_main (this=0x5894a78) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [22] #5 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x7fc04c0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [22] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [22] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [22] [22] Thread 12 (Thread 0x7ffd5cfff600 (LWP 565380) "s3d.x"): [22] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [22] #2 0x00007fffdf9be398 in Realm::Doorbell::wait (this=0x7ffd5cffeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [22] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5bd3f28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [22] #4 0x00007fffdf96068b in Realm::Hip::ContextSynchronizer::thread_main (this=0x5bd3f08) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [22] #5 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x7fc0480013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [22] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [22] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [22] [22] Thread 11 (Thread 0x7ff0743f9600 (LWP 565373) "s3d.x"): [22] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ff0743f8ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [22] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [22] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5a47660) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [22] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5a47660, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [22] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5a47660) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [22] #6 0x00007fffdf941ac2 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5a47660) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [22] #7 Realm::Thread::thread_entry_wrapper (obj=0x5a47660) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [22] #8 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x538d440) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [22] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [22] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [22] [22] Thread 10 (Thread 0x7ff0745fd600 (LWP 565370) "s3d.x"): [22] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ff0745fcba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [22] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [22] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x57febd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [22] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x57febd0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [22] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x57febd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [22] #6 0x00007fffdf941ac2 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x57febd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [22] #7 Realm::Thread::thread_entry_wrapper (obj=0x57febd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [22] #8 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x80ecd80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [22] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [22] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [22] [22] Thread 9 (Thread 0x7ffd7d59f600 (LWP 565367) "s3d.x"): [22] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd7d59eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [22] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [22] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bc5c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [22] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bc5c0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [22] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bc5c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [22] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [22] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [22] #8 0x0000000000000000 in ?? () [22] [22] Thread 8 (Thread 0x7ffd8163f600 (LWP 565363) "s3d.x"): [22] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd8163eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [22] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [22] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56bc0c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [22] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56bc0c0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [22] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56bc0c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [22] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [22] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [22] #8 0x0000000000000000 in ?? () [22] [22] Thread 7 (Thread 0x7ffd8583f600 (LWP 565359) "s3d.x"): [22] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd8583eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [22] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [22] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56d5b90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [22] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56d5b90, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [22] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56d5b90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [22] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [22] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [22] #8 0x0000000000000000 in ?? () [22] [22] Thread 6 (Thread 0x7ffd85a0f600 (LWP 565355) "s3d.x"): [22] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd85a0eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [22] #2 0x00007fffdf9373ef in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [22] #3 0x00007fffdf9374a8 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x58f10c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [22] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x58f10c0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [22] #5 0x00007fffdf93e533 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x58f10c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [22] #6 0x00007fffdf94a01a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [22] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [22] #8 0x0000000000000000 in ?? () [22] [22] Thread 5 (Thread 0x7ffd615ff600 (LWP 565351) "s3d.x"): [22] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [22] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [22] #2 0x00007fffe088c9d4 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #3 0x00007fffe088d027 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #4 0x00007fffe0890a2e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #5 0x00007fffdf464fc9 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #6 0x00007fffdf4650ba in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #7 0x00007fffe0886672 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #8 0x00007fffe08867ac in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #9 0x00007fffe087c4e0 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #10 0x00007fffe087c9bf in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #11 0x00007fffe087cdbe in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #12 0x00007fffe0896aec in gasneti_AMPSHMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #13 0x00007fffe087c4db in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/pressure_wave_case/../..//legion/language/build/lib/librealm.so.1 [22] #14 0x00007fffe087b066 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [22] #15 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [22] #16 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [22] #17 0x00007fffdf9a34cb in Realm::GASNetEXPoller::do_work (this=0x5267e30, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [22] #18 0x00007fffdf8209d9 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd615fd5c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [22] #19 0x00007fffdf821271 in Realm::BackgroundWorkThread::main_loop (this=0x5716770) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [22] #20 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x5715b20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [22] #21 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [22] #22 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [22] [22] Thread 4 (Thread 0x7ffd717f3600 (LWP 565349) "s3d.x"): [22] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd717f2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [22] #2 0x00007fffdf821bc5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [22] #3 Realm::BackgroundWorkThread::main_loop (this=0x5716580) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [22] #4 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x5716890) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [22] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [22] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [22] [22] Thread 3 (Thread 0x7ffd77905600 (LWP 565346) "s3d.x"): [22] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [22] #1 0x00007fffdf9bdc85 in Realm::Doorbell::wait_slow (this=0x7ffd77904ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [22] #2 0x00007fffdf821bc5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [22] #3 Realm::BackgroundWorkThread::main_loop (this=0x5715af0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [22] #4 0x00007fffdf944749 in Realm::KernelThread::pthread_entry (data=0x5716660) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [22] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [22] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [22] [22] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 565335) "s3d.x"): [22] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [22] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [22] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [22] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [22] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [22] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [22] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [22] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [22] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [22] [22] Thread 1 (Thread 0x7fffd5a20600 (LWP 565316) "s3d.x"): [22] #0 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [22] #1 0x00007fffe65aa548 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 [22] #2 0x00007fffdf83d71d in Realm::BarrierImpl::external_wait (this=0x80a4450, gen_needed=2, poisoned=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/barrier_impl.cc:1281 [22] #3 0x00007fffdf82ce75 in Realm::Event::external_wait_faultaware (this=0x40c4c50, poisoned=@0x7fffffff3920: .FALSE.) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/event_impl.cc:282 [22] #4 0x00007fffe174cf75 in Legion::Internal::LgEvent::wait_faultaware (this=0x40c4c50, poisoned=@0x7fffffff3920: .FALSE., from_app=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3459 [22] #5 0x00007fffe1ad0dfa in Legion::Internal::ApEvent::wait_faultignorant (this=0x40c4c50) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:2966 [22] #6 Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion (this=0x40c4c40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:7112 [22] #7 0x00000000005da171 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:229 [22] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [22] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [22] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [22] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [22] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [22] [Inferior 1 (process 565316) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit srun: error: frontier06004: task 14: Exited with exit code 255 srun: error: frontier06387: task 26: Exited with exit code 255 srun: error: frontier06260: task 22: Exited with exit code 255 aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit srun: error: frontier06515: tasks 29,31: Exited with exit code 255 srun: error: frontier06004: tasks 13,15: Exited with exit code 255 srun: error: frontier06260: task 21: Exited with exit code 255 srun: error: frontier06387: task 27: Exited with exit code 255 srun: error: frontier05621: tasks 0-2: Exited with exit code 255 srun: error: frontier05876: tasks 8-10: Exited with exit code 255 srun: error: frontier05749: tasks 5-7: Exited with exit code 255 srun: error: frontier06132: tasks 17-19: Exited with exit code 255 srun: error: frontier06260: task 20: Exited with exit code 255 srun: error: frontier06387: tasks 24-25: Exited with exit code 255 srun: error: frontier06260: task 23: Exited with exit code 255 srun: error: frontier06515: task 28: Exited with exit code 255 srun: error: frontier06004: task 12: Exited with exit code 255 srun: error: frontier05621: task 3: Exited with exit code 255 srun: error: frontier06132: task 16: Exited with exit code 255 srun: error: frontier05749: task 4: Exited with exit code 255 srun: error: frontier05876: task 11: Exited with exit code 255