============================================================================== Hello, and welcome to S3D! ============================================================================== ============================================================== --------------------S3D BUILD CONFIGURATION----------------- Build directory: /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/fortran/ammonia ------------------------------------------------------ S3D Options: Using mixture averaged transport: ON Using ARK integration: OFF Using chemical mechanism: ammonia Using srcs for timescale analysis: OFF ------------------------------------------------------ Defines: SYSTEMCALLWONTWORK SAVEFILEINSEPDIR ARCH_OPTERON MPIIO BLOCKIO MIXAVG FROZENFEED ------------------------------------------------------ FORTRAN COMPILER FLAGS: -cpp -fdefault-double-8 -fdefault-real-8 -fno-range-check -ffree-line-length-none -fallow-argument-mismatch C COMPILER FLAGS: ------------------------------------------------------ S3D Version: ------------------------------------------------------ Chemistry routine header: c**************************************************************************** c ... DO NOT EDIT. AUTOMATICALLY GENERATED FILE. c**************************************************************************** c ... c ... CKWYP-compatible GETRATES: return molar production rates for all c ... species given pressure (P), temperature (T), and mass fractions (Y). c ... c ... Generated by cgetrates 1.0b c ... c**************************************************************************** subroutine getrates(P,T,Y,ICKWRK,RCKWRK,WSPL) ------------------------------------------------------ Transport routine header: !Standard version of tranlib.f. 18Feb 2009 ! (Changes by Mark Fahey, ORNL) ! (Update by Evatt Hawkes, SNL) SUBROUTINE MCABS ------------------------------------------------------ ============================================================== ------------------------------------------------------------------------------ this run started at: date = 06/13/2025 time = 12:18:59 ------------------------------------------------------------------------------ initializing chemkin module... opening chemkin link file... setting chemkin work array lengths... allocating chemkin work arrays... filling chemkin work arrays based on chemkin link file... CKLIB: CHEMKIN-III GAS-PHASE CHEMICAL KINETICS LIBRARY, DOUBLE PRECISION Vers. 5.36 2000/12/15 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 the following was extracted from the ../input/chem.asc file: element names are as follows: index name 1 H 2 O 3 N species names are as follows: index name 1 H2 2 NH3 3 NO 4 N2O 5 O2 6 H 7 O 8 OH 9 HO2 10 H2O 11 H2O2 12 NO2 13 HNO 14 N 15 NNH 16 NH2 17 NH 18 H2NO 19 N2 species molecular weights are as follows: index name value (kg/mol) 1 H2 2.01594E-03 2 NH3 1.70306E-02 3 NO 3.00061E-02 4 N2O 4.40128E-02 5 O2 3.19988E-02 6 H 1.00797E-03 7 O 1.59994E-02 8 OH 1.70074E-02 9 HO2 3.30068E-02 10 H2O 1.80153E-02 11 H2O2 3.40147E-02 12 NO2 4.60055E-02 13 HNO 3.10141E-02 14 N 1.40067E-02 15 NNH 2.90214E-02 16 NH2 1.60226E-02 17 NH 1.50147E-02 18 H2NO 3.20220E-02 19 N2 2.80134E-02 ------------------------------------------------------------------------------ initializing param module... number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 number of reaction third-body reactions = 11 ------------------------------------------------------------------------------ initializing topology module... Initialized topology module, npx = 2 npy = 1 npz = 4 npes = 8 ------------------------------------------------------------------------------ initializing reference module... the various reference values are as follows: universal gas constant (J/mol-K) = 8.31400E+00 freestream temperature (K) = 8.01900E+02 reference ratio of specifice heats = 1.36400E+00 reference speed of sound (m/s) = 5.28000E+02 reference density (kg/m^3) = 1.22300E+01 reference conductivity (W/m-K) = 5.76900E-02 reference temperature (K) = 2.91892E+02 reference pressure (Pa) = 3.40953E+06 standard atmospheric pressure (Pa) = 1.01325E+05 reference time (s) = 1.25428E-06 reference specific heat (J/kg-K) = 9.55094E+02 reference length (m) = 6.62258E-04 reference viscosity (Pa-s) = 4.27649E+00 acoustic Reynolds number = 1.00000E+05 Mach number = 1.00000E-03 convective Reynolds number = 1.00000E+02 ------------------------------------------------------------------------------ initializing ERK module... ERK module information is as follows: rk_method = -64 cont_switch = 0 tstep_init (seconds) = 5.00E-10 tstep_min (seconds) = 5.00E-10 tstep_max (seconds) = 5.00E-10 rk_rtol = 1.00E-03 rk_atol = 1.00E-15 cont_safety = 9.00E-01 k_I = 2.00E-01 k_P = 2.00E-01 k_D = 5.00E-02 k_D2 = 0.00E+00 cont_factor = 3.00E+00 cfl_switch = 1 i_time_cont = 20 cont_n_reg = 1 controller type = NONE rk_tol = 1.00E-12 using RK4(3)6[2R+]C ERK scheme! ------------------------------------------------------------------------------ initializing transport module... opening mixavg.in ... Lagging is switched on WARNING - with coefficient lagging, theoretical temporal convergence order may not be observed opening transport link file... setting lengths of transport work arrays... allocating transport work arrays... creating transport work arrays from transport link file... TRANLIB: CHEMKIN-III MULTICOMPONENT TRANSPORT LIBRARY, DOUBLE PRECISION Vers. 4.6 1999/06/03 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. ------------------------------------------------------------------------------ initializing filter module... using explicit filter of order 10 filter will be applied every 10 time steps ------------------------------------------------------------------------------ initializing derivative module... ------------------------------------------------------------------------------ initializing thermchem module... ------------------------------------------------------------------------------ initializing grid module... ------------------------------------------------------------------------------ initializing boundary condition module... ------------------------------------------------------------------------------ *** WARNING: ofi-conduit failed to configure FI_CXI_* environment variables due to prior conflicting settings. This may lead to unstable behavior and/or degraded performance. If you did not intentionally set FI_CXI_RX_MATCH_MODE='software', FI_CXI_RDZV_THRESHOLD='16384', or FI_CXI_RDZV_GET_MIN then this condition may have resulted from initializing MPI prior to initialization of GASNet. For more information on that scenario, please see "Limits to MPI interoperability" in the ofi-conduit README. Legion Init: Mechanism = 'ammonia' Local Grid Size = (128, 128, 256) Global Grid Size = (256, 128, 1024) Proc Grid Size = ( 2, 1, 4) Proc ID = ( 0, 0, 0) nvar_tot = 23, n_spec = 19, n_reg = 3, n_scalar = 0 iorder = 8, iforder = 10, vary = XYZ fuse fields set to 200 from environment set mapper mode to 'allgpu' from environment RHST_ARGS: -ll:gpu 2 -ll:cpu 1 -ll:util 3 -ll:bgwork 3 -ll:fsize 16384 -ll:csize 4096 -ll:zsize 1024 -ll:stacksize 16 -logfile run_%.log -hip:hostreg 0 -lg:no_transitive_reduction 1 -lg:prof 8 -lg:prof_logfile prof_%.gz -level barrier=1 -ll:force_kthreads 1 REGULAR STENCIL !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!! YOU ARE PROFILING IN DEBUG MODE !!! !!! SERIOUS PERFORMANCE DEGRADATION WILL OCCUR!!! !!! COMPILE WITH DEBUG=0 FOR PROFILING !!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SLEEPING FOR 5 SECONDS SO YOU READ THIS WARNING... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!! YOU ARE PROFILING IN DEBUG MODE !!! !!! SERIOUS PERFORMANCE DEGRADATION WILL OCCUR!!! !!! COMPILE WITH DEBUG=0 FOR PROFILING !!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SLEEPING FOR 5 SECONDS SO YOU READ THIS WARNING... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!! YOU ARE PROFILING IN DEBUG MODE !!! !!! SERIOUS PERFORMANCE DEGRADATION WILL OCCUR!!! !!! COMPILE WITH DEBUG=0 FOR PROFILING !!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SLEEPING FOR 5 SECONDS SO YOU READ THIS WARNING... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!! YOU ARE PROFILING IN DEBUG MODE !!! !!! SERIOUS PERFORMANCE DEGRADATION WILL OCCUR!!! !!! COMPILE WITH DEBUG=0 FOR PROFILING !!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SLEEPING FOR 5 SECONDS SO YOU READ THIS WARNING... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!! YOU ARE PROFILING IN DEBUG MODE !!! !!! SERIOUS PERFORMANCE DEGRADATION WILL OCCUR!!! !!! COMPILE WITH DEBUG=0 FOR PROFILING !!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SLEEPING FOR 5 SECONDS SO YOU READ THIS WARNING... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!! YOU ARE PROFILING IN DEBUG MODE !!! !!! SERIOUS PERFORMANCE DEGRADATION WILL OCCUR!!! !!! COMPILE WITH DEBUG=0 FOR PROFILING !!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SLEEPING FOR 5 SECONDS SO YOU READ THIS WARNING... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!! YOU ARE PROFILING IN DEBUG MODE !!! !!! SERIOUS PERFORMANCE DEGRADATION WILL OCCUR!!! !!! COMPILE WITH DEBUG=0 FOR PROFILING !!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SLEEPING FOR 5 SECONDS SO YOU READ THIS WARNING... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!! YOU ARE PROFILING IN DEBUG MODE !!! !!! SERIOUS PERFORMANCE DEGRADATION WILL OCCUR!!! !!! COMPILE WITH DEBUG=0 FOR PROFILING !!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !WARNING WARNING WARNING WARNING WARNING WARNING! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SLEEPING FOR 5 SECONDS SO YOU READ THIS WARNING... initializing variables... Host: frontier00069 Rank: 0 PID: 195743 Time: 12:19:13 STARTING MAIN TASK! initializing bomb (stand back) Mixture fraction module successfully initialized... storing hard inflow conditions... ------------------------------------------------------------------------------ continuing with initialization... ------------------------------------------------------------------------------ storing hard inflow conditions... maximum speed of sound = 839.70 m/s mimimum speed of sound = 839.70 m/s maximum acoustic time = 2.06E-05 sec minimum acoustic time = 2.06E-05 sec initialization completed for time = 0.00E+00 ------------------------------------------------------------------------------ writing save files for: i_time = 0, time = 0.0000E+00 (sec) precision is d s3d.x: /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_profiling.cc:740: void Legion::Internal::LegionProfInstance::record_barrier_use(Legion::Internal::LgEvent, Legion::UniqueID): Assertion `arrival_info.fevent.exists()' failed. s3d.x: /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_profiling.cc:740: void Legion::Internal::LegionProfInstance::record_barrier_use(Legion::Internal::LgEvent, Legion::UniqueID): Assertion `arrival_info.fevent.exists()' failed. *** Caught a fatal signal (proc 6): SIGABRT(6) *** NOTICE (proc 6): We recommend linking the debug version of GASNet to assist you in resolving this application issue. s3d.x: /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_profiling.cc:740: void Legion::Internal::LegionProfInstance::record_barrier_use(Legion::Internal::LgEvent, Legion::UniqueID): Assertion `arrival_info.fevent.exists()' failed. *** Caught a fatal signal (proc 7): SIGABRT(6) *** NOTICE (proc 7): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Caught a fatal signal (proc 5): SIGABRT(6) *** NOTICE (proc 5): We recommend linking the debug version of GASNet to assist you in resolving this application issue. [5] Invoking GDB for backtrace... [6] Invoking GDB for backtrace... [7] Invoking GDB for backtrace... [7] /usr/bin/gdb -nx -batch -x /tmp/gasnet_NsLbJV '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_2/s3d.x' 181779 [7] [New LWP 181794] [7] [New LWP 181806] [7] [New LWP 181809] [7] [New LWP 181813] [7] [New LWP 181823] [7] [New LWP 181827] [7] [New LWP 181831] [7] [New LWP 181835] [7] [New LWP 181839] [7] [New LWP 181857] [7] [New LWP 181858] [7] [New LWP 181859] [7] [New LWP 181865] [7] [Thread debugging using libthread_db enabled] [7] Using host libthread_db library "/lib64/libthread_db.so.1". [7] 0x00007fffe65a6cce in __futex_abstimed_wait_common () from /lib64/libc.so.6 [7] Id Target Id Frame [7] * 1 Thread 0x7fffd18c5700 (LWP 181779) "s3d.x" 0x00007fffe65a6cce in __futex_abstimed_wait_common () from /lib64/libc.so.6 [7] 2 Thread 0x7fffce5ff6c0 (LWP 181794) "s3d.x" 0x00007fffe662cf9b in ioctl () from /lib64/libc.so.6 [7] 3 Thread 0x7ffd27504700 (LWP 181806) "s3d.x" Realm::IntrusiveList::empty (this=0x529fa50) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/lists.inl:143 [7] 4 Thread 0x7ffd213f2700 (LWP 181809) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] 5 Thread 0x7ffd123fe700 (LWP 181813) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] 6 Thread 0x7ff01eff9700 (LWP 181823) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] 7 Thread 0x7ff01dff4700 (LWP 181827) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] 8 Thread 0x7fc0167fe700 (LWP 181831) "s3d.x" 0x00007fffe661a3a7 in wait4 () from /lib64/libc.so.6 [7] 9 Thread 0x7ff0247fd700 (LWP 181835) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] 10 Thread 0x7ff0245f8700 (LWP 181839) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] 11 Thread 0x7fc0053fd700 (LWP 181857) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] 12 Thread 0x7ffd0ddfe700 (LWP 181858) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] 13 Thread 0x7ffd097fe700 (LWP 181859) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] 14 Thread 0x7ff01fffe700 (LWP 181865) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] [7] Thread 14 (Thread 0x7ff01fffe700 (LWP 181865) "s3d.x"): [7] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff01fffdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff01fffdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdc90c51d in Realm::FIFOCondVar::wait (this=0x7ff01fffc870) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:1027 [7] #4 0x00007fffdc85aef3 in Realm::KernelThreadTaskScheduler::worker_sleep (this=0x5b704b0, switch_to=0x8071240) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1472 [7] #5 0x00007fffdc859de7 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5b704b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1200 [7] #6 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5b704b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [7] #7 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x5b704b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #8 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fbff40076d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #9 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #10 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 13 (Thread 0x7ffd097fe700 (LWP 181859) "s3d.x"): [7] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd097fdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd097fdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdc90c1e1 in Realm::UnfairCondVar::wait (this=0x5ab2ae8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [7] #4 0x00007fffdc88bff8 in Realm::Hip::ContextSynchronizer::thread_main (this=0x5ab2ab0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [7] #5 0x00007fffdc8a304c in Realm::Thread::thread_entry_wrapper (obj=0x5ab2ab0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #6 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fbfe80013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #7 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #8 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 12 (Thread 0x7ffd0ddfe700 (LWP 181858) "s3d.x"): [7] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd0ddfdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd0ddfdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdc90c1e1 in Realm::UnfairCondVar::wait (this=0x558a708) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [7] #4 0x00007fffdc88bff8 in Realm::Hip::ContextSynchronizer::thread_main (this=0x558a6d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [7] #5 0x00007fffdc8a304c in Realm::Thread::thread_entry_wrapper (obj=0x558a6d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #6 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fbfe00013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #7 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #8 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 11 (Thread 0x7fc0053fd700 (LWP 181857) "s3d.x"): [7] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7fc0053fcca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7fc0053fcca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x3ff5320, old_counter=37) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [7] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x3ff5160, old_work_counter=37) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [7] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x3ff5160, old_work_counter=37) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [7] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [7] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [7] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fbffc082c90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 10 (Thread 0x7ff0245f8700 (LWP 181839) "s3d.x"): [7] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff0245f7ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff0245f7ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x59b5c00, old_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [7] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x59b5a40, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [7] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x59b5a40, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [7] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x59b5a40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [7] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x59b5a40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [7] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x59b5a40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x8071e60) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 9 (Thread 0x7ff0247fd700 (LWP 181835) "s3d.x"): [7] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff0247fcca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff0247fcca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x5fffc90, old_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [7] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x5fffad0, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [7] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x5fffad0, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [7] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5fffad0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [7] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5fffad0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [7] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x5fffad0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x8071850) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 8 (Thread 0x7fc0167fe700 (LWP 181831) "s3d.x"): [7] #0 0x00007fffe661a3a7 in wait4 () from /lib64/libc.so.6 [7] #1 0x00007fffe656b6cb in do_system () from /lib64/libc.so.6 [7] #2 0x00007fffdd833854 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [7] #3 0x00007fffdd833ea7 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [7] #4 0x00007fffdd8378ae in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [7] #5 0x00007fffdc1b9afd in gasneti_defaultSignalHandler () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [7] #6 [7] #7 0x00007fffe65ac49c in __pthread_kill_implementation () from /lib64/libc.so.6 [7] #8 0x00007fffe655a8c2 in raise () from /lib64/libc.so.6 [7] #9 0x00007fffe654264f in abort () from /lib64/libc.so.6 [7] #10 0x00007fffe6542567 in __assert_fail_base.cold () from /lib64/libc.so.6 [7] #11 0x00007fffe6552bb2 in __assert_fail () from /lib64/libc.so.6 [7] #12 0x00007fffe0aec48a in Legion::Internal::LegionProfInstance::record_barrier_use (this=0x7fbff4000ca0, bar=..., uid=0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_profiling.cc:740 [7] #13 0x00007fffe0aefedd in Legion::Internal::LegionProfInstance::record_event_wait (this=0x7fbff4000ca0, event=..., bt=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_profiling.cc:1517 [7] #14 0x00007fffe0e2c293 in Legion::Internal::LgEvent::record_event_wait (this=0x7fc01678a080, bt=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:146 [7] #15 0x00007fffe068f9de in Legion::Internal::LgEvent::wait (this=0x7fc01678a080) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3357 [7] #16 0x00007fffe08c733e in Legion::Internal::ReplicateContext::perform_semantic_attach (this=0x7fc00270a040, func=0x7fffe1e7fe17 'attach_semantic_information\000', kind=79, arg=0x7fc01678a108, arglen=12, tag=2, buffer=0x7fffe57769c6, size=5, is_mutable=false, global=@0x7fc01678a12f: .FALSE., arg2=0x0, arg2len=0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_context.cc:13545 [7] #17 0x00007fffe0e93ebd in Legion::Internal::Runtime::attach_semantic_information (this=0x5709e60, handle=..., tag=2, buffer=0x7fffe57769c6, size=5, is_mutable=false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:24048 [7] #18 0x00007fffe09c17ae in Legion::Runtime::attach_semantic_information (this=0x5b786c0, handle=..., tag=2, buffer=0x7fffe57769c6, size=5, is_mut=false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion.cc:7000 [7] #19 0x00007fffe07f1812 in legion_index_space_attach_semantic_information (runtime_=..., handle_=..., tag=2, buffer=0x7fffe57769c6, size=5, is_mutable=false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_c.cc:799 [7] #20 0x00007fffe5679ae3 in $
() from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//build/ammonia/libregent_tasks.so [7] #21 0x00007fffe5679608 in $__regent_task_main_primary () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//build/ammonia/libregent_tasks.so [7] #22 0x00007fffdc7df147 in Realm::LocalTaskProcessor::execute_task (this=0x57c1790, func_id=1511, task_args=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/proc_impl.cc:1188 [7] #23 0x00007fffdc856d98 in Realm::Task::execute_on_processor (this=0x7fc002715aa0, p=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:330 [7] #24 0x00007fffdc85ad56 in Realm::KernelThreadTaskScheduler::execute_task (this=0x5b704b0, task=0x7fc002715aa0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1424 [7] #25 0x00007fffdc859b91 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5b704b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1163 [7] #26 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5b704b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [7] #27 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x5b704b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #28 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x8071240) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #29 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #30 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 7 (Thread 0x7ff01dff4700 (LWP 181827) "s3d.x"): [7] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff01dff3ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff01dff3ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x5b700e0, old_counter=28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [7] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x5b6ff20, old_work_counter=28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [7] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x5b6ff20, old_work_counter=28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [7] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5b6ff20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [7] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5b6ff20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [7] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x5b6ff20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fa7750) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 6 (Thread 0x7ff01eff9700 (LWP 181823) "s3d.x"): [7] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff01eff8ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff01eff8ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x56997a0, old_counter=28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [7] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x56995e0, old_work_counter=28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [7] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x56995e0, old_work_counter=28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [7] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56995e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [7] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x56995e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [7] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x56995e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fefea0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 5 (Thread 0x7ffd123fe700 (LWP 181813) "s3d.x"): [7] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd123fdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd123fdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdc698d17 in Realm::BackgroundWorkThread::main_loop (this=0x5b6f1a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [7] #4 0x00007fffdc69c174 in Realm::Thread::thread_entry_wrapper (obj=0x5b6f1a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #5 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x59e5d10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #6 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #7 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 4 (Thread 0x7ffd213f2700 (LWP 181809) "s3d.x"): [7] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [7] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd213f1ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [7] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd213f1ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [7] #3 0x00007fffdc698d17 in Realm::BackgroundWorkThread::main_loop (this=0x5694da0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [7] #4 0x00007fffdc69c174 in Realm::Thread::thread_entry_wrapper (obj=0x5694da0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #5 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x5949140) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #6 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #7 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 3 (Thread 0x7ffd27504700 (LWP 181806) "s3d.x"): [7] #0 Realm::IntrusiveList::empty (this=0x529fa50) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/lists.inl:143 [7] #1 0x00007fffdc8e10c8 in Realm::GASNetEXPoller::do_work (this=0x529f960, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2862 [7] #2 0x00007fffdc69af3b in Realm::BackgroundWorkManager::Worker::do_work (this=0x7ffd27502110, max_time_in_ns=-1, interrupt_flag=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [7] #3 0x00007fffdc698bb3 in Realm::BackgroundWorkThread::main_loop (this=0x5694b90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [7] #4 0x00007fffdc69c174 in Realm::Thread::thread_entry_wrapper (obj=0x5694b90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [7] #5 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x5882ed0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [7] #6 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #7 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 2 (Thread 0x7fffce5ff6c0 (LWP 181794) "s3d.x"): [7] #0 0x00007fffe662cf9b in ioctl () from /lib64/libc.so.6 [7] #1 0x00007fffd39786d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #2 0x00007fffd3971a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #3 0x00007fffd3905f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #4 0x00007fffd38df92e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #5 0x00007fffd38f7b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #6 0x00007fffd38a99e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [7] #7 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [7] #8 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [7] [7] Thread 1 (Thread 0x7fffd18c5700 (LWP 181779) "s3d.x"): [7] #0 0x00007fffe65a6cce in __futex_abstimed_wait_common () from /lib64/libc.so.6 [7] #1 0x00007fffe65a9910 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 [7] #2 0x00007fffdc90c622 in Realm::KernelCondVar::wait (this=0x7fee078) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:1087 [7] #3 0x00007fffdc6bca95 in Realm::BarrierImpl::external_wait (this=0x7fedfa0, gen_needed=2, poisoned=@0x7fffffff398f: false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/barrier_impl.cc:994 [7] #4 0x00007fffdc6a63e7 in Realm::Event::external_wait_faultaware (this=0x410df90, poisoned=@0x7fffffff398f: false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/event_impl.cc:282 [7] #5 0x00007fffe0907fb4 in Legion::Internal::LgEvent::wait_faultaware (this=0x410df90, poisoned=@0x7fffffff398f: false, from_app=true) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3459 [7] #6 0x00007fffe0907b2a in Legion::Internal::ApEvent::wait_faultignorant (this=0x410df90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:2966 [7] #7 0x00007fffe0e4c784 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion (this=0x410df80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:7112 [7] #8 0x00007fffe09aee65 in Legion::LegionHandshake::ext_wait_on_legion (this=0x4111260) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion.cc:2285 [7] #9 0x00000000005da171 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:229 [7] #10 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [7] #11 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [7] #12 0x00007fffe6543eec in __libc_start_call_main () from /lib64/libc.so.6 [7] #13 0x00007fffe6543fb5 in __libc_start_main_impl () from /lib64/libc.so.6 [7] #14 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [7] [Inferior 1 (process 181779) detached] [5] /usr/bin/gdb -nx -batch -x /tmp/gasnet_BCjgpr '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_2/s3d.x' 181777 [5] [New LWP 181796] [5] [New LWP 181807] [5] [New LWP 181811] [5] [New LWP 181816] [5] [New LWP 181824] [5] [New LWP 181828] [5] [New LWP 181832] [5] [New LWP 181836] [5] [New LWP 181840] [5] [New LWP 181848] [5] [New LWP 181849] [5] [New LWP 181852] [5] [New LWP 181864] [5] [Thread debugging using libthread_db enabled] [5] Using host libthread_db library "/lib64/libthread_db.so.1". [5] 0x00007fffe65a6cce in __futex_abstimed_wait_common () from /lib64/libc.so.6 [5] Id Target Id Frame [5] * 1 Thread 0x7fffd18c5700 (LWP 181777) "s3d.x" 0x00007fffe65a6cce in __futex_abstimed_wait_common () from /lib64/libc.so.6 [5] 2 Thread 0x7fffce5ff6c0 (LWP 181796) "s3d.x" 0x00007fffe662cf9b in ioctl () from /lib64/libc.so.6 [5] 3 Thread 0x7ffd27504700 (LWP 181807) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] 4 Thread 0x7ffd213f2700 (LWP 181811) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] 5 Thread 0x7ffd123fe700 (LWP 181816) "s3d.x" _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1316 [5] 6 Thread 0x7ff01eff9700 (LWP 181824) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] 7 Thread 0x7ff01dff4700 (LWP 181828) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] 8 Thread 0x7fc0167fe700 (LWP 181832) "s3d.x" 0x00007fffe661a3a7 in wait4 () from /lib64/libc.so.6 [5] 9 Thread 0x7ff0247fd700 (LWP 181836) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] 10 Thread 0x7ff0245f8700 (LWP 181840) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] 11 Thread 0x7ffd097fe700 (LWP 181848) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] 12 Thread 0x7ffd0ddfe700 (LWP 181849) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] 13 Thread 0x7fc0157f9700 (LWP 181852) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] 14 Thread 0x7ff01fffe700 (LWP 181864) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] [5] Thread 14 (Thread 0x7ff01fffe700 (LWP 181864) "s3d.x"): [5] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff01fffdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [5] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff01fffdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [5] #3 0x00007fffdc90c51d in Realm::FIFOCondVar::wait (this=0x7ff01fffc870) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:1027 [5] #4 0x00007fffdc85aef3 in Realm::KernelThreadTaskScheduler::worker_sleep (this=0x3ff5160, switch_to=0x7fef7d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1472 [5] #5 0x00007fffdc859de7 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1200 [5] #6 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [5] #7 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #8 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fc0040076d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #9 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #10 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 13 (Thread 0x7fc0157f9700 (LWP 181852) "s3d.x"): [5] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7fc0157f8ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [5] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7fc0157f8ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [5] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x5bbe6f0, old_counter=45) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [5] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x5bbe530, old_work_counter=45) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [5] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x5bbe530, old_work_counter=45) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [5] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5bbe530) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [5] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5bbe530) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [5] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x5bbe530) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fc01270fa30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 12 (Thread 0x7ffd0ddfe700 (LWP 181849) "s3d.x"): [5] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd0ddfdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [5] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd0ddfdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [5] #3 0x00007fffdc90c1e1 in Realm::UnfairCondVar::wait (this=0x5c29b78) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [5] #4 0x00007fffdc88bff8 in Realm::Hip::ContextSynchronizer::thread_main (this=0x5c29b40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [5] #5 0x00007fffdc8a304c in Realm::Thread::thread_entry_wrapper (obj=0x5c29b40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #6 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fbff80013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #7 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #8 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 11 (Thread 0x7ffd097fe700 (LWP 181848) "s3d.x"): [5] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd097fdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [5] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd097fdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [5] #3 0x00007fffdc90c1e1 in Realm::UnfairCondVar::wait (this=0x55d5418) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [5] #4 0x00007fffdc88bff8 in Realm::Hip::ContextSynchronizer::thread_main (this=0x55d53e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [5] #5 0x00007fffdc8a304c in Realm::Thread::thread_entry_wrapper (obj=0x55d53e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #6 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fbffc0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #7 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #8 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 10 (Thread 0x7ff0245f8700 (LWP 181840) "s3d.x"): [5] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff0245f7ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [5] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff0245f7ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [5] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x57ea7c0, old_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [5] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x57ea600, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [5] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x57ea600, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [5] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x57ea600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [5] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x57ea600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [5] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x57ea600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x80714f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 9 (Thread 0x7ff0247fd700 (LWP 181836) "s3d.x"): [5] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff0247fcca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [5] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff0247fcca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [5] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x59cc8a0, old_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [5] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x59cc6e0, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [5] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x59cc6e0, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [5] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x59cc6e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [5] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x59cc6e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [5] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x59cc6e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fefde0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 8 (Thread 0x7fc0167fe700 (LWP 181832) "s3d.x"): [5] #0 0x00007fffe661a3a7 in wait4 () from /lib64/libc.so.6 [5] #1 0x00007fffe656b6cb in do_system () from /lib64/libc.so.6 [5] #2 0x00007fffdd833854 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [5] #3 0x00007fffdd833ea7 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [5] #4 0x00007fffdd8378ae in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [5] #5 0x00007fffdc1b9afd in gasneti_defaultSignalHandler () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [5] #6 [5] #7 0x00007fffe65ac49c in __pthread_kill_implementation () from /lib64/libc.so.6 [5] #8 0x00007fffe655a8c2 in raise () from /lib64/libc.so.6 [5] #9 0x00007fffe654264f in abort () from /lib64/libc.so.6 [5] #10 0x00007fffe6542567 in __assert_fail_base.cold () from /lib64/libc.so.6 [5] #11 0x00007fffe6552bb2 in __assert_fail () from /lib64/libc.so.6 [5] #12 0x00007fffe0aec48a in Legion::Internal::LegionProfInstance::record_barrier_use (this=0x7fc004000ca0, bar=..., uid=0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_profiling.cc:740 [5] #13 0x00007fffe0aefedd in Legion::Internal::LegionProfInstance::record_event_wait (this=0x7fc004000ca0, event=..., bt=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_profiling.cc:1517 [5] #14 0x00007fffe0e2c293 in Legion::Internal::LgEvent::record_event_wait (this=0x7fc01678a080, bt=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:146 [5] #15 0x00007fffe068f9de in Legion::Internal::LgEvent::wait (this=0x7fc01678a080) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3357 [5] #16 0x00007fffe08c733e in Legion::Internal::ReplicateContext::perform_semantic_attach (this=0x7fc00a8174e0, func=0x7fffe1e7fe17 'attach_semantic_information\000', kind=79, arg=0x7fc01678a108, arglen=12, tag=2, buffer=0x7fffe57769c6, size=5, is_mutable=false, global=@0x7fc01678a12f: .FALSE., arg2=0x0, arg2len=0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_context.cc:13545 [5] #17 0x00007fffe0e93ebd in Legion::Internal::Runtime::attach_semantic_information (this=0x5ad3a30, handle=..., tag=2, buffer=0x7fffe57769c6, size=5, is_mutable=false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:24048 [5] #18 0x00007fffe09c17ae in Legion::Runtime::attach_semantic_information (this=0x59c0810, handle=..., tag=2, buffer=0x7fffe57769c6, size=5, is_mut=false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion.cc:7000 [5] #19 0x00007fffe07f1812 in legion_index_space_attach_semantic_information (runtime_=..., handle_=..., tag=2, buffer=0x7fffe57769c6, size=5, is_mutable=false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_c.cc:799 [5] #20 0x00007fffe5679ae3 in $
() from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//build/ammonia/libregent_tasks.so [5] #21 0x00007fffe5679608 in $__regent_task_main_primary () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//build/ammonia/libregent_tasks.so [5] #22 0x00007fffdc7df147 in Realm::LocalTaskProcessor::execute_task (this=0x5613810, func_id=1509, task_args=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/proc_impl.cc:1188 [5] #23 0x00007fffdc856d98 in Realm::Task::execute_on_processor (this=0x7fc00a822b90, p=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:330 [5] #24 0x00007fffdc85ad56 in Realm::KernelThreadTaskScheduler::execute_task (this=0x3ff5160, task=0x7fc00a822b90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1424 [5] #25 0x00007fffdc859b91 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1163 [5] #26 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [5] #27 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #28 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fef7d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #29 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #30 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 7 (Thread 0x7ff01dff4700 (LWP 181828) "s3d.x"): [5] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff01dff3ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [5] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff01dff3ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [5] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x599b5d0, old_counter=36) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [5] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x599b410, old_work_counter=36) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [5] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x599b410, old_work_counter=36) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [5] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x599b410) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [5] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x599b410) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [5] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x599b410) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fa6f80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 6 (Thread 0x7ff01eff9700 (LWP 181824) "s3d.x"): [5] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff01eff8ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [5] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff01eff8ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [5] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x5b6c580, old_counter=36) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [5] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x5b6c3c0, old_work_counter=36) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [5] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x5b6c3c0, old_work_counter=36) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [5] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5b6c3c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [5] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5b6c3c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [5] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x5b6c3c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fa6b80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 5 (Thread 0x7ffd123fe700 (LWP 181816) "s3d.x"): [5] #0 _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1316 [5] #1 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [5] #2 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [5] #3 0x00007fffdc8e14bc in Realm::GASNetEXPoller::do_work (this=0x529fe50, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [5] #4 0x00007fffdc69af3b in Realm::BackgroundWorkManager::Worker::do_work (this=0x7ffd123fc110, max_time_in_ns=-1, interrupt_flag=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [5] #5 0x00007fffdc698bb3 in Realm::BackgroundWorkThread::main_loop (this=0x5a143d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [5] #6 0x00007fffdc69c174 in Realm::Thread::thread_entry_wrapper (obj=0x5a143d0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #7 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x56230a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #8 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #9 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 4 (Thread 0x7ffd213f2700 (LWP 181811) "s3d.x"): [5] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd213f1ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [5] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd213f1ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [5] #3 0x00007fffdc698d17 in Realm::BackgroundWorkThread::main_loop (this=0x5a14070) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [5] #4 0x00007fffdc69c174 in Realm::Thread::thread_entry_wrapper (obj=0x5a14070) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #5 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x5bc9580) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #6 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #7 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 3 (Thread 0x7ffd27504700 (LWP 181807) "s3d.x"): [5] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [5] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd27503ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [5] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd27503ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [5] #3 0x00007fffdc698d17 in Realm::BackgroundWorkThread::main_loop (this=0x5654fd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [5] #4 0x00007fffdc69c174 in Realm::Thread::thread_entry_wrapper (obj=0x5654fd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [5] #5 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x5953dc0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [5] #6 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #7 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 2 (Thread 0x7fffce5ff6c0 (LWP 181796) "s3d.x"): [5] #0 0x00007fffe662cf9b in ioctl () from /lib64/libc.so.6 [5] #1 0x00007fffd39786d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [5] #2 0x00007fffd3971a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [5] #3 0x00007fffd3905f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [5] #4 0x00007fffd38df92e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [5] #5 0x00007fffd38f7b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [5] #6 0x00007fffd38a99e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [5] #7 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [5] #8 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [5] [5] Thread 1 (Thread 0x7fffd18c5700 (LWP 181777) "s3d.x"): [5] #0 0x00007fffe65a6cce in __futex_abstimed_wait_common () from /lib64/libc.so.6 [5] #1 0x00007fffe65a9910 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 [5] #2 0x00007fffdc90c622 in Realm::KernelCondVar::wait (this=0x7fed778) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:1087 [5] #3 0x00007fffdc6bca95 in Realm::BarrierImpl::external_wait (this=0x7fed6a0, gen_needed=2, poisoned=@0x7fffffff398f: false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/barrier_impl.cc:994 [5] #4 0x00007fffdc6a63e7 in Realm::Event::external_wait_faultaware (this=0x410ea20, poisoned=@0x7fffffff398f: false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/event_impl.cc:282 [5] #5 0x00007fffe0907fb4 in Legion::Internal::LgEvent::wait_faultaware (this=0x410ea20, poisoned=@0x7fffffff398f: false, from_app=true) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3459 [5] #6 0x00007fffe0907b2a in Legion::Internal::ApEvent::wait_faultignorant (this=0x410ea20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:2966 [5] #7 0x00007fffe0e4c784 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion (this=0x410ea10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:7112 [5] #8 0x00007fffe09aee65 in Legion::LegionHandshake::ext_wait_on_legion (this=0x4111260) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion.cc:2285 [5] #9 0x00000000005da171 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:229 [5] #10 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [5] #11 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [5] #12 0x00007fffe6543eec in __libc_start_call_main () from /lib64/libc.so.6 [5] #13 0x00007fffe6543fb5 in __libc_start_main_impl () from /lib64/libc.so.6 [5] #14 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [5] [Inferior 1 (process 181777) detached] [6] /usr/bin/gdb -nx -batch -x /tmp/gasnet_fsGxTh '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_2/s3d.x' 181778 [6] [New LWP 181793] [6] [New LWP 181808] [6] [New LWP 181812] [6] [New LWP 181815] [6] [New LWP 181822] [6] [New LWP 181830] [6] [New LWP 181834] [6] [New LWP 181838] [6] [New LWP 181860] [6] [New LWP 181861] [6] [New LWP 181862] [6] [New LWP 181863] [6] [New LWP 181867] [6] [Thread debugging using libthread_db enabled] [6] Using host libthread_db library "/lib64/libthread_db.so.1". [6] 0x00007fffe65a6cce in __futex_abstimed_wait_common () from /lib64/libc.so.6 [6] Id Target Id Frame [6] * 1 Thread 0x7fffd18c5700 (LWP 181778) "s3d.x" 0x00007fffe65a6cce in __futex_abstimed_wait_common () from /lib64/libc.so.6 [6] 2 Thread 0x7fffce5ff6c0 (LWP 181793) "s3d.x" 0x00007fffe662cf9b in ioctl () from /lib64/libc.so.6 [6] 3 Thread 0x7ffd27504700 (LWP 181808) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] 4 Thread 0x7ffd213f2700 (LWP 181812) "s3d.x" 0x00007fffd355759b in cxip_evtq_progress (evtq=0x52f4510) at prov/cxi/src/cxip_evtq.c:375 [6] 5 Thread 0x7ffd123fe700 (LWP 181815) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] 6 Thread 0x7ff01eff9700 (LWP 181822) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] 7 Thread 0x7fc0167fe700 (LWP 181830) "s3d.x" 0x00007fffe661a3a7 in wait4 () from /lib64/libc.so.6 [6] 8 Thread 0x7ff0247fd700 (LWP 181834) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] 9 Thread 0x7ff0245f8700 (LWP 181838) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] 10 Thread 0x7ff01fffe700 (LWP 181860) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] 11 Thread 0x7fc00d3fd700 (LWP 181861) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] 12 Thread 0x7ffd0ddfe700 (LWP 181862) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] 13 Thread 0x7ffd097fe700 (LWP 181863) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] 14 Thread 0x7ff01dff4700 (LWP 181867) "s3d.x" 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] [6] Thread 14 (Thread 0x7ff01dff4700 (LWP 181867) "s3d.x"): [6] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff01dff3ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [6] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff01dff3ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [6] #3 0x00007fffdc90c51d in Realm::FIFOCondVar::wait (this=0x7ff01dff2870) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:1027 [6] #4 0x00007fffdc85aef3 in Realm::KernelThreadTaskScheduler::worker_sleep (this=0x5510f30, switch_to=0x7ff0500) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1472 [6] #5 0x00007fffdc859de7 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5510f30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1200 [6] #6 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5510f30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [6] #7 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x5510f30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #8 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fbff8e67450) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #9 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #10 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 13 (Thread 0x7ffd097fe700 (LWP 181863) "s3d.x"): [6] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd097fdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [6] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd097fdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [6] #3 0x00007fffdc90c1e1 in Realm::UnfairCondVar::wait (this=0x582da58) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [6] #4 0x00007fffdc88bff8 in Realm::Hip::ContextSynchronizer::thread_main (this=0x582da20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [6] #5 0x00007fffdc8a304c in Realm::Thread::thread_entry_wrapper (obj=0x582da20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #6 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fbfec0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #7 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #8 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 12 (Thread 0x7ffd0ddfe700 (LWP 181862) "s3d.x"): [6] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd0ddfdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [6] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd0ddfdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [6] #3 0x00007fffdc90c1e1 in Realm::UnfairCondVar::wait (this=0x5a4d188) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [6] #4 0x00007fffdc88bff8 in Realm::Hip::ContextSynchronizer::thread_main (this=0x5a4d150) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [6] #5 0x00007fffdc8a304c in Realm::Thread::thread_entry_wrapper (obj=0x5a4d150) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #6 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fc0080832a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #7 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #8 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 11 (Thread 0x7fc00d3fd700 (LWP 181861) "s3d.x"): [6] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7fc00d3fcca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [6] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7fc00d3fcca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [6] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x5510b60, old_counter=36) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [6] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x55109a0, old_work_counter=36) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [6] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x55109a0, old_work_counter=36) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [6] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x55109a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [6] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x55109a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [6] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x55109a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fbff8e20190) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 10 (Thread 0x7ff01fffe700 (LWP 181860) "s3d.x"): [6] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff01fffdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [6] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff01fffdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [6] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x3ff5320, old_counter=46) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [6] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x3ff5160, old_work_counter=46) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [6] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x3ff5160, old_work_counter=46) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [6] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [6] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [6] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x3ff5160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fbff4083030) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 9 (Thread 0x7ff0245f8700 (LWP 181838) "s3d.x"): [6] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff0245f7ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [6] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff0245f7ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [6] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x5623220, old_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [6] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x5623060, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [6] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x5623060, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [6] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5623060) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [6] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5623060) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [6] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x5623060) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x8072240) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 8 (Thread 0x7ff0247fd700 (LWP 181834) "s3d.x"): [6] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff0247fcca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [6] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff0247fcca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [6] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x5b69b10, old_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [6] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x5b69950, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [6] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x5b69950, old_work_counter=1) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [6] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5b69950) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [6] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5b69950) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [6] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x5b69950) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7ff0b10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 7 (Thread 0x7fc0167fe700 (LWP 181830) "s3d.x"): [6] #0 0x00007fffe661a3a7 in wait4 () from /lib64/libc.so.6 [6] #1 0x00007fffe656b6cb in do_system () from /lib64/libc.so.6 [6] #2 0x00007fffdd833854 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [6] #3 0x00007fffdd833ea7 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [6] #4 0x00007fffdd8378ae in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [6] #5 0x00007fffdc1b9afd in gasneti_defaultSignalHandler () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [6] #6 [6] #7 0x00007fffe65ac49c in __pthread_kill_implementation () from /lib64/libc.so.6 [6] #8 0x00007fffe655a8c2 in raise () from /lib64/libc.so.6 [6] #9 0x00007fffe654264f in abort () from /lib64/libc.so.6 [6] #10 0x00007fffe6542567 in __assert_fail_base.cold () from /lib64/libc.so.6 [6] #11 0x00007fffe6552bb2 in __assert_fail () from /lib64/libc.so.6 [6] #12 0x00007fffe0aec48a in Legion::Internal::LegionProfInstance::record_barrier_use (this=0x7fbff8e20ea0, bar=..., uid=0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_profiling.cc:740 [6] #13 0x00007fffe0aefedd in Legion::Internal::LegionProfInstance::record_event_wait (this=0x7fbff8e20ea0, event=..., bt=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_profiling.cc:1517 [6] #14 0x00007fffe0e2c293 in Legion::Internal::LgEvent::record_event_wait (this=0x7fc01678a080, bt=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:146 [6] #15 0x00007fffe068f9de in Legion::Internal::LgEvent::wait (this=0x7fc01678a080) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3357 [6] #16 0x00007fffe08c733e in Legion::Internal::ReplicateContext::perform_semantic_attach (this=0x7fbffc040b00, func=0x7fffe1e7fe17 'attach_semantic_information\000', kind=79, arg=0x7fc01678a108, arglen=12, tag=2, buffer=0x7fffe57769c6, size=5, is_mutable=false, global=@0x7fc01678a12f: .FALSE., arg2=0x0, arg2len=0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_context.cc:13545 [6] #17 0x00007fffe0e93ebd in Legion::Internal::Runtime::attach_semantic_information (this=0x5709620, handle=..., tag=2, buffer=0x7fffe57769c6, size=5, is_mutable=false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:24048 [6] #18 0x00007fffe09c17ae in Legion::Runtime::attach_semantic_information (this=0x596c800, handle=..., tag=2, buffer=0x7fffe57769c6, size=5, is_mut=false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion.cc:7000 [6] #19 0x00007fffe07f1812 in legion_index_space_attach_semantic_information (runtime_=..., handle_=..., tag=2, buffer=0x7fffe57769c6, size=5, is_mutable=false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_c.cc:799 [6] #20 0x00007fffe5679ae3 in $
() from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//build/ammonia/libregent_tasks.so [6] #21 0x00007fffe5679608 in $__regent_task_main_primary () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//build/ammonia/libregent_tasks.so [6] #22 0x00007fffdc7df147 in Realm::LocalTaskProcessor::execute_task (this=0x57d2eb0, func_id=1510, task_args=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/proc_impl.cc:1188 [6] #23 0x00007fffdc856d98 in Realm::Task::execute_on_processor (this=0x7fc012751cd0, p=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:330 [6] #24 0x00007fffdc85ad56 in Realm::KernelThreadTaskScheduler::execute_task (this=0x5510f30, task=0x7fc012751cd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1424 [6] #25 0x00007fffdc859b91 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5510f30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1163 [6] #26 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5510f30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [6] #27 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x5510f30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #28 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7ff0500) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #29 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #30 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 6 (Thread 0x7ff01eff9700 (LWP 181822) "s3d.x"): [6] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ff01eff8ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [6] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ff01eff8ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [6] #3 0x00007fffdc857f64 in Realm::ThreadedTaskScheduler::WorkCounter::wait_for_work (this=0x552b320, old_counter=35) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:722 [6] #4 0x00007fffdc85a2b4 in Realm::ThreadedTaskScheduler::wait_for_work (this=0x552b160, old_work_counter=35) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [6] #5 0x00007fffdc85b1aa in Realm::KernelThreadTaskScheduler::wait_for_work (this=0x552b160, old_work_counter=35) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1531 [6] #6 0x00007fffdc85a10d in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x552b160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [6] #7 0x00007fffdc85a1cc in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x552b160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [6] #8 0x00007fffdc86134e in Realm::Thread::thread_entry_wrapper (obj=0x552b160) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #9 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x7fa7870) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #10 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #11 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 5 (Thread 0x7ffd123fe700 (LWP 181815) "s3d.x"): [6] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd123fdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [6] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd123fdca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [6] #3 0x00007fffdc698d17 in Realm::BackgroundWorkThread::main_loop (this=0x562c3b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [6] #4 0x00007fffdc69c174 in Realm::Thread::thread_entry_wrapper (obj=0x562c3b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #5 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x56d83c0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #6 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #7 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 4 (Thread 0x7ffd213f2700 (LWP 181812) "s3d.x"): [6] #0 0x00007fffd355759b in cxip_evtq_progress (evtq=0x52f4510) at prov/cxi/src/cxip_evtq.c:375 [6] #1 0x00007fffd3525a00 in cxip_ep_progress (fid=) at prov/cxi/src/cxip_ep.c:183 [6] #2 0x00007fffd352f219 in cxip_util_cq_progress (util_cq=0x52a5640) at prov/cxi/src/cxip_cq.c:123 [6] #3 0x00007fffd3501f00 in ofi_cq_readfrom (cq_fid=0x52a5640, buf=0x7ffd213ef2e0, count=64, src_addr=0x0) at prov/util/src/util_cq.c:270 [6] #4 0x00007fffdd82d30c in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [6] #5 0x00007fffdd82d62c in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [6] #6 0x00007fffdd823360 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [6] #7 0x00007fffdd81fc1d in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [6] #8 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [6] #9 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [6] #10 0x00007fffdc8e14bc in Realm::GASNetEXPoller::do_work (this=0x529feb0, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [6] #11 0x00007fffdc69af3b in Realm::BackgroundWorkManager::Worker::do_work (this=0x7ffd213f0110, max_time_in_ns=-1, interrupt_flag=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [6] #12 0x00007fffdc698bb3 in Realm::BackgroundWorkThread::main_loop (this=0x562c030) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [6] #13 0x00007fffdc69c174 in Realm::Thread::thread_entry_wrapper (obj=0x562c030) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #14 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x5bd9aa0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #15 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #16 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 3 (Thread 0x7ffd27504700 (LWP 181808) "s3d.x"): [6] #0 0x00007fffe662f709 in syscall () from /lib64/libc.so.6 [6] #1 0x00007fffdc90afbb in Realm::Doorbell::wait_slow (this=0x7ffd27503ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [6] #2 0x00007fffdc69b2d1 in Realm::Doorbell::wait (this=0x7ffd27503ca0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [6] #3 0x00007fffdc698d17 in Realm::BackgroundWorkThread::main_loop (this=0x562bdb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [6] #4 0x00007fffdc69c174 in Realm::Thread::thread_entry_wrapper (obj=0x562bdb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [6] #5 0x00007fffdc86d4ab in Realm::KernelThread::pthread_entry (data=0x5a8d4a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [6] #6 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #7 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 2 (Thread 0x7fffce5ff6c0 (LWP 181793) "s3d.x"): [6] #0 0x00007fffe662cf9b in ioctl () from /lib64/libc.so.6 [6] #1 0x00007fffd39786d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [6] #2 0x00007fffd3971a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [6] #3 0x00007fffd3905f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [6] #4 0x00007fffd38df92e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [6] #5 0x00007fffd38f7b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [6] #6 0x00007fffd38a99e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [6] #7 0x00007fffe65aa60c in start_thread () from /lib64/libc.so.6 [6] #8 0x00007fffe6631aa8 in clone3 () from /lib64/libc.so.6 [6] [6] Thread 1 (Thread 0x7fffd18c5700 (LWP 181778) "s3d.x"): [6] #0 0x00007fffe65a6cce in __futex_abstimed_wait_common () from /lib64/libc.so.6 [6] #1 0x00007fffe65a9910 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 [6] #2 0x00007fffdc90c622 in Realm::KernelCondVar::wait (this=0x7fee4a8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:1087 [6] #3 0x00007fffdc6bca95 in Realm::BarrierImpl::external_wait (this=0x7fee3d0, gen_needed=2, poisoned=@0x7fffffff398f: false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/barrier_impl.cc:994 [6] #4 0x00007fffdc6a63e7 in Realm::Event::external_wait_faultaware (this=0x40fb500, poisoned=@0x7fffffff398f: false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/event_impl.cc:282 [6] #5 0x00007fffe0907fb4 in Legion::Internal::LgEvent::wait_faultaware (this=0x40fb500, poisoned=@0x7fffffff398f: false, from_app=true) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3459 [6] #6 0x00007fffe0907b2a in Legion::Internal::ApEvent::wait_faultignorant (this=0x40fb500) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:2966 [6] #7 0x00007fffe0e4c784 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion (this=0x40fb4f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:7112 [6] #8 0x00007fffe09aee65 in Legion::LegionHandshake::ext_wait_on_legion (this=0x410dc70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion.cc:2285 [6] #9 0x00000000005da171 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:229 [6] #10 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [6] #11 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [6] #12 0x00007fffe6543eec in __libc_start_call_main () from /lib64/libc.so.6 [6] #13 0x00007fffe6543fb5 in __libc_start_main_impl () from /lib64/libc.so.6 [6] #14 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [6] [Inferior 1 (process 181778) detached] srun: error: frontier00075: tasks 5-6: Aborted (core dumped) srun: Terminating StepId=3514308.0 slurmstepd: error: *** STEP 3514308.0 ON frontier00069 CANCELLED AT 2025-06-13T12:20:28 *** *** Caught a signal (proc 0): SIGTERM(15) *** Caught a signal (proc 4): SIGTERM(15) *** Caught a signal (proc 1): SIGTERM(15) *** Caught a signal (proc 2): SIGTERM(15) *** Caught a signal (proc 3): SIGTERM(15) *** FATAL ERROR (proc 3): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** FATAL ERROR (proc 3): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -86908368(Unspecified error) *** NOTICE (proc 3): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 3): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** NOTICE (proc 3): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 3): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu srun: error: frontier00075: task 7: Aborted (core dumped) aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit srun: error: frontier00069: tasks 0-2: Exited with exit code 255 srun: error: frontier00075: task 4: Exited with exit code 255 srun: error: frontier00069: task 3: Aborted (core dumped)