============================================================================== Hello, and welcome to S3D! ============================================================================== ============================================================== --------------------S3D BUILD CONFIGURATION----------------- Build directory: /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/fortran/ammonia ------------------------------------------------------ S3D Options: Using mixture averaged transport: ON Using ARK integration: OFF Using chemical mechanism: ammonia Using srcs for timescale analysis: OFF ------------------------------------------------------ Defines: SYSTEMCALLWONTWORK SAVEFILEINSEPDIR ARCH_OPTERON MPIIO BLOCKIO MIXAVG FROZENFEED ------------------------------------------------------ FORTRAN COMPILER FLAGS: -cpp -fdefault-double-8 -fdefault-real-8 -fno-range-check -ffree-line-length-none -fallow-argument-mismatch C COMPILER FLAGS: ------------------------------------------------------ S3D Version: ------------------------------------------------------ Chemistry routine header: c**************************************************************************** c ... DO NOT EDIT. AUTOMATICALLY GENERATED FILE. c**************************************************************************** c ... c ... CKWYP-compatible GETRATES: return molar production rates for all c ... species given pressure (P), temperature (T), and mass fractions (Y). c ... c ... Generated by cgetrates 1.0b c ... c**************************************************************************** subroutine getrates(P,T,Y,ICKWRK,RCKWRK,WSPL) ------------------------------------------------------ Transport routine header: !Standard version of tranlib.f. 18Feb 2009 ! (Changes by Mark Fahey, ORNL) ! (Update by Evatt Hawkes, SNL) SUBROUTINE MCABS ------------------------------------------------------ ============================================================== ------------------------------------------------------------------------------ this run started at: date = 06/30/2025 time = 10:26:41 ------------------------------------------------------------------------------ initializing chemkin module... opening chemkin link file... setting chemkin work array lengths... allocating chemkin work arrays... filling chemkin work arrays based on chemkin link file... CKLIB: CHEMKIN-III GAS-PHASE CHEMICAL KINETICS LIBRARY, DOUBLE PRECISION Vers. 5.36 2000/12/15 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 the following was extracted from the ../input/chem.asc file: element names are as follows: index name 1 H 2 O 3 N species names are as follows: index name 1 H2 2 NH3 3 NO 4 N2O 5 O2 6 H 7 O 8 OH 9 HO2 10 H2O 11 H2O2 12 NO2 13 HNO 14 N 15 NNH 16 NH2 17 NH 18 H2NO 19 N2 species molecular weights are as follows: index name value (kg/mol) 1 H2 2.01594E-03 2 NH3 1.70306E-02 3 NO 3.00061E-02 4 N2O 4.40128E-02 5 O2 3.19988E-02 6 H 1.00797E-03 7 O 1.59994E-02 8 OH 1.70074E-02 9 HO2 3.30068E-02 10 H2O 1.80153E-02 11 H2O2 3.40147E-02 12 NO2 4.60055E-02 13 HNO 3.10141E-02 14 N 1.40067E-02 15 NNH 2.90214E-02 16 NH2 1.60226E-02 17 NH 1.50147E-02 18 H2NO 3.20220E-02 19 N2 2.80134E-02 ------------------------------------------------------------------------------ initializing param module... number of elements in reaction mechansim = 3 number of species in reaction mechansim = 19 number of steps in reaction mechansims = 63 number of reaction third-body reactions = 11 ------------------------------------------------------------------------------ initializing topology module... Initialized topology module, npx = 4 npy = 4 npz = 4 npes = 64 ------------------------------------------------------------------------------ initializing reference module... the various reference values are as follows: universal gas constant (J/mol-K) = 8.31400E+00 freestream temperature (K) = 8.01900E+02 reference ratio of specifice heats = 1.36400E+00 reference speed of sound (m/s) = 5.28000E+02 reference density (kg/m^3) = 1.22300E+01 reference conductivity (W/m-K) = 5.76900E-02 reference temperature (K) = 2.91892E+02 reference pressure (Pa) = 3.40953E+06 standard atmospheric pressure (Pa) = 1.01325E+05 reference time (s) = 1.25428E-06 reference specific heat (J/kg-K) = 9.55094E+02 reference length (m) = 6.62258E-04 reference viscosity (Pa-s) = 4.27649E+00 acoustic Reynolds number = 1.00000E+05 Mach number = 1.00000E-03 convective Reynolds number = 1.00000E+02 ------------------------------------------------------------------------------ initializing ERK module... ERK module information is as follows: rk_method = -64 cont_switch = 0 tstep_init (seconds) = 5.00E-10 tstep_min (seconds) = 5.00E-10 tstep_max (seconds) = 5.00E-10 rk_rtol = 1.00E-03 rk_atol = 1.00E-15 cont_safety = 9.00E-01 k_I = 2.00E-01 k_P = 2.00E-01 k_D = 5.00E-02 k_D2 = 0.00E+00 cont_factor = 3.00E+00 cfl_switch = 1 i_time_cont = 20 cont_n_reg = 1 controller type = NONE rk_tol = 1.00E-12 using RK4(3)6[2R+]C ERK scheme! ------------------------------------------------------------------------------ initializing transport module... opening mixavg.in ... Lagging is switched on WARNING - with coefficient lagging, theoretical temporal convergence order may not be observed opening transport link file... setting lengths of transport work arrays... allocating transport work arrays... creating transport work arrays from transport link file... TRANLIB: CHEMKIN-III MULTICOMPONENT TRANSPORT LIBRARY, DOUBLE PRECISION Vers. 4.6 1999/06/03 Copyright 1995, Sandia Corporation. The U.S. Government retains a limited license in this software. ------------------------------------------------------------------------------ initializing filter module... using explicit filter of order 10 filter will be applied every 10 time steps ------------------------------------------------------------------------------ initializing derivative module... ------------------------------------------------------------------------------ initializing thermchem module... ------------------------------------------------------------------------------ initializing grid module... ------------------------------------------------------------------------------ initializing boundary condition module... ------------------------------------------------------------------------------ *** WARNING: ofi-conduit failed to configure FI_CXI_* environment variables due to prior conflicting settings. This may lead to unstable behavior and/or degraded performance. If you did not intentionally set FI_CXI_RX_MATCH_MODE='software', FI_CXI_RDZV_THRESHOLD='16384', or FI_CXI_RDZV_GET_MIN then this condition may have resulted from initializing MPI prior to initialization of GASNet. For more information on that scenario, please see "Limits to MPI interoperability" in the ofi-conduit README. Legion Init: Mechanism = 'ammonia' Local Grid Size = ( 64, 64, 256) Global Grid Size = (256, 256, 1024) Proc Grid Size = ( 4, 4, 4) Proc ID = ( 0, 0, 0) nvar_tot = 23, n_spec = 19, n_reg = 3, n_scalar = 0 iorder = 8, iforder = 10, vary = XYZ fuse fields set to 200 from environment set mapper mode to 'allgpu' from environment RHST_ARGS: -ll:gpu 2 -ll:cpu 1 -ll:util 3 -ll:bgwork 3 -ll:fsize 16384 -ll:csize 4096 -ll:zsize 1024 -ll:stacksize 16 -logfile run_%.log -hip:hostreg 0 -lg:no_transitive_reduction 1 -lg:prof 64 -lg:prof_logfile prof_%.gz REGULAR STENCIL initializing variables... Host: frontier00125 Rank: 0 PID: 1574430 Time: 10:26:46 STARTING MAIN TASK! initializing bomb (stand back) Mixture fraction module successfully initialized... storing hard inflow conditions... ------------------------------------------------------------------------------ continuing with initialization... ------------------------------------------------------------------------------ storing hard inflow conditions... maximum speed of sound = 839.70 m/s mimimum speed of sound = 839.70 m/s maximum acoustic time = 2.06E-05 sec minimum acoustic time = 2.06E-05 sec initialization completed for time = 0.00E+00 ------------------------------------------------------------------------------ writing save files for: i_time = 0, time = 0.0000E+00 (sec) precision is d *** Caught a fatal signal (proc 40): SIGSEGV(11) *** NOTICE (proc 40): We recommend linking the debug version of GASNet to assist you in resolving this application issue. [40] Invoking GDB for backtrace... [40] /usr/bin/gdb -nx -batch -x /tmp/gasnet_baVHWL '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_16/s3d.x' 318182 [40] [New LWP 318201] [40] [New LWP 318210] [40] [New LWP 318214] [40] [New LWP 318220] [40] [New LWP 318223] [40] [New LWP 318225] [40] [New LWP 318229] [40] [New LWP 318232] [40] [New LWP 318234] [40] [New LWP 318235] [40] [New LWP 318252] [40] [New LWP 318253] [40] [Thread debugging using libthread_db enabled] [40] Using host libthread_db library "/lib64/libthread_db.so.1". [40] 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [40] Id Target Id Frame [40] * 1 Thread 0x7fffd5a20600 (LWP 318182) "s3d.x" 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [40] 2 Thread 0x7fffcf5ff6c0 (LWP 318201) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [40] 3 Thread 0x7ffd783fc600 (LWP 318210) "s3d.x" 0x00007fffdf820a2d in __rdtsc () at /usr/lib64/gcc/x86_64-suse-linux/13/include/ia32intrin.h:114 [40] 4 Thread 0x7ffd721f3600 (LWP 318214) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] 5 Thread 0x7ffd631ff600 (LWP 318220) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] 6 Thread 0x7ffd8280f600 (LWP 318223) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] 7 Thread 0x7ffd8261f600 (LWP 318225) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [40] 8 Thread 0x7ffd82613600 (LWP 318229) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] 9 Thread 0x7ffd8243f600 (LWP 318232) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] 10 Thread 0x7ff0743ff600 (LWP 318234) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] 11 Thread 0x7ff06c57d600 (LWP 318235) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] 12 Thread 0x7ffd5ebff600 (LWP 318252) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] 13 Thread 0x7ffd5a5ff600 (LWP 318253) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] [40] Thread 13 (Thread 0x7ffd5a5ff600 (LWP 318253) "s3d.x"): [40] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [40] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [40] #3 Realm::UnfairCondVar::wait (this=this@entry=0x587bdc8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [40] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x587bda8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [40] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0480013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [40] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [40] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [40] [40] Thread 12 (Thread 0x7ffd5ebff600 (LWP 318252) "s3d.x"): [40] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5ebfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [40] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5ebfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [40] #3 Realm::UnfairCondVar::wait (this=this@entry=0x584c168) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [40] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x584c148) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [40] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [40] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [40] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [40] [40] Thread 11 (Thread 0x7ff06c57d600 (LWP 318235) "s3d.x"): [40] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff06c57cba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [40] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [40] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5a280e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [40] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5a280e0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [40] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5a280e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [40] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5a280e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [40] #7 Realm::Thread::thread_entry_wrapper (obj=0x5a280e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [40] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x82ccf20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [40] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [40] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [40] [40] Thread 10 (Thread 0x7ff0743ff600 (LWP 318234) "s3d.x"): [40] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff0743feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [40] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [40] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x55c31e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [40] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x55c31e0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [40] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x55c31e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [40] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x55c31e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [40] #7 Realm::Thread::thread_entry_wrapper (obj=0x55c31e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [40] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5890c90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [40] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [40] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [40] [40] Thread 9 (Thread 0x7ffd8243f600 (LWP 318232) "s3d.x"): [40] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8243eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [40] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [40] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56e8600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [40] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56e8600, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [40] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56e8600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [40] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [40] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [40] #8 0x0000000000000000 in ?? () [40] [40] Thread 8 (Thread 0x7ffd82613600 (LWP 318229) "s3d.x"): [40] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd82612ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [40] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [40] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56e7020) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [40] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56e7020, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [40] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56e7020) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [40] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [40] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [40] #8 0x0000000000000000 in ?? () [40] [40] Thread 7 (Thread 0x7ffd8261f600 (LWP 318225) "s3d.x"): [40] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [40] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [40] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [40] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [40] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [40] #5 0x00007fffdf46fa28 in gasneti_defaultSignalHandler () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [40] #6 [40] #7 0x00007fffe185550b in Legion::Internal::LegionProfInstance::record_mapper_call (this=0x0, mapper=0, mapper_proc=..., kind=Legion::Internal::CONFIGURE_CONTEXT_CALL, uid=744, start=6539245857, stop=6546615985) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_profiling.cc:1457 [40] #8 0x00007fffe19ceab6 in Legion::Internal::SerializingManager::finish_mapper_call (this=0x584df20, info=0x7fc03fffd670) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/mapper_manager.cc:1156 [40] #9 0x00007fffe19cbde4 in Legion::Internal::MappingCallInfo::~MappingCallInfo (this=0x7fc03fffd670, __in_chrg=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/mapper_manager.cc:58 [40] #10 0x00007fffe19cd227 in Legion::Internal::MapperManager::invoke_configure_context (this=0x584df20, task=0x7fc05a79cf00, output=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/mapper_manager.cc:645 [40] #11 0x00007fffe18f7bdd in Legion::Internal::SingleTask::configure_execution_context (this=0x7fc05a79cf00, context_configuration=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_tasks.cc:3613 [40] #12 0x00007fffe190903f in Legion::Internal::SingleTask::create_execution_context (this=0x7fc05a79cf00, v=0x49c6580, launch_events=..., inline_task=false, leaf_task=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_tasks.cc:5655 [40] #13 0x00007fffe1909de2 in Legion::Internal::SingleTask::launch_task (this=0x7fc05a79cf00, inline_task=false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_tasks.cc:5176 [40] #14 0x00007fffe18dd22f in Legion::Internal::SliceTask::launch_task (this=0x7fc05a712fe0, inline_task=false) at /usr/include/c++/13/bits/stl_vector.h:1126 [40] #15 0x00007fffe1910dda in Legion::Internal::PointTask::perform_mapping (must_epoch_owner=, args=, this=0x7fc05a79cf00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_tasks.cc:7926 [40] #16 Legion::Internal::PointTask::perform_mapping (this=0x7fc05a79cf00, must_epoch_owner=, args=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_tasks.cc:7917 [40] #17 0x00007fffe18ed489 in Legion::Internal::SingleTask::trigger_mapping (this=0x7fc05a79cf00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_tasks.cc:2476 [40] #18 0x00007fffe1aeb1be in Legion::Internal::Runtime::legion_runtime_task (args=0x7fc05a79e230, arglen=12, userdata=, userlen=, p=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:36276 [40] #19 0x00007fffdf93f7ab in Realm::Task::execute_on_processor (this=0x7fc05a79e0b0, p=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:330 [40] #20 0x00007fffdf93f8a6 in Realm::UserThreadTaskScheduler::execute_task (this=, task=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1690 [40] #21 0x00007fffdf93dc8f in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56e6d30) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1161 [40] #22 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [40] #23 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [40] #24 0x0000000000000000 in ?? () [40] [40] Thread 6 (Thread 0x7ffd8280f600 (LWP 318223) "s3d.x"): [40] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8280eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [40] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [40] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5ff9610) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [40] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5ff9610, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [40] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5ff9610) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [40] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [40] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [40] #8 0x0000000000000000 in ?? () [40] [40] Thread 5 (Thread 0x7ffd631ff600 (LWP 318220) "s3d.x"): [40] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd631feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [40] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [40] #3 Realm::BackgroundWorkThread::main_loop (this=0x56a9520) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [40] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x56a9810) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [40] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [40] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [40] [40] Thread 4 (Thread 0x7ffd721f3600 (LWP 318214) "s3d.x"): [40] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [40] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd721f2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [40] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [40] #3 Realm::BackgroundWorkThread::main_loop (this=0x56c3ed0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [40] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x56a9600) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [40] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [40] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [40] [40] Thread 3 (Thread 0x7ffd783fc600 (LWP 318210) "s3d.x"): [40] #0 0x00007fffdf820a2d in __rdtsc () at /usr/lib64/gcc/x86_64-suse-linux/13/include/ia32intrin.h:114 [40] #1 Realm::Clock::raw_cpu_tsc () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/timers.inl:127 [40] #2 Realm::Clock::native_time () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/timers.inl:91 [40] #3 Realm::Clock::current_time_in_nanoseconds (absolute=true) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/timers.inl:76 [40] #4 Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd783fa5c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:606 [40] #5 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x5b9dc10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [40] #6 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x56a93f0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [40] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [40] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [40] [40] Thread 2 (Thread 0x7fffcf5ff6c0 (LWP 318201) "s3d.x"): [40] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [40] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [40] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [40] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [40] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [40] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [40] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [40] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [40] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [40] [40] Thread 1 (Thread 0x7fffd5a20600 (LWP 318182) "s3d.x"): [40] #0 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [40] #1 0x00007fffe65aa548 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 [40] #2 0x00007fffdf83d7fd in Realm::BarrierImpl::external_wait (this=0x8283c20, gen_needed=2, poisoned=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/barrier_impl.cc:1166 [40] #3 0x00007fffdf82cea5 in Realm::Event::external_wait_faultaware (this=0x40d8760, poisoned=@0x7fffffff3950: false) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/event_impl.cc:282 [40] #4 0x00007fffe174cf75 in Legion::Internal::LgEvent::wait_faultaware (this=0x40d8760, poisoned=@0x7fffffff3950: false, from_app=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3459 [40] #5 0x00007fffe1ad0dfa in Legion::Internal::ApEvent::wait_faultignorant (this=0x40d8760) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:2966 [40] #6 Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion (this=0x40d8750) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:7112 [40] #7 0x00000000005da171 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:229 [40] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [40] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [40] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [40] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [40] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [40] [Inferior 1 (process 318182) detached] srun: error: frontier05631: task 40: Segmentation fault (core dumped) srun: Terminating StepId=3546542.0 slurmstepd: error: *** STEP 3546542.0 ON frontier00125 CANCELLED AT 2025-06-30T10:27:38 *** *** Caught a signal (proc 41): SIGTERM(15) *** Caught a signal (proc 42): SIGTERM(15) *** Caught a signal (proc 43): SIGTERM(15) *** Caught a signal (proc 0): SIGTERM(15) *** Caught a signal (proc 1): SIGTERM(15) *** Caught a signal (proc 2): SIGTERM(15) *** Caught a signal (proc 3): SIGTERM(15) *** Caught a signal (proc 4): SIGTERM(15) *** Caught a signal (proc 28): SIGTERM(15) *** Caught a signal (proc 5): SIGTERM(15) *** Caught a signal (proc 29): SIGTERM(15) *** Caught a signal (proc 6): SIGTERM(15) *** Caught a signal (proc 30): SIGTERM(15) *** Caught a signal (proc 7): SIGTERM(15) *** Caught a signal (proc 48): SIGTERM(15) *** Caught a signal (proc 31): SIGTERM(15) *** Caught a signal (proc 49): SIGTERM(15) *** Caught a signal (proc 50): SIGTERM(15) *** Caught a signal (proc 32): SIGTERM(15) *** Caught a signal (proc 51): SIGTERM(15) *** Caught a signal (proc 8): SIGTERM(15) *** Caught a signal (proc 12): SIGTERM(15) *** Caught a signal (proc 9): SIGTERM(15) *** Caught a signal (proc 33): SIGTERM(15) *** Caught a signal (proc 13): SIGTERM(15) *** Caught a signal (proc 10): SIGTERM(15) *** Caught a signal (proc 34): SIGTERM(15) *** Caught a signal (proc 14): SIGTERM(15) *** Caught a signal (proc 35): SIGTERM(15) *** Caught a signal (proc 60): SIGTERM(15) *** Caught a signal (proc 20): SIGTERM(15) *** Caught a signal (proc 24): SIGTERM(15) *** Caught a signal (proc 61): SIGTERM(15) *** Caught a signal (proc 21): SIGTERM(15) *** Caught a signal (proc 11): SIGTERM(15) *** Caught a signal (proc 25): SIGTERM(15) *** Caught a signal (proc 62): SIGTERM(15) *** Caught a signal (proc 15): SIGTERM(15) *** Caught a signal (proc 22): SIGTERM(15) *** Caught a signal (proc 26): SIGTERM(15) *** Caught a signal (proc 23): SIGTERM(15) *** Caught a signal (proc 27): SIGTERM(15) *** Caught a signal (proc 52): SIGTERM(15) *** Caught a signal (proc 53): SIGTERM(15) *** FATAL ERROR (proc 32): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 32): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 32): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** Caught a signal (proc 54): SIGTERM(15) *** Caught a signal (proc 16): SIGTERM(15) *** Caught a signal (proc 63): SIGTERM(15) *** Caught a signal (proc 17): SIGTERM(15) [32] Invoking GDB for backtrace... *** Caught a signal (proc 55): SIGTERM(15) *** Caught a signal (proc 18): SIGTERM(15) *** Caught a signal (proc 19): SIGTERM(15) *** Caught a signal (proc 56): SIGTERM(15) *** Caught a signal (proc 57): SIGTERM(15) *** Caught a signal (proc 58): SIGTERM(15) *** Caught a signal (proc 59): SIGTERM(15) *** FATAL ERROR (proc 24): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 24): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 24): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** FATAL ERROR (proc 24): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -3(No such process) *** Details for bug reporting (proc 24): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu [24] Invoking GDB for backtrace... *** Caught a signal (proc 36): SIGTERM(15) *** FATAL ERROR (proc 36): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 36): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 36): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** FATAL ERROR (proc 36): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -1916734016(Unspecified error) *** Details for bug reporting (proc 36): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** Caught a signal (proc 44): SIGTERM(15) [36] Invoking GDB for backtrace... *** Caught a signal (proc 45): SIGTERM(15) *** Caught a signal (proc 37): SIGTERM(15) *** Caught a signal (proc 46): SIGTERM(15) *** Caught a signal (proc 38): SIGTERM(15) *** FATAL ERROR (proc 38): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 38): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 38): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** FATAL ERROR (proc 38): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -1(Operation not permitted) *** Details for bug reporting (proc 38): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu *** Caught a signal (proc 47): SIGTERM(15) [38] Invoking GDB for backtrace... *** Caught a signal (proc 39): SIGTERM(15) *** FATAL ERROR (proc 39): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 39): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 39): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu [39] Invoking GDB for backtrace... *** FATAL ERROR (proc 8): in gasnetc_ofi_tx_poll_one() at subrank/gasnet/GASNet-2024.5.0/ofi-conduit/gasnet_ofi.c:2171: fi_cq_read for tx_poll failed with error: -107(Transport endpoint is not connected) *** NOTICE (proc 8): We recommend linking the debug version of GASNet to assist you in resolving this application issue. *** Details for bug reporting (proc 8): config=RELEASE=2024.5.0,SPEC=1.20,PTR=64bit,nodebug,PAR,timers_native,membars_native,atomics_native,atomic32_native,atomic64_native compiler=GNU/13.2.1 sys=x86_64-pc-linux-gnu [8] Invoking GDB for backtrace... [8] /usr/bin/gdb -nx -batch -x /tmp/gasnet_41kCIF '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_16/s3d.x' 254040 [8] [New LWP 254059] [8] [New LWP 254063] [8] [New LWP 254068] [8] [New LWP 254072] [8] [New LWP 254076] [8] [New LWP 254079] [8] [New LWP 254081] [8] [New LWP 254083] [8] [New LWP 254087] [8] [New LWP 254091] [8] [New LWP 254105] [8] [New LWP 254106] [8] [Thread debugging using libthread_db enabled] [8] Using host libthread_db library "/lib64/libthread_db.so.1". [8] 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [8] Id Target Id Frame [8] * 1 Thread 0x7fffd5a20600 (LWP 254040) "s3d.x" 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [8] 2 Thread 0x7fffcf5ff6c0 (LWP 254059) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [8] 3 Thread 0x7ffd723f3600 (LWP 254063) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] 4 Thread 0x7ffd633ff600 (LWP 254068) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] 5 Thread 0x7ffd5edff600 (LWP 254072) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [8] 6 Thread 0x7ffd8270f600 (LWP 254076) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] 7 Thread 0x7ffd8243f600 (LWP 254079) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] 8 Thread 0x7ffd7e31f600 (LWP 254081) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] 9 Thread 0x7ffd7e313600 (LWP 254083) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] 10 Thread 0x7ff0745ff600 (LWP 254087) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] 11 Thread 0x7ff0743fb600 (LWP 254091) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] 12 Thread 0x7ffd5a7ff600 (LWP 254105) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] 13 Thread 0x7ffd561ff600 (LWP 254106) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] [8] Thread 13 (Thread 0x7ffd561ff600 (LWP 254106) "s3d.x"): [8] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd561feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [8] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd561feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [8] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5a12388) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [8] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5a12368) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [8] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc04c0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [8] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [8] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [8] [8] Thread 12 (Thread 0x7ffd5a7ff600 (LWP 254105) "s3d.x"): [8] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [8] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [8] #3 Realm::UnfairCondVar::wait (this=this@entry=0x55adc28) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [8] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x55adc08) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [8] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0480013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [8] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [8] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [8] [8] Thread 11 (Thread 0x7ff0743fb600 (LWP 254091) "s3d.x"): [8] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff0743faba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [8] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [8] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5c0b960) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [8] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5c0b960, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [8] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5c0b960) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [8] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5c0b960) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [8] #7 Realm::Thread::thread_entry_wrapper (obj=0x5c0b960) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [8] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5bba5a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [8] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [8] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [8] [8] Thread 10 (Thread 0x7ff0745ff600 (LWP 254087) "s3d.x"): [8] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff0745feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [8] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [8] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5ca2af0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [8] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5ca2af0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [8] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5ca2af0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [8] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5ca2af0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [8] #7 Realm::Thread::thread_entry_wrapper (obj=0x5ca2af0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [8] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5bb9c80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [8] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [8] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [8] [8] Thread 9 (Thread 0x7ffd7e313600 (LWP 254083) "s3d.x"): [8] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7e312ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [8] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [8] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5673360) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [8] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5673360, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [8] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5673360) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [8] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [8] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [8] #8 0x0000000000000000 in ?? () [8] [8] Thread 8 (Thread 0x7ffd7e31f600 (LWP 254081) "s3d.x"): [8] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7e31eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [8] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [8] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56e8760) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [8] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56e8760, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [8] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56e8760) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [8] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [8] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [8] #8 0x0000000000000000 in ?? () [8] [8] Thread 7 (Thread 0x7ffd8243f600 (LWP 254079) "s3d.x"): [8] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8243eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [8] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [8] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56e7180) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [8] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56e7180, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [8] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56e7180) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [8] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [8] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [8] #8 0x0000000000000000 in ?? () [8] [8] Thread 6 (Thread 0x7ffd8270f600 (LWP 254076) "s3d.x"): [8] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8270eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [8] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [8] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56e6e90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [8] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56e6e90, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [8] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56e6e90) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [8] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [8] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [8] #8 0x0000000000000000 in ?? () [8] [8] Thread 5 (Thread 0x7ffd5edff600 (LWP 254072) "s3d.x"): [8] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [8] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [8] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #5 0x00007fffdf464ff7 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #6 0x00007fffdf4650e8 in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #7 0x00007fffe08861e2 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #8 0x00007fffe088631c in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #9 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #10 0x00007fffe087c52f in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #11 0x00007fffe087c92e in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #12 0x00007fffe0886dc0 in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #13 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [8] #14 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [8] #15 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [8] #16 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [8] #17 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x52671a0, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [8] #18 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd5edfd5c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [8] #19 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x5b780b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [8] #20 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5b783e0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [8] #21 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [8] #22 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [8] [8] Thread 4 (Thread 0x7ffd633ff600 (LWP 254068) "s3d.x"): [8] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd633feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [8] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [8] #3 Realm::BackgroundWorkThread::main_loop (this=0x5b77ea0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [8] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5b78190) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [8] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [8] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [8] [8] Thread 3 (Thread 0x7ffd723f3600 (LWP 254063) "s3d.x"): [8] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [8] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd723f2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [8] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [8] #3 Realm::BackgroundWorkThread::main_loop (this=0x5677250) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [8] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5b77f80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [8] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [8] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [8] [8] Thread 2 (Thread 0x7fffcf5ff6c0 (LWP 254059) "s3d.x"): [8] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [8] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [8] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [8] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [8] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [8] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [8] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [8] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [8] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [8] [8] Thread 1 (Thread 0x7fffd5a20600 (LWP 254040) "s3d.x"): [8] #0 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [8] #1 0x00007fffe65aa548 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 [8] #2 0x00007fffdf83d7fd in Realm::BarrierImpl::external_wait (this=0x82889c0, gen_needed=2, poisoned=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/barrier_impl.cc:1166 [8] #3 0x00007fffdf82cea5 in Realm::Event::external_wait_faultaware (this=0x40d8760, poisoned=@0x7fffffff3950: .FALSE.) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/event_impl.cc:282 [8] #4 0x00007fffe174cf75 in Legion::Internal::LgEvent::wait_faultaware (this=0x40d8760, poisoned=@0x7fffffff3950: .FALSE., from_app=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3459 [8] #5 0x00007fffe1ad0dfa in Legion::Internal::ApEvent::wait_faultignorant (this=0x40d8760) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:2966 [8] #6 Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion (this=0x40d8750) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:7112 [8] #7 0x00000000005da171 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:229 [8] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [8] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [8] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [8] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [8] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [8] [Inferior 1 (process 254040) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit [32] /usr/bin/gdb -nx -batch -x /tmp/gasnet_pi5nyn '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_16/s3d.x' 261607 [32] [New LWP 261626] [32] [New LWP 261635] [32] [New LWP 261637] [32] [New LWP 261638] [32] [New LWP 261648] [32] [New LWP 261649] [32] [New LWP 261652] [32] [New LWP 261656] [32] [New LWP 261657] [32] [New LWP 261660] [32] [New LWP 261675] [32] [New LWP 261676] [32] [Thread debugging using libthread_db enabled] [32] Using host libthread_db library "/lib64/libthread_db.so.1". [32] 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [32] Id Target Id Frame [32] * 1 Thread 0x7fffd5a20600 (LWP 261607) "s3d.x" 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [32] 2 Thread 0x7fffcf5ff6c0 (LWP 261626) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [32] 3 Thread 0x7ffd783fc600 (LWP 261635) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] 4 Thread 0x7ffd721f3600 (LWP 261637) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] 5 Thread 0x7ffd631ff600 (LWP 261638) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [32] 6 Thread 0x7ffd8280f600 (LWP 261648) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] 7 Thread 0x7ffd8261f600 (LWP 261649) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] 8 Thread 0x7ffd82613600 (LWP 261652) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] 9 Thread 0x7ffd8243f600 (LWP 261656) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] 10 Thread 0x7ff0743ff600 (LWP 261657) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] 11 Thread 0x7ff06c57d600 (LWP 261660) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] 12 Thread 0x7ffd5ebff600 (LWP 261675) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] 13 Thread 0x7ffd5a5ff600 (LWP 261676) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] [32] Thread 13 (Thread 0x7ffd5a5ff600 (LWP 261676) "s3d.x"): [32] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [32] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a5feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [32] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5932458) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [32] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5932438) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [32] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0400013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [32] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [32] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [32] [32] Thread 12 (Thread 0x7ffd5ebff600 (LWP 261675) "s3d.x"): [32] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5ebfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [32] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5ebfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [32] #3 Realm::UnfairCondVar::wait (this=this@entry=0x5551aa8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [32] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x5551a88) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [32] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0480013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [32] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [32] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [32] [32] Thread 11 (Thread 0x7ff06c57d600 (LWP 261660) "s3d.x"): [32] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff06c57cba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [32] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [32] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5a4e430) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [32] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5a4e430, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [32] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5a4e430) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [32] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5a4e430) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [32] #7 Realm::Thread::thread_entry_wrapper (obj=0x5a4e430) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [32] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x589abc0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [32] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [32] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [32] [32] Thread 10 (Thread 0x7ff0743ff600 (LWP 261657) "s3d.x"): [32] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff0743feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [32] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [32] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5ca7b00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [32] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5ca7b00, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [32] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5ca7b00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [32] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5ca7b00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [32] #7 Realm::Thread::thread_entry_wrapper (obj=0x5ca7b00) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [32] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x589d2b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [32] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [32] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [32] [32] Thread 9 (Thread 0x7ffd8243f600 (LWP 261656) "s3d.x"): [32] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8243eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [32] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [32] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x57b0d10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [32] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x57b0d10, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [32] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x57b0d10) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [32] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [32] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [32] #8 0x0000000000000000 in ?? () [32] [32] Thread 8 (Thread 0x7ffd82613600 (LWP 261652) "s3d.x"): [32] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd82612ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [32] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [32] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56ec680) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [32] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56ec680, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [32] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56ec680) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [32] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [32] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [32] #8 0x0000000000000000 in ?? () [32] [32] Thread 7 (Thread 0x7ffd8261f600 (LWP 261649) "s3d.x"): [32] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8261eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [32] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [32] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56eb0a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [32] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56eb0a0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [32] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56eb0a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [32] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [32] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [32] #8 0x0000000000000000 in ?? () [32] [32] Thread 6 (Thread 0x7ffd8280f600 (LWP 261648) "s3d.x"): [32] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8280eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [32] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [32] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56eadb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [32] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56eadb0, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [32] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56eadb0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [32] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [32] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [32] #8 0x0000000000000000 in ?? () [32] [32] Thread 5 (Thread 0x7ffd631ff600 (LWP 261638) "s3d.x"): [32] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [32] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [32] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #5 0x00007fffdf464ff7 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #6 0x00007fffdf4650e8 in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #7 0x00007fffe08861e2 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #8 0x00007fffe088631c in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #9 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #10 0x00007fffe087c52f in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #11 0x00007fffe087c92e in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #12 0x00007fffe0886dc0 in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #13 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [32] #14 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [32] #15 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [32] #16 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [32] #17 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x526b050, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [32] #18 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd631fd5c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [32] #19 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x5ad9d60) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [32] #20 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5ada090) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [32] #21 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [32] #22 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [32] [32] Thread 4 (Thread 0x7ffd721f3600 (LWP 261637) "s3d.x"): [32] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd721f2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [32] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [32] #3 Realm::BackgroundWorkThread::main_loop (this=0x59d78b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [32] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5ad9e40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [32] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [32] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [32] [32] Thread 3 (Thread 0x7ffd783fc600 (LWP 261635) "s3d.x"): [32] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [32] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd783fbba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [32] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [32] #3 Realm::BackgroundWorkThread::main_loop (this=0x57e5850) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [32] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5ad9c70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [32] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [32] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [32] [32] Thread 2 (Thread 0x7fffcf5ff6c0 (LWP 261626) "s3d.x"): [32] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [32] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [32] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [32] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [32] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [32] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [32] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [32] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [32] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [32] [32] Thread 1 (Thread 0x7fffd5a20600 (LWP 261607) "s3d.x"): [32] #0 0x00007fffe65a7c1e in __futex_abstimed_wait_common () from /lib64/libc.so.6 [32] #1 0x00007fffe65aa548 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 [32] #2 0x00007fffdf83d7fd in Realm::BarrierImpl::external_wait (this=0x589b520, gen_needed=2, poisoned=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/barrier_impl.cc:1166 [32] #3 0x00007fffdf82cea5 in Realm::Event::external_wait_faultaware (this=0x40d8760, poisoned=@0x7fffffff3950: .FALSE.) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/event_impl.cc:282 [32] #4 0x00007fffe174cf75 in Legion::Internal::LgEvent::wait_faultaware (this=0x40d8760, poisoned=@0x7fffffff3950: .FALSE., from_app=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3459 [32] #5 0x00007fffe1ad0dfa in Legion::Internal::ApEvent::wait_faultignorant (this=0x40d8760) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:2966 [32] #6 Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion (this=0x40d8750) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:7112 [32] #7 0x00000000005da171 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:229 [32] #8 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [32] #9 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [32] #10 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [32] #11 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [32] #12 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [32] [Inferior 1 (process 261607) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit srun: error: frontier03072: task 8: Exited with exit code 255 [39] /usr/bin/gdb -nx -batch -x /tmp/gasnet_Dcs6Qv '/lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/ammonia_16/s3d.x' 252482 [39] [New LWP 252498] [39] [New LWP 252508] [39] [New LWP 252512] [39] [New LWP 252516] [39] [New LWP 252520] [39] [New LWP 252524] [39] [New LWP 252527] [39] [New LWP 252533] [39] [New LWP 252536] [39] [New LWP 252539] [39] [New LWP 252543] [39] [New LWP 252544] [39] [Thread debugging using libthread_db enabled] [39] Using host libthread_db library "/lib64/libthread_db.so.1". [39] 0x00007fffe65a7ecc in __lll_lock_wait () from /lib64/libc.so.6 [39] Id Target Id Frame [39] * 1 Thread 0x7fffd5a20600 (LWP 252482) "s3d.x" 0x00007fffe65a7ecc in __lll_lock_wait () from /lib64/libc.so.6 [39] 2 Thread 0x7fffd35ff6c0 (LWP 252498) "s3d.x" 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [39] 3 Thread 0x7ffd77f05600 (LWP 252508) "s3d.x" 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [39] 4 Thread 0x7ffd71df3600 (LWP 252512) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] 5 Thread 0x7ffd62dff600 (LWP 252516) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] 6 Thread 0x7ffd8600f600 (LWP 252520) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] 7 Thread 0x7ffd85e3f600 (LWP 252524) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] 8 Thread 0x7ffd81c3f600 (LWP 252527) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] 9 Thread 0x7ffd7db9f600 (LWP 252533) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] 10 Thread 0x7ff06c37d600 (LWP 252536) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] 11 Thread 0x7fc0647fd600 (LWP 252539) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] 12 Thread 0x7ffd5e7ff600 (LWP 252543) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] 13 Thread 0x7ffd5a1ff600 (LWP 252544) "s3d.x" 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] [39] Thread 13 (Thread 0x7ffd5a1ff600 (LWP 252544) "s3d.x"): [39] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [39] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5a1feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [39] #3 Realm::UnfairCondVar::wait (this=this@entry=0x591d8f8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [39] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x591d8d8) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [39] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc0440013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [39] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [39] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [39] [39] Thread 12 (Thread 0x7ffd5e7ff600 (LWP 252543) "s3d.x"): [39] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=this@entry=0x7ffd5e7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [39] #2 0x00007fffdf9bdf08 in Realm::Doorbell::wait (this=0x7ffd5e7feba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [39] #3 Realm::UnfairCondVar::wait (this=this@entry=0x57ef258) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:948 [39] #4 0x00007fffdf9601fb in Realm::Hip::ContextSynchronizer::thread_main (this=0x57ef238) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/hip/hip_module.cc:587 [39] #5 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x7fc03c0013a0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [39] #6 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [39] #7 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [39] [39] Thread 11 (Thread 0x7fc0647fd600 (LWP 252539) "s3d.x"): [39] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7fc0647fcba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [39] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [39] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x577de40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [39] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x577de40, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [39] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x577de40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [39] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x577de40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [39] #7 Realm::Thread::thread_entry_wrapper (obj=0x577de40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [39] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x82bd410) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [39] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [39] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [39] [39] Thread 10 (Thread 0x7ff06c37d600 (LWP 252536) "s3d.x"): [39] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ff06c37cba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [39] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [39] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x5c8f400) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [39] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x5c8f400, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [39] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x5c8f400) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [39] #6 0x00007fffdf941632 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x5c8f400) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1275 [39] #7 Realm::Thread::thread_entry_wrapper (obj=0x5c8f400) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.inl:97 [39] #8 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x587dd20) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [39] #9 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [39] #10 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [39] [39] Thread 9 (Thread 0x7ffd7db9f600 (LWP 252533) "s3d.x"): [39] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd7db9eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [39] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [39] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x552a630) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [39] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x552a630, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [39] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x552a630) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [39] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [39] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [39] #8 0x0000000000000000 in ?? () [39] [39] Thread 8 (Thread 0x7ffd81c3f600 (LWP 252527) "s3d.x"): [39] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd81c3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [39] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [39] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56d1750) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [39] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56d1750, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [39] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56d1750) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [39] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [39] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [39] #8 0x0000000000000000 in ?? () [39] [39] Thread 7 (Thread 0x7ffd85e3f600 (LWP 252524) "s3d.x"): [39] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd85e3eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [39] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [39] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56d0170) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [39] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56d0170, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [39] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56d0170) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [39] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [39] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [39] #8 0x0000000000000000 in ?? () [39] [39] Thread 6 (Thread 0x7ffd8600f600 (LWP 252520) "s3d.x"): [39] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd8600eba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [39] #2 0x00007fffdf936f5f in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [39] #3 0x00007fffdf937018 in Realm::ThreadedTaskScheduler::wait_for_work (old_work_counter=, this=0x56cfe80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1297 [39] #4 Realm::ThreadedTaskScheduler::wait_for_work (this=0x56cfe80, old_work_counter=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1278 [39] #5 0x00007fffdf93e0a3 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x56cfe80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/tasks.cc:1263 [39] #6 0x00007fffdf949b8a in Realm::UserThread::uthread_entry () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:1428 [39] #7 0x00007fffe657227e in ?? () from /lib64/libc.so.6 [39] #8 0x0000000000000000 in ?? () [39] [39] Thread 5 (Thread 0x7ffd62dff600 (LWP 252516) "s3d.x"): [39] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd62dfeba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [39] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [39] #3 Realm::BackgroundWorkThread::main_loop (this=0x5823e80) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [39] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x58241b0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [39] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [39] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [39] [39] Thread 4 (Thread 0x7ffd71df3600 (LWP 252512) "s3d.x"): [39] #0 0x00007fffe662ff99 in syscall () from /lib64/libc.so.6 [39] #1 0x00007fffdf9bd7f5 in Realm::Doorbell::wait_slow (this=0x7ffd71df2ba0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.cc:263 [39] #2 0x00007fffdf821bf5 in Realm::Doorbell::wait (this=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/mutex.inl:81 [39] #3 Realm::BackgroundWorkThread::main_loop (this=0x5823c70) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:144 [39] #4 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5823f60) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [39] #5 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [39] #6 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [39] [39] Thread 3 (Thread 0x7ffd77f05600 (LWP 252508) "s3d.x"): [39] #0 0x00007fffe661ac37 in wait4 () from /lib64/libc.so.6 [39] #1 0x00007fffe656c64b in do_system () from /lib64/libc.so.6 [39] #2 0x00007fffe088c544 in gasneti_system_redirected () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #3 0x00007fffe088cb97 in gasneti_bt_gdb () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #4 0x00007fffe089059e in gasneti_print_backtrace () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #5 0x00007fffdf464ff7 in gasneti_error_abort () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #6 0x00007fffdf4650e8 in _gasneti_fatalerror () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #7 0x00007fffe08861e2 in gasnetc_ofi_tx_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #8 0x00007fffe088631c in gasnetc_ofi_poll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #9 0x00007fffe087c050 in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #10 0x00007fffe087c52f in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #11 0x00007fffe087c92e in gasnetc_exit_reqh () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #12 0x00007fffe089665c in gasneti_AMPSHMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #13 0x00007fffe087c04b in gasnetc_AMPoll () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #14 0x00007fffe087abd6 in _gasneti_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1315 [39] #15 _gasnet_AMPoll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/gasnet/release/include/gasnet_help.h:1528 [39] #16 Realm::GASNetEXWrapper::gex_am_poll () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201 [39] #17 0x00007fffdf9a303b in Realm::GASNetEXPoller::do_work (this=0x525e790, work_until=...) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955 [39] #18 0x00007fffdf820a09 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x7ffd77f035c0, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:600 [39] #19 0x00007fffdf8212a1 in Realm::BackgroundWorkThread::main_loop (this=0x5823c40) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/bgwork.cc:103 [39] #20 0x00007fffdf9442b9 in Realm::KernelThread::pthread_entry (data=0x5823d50) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/threads.cc:852 [39] #21 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [39] #22 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [39] [39] Thread 2 (Thread 0x7fffd35ff6c0 (LWP 252498) "s3d.x"): [39] #0 0x00007fffe662d82b in ioctl () from /lib64/libc.so.6 [39] #1 0x00007fffd6ca96d8 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [39] #2 0x00007fffd6ca2a4e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [39] #3 0x00007fffd6c36f94 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [39] #4 0x00007fffd6c1092e in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [39] #5 0x00007fffd6c28b12 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [39] #6 0x00007fffd6bda9e7 in ?? () from /opt/rocm-6.0.0/lib/libhsa-runtime64.so.1 [39] #7 0x00007fffe65aaf6c in start_thread () from /lib64/libc.so.6 [39] #8 0x00007fffe6632338 in clone3 () from /lib64/libc.so.6 [39] [39] Thread 1 (Thread 0x7fffd5a20600 (LWP 252482) "s3d.x"): [39] #0 0x00007fffe65a7ecc in __lll_lock_wait () from /lib64/libc.so.6 [39] #1 0x00007fffe65ae615 in pthread_mutex_lock@@GLIBC_2.2.5 () from /lib64/libc.so.6 [39] #2 0x00007fffe087c366 in gasnetc_exit () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #3 0x00007fffdf46f9ed in gasneti_defaultSignalHandler () from /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/test_cases/0d_ammonia/../..//legion/language/build/lib/librealm.so.1 [39] #4 [39] #5 0x00007fffe65acdfc in __pthread_kill_implementation () from /lib64/libc.so.6 [39] #6 0x00007fffe655b842 in raise () from /lib64/libc.so.6 [39] #7 [39] #8 0x00007fffe65a7c1c in __futex_abstimed_wait_common () from /lib64/libc.so.6 [39] #9 0x00007fffe65aa548 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6 [39] #10 0x00007fffdf83d7fd in Realm::BarrierImpl::external_wait (this=0x587e680, gen_needed=2, poisoned=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/barrier_impl.cc:1166 [39] #11 0x00007fffdf82cea5 in Realm::Event::external_wait_faultaware (this=0x40d7fe0, poisoned=@0x7fffffff3950: .FALSE.) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/realm/event_impl.cc:282 [39] #12 0x00007fffe174cf75 in Legion::Internal::LgEvent::wait_faultaware (this=0x40d7fe0, poisoned=@0x7fffffff3950: .FALSE., from_app=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:3459 [39] #13 0x00007fffe1ad0dfa in Legion::Internal::ApEvent::wait_faultignorant (this=0x40d7fe0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/legion_types.h:2966 [39] #14 Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion (this=0x40d7fd0) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/legion/runtime/legion/runtime.cc:7112 [39] #15 0x00000000005da171 in solve_driver (io=6) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/solve_driver.f90:229 [39] #16 0x00000000005d9a62 in s3d () at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:149 [39] #17 0x000000000040422d in main (argc=, argv=) at /lustre/orion/cmb103/scratch/seshuy/legion_s3d_subrank/s3d/source/drivers/main.f90:8 [39] #18 0x00007fffe6544e6c in __libc_start_call_main () from /lib64/libc.so.6 [39] #19 0x00007fffe6544f35 in __libc_start_main_impl () from /lib64/libc.so.6 [39] #20 0x0000000000404261 in _start () at ../sysdeps/x86_64/start.S:115 [39] [Inferior 1 (process 252482) detached] gasnet_exit(): ERROR: signal 6 received during exit... goodbye. [coordinating shutdown] aborting job: GASNet abnormal exit srun: error: frontier05247: task 32: Exited with exit code 255 srun: error: frontier05504: task 39: Exited with exit code 255 aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit aborting job: GASNet abnormal exit srun: error: frontier05504: task 37: Exited with exit code 255 srun: error: frontier05631: tasks 42-43: Exited with exit code 255 srun: error: frontier05247: tasks 34-35: Exited with exit code 255 srun: error: frontier04992: tasks 25-27: Exited with exit code 255 srun: error: frontier03072: tasks 10-11: Exited with exit code 255 srun: error: frontier05120: tasks 29-31: Exited with exit code 255 srun: error: frontier07039: tasks 56,58-59: Exited with exit code 255 srun: error: frontier03840: tasks 12-14: Exited with exit code 255 srun: error: frontier04607: tasks 20-21,23: Exited with exit code 255 srun: error: frontier06782: tasks 52-53,55: Exited with exit code 255 srun: error: frontier00125: tasks 1-3: Exited with exit code 255 srun: error: frontier02432: tasks 4-6: Exited with exit code 255 srun: error: frontier04223: tasks 17-19: Exited with exit code 255 srun: error: frontier06143: tasks 44-45,47: Exited with exit code 255 srun: error: frontier06526: tasks 49-51: Exited with exit code 255 srun: error: frontier07293: tasks 61-63: Exited with exit code 255 srun: error: frontier05631: task 41: Exited with exit code 255 srun: error: frontier05247: task 33: Exited with exit code 255 srun: error: frontier03072: task 9: Exited with exit code 255 srun: error: frontier05120: task 28: Exited with exit code 255 srun: error: frontier07039: task 57: Exited with exit code 255 srun: error: frontier06526: task 48: Exited with exit code 255 srun: error: frontier02432: task 7: Exited with exit code 255 srun: error: frontier04607: task 22: Exited with exit code 255 srun: error: frontier06782: task 54: Exited with exit code 255 srun: error: frontier06143: task 46: Exited with exit code 255 srun: error: frontier04223: task 16: Exited with exit code 255 srun: error: frontier03840: task 15: Exited with exit code 255 srun: error: frontier00125: task 0: Exited with exit code 255 srun: error: frontier07293: task 60: Exited with exit code 255 srun: error: frontier05504: task 36: Killed srun: error: frontier05504: task 38: Killed srun: error: frontier04992: task 24: Aborted (core dumped) srun: Force Terminated StepId=3546542.0