%MSG-i AlpakaService: (NoModuleName) 07-Nov-2025 08:05:40 PST pre-events AlpakaServiceSerialSync succesfully initialised. Found 1 device: - AMD EPYC 7763 64-Core Processor %MSG %MSG-i CUDAService: (NoModuleName) 07-Nov-2025 08:05:40 PST pre-events CUDA runtime version 12.9, driver version 12.9, NVIDIA driver version 575.51.03 CUDA device 0: NVIDIA L4 (sm_89) CUDA device 1: NVIDIA A30 (sm_80) %MSG %MSG-i AlpakaService: (NoModuleName) 07-Nov-2025 08:05:40 PST pre-events AlpakaServiceCudaAsync succesfully initialised. Found 2 devices: - NVIDIA L4 - NVIDIA A30 %MSG 07-Nov-2025 08:06:20 PST Initiating request to open file file:/data2/mmasciov/LST/output_TTPU_Phase2_L1T.root 07-Nov-2025 08:06:50 PST Successfully opened file file:/data2/mmasciov/LST/output_TTPU_Phase2_L1T.root Begin processing the 1st record. Run 1, Event 9002, LumiSection 10 on stream 0 at 07-Nov-2025 08:24:27.929 PST A fatal system signal has occurred: segmentation violation The following is the call stack containing the origin of the signal. Fri Nov 7 08:24:37 PST 2025 Thread 7 (Thread 0x7f00339ff700 (LWP 1178513) "cmsRun"): #0 0x00007f006dd07e42 in waitpid () from /lib64/libc.so.6 #1 0x00007f00670cb8f8 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/pluginFWCoreServicesPlugins.so #2 0x00007f00670cd662 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/pluginFWCoreServicesPlugins.so #3 0x00007f006e0e2204 in std::execute_native_thread_routine (__p=0x7f0067794fb0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:104 #4 0x00007f006d0081ca in start_thread () from /lib64/libpthread.so.0 #5 0x00007f006dc398d3 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f002d7ff700 (LWP 1178519) "cuda0000300000d"): #0 0x00007f006dd32ac1 in poll () from /lib64/libc.so.6 #1 0x00007f002db5f217 in ?? () from /lib64/libcuda.so.1 #2 0x00007f002dc4455a in ?? () from /lib64/libcuda.so.1 #3 0x00007f002db4f243 in ?? () from /lib64/libcuda.so.1 #4 0x00007f006d0081ca in start_thread () from /lib64/libpthread.so.0 #5 0x00007f006dc398d3 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f00245bd700 (LWP 1178526) "cuda-EvtHandlr"): #0 0x00007f006dd32ac1 in poll () from /lib64/libc.so.6 #1 0x00007f002db5f217 in ?? () from /lib64/libcuda.so.1 #2 0x00007f002dc4455a in ?? () from /lib64/libcuda.so.1 #3 0x00007f002db4f243 in ?? () from /lib64/libcuda.so.1 #4 0x00007f006d0081ca in start_thread () from /lib64/libpthread.so.0 #5 0x00007f006dc398d3 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f00215de700 (LWP 1178527) "cuda-EvtHandlr"): #0 0x00007f006dd32ac1 in poll () from /lib64/libc.so.6 #1 0x00007f002db5f217 in ?? () from /lib64/libcuda.so.1 #2 0x00007f002dc4455a in ?? () from /lib64/libcuda.so.1 #3 0x00007f002db4f243 in ?? () from /lib64/libcuda.so.1 #4 0x00007f006d0081ca in start_thread () from /lib64/libpthread.so.0 #5 0x00007f006dc398d3 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7eff4fbff700 (LWP 1286434) "edm async pool"): #0 0x00007f006d00e371 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f006f0483e8 in edm::impl::WaitingThread::threadLoop() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libFWCoreConcurrency.so #2 0x00007f006e0e2204 in std::execute_native_thread_routine (__p=0x7eff528e9ae0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:104 #3 0x00007f006d0081ca in start_thread () from /lib64/libpthread.so.0 #4 0x00007f006dc398d3 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7eff4edff700 (LWP 1286435) "edm async pool"): #0 0x00007f006dd32ac1 in poll () from /lib64/libc.so.6 #1 0x00007f00670cd88e in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/pluginFWCoreServicesPlugins.so #2 0x00007f00670cda93 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/pluginFWCoreServicesPlugins.so #3 #4 0x00007efff97beb7a in mkfit::MkFitter::reFitIndices(mkfit::EventOfHits const&, int, int) () from /home/data2/legianni/code-recovery/fitReview-forPR/test169+Fit/redo/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libRecoTrackerMkFitCore.so #5 0x00007efff9793efc in mkfit::MkBuilder::fit_tracks(mkfit::MkFitter*, int, std::vector >, int, int, std::map >, std::less, std::allocator > > > >*) () from /home/data2/legianni/code-recovery/fitReview-forPR/test169+Fit/redo/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libRecoTrackerMkFitCore.so #6 0x00007efff9795605 in mkfit::MkBuilder::fittracks() () from /home/data2/legianni/code-recovery/fitReview-forPR/test169+Fit/redo/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libRecoTrackerMkFitCore.so #7 0x00007efff9e19d6f in mkfit::run_MkFitFit(mkfit::TrackerInfo const&, mkfit::IterationConfig const&, mkfit::EventOfHits const&, mkfit::MkBuilder&, std::vector > const&, std::vector >&, std::function) () from /home/data2/legianni/code-recovery/fitReview-forPR/test169+Fit/redo/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libRecoTrackerMkFitCMS.so #8 0x00007efff9acd7f3 in tbb::detail::d1::task_arena_function::operator()() const () from /home/data2/legianni/code-recovery/fitReview-forPR/test169+Fit/redo/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/pluginRecoTrackerMkFitPlugins.so #9 0x00007f006edc1251 in operator() (__closure=) at src/tbb/arena.cpp:944 #10 tbb::detail::d0::try_call_proxy >::on_completion > (on_completion_body=..., this=) at src/tbb/../../include/oneapi/tbb/detail/_template_helpers.h:234 #11 tbb::detail::r1::isolate_within_arena (d=..., isolation=) at src/tbb/arena.cpp:945 #12 0x00007efff9ad4026 in MkFitFitProducer::produce(edm::StreamID, edm::Event&, edm::EventSetup const&) const () from /home/data2/legianni/code-recovery/fitReview-forPR/test169+Fit/redo/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/pluginRecoTrackerMkFitPlugins.so #13 0x00007f006ec49531 in edm::global::EDProducerBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libFWCoreFramework.so #14 0x00007f006ec41597 in edm::WorkerT::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libFWCoreFramework.so #15 0x00007f006ebcfab2 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits::Context const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libFWCoreFramework.so #16 0x00007f006ebcffc7 in edm::Worker::RunModuleTask >::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libFWCoreFramework.so #17 0x00007f006f046e64 in tbb::detail::d2::function_task::execute(tbb::detail::d1::execution_data&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libFWCoreConcurrency.so #18 0x00007f006edd91fb in tbb::detail::r1::task_dispatcher::local_wait_for_all (t=, waiter=..., this=0x7f006bbb3b80) at src/tbb/task_dispatcher.h:344 #19 tbb::detail::r1::task_dispatcher::local_wait_for_all (t=0x0, waiter=..., this=0x7f006bbb3b80) at src/tbb/task_dispatcher.h:487 #20 tbb::detail::r1::arena::process (tls=..., this=) at src/tbb/arena.cpp:216 #21 tbb::detail::r1::thread_dispatcher_client::process (td=..., this=) at src/tbb/thread_dispatcher_client.h:41 #22 tbb::detail::r1::thread_dispatcher::process (this=, j=...) at src/tbb/thread_dispatcher.cpp:195 #23 0x00007f006edccd25 in tbb::detail::r1::rml::private_worker::run (this=0x7f006b84f100) at src/tbb/private_server.cpp:271 #24 tbb::detail::r1::rml::private_worker::thread_routine (arg=0x7f006b84f100) at src/tbb/private_server.cpp:221 #25 0x00007f006d0081ca in start_thread () from /lib64/libpthread.so.0 #26 0x00007f006dc398d3 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f006e8c6580 (LWP 1178488) "cmsRun"): #0 0x00007f006dd08098 in nanosleep () from /lib64/libc.so.6 #1 0x00007f006dd07f9e in sleep () from /lib64/libc.so.6 #2 0x00007f00670cb78c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/pluginFWCoreServicesPlugins.so #3 #4 0x00007f006dc38b8b in sched_yield () from /lib64/libc.so.6 #5 0x00007f006edd1aeb in __gthread_yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/include/c++/13.4.0/x86_64-redhat-linux-gnu/bits/gthr-default.h:693 #6 std::this_thread::yield () at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/include/c++/13.4.0/bits/std_thread.h:366 #7 tbb::detail::r1::stealing_loop_backoff::pause (this=0x7fffe39eba88) at src/tbb/scheduler_common.h:294 #8 tbb::detail::r1::waiter_base::pause (this=0x7fffe39eba80) at src/tbb/waiters.h:37 #9 tbb::detail::r1::external_waiter::pause (this=0x7fffe39eba80) at src/tbb/waiters.h:170 #10 tbb::detail::r1::task_dispatcher::receive_or_steal_task (ctxguard=..., critical_allowed=, fifo_allowed=, isolation=, waiter=..., ed=..., tls=..., this=) at src/tbb/task_dispatcher.h:237 #11 tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=0x0, this=) at src/tbb/task_dispatcher.h:373 #12 tbb::detail::r1::task_dispatcher::local_wait_for_all (waiter=..., t=, this=) at src/tbb/task_dispatcher.h:487 #13 tbb::detail::r1::task_dispatcher::execute_and_wait (t=, wait_ctx=..., w_ctx=...) at src/tbb/task_dispatcher.cpp:169 #14 0x00007f006eb48e5f in edm::FinalWaitingTask::wait() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libFWCoreFramework.so #15 0x00007f006eb5e6d9 in edm::EventProcessor::processRuns() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libFWCoreFramework.so #16 0x00007f006eb584e1 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc13/cms/cmssw/CMSSW_16_0_X_2025-11-05-1100/lib/el8_amd64_gcc13/libFWCoreFramework.so #17 0x000000000040916d in tbb::detail::d1::task_arena_function::operator()() const () #18 0x00007f006edc08c2 in tbb::detail::r1::task_arena_impl::execute (ta=..., d=...) at src/tbb/arena.cpp:860 #19 0x000000000040abfe in main::{lambda()#1}::operator()() const () #20 0x0000000000405268 in main () Current Modules: Module: MkFitFitProducer:hltInitialStepTrackCandidatesMkFitFit (crashed) Module: none A fatal system signal has occurred: segmentation violation