LArSoft Offline Leads meeting Notes – June 12, 2025

Attendees: Tom Junk, Dominic Brailsford, Guiseppe Cerati, Erica Snider, Katherine Lato

Executive summary

• Project effort profile continues to be very thin
• Spack migration
  • Migration requires v1.0. That is due out at the end of June. Spack team may need some additional time to verify that it works as needed. Can then begin migration of experiments.
  • Have already migrated the external products needed by LArSoft. Experiments may need to Spackify external products not integrated into LArSoft. 
  • Will start providing both Spack and UPS builds of LArSoft once v1.0 is verified.
  • Will set a deadline on experiments completing their migrations. Expect this to be around two months.
  • Turning off UPS will follow experiment migrations and update of CI system to use Spack.
• Preparing to migrate to Phlex
  • Plan calls for Phlex to test high risk features by end of 2025, and to be ready to integrate DUNE FD algorithms by end of 2028.
  • No specific guidance on what migrating algorithms from LArSoft to Phlex will require, except that complying with LArSoft coding guidelines will minimize the migration effort required.
  • Phlex team will use a DUNE workflow as part of their demonstrations / validation. Expect them to reach out about that.
  • At the time of migration, there will be a “legacy” branch in LArSoft to accommodate legacy experiments and workflows that cannot / should not migrate to Phlex. art will receive maintenance support, but will  remain feature-frozen leading up to migration.
• Geant4 update to v11.2+
  • NOTE:  the current version of Geant4 in LArSoft is blocking updates to Root and art. So want to migrate,
  • Checking on whether current behavior can be replicated with the new Geant4.
  • v11.2+ offers improved physics modeling (eg, for low energy interactions and hadron production) and new interfaces that can improve memory management and much better optimize CPU performance.
• GENIE update to v3.06
  • NOTE:  the current version of GENIE in LArSoft is blocking updates to Root and art. So want to migrate.
  • GENIE team has indicated that the behavior of existing SBN+DUNE tunes will remain the same after the upgrade.
• Code clean-up
  • SciSoft will add monitoring to art jobs to track library usage. Will use this data to help eliminate unused code from the repositories. 
  • Experiments are encouraged to do the same with their repositories.

Detailed Notes:

• Project effort profile continues to be very thin. Team is mostly just addressing tickets and PRs.
  • Once we move to Spack, expect that release builds will require less effort. Won’t have to modify everything above when doing a release.
• Spack migration
  • Spack v1.0 is due out at the end of June. (Previously was estimated to be June 13.) Will need to check in with the spack team at release time as to suitability as base for migration. The delay might be helpful in that more likely that features we care about will be included in v1.0.  Either way, expect to be in a position to start regular Spack builds of LArSoft soon. So…
  • What if any work has been done in preparation for Spack v1.0?
    • Have experiments identified a version of Spack to use?
    • Have experiments been working on any experiment-specific recipes that will be needed under Spack?

• If the new version drops tomorrow, how long before LArSoft is routinely being built with that? How long before the experiments can be using it? Have to use it?
  • Plan is to start making routine releases right away if there are no problems, then it’s up to the experiments how soon they use it.
  • There is a migration period when both UPS and Spack builds of LArSoft will be created. Experiments should migrate during this time. We’d like to keep this period as short as possible, but expect it may take up to a few months.
  • [Note that the CI system also need to be updated to use Spack. That will be required before we stop building under UPS. CSAID is working on a plan to do this work.]
  • Once the experiments are all building with Spack, SciSoft will stop making UPS versions
  • Experiments expressed some interesß in having a deadline imposed. OK.
• Some have experienced issues with development environment under Spack
  • There are still some issues with underlying pre-release versions of Spack, which could be the source of problems. Expect those to be resolved with v1.0. If not, report a bug / request support.
    • One noted that experiment development won’t be in a position to build a package (??).
    • These problems may be a barrier to people otherwise available to work on migration prior to v1.0
    • mu2e has experienced problems. They were an earlier adopter of Spack. May have made things more difficult for themselves by working from the head. This is why Spack team recommended choosing a base release of Spack to work with. Even then, however, there may be issues. [Strategy would be to implement work-arounds in those particular cases. Not a good situation, but may provide a stable solution.]
• Expect any issue with external packages in moving to the new system?
  • Don’t think so. Package authors [or customers] need to provide recipes, but once we have those, we should be fine.
• Is there something experiments will need to do that’s different from what they do now? Eg, with Wirecell? Their system is integrated into UPS. Is there anything that changes on the external party’s side?
  • Right now, people hand Lynn the code and she UPS-ifies it. That step won’t be necessary. [Product authors need to provide recipes. We have Spack builds of LArSoft, so presumably this has been done for WireCell]
• Lynn aligned versions underneath LArSoft. Worked to straighten out competing versions.
  • How it works now is complicated.
  • The external package situation will be better under Spack. We just take their recipes and libraries and there’s no need to repackage anything. This work now goes into creating well-defined Spack environment layers that have dependencies aligned. If changes are needed, then a new environment will be created. The plan is to gather all dependencies from the experiments, then make them all available. [Generally the OR of all requirements]
• Someone needs to judge what is compatible. Have to decide if it’s ok to move to a new version of something one experiment needs that others don’t.
  • [LArSoft will put out a single “standard release” at a time.] If the experiments want to do something different, that’s ok, but they have to do that work themselves.
  • It’s all layered, and such differences will be isolated from the rest of LArSoft.
• Have the external packages already supplied recipes or do they need to be chased down?
  • Those needed by LArSoft are all ready. We won’t get the fuil list until all the experiments migrate
  • Wirecell and Pandora?
    • We have a few builds of LArSoft so that’s an existence proof of that we have those.
  • Sometimes packages provide things inside that are supplied by other packages. Have to catch that if people have copied them.
• Have experiments been working on recipes? 
  • They are waiting, but already have working recipes in some cases.
  • SBND and ICARUS have work-arounds for various issues, but are using Spack builds at SC centers. (on Polaris) Team wanted to do work over the summer. (?)
  • Is it also running Spine?
    • Full production, so yes (??).
  • Is the entire workflow running on Polaris, not just the Spine?
    • Yes. Are not leaving out any packages. Correct.

• Previously discussed the need to migrate  from art to Phlex at some point
  • What, if any, work or planning that has been performed in preparation for a migration to phlex. Have no new information yet on specifics of the refactoring that might be needed, but it is important to start planning this now. 
  • Has there been any thought about re-factoring to comply with LArSoft coding guidelines? (eg, No framework interaction in the algorithm code.)
  • Phlex migration team will use DUNE workflows as part of the early demo and feature validation. Phlex developers should have reached out to identify a suitable candidate workflow. Have they?
  • Milestone for 2028 is for Phlex to be ready for FD algorithm integration
    • DUNE has been thinking about refactoring. Also interested in culling dead code. But to knowledge, Phlex team has not reached out to DUNE re workflow
  • What are the plans for legacy experiments with art? Don’t want to make a decision now, but has that been a part of the planning? Wait for official word?
    • art will continue to receive maintenance support, but there will not be efforts to introduce new features in art. As OS’s evolve, we need to ask, at what point can we no longer bring forward legacy systems with art? That may determine the art EOL
    • Do we have the same issue with new features in LArSoft?
      • Expect there to be a “legacy” branch in LArSoft. Not clear that the code on the head will remain backwards compatible. It depends on how things are handled, particularly with respect to services and associations (art::Ptr, art::Assn) to see if they will be compatible. Might need to port things, which will complicate any back-porting to the legacy branch
    • Are we concerned about legacy projects not bringing their work forward to the new system?
      • Yes. But we probably want to see what a migration to Phlex looks like. The design is focused on issues to address long-term needs, but there is also an acute awareness of the migration problem. They’re not just focusing on what it ought to look like in the long term. Making a conscious effort to make it easy to bring code forward. But won’t know more until later.
    • At what point will there be an official dialogue opened up?
      • Were hoping that the Phlex team had already reached out. If they haven’t, then it is probably because they are probably not quite ready for that step yet.
    • Can we get interns to get LLM models doing the migration for us? Some exploration might be worth doing.
      • Agree. [The Spack migration and SciSoft teams have actually already discussed this point. Expect to hear more on it]

• Geant4 validation / migration planning?
  • Geant4 team suggests that we rewrite the Geant4 interface in LArSoft to v11. DUNE might benefit from that, given better memory management capabilities of Geant4 v11. And you get Opticks / Celeritas integration for free once those happen.
    • DUNE does struggle with photon tracking
    • DUNE/SBND (??) also interested in saving memory. We’d be happy to enable Geant4 v11 if it doesn’t cause problems. For the low-energy work it may be more significant.
    • The low-energy people are interested in this.
  • What are the main benefits of moving?
    • You get all the updated physics list. The beam simulation people should be interested in this. Lot of benchmarking related to that. Now we’re at 10.4 for the beam sim (so not LArSoft). Low energy side should be interested as well.
    • Do BNB and NuMI use the same beam simulation?
      • They’re all Geant4, but they use different driver programs
  • Are the changes opt-in? [Assuming here that the question is, “can we replicate existing results with the new Geant4?”]
    • Not sure. Probably not. But will check.

• GENIE v3.06 validation / migration planning
  • Anyone thinking about this?
  • If we can opt in, then possible.
    • Yes, we expect so. The base model for SBN + DUNE is expected to behave the same. Steven Gardiner is presenting to SBND soon on exactly this question. Will share the slides with the offline leads when available.
  • From the chat from  Steven: Probably we want to update Genie, but haven’t actually discussed it. Changes aren’t big for stuff SBN cares about
  • What is the current GENIE in LArSoft? 3.04.02a

• Has the Geant4 talk series been useful? Why or why not? If not, what would be more useful?
  • Good technical details

• Code clean-up. 
  • SciSoft proposes attaching reporting to fw jobs to see what things are actually loaded. Same for config files. 
  • Results will be used to prune repositories. 
  • We need to do this (prune repositories) before any phlex migration
  • No comments from experiments.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.