All posts by klato

April 2025

leads

Offline leads meeting notes – 4/30/25

Attendees: Thomas Junk, Giuseppe Cerati, Erica Snider, Katherine Lato

Executive summary:

  • Let us know what your plans are with respect to validating / migrating to Geant4 11.2 or 11.3
  • Let us know what your plans are with respect to validating / migrating to GENIE 3.06
  • Select version of Spack to work with and work on Spack recipes for experiment code in anticipation of Spack v1.0
  • Review the L2 managers and experiment contacts listed here:

Erica opens meeting on following topics

  • Phlex migration
    • A migration will be necessary at some point as support shifts away from art to Phlex.
    • Have started thinking about this, but have no concrete plans as we are too early in Phlex development.
    • That said, following LArSoft published coding guidelines and principles, in particular separation of interface from algorithm code, will make migration fairly straight-forward and minimize effort needed.
    • Suggest experiments identify important production code for compliance with coding guidelines
    • Noted that “separation” allows some things to bleed over:  elements of event data model, message-facility, FHiCL parameter sets, etc. These may change under Phlex
      • Tom: Converting DAC code to offline (a job that gets handed to me), they use an ATLAS thing in place of message-facility. I have to swap in message things. Not hard, but I don’t know how to automate it.
      • Erica: Maybe abstracting the interface can solve this?
      • Tom: A lot of people use C-out? General issue, if there’s a tool that is provided, do we have to use it?
      • Erica:  Depends. For debugging, can use whatever. Otherwise, you need to think about the objective for the code you have. If LArSoft code, it needs to work for multiple experiments, so a common tool / facility / interface is needed.
      • Tom: Are experiments other than DUNE excited about Phlex?
      • DUNE is the only one arguing they need it, so expect not. Part of the Phlex plan is that it be able to replicate as closely as possible what you’re able to do in art. Don’t know what this means yet, but hope is that it minimizes pain of migration.
  • Geant4 update?
    • V10_20_00rc0 = Geant4 11.2.p02 (from Nov 21, 2024). Have heard of no plans or efforts to validate.
    • Please let us know what your plans are. If you don’t want to migrate, let us know that too.
  • New Geant4 patch release available, 11.3.2
  • It would be useful to be using a release that is supported by the Geant4 team. They have also been working hard to improve hadronic interaction modeling, which should be of interest to people working on neutrino flux predictions
  • Before we migrate, will need to decide how to deal Legacy experiments (MicroBooNE…), since Legacy LArG4 will no longer be supported.
  • Reminder about on-going weekly G4 series:
    • Intro / Documentation / General Geant4 LArSoft interface (LArG4) overview – 04/01
    • Physics Overview – 04/08
    • Geometry/Materials Overview – 04/15
    • Hadronic Physics Evolution/Optimization and Validation – 04/22
    • Standard/Low EM/Optical Physics Part 1 – 04/29
    • Standard/Low EM/Optical Physics, Part 2 – 05/06  <<===== Next
    • Hadronic Physics, High Energy – 05/13
    • Hadronic Physics, Low energy, including High Precision Neutron Physics – 05/20
    • Physics Lists, including Constructors and Factories – 05/27
    • (Processing) Kernel Overview – 06/03
    • Geometry (Advanced Topics) – 06/10
    • Propagation in the Magnetic Field – 06/17 
  • Genie update to 3.06 (??)
        • Have a release candidate:  v10_21_00rc0:  GENIE 3.06.00 (from Mar 25, 2025)
        • Experiment plans?
          • Ideally decouple GENIE, but not possible until we migrate to Spack. 
            • Hope this capability to appear quickly after Spack migration
          • Intent is the version of GENIE will be flexible. LArSoft should be able to be tested with different versions of GENIE.
        • Please let us know what your plans are.
  • Need updates to contact lists
    • Please review and let us know of any additions / changes, Name + role
  • Spack migration plan pre-discussion
    • Discuss action items for experiments
    • Note that there will not be a recipe tutorial. Too much doc on the web
    • Note that it does not address a CI solution under Spack. In discussion with Adam, believe we should be able to migrate current system at low cost, so expect that to be the default until a decision on GitHub Actions solution 
      • Tom: User has onus to build the right chain?
      • E: Right. MPD is a development tool. Building releases, don’t need MPD. In principle, could do that now.
      • T: I can build Spack in DUNE without MPD, but I have to build everything.
    • Spack migration plan
      • Have an outline for migration plan developed during recent discussion with Spack migration lead. The basic plan:
      • Spack development phase
        • First production release will be v1.0. Need to wait for this to be released before formal migration can occur.
          • Timescale is summer, as in mid June. As of writing, appear to still be on track
        • Write, validate document that describes detailed, technical procedure for creating a release of LArSoft using a well defined set of tools and steps
          • Presumably trivially extensible to experiment code
        • Write, validate documentation on MPD and Fermilab Spack environment
          • Separately target release managers and end users
      • Pre Spack v1.0 work for experiments
        • Select a Spack release on which to base work. Spack team will offer recommendations
        • Create experiment-specific repository for base experiment code recipes
        • Prepare and test recipes for experiment code and dependencies for which recipes not already available
          • Rely on Spack documentation and tutorials for this.
            • Spack migration team claims online materials are excellent, and sufficient for otherwise experienced people coming in fresh. Consequently, the migration team plans no tutorials or additional documentation beyond that noted above.
          • To make fixes, submit PR for recipes either to Spack, LArSoft or experiment recipe repositories, as appropriate
            • Tom: Merging pull requests…some work required to make [recipes?/spack?]  work
            • E: Objective for v1.0 is to separate Spack and recipe repositories. So plan is to get all that fixed prior to v1.0
            • T: things are big, and stuff will break on the head of the product recipes
            • E: It’s possible. So should try to establish releases for recipe repositories. It is also possible that Spack v1.0 isn’t what we’ll need. It’s 1.1 or 1.2, … Will just need to wait and see
            • T: Can’t believe we’re the only people seeing that things aren’t always right.
            • E: These things affect the entire community. [??]
            • E: [??] if those aren’t sufficient, let us know. Can submit PRs to Spack, LArSoft, wherever it’s appropriate.
            • T: Spack didn’t seem to be exactly designed to do what we’re doing. Usability is an issue sometimes. Default configuration scope is home directory, which means that if two things are going at the same time, one project could interfere with the other.
            • E: If no one raised an alarm about it at the right level, then it won’t be addressed.
        • Identify cases where migration to AL9 is not possible
          • Define procedures for handling these presumably rare cases
      • Post Spack v1.0 work
        • LArSoft: 
          • Make  AL9 releases of LArSoft under Spack based on release policy [which is basically the same as it is now]
          • In parallel, make SL7 releases under UPS
          • Work with experiments to define the time window during which SL7 builds will be released
            • CSAID may put a timeline on phase-out of SL7 containers
        • Experiments
          • Complete necessary recipe changes
          • Start building AL9 releases of experiment code using Spack
            • Validate software development procedures
            • Validate releases under AL9
          • In parallel, create SL7 releases under UPS
          • When AL9 validations are completed, cease SL7/UPS builds of experiment code
        • LArSoft
          • Cease regular SL7 builds when all experiments have migrated
  • Discussion
    • Starting now, if you have cycles to work on migration, you should choose a version of Spack and work on recipes.
    • G: For SBN, there have been Spack builds for running on Polaris. I need to figure out the setup, but given this, things should be straight-forward.
    • E: It is possible that the recipes have changed, but hopefully it will be straight-forward.
    • G: What needs to happen between now and June.
    • E: Choose a Spack release,  create a repository for your recipes, make sure the recipes work for your code and that version of Spack. (No cases in SBN where AL9 migration is impossible.)
    • G: What needs to happen on LArSoft side?
    • E: We’ve already built with AL9 and have a Spack recipe repository, and LArSoft is one of the test cases for Spack migration. So as soon as v1.0 is ready, I think LArSoft will be ready to begin building standard releases
    • G: Is the development environment ready too?
    • E: Part of Spack v1.0 development is to address needs of MPD, so Spack v1.0 will deliver the development environment with MPD.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

November 2024

leads

Offline Leads Meeting – 11/20/24

Present:  Steven Gardiner, V. Hewes, Tom Junk, Herb Greenlee, Erica Snider, Katherine Lato

Agenda:  discussed draft 2025 work plan

  • Note effort issues:  Hans retired, SciSoft team working on DUNE event processing framework, Lynn, who retired several years ago, will leave once Spack is deployed.
  • Changes from last year
    • Split item 1 into two to better differentiate the objectives, milestones, progress. Substance largely unchanged.
      • 1) Focus will be HPC + GPU. Expect Marc P to return to this work after Spack deployment
      • 2) Multi-threading and thread safety. Most of the work here that we had identified has concluded
    • Spack: expect this to conclude next CY. Hopefully early, but we will see
      • Added the next major milestones
        • Spack ready to deploy
        • Approval of transition plan
    • Enabling pixel detectors in LArSoft
      • Completed, so removed. Experiment(s) must now implement pixel readout geometry interface.
    • Event generator re-factoring
      • Same. Waiting for Spack. Added the next generators to be integrated in this way after GENIE
    • Infrastructure updates
      • Added refactoring hit-finding code to allow experiment-dependent handling of long pulses.
      • Also includes non-planar cathode geometries + TPC/cryostat dep drift velocities and electron lifetimes
      • Also have item to review case of jumpered and staggered readout wires within a single logical TPC, as is the case with SBND.
    • Documentation item largely unchanged
    • Consultation on larg4 transition still there, but believe most experiments are there??
    • Long-term priorities
      • Dropped a few items there: SBND data reduction strategies, common framework for data prep
      • Added participation in planning related to migration to new DUNE event processing framework, and consulting on development of pixel readout geometry

Discussion: 

The plan covers what people expected to see. It appears that once we have Spack, there are tools to make building code easier. Spack makes it easier to incorporate Python Packages, easier to have things in the same ecosystem. 

Is there a coordinated effort to standardize how experiments put their specific code bases on top of LArSoft and the art suite and how they build that code? Yes. Marc and his team are working toward a model with layered environments [and the CI system] that we hope the experiments will adopt. Part of the model will be:  keep your (experiment) code current; use our model as much as possible; keep CI tests up to date.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

August 2024

leads

Offline Leads Meeting Notes: 8/14/24

Attendees: Michael Kirby, Thomas Junk, Erica Snider, Katherine Lato

LArSoft status:

The main focus of the meeting was on the status of Spack. 

  • As of the time of meeting, we did not have much new information since the August 6th LArSoft Coordination Meeting aside from the following update from Marc Paterno:
  • The team is working to finalize a plan, with an associated document, for how we are to provide pre-built software through a Spack environment, what we call “standard builds”. This takes the place of UPS packages grouped into “umbrella products” or “distributions”. The document also describes new procedures and processes for managing releases, and the division of responsibilities between experiments and the SciSoft team should experiments wish to exploit the added flexibility in building releases that Spack provides.
  • The team recently started testing a method for building standard releases that avoids issues of Spack “re-concretizing” already-built products. This re-concretizing is the last significant issue with Spack that is blocking migration.
    • [Discussed this point with Marc after the meeting. The candidate solution involves creating a sub-environment of low-level libraries, then telling Spack that they are not re-buildable. This approach worked in a small scale test, and is now being tested on a large software suite (not LArSoft).]
  • The team has developed a beta version of a download-and-setup script that with only two commands installs spack and sets up a personal environment ready for development

 

  • Q:  Is there  “added flexibility” because it allows experiments to build against any compatible version of any dependent product?
    • A:  Essentially yes. UPS locks you into specific versions. Spack does not. This is something DUNE has been requesting for a long time. It has many benefits, but also some notable costs. 
    • Along the lines of added flexibility, there has long been a work plan item to decouple the version of GENIE from LArSoft. We will complete this work after the migration. Can then compare two versions of GENIE in exactly the same LArSoft release.
  • Q:  Is progress on the migration moving faster since moving the project into DSSL?
    • A:  [Update since the meeting:  as noted above, there is potentially significant progress on solving the main problem that has plagued Spack for a long time. Also, there has been progress in recent weeks in several areas that are relevant to LArSoft. So maybe.]

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

April 2024

leads

Offline Leads Meeting Notes 4/24/24

Attendees: Tom Junk, Giuseppe Cerat, Herb Greenlee, Steven Gardiner, Erica Snider, Katherine Lato

  1. New geometry validation status

    • Is anyone working on this?
    • Tom: started a build with RC0. Easier to merge than expected. Someone else was working on RC1. It builds and runs, but no one has checked the output.
    • The validation is fairly simple. Do not expect any changes in results, so just see that nothing changes. 
      • Given this, we will adopt a new acceptance protocol:  one completed validation, then proceed with the migration. 

  2. Erica has completed the LArSoft portion of SpaceCharge changes requested by SBN. 

    • Joseph Zennamo is responsible for the experiment side. Mike Mooney will be making the necessary changes to SpaceCharge service, and will do so for all experiments. Joseph has been busy with SBND commissioning, but said he will get to this soon.
    • Once SpaceCharge changes proceed, LArSoft will make changes necessary for TPC-dependent electric fields (nominal) and electron lifetimes.
      • Tom commented on possibly spatial dependence of lifetime. A resolution issue.
      • Erica:  Would like to see the data that says it’s important enough to matter.
      • Tom:  believes they have ProtoDUNE data that says it does not.

  3. Changes to recob::Hit coming

    • The GausHitFinder is also changing. Will add code to draw boundaries between overlapping hits, and use the smaller sums to fill the new data members
    • At present, the new algorithm is built into the hit finder code. If a change is needed to diversify the solution, then it will be made into a tool. That was the promise from the algorithm authors [at DUNE].  
    • Giuseppe commented to be sure that new code does not affect the multi-threading capabilities.
      • Are reviewing the PRs now, so will check.

  4. Note about services that are either going away or being transferred to other places (which could include the experiments)

    • Has been mentioned to experiments, hopefully at FIFE meetings
    • The operations budget has been stressed, so many systems are being affected.
    • Things like POMS, SciSoft web service, CI system, Spack may be impacted.
    • Bringing this up because it (a) impacts all experiments, and (b) may impact LArSoft support in some way. 
    • DSSL does not want to see loss of support for any of these critical resources. Giuseppe echoed that CSAID has worked hard for a long time to get everyone consolidated into using these common solutions, so it would be a bad thing to turn around and tell people that they needed experiment specific solutions from now on.
      • Exactly…
      • May need to organize meetings with current support people, understand how and if support can be moved to DSSL

  5. Spack status

    • Kyle has been working on the development environment part of this. Will report at the Coordination meeting after this next one. Waiting to hear details about a potential solution from Sandia NL.
    • More than just packaging is required. Spack environment needed. 

  6. Round-table

    • SBN Data/Infrastructure (Giuseppe)
      •  SBN has recently cut a production release, but realized that some other changes in larsim are needed
        • Decided to update production to newer LArSoft base release – now v09_89_01.
        • Tracy wrote an email about this last week. Everyone is on board as far as I can tell.
      •  SBND also needs a new GENIE release to fix a timing problem for “dirt” events.
        •  The new GENIE release has been tagged, but not yet distributed in UPS. Eventually this will need to be picked up by the SBN production release
        • Erica:  So SBN will be on a separate version of GENIE?
        • No, hope not.
        • Mentioned GENIE version decoupling work that is on-going. Have not completed this yet, so will either need to isolate any special GENIE version to a production branch, or bring all of LArSoft along.
      • Hosted a first spack tutorial at SBN Analysis Infrastructure meeting
        • Idea within SBN was to have separate follow-ups for different communities
        • Building and distributing code under Spack for release managers and similar
        • When Spack development solution ready, then another tutorial on that for a larger audience.
        • Need to know the timeline for things
        • Erica: Sounds like a good plan.
    • MicroBooNE (Herb)
      • LArSoft will stop building under SL7, right?
        • Erica:  We cannot run SL7 at all, though we should be able to provide builds using containers, for use within containers.  You can assume we would continue to build for the platforms experiments required, at least for the time being, so will provide SL7 builds if MicroBooNE needs those. 
        • Is MicroBooNE committed to SL7 for the foreseeable future? Yes for MCC9.
      • Will MicroBooNE remain at MCC9 forever? Seems that some parts of MicroBooNE will
        • Discussed MCC10. Adoption has been very slow. Even people working on DL code have been back-porting into older releases. Herb has encouraged them to use integration releases (ie, MCC10), but the developers seem reluctant.
        • Giuseppe mentioned that MCC10 might require huge validation investment, which is too much for individual developers. 
        • Herb mentioned that MCC10 workflows run, but that no one has looked at the results.
        • So seems MCC9 will remain a thing for a long time
      • In general, MicroBooNE needs more hand holding for Spack. Herb has looked at the tutorials and they don’t answer the questions he has.
    • DUNE
      • Exactly when are we going to start building routinely with Spack?
      • Erica will check with SciSoft team
      • Should be happening on the CI system, right? Yes, in principle. 
      • Have to keep a version of all the pieces in the UPS products we have. I did that this week. DUNE dac people have a development environment that I don’t understand. Could ask them how they do Spack? Problem is multiple repositories. 

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

January 30, 2024

leads

The January 2024 LArSoft Offline Leads status update was handled via email and a google document. 

LArSoft – Erica Snider

  • Work is proceeding to accelerate DUNE physics processing using GPUs for select LArSoft algorithms, starting with the PDFastSimPAR module. The plan is to first  optimize serial execution, then parallelize on CPUs, and if needed to achieve performance goals, parallelize on GPUs. There will be update reports at both LArSoft Coordination Meeting and within DUNE FD reco group this month.
  • A new machine learning algorithm, NuGraph2, is available within LArSoft. The algorithm performs hit classification and clustering using a GNN, and is being developed for use in MicroBooNE, and will be used in ICARUS later. Details were discussed at the Dec 12 LCM. Documentation on the algorithm and how to use it will be posted to LArSoft.org in coming weeks. 
  • A Spack build of LArSoft became available last month. SciSoft is planning to provide a tutorial on how to build experiment code under Spack using this new release. Experiments wishing to participate will need to use machines under AL9 to run the builds.
  • Geometry re-factoring:  working on completing documentation, the last step needed prior to releasing it as a LArSoft v10 release candidate for experiment validation.

DUNE – Heidi Schellman, Tingjun Yang, Tom Junk

DUNE will start work this period on the Spack migration now that the LArSoft Spack build was made available.  We have already been in contact with Steve White, Marc Mengel, Patrick Gartung and Kyle Knoepfel on the subject.  We will need to keep SL7 build nodes and interactive nodes as long as possible – there have been proposals to shut them down in March but we would like them longer.  We also are attending the container task force meetings and have a container that works for builds and will test the current worker node container again.

LArSoft comment:  The project pushed back on the original March 20 suggestion, and SCF Dept Head agreed to allow at least some build nodes to remain up beyond that date. A suggested compromise shutdown date is mid-May. LArSoft (tentatively) responded that this would be acceptable. EOL is end of June.

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee

No Report

SBND – Andrzej Szelc

No Report

SBN Data/Infrastructure – Steven J. Gardiner, Giuseppe B. Cerati

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

November 15, 2023

leads

LArSoft Offline Leads 11/15/23 Meeting

Attendees: Herb Greenlee, Tom Junk, Will Foreman, Giuseppi Cerati, Erica Snider, Katherine Lato

At the meeting, we went over the draft of the 2024 work plan for LArSoft. 

Number one priority continues to be multi-threading and High Performance Computing (HPC) with several people working on this in 2024. Notably, SciSoft has effort available to make algorithms run on GPUs. Requirement is that they be relatively slow, in LArSoft, and amenable to acceleration with a GPU. There was a follow-on question about GPU as a service. This is described at: https://larsoft.org/using-gpu-as-a-service-in-larsoft/. SciSoft believes this is ready to run at scale, so just need to work out the logistics of spinning up the GPU server somewhere.

Spack migration has a deadline of Q2 2024 in order to provide time for experiments to complete their migrations in advance of the June 2024 SL7 EOL.  AL9 requires Spack, since there will be no UPS support. This work seems close, though we still do not have a detailed timeline to completion. Comment:  Experiments have changes to make, so do not want to be pinched for time. Response:  Given that all experiment code has been migrated to cetmodules, the procedure for getting from there to a Spack build should be straight-forward. Expect that any issues will be related to getting the experiment-specific product stacks under Spack. Q:  Containers should provide some cover? Yes, should provide a buffer to the June 2024 EOL. Generally, the process will be that AL9 and SL7 will co-exist for some time to allow experiments to migrate. Spack has to come first. 

Support for multi-experiment event display has been on the wish list for a while so that upper management understands that multiple experiments have requested this.

A new item in this year’s work plan are updates to the LArSoft infrastructure. These include, but are not limited to:

  • Sampling frequencies vary across TPCs in protoDUNE, while LArSoft supports only a single value.
  • Support for non-planar cathode geometries to facilitate tracking across non-planar cathodes. 
  • Support for TPC-dependent drift velocities and electron lifetimes. 

Appendix B is a short summary of our major observations from one-on-one meetings with each experiment in September and October of 2023. Common items include: 

  1. Event display that is useful in the current environment.
  2. HPC.
  3. Faster processing.
  4. Event generators

Round Robin:

  • SBND:  Will Foreman
    • working to integrate blip reco into SBND code. Once completed, will migrate to LArSoft. But first want to make sure it is working in SBND
  • MicroBooNE:  all question answered during work plan discussion
  • SBN: Giuseppi 
    • SBND has allocation to run on Polaris at Argonne. Will be running simulation. Running in a container. First asked about Spack, but was not available, so opted for container. That has been demonstrated to work.
  • DUNE:  will send comments later.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.