May 2022 Meeting Notes

leads

Offline Leads – May 25, 2022

Attendees: Chris Backhouse, Wesley Ketchum, Herb Greenlee, Joseph Zennamo, Tom Junk, Tingjun Yang, Erica Snider, Katherine Lato

LArSoft – Erica Snider

  • The migration of the Redmine LArSoft Wiki to GitHub Pages has been completed and is now available at https://larsoft.github.io/LArSoftWiki/. Among other things, this move should in principle allow search engines to index the LArSoft documentation, as was possible before the Fermilab web servers were put behind the Fermilab SSO. To date, however, Google searches do not find the LArSoft GitHub wiki.
    • Note: bing.com and duckduckgo.com do find LArSoft wiki pages on GitHub.
    • Should you have an edit or other content suggestion, you may let us know via issue tickets (which are still in Redmine), pull-requests on the LArSoft/larsoft.github.io repository in GitHub, or email to scisoft-team@fnal.gov.
  • Thread safety and multi-threading work:  Mike Wang has been working on a simplified DUNE SNB processing workflow that uses the 1D deconvolution in CalData instead of WireCell. The steps are:
    • CalData
    • GausHit
    • SPSolve
    • HitFD
    • TrajCluster
    • PMTrackTC
  • The CalData stage has been modified to use a thread-safe implementation of LArFFT written by Mike.  The GausHit hit finding stage is based on Mike’s implementation of a Levenberg-Marquardt fitter that Guiseppe Cerati and Sophie Berkman ported to LArSoft.  Mike is currently looking at the SPSolve stage, which is the 3D space point solver that also performs some disambiguation of hits.  Aside from making this stage thread safe, there is opportunity to incorporate multi-threading within the module as is done in the GausHit stage.
    • Mike is currently validating that multi-threaded and single-threaded execution yield the same results. As of May 16,  there are differences, so he was working to identify the source.
  • Work requested by SBND to implement legacy LArG4 behavior has been completed. Though missing the SBND deadline, the work is now on the head of develop, so will be available for future releases. A proposal for a long-term solution has been advanced. A follow-up discussion is needed to determine how to proceed from this point.
  • Spack migration
    • The Phase 1 migration of all LArSoft repositories to cetmodules is nearly completed. The process was relatively straight-forward in almost all cases. larrecodnn required some extra attention, which is not unusual given that it touches TensorFlow.
    • Expect work toward Phase 2 will begin shortly. Unlike Phase 1
      • The tools and procedures for building will change
      • Everything changes at once – we wake up one day, and everything will be different. There are no staged changes
    • Do experiments plan to follow this migration? (It might be required, but not yet sure.)
    • Q:  How will builds of external packages maintained by experiments work once we migrate?
      • Currently packaging a number of experiment code dependencies  in UPS
      • A:  not entirely sure. Will discuss this within the SciSoft team. SciSoft will in general provide assistance with migrating experiment code and external dependencies.
    • The details of the migration are not yet known, but will feature
      • A migration path that allows for testing of all relevant code prior to the switch
      • An education campaign for users
      • Assistance with migrating any experiment code needed 
      • Timelines developed in close consultation and agreement with experiments
    • Q:  What about legacy branches, eg, the MicroBooNE MCC9 series?
      • Presumably, things that live in the legacy world will continue to work within the legacy environment. Will not obviously even complicate back-porting code, since that usually does not involve elements of the build systems
    • A discussion with Chris about what NOvA is doing ensued, since they are art users. Not likely to be directly relevant to LArSoft, though could affect the timeline if an associated migration is needed.
    • Wes:  for SBN, there is the question of how to manage the Spack transition for the online environment
      • The DAQ is built on art-daq. Everything relies on relocatable UPS packages
      • Erica:  so this suggests that transition needs to happen during a beam downtime?
      • Not necessarily. Noted that they can operate in a legacy mode for some time, but best not to be in that position long-term. So just needs to be coordinated with SBN DAQ
      • Next SBN downtime:  Early July through mid-Sept, full beam back early Oct. (we think)
      • Wes will be working with DAQ people to discuss this.

Experiments:

  • DUNE
    • Tom: 
      • ProtoDUNE 2 takes data next year. Will use the DUNE DAQ. Wes knows more of the details.
      • DUNE can probably find people to do the Spack migration on the timescale we want, provided that the legacy system is available to everyone else. DUNE may require help with Phase 2 Spack migration if they get stuck.  
        • Erica:  This is expected. SciSoft will provide support.
    • Tingjun:
      • Discussing with LArSoft team about supporting simulation and detector.
        • Erica:  excited to have effort from the experiment directed into this long-standing interest for LArSoft. Will be working with the experiment to develop a plan for the necessary changes. That will involve changes to the geometry, so need to be clever so as to minimize the disruption from that.
        • Tom:  commented on possibility of dueling software stacks. Do not want to disrupt ability to continue code development
        • Erica:  prefer integration, but may be places were dueling stacks may occur. Want to minimize that unless there are clear gains. 
        • Note:  There is already a PR https://github.com/LArSoft/larsim/pull/94 and a resolved redmine ticket to address some of the issues related to this work: https://cdcvs.fnal.gov/redmine/issues/26961
  • ArgoNeuT:  Tingjun
    • ArgoNeuT imported the CVN product, which provides DL tools to select neutrino events.
      • Originally added to DUNE, and copied it from there into ArgoNeuT, but there was an issue trying to build it in ArgoNeuT. Once difference with DUNE usage was that DUNE followed the update to use the Triton package with CVN, but ArgoNeuT did not.
      • One idea to resolve this is to move the common code to LArSoft and only have the experiment specific part in each experiment repository. 
      • Working on it now, may need support from LArSoft.

 

  • SBN:  
    • Joseph
      • Chris Backhouse is taking over the SBND side of SBN for Joseph. 
      • SBN just launched the first large scale production of beam exposure. That’s been going well. 
      • Moving on to the next stage, at-scale production similar to one year’s exposure. 
        • Chris will be taking the lead on this. 
        • Maybe 100 million events. 
        • This will probably strain our systems, already see the need for performance and workflow improvements. 
          • Have been focusing on running all the basics that are needed at scale, but now performance upgrades are needed. 
          • Hope to adopt 2D deconvolution, overlay workflows, etc. 
          • Long-term, need to come back to understand how to lower [delta-ray] production thresholds in Geant4. Those improve fidelity, but come with a steep performance impact. 
    • Wes
      • We need to go through and plan the next production, software updates. Might be a number of requests that come in related to that. 
        • Eg., different lifetimes in different cryostats. 

 

  • MicroBoone: Herb
    • Making an effort to integrate MCC9 updates into develop. 
      • May require updating to refactored LArG4 framework. Would require help or advice for this migration
        • Otherwise, worried that they will be left behind. For instance, the new light simulation is probably the  most important new development that would be useful to MicroBooNE. This is in the new LArG4.
        • Erica:   very glad to hear that this is in the plan for MicroBooNE. The project will provide whatever assistance is needed. 
    • Worried about whether redmine will go away. 
      • MicroBooNE has not migrated to GitHub, and has not been pushing that.
      • One advantage of Redmine is that it has one landing page with links for wiki, repository, and issues. Would need to work to replicate this on GitHub. Otherwise people need to look in multiple places for everything.
        • Erica:  so far no indication there is an EOL date for Redmine. Will bring the question to Jim Amundson at next meeting with him.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.