August 2022 Meeting Notes

leads

Offline Leads Meeting Notes August 31st

Attendees: Herbert Greenlee, Tingjun Yang, Tracy Usher, Erica Snider, Katherine Lato

LArSoft Update:

  1. Clang-format for repositories

We want to move LArSoft repositories to a standard clang-format to have a standard across repositories. Kyle produced some examples. https://github.com/knoepfel/larreco/tree/clang-format-example Please look at the example and provide feedback.

Herb: I assume there are a ton of badly formatted files. Would these be cleaned up?

Erica: Yes. We envision there would be a number of changes in each repository that don’t change the code, just the format. Then just need to be disciplined so that real changes can be seen.

Herb: Is this run automatically?


Erica: There’s a codechecks phase in the auto-CI workflow triggered by GitHub, and running clang-format has been proposed to be part of that. Details haven’t been discussed. Codechecks fixing it for you is one possible mode we could use, but originally envisioned just as a check. If people wanted to do it manually, that would be great. Would you rather have it be automatic?

Herb: I don’t have a strong opinion either way. It’s probably a good idea to have a standard format. 

Herb: New topic:  Any thoughts about enforcing ‘consumes’ and ‘produces’ in art modules?

Erica: ‘produces’ has been enforced for some time. [after the meeting]  ‘consumes’ has been recommended for a long time, but is not enforced by default. One can enable it with a command line option:  

lar -c <my config> –errorOnMissingConsumes=true …

So MicroBooNE (or any experiment) could choose to enforce this independently of anyone else. 

There is also an option to find out what ‘consumes’ statements are missing:

lar -c <my configs> -M mt_diagnostics.txt …

 2. LArSoft Multi-threading workshop

  • https://docs.google.com/document/d/1QCo5GUQ5Js9iU8iodZzXvkK2uNjdQC8viDGHPg5xQgM/edit 
  • Seems to be some consensus that this could be a good thing, but still do not have a clear picture of the goals or program. If we are to proceed with this, we need some follow-up from experiments regarding what they want from the workshop, and what problems we should try to address during the working time. Please try to provide feedback on that.

3. On-going multi-threading work

  • Mike Wang continues to find differences in the output of the DUNE workflow he is working on depending upon whether it is executed in single-threaded or multi-threaded mode. He is working to understand the cause. Previously found an art bug that caused event number mappings to change.

Herb: What kind of multi-threading is he doing?

Erica: A couple of types. Events are being run in parallel using the multi-threading features of art, and hits are being reconstructed within each event using multi-threading in the GausHitFinder code. Both use art and GausHitFinder use TBB.

  1. Google searches of LArSoft wiki on GitHub
    • Have had success in getting at least some pages indexed by google, and getting google hits within larsoft.github.io. 
    • People should try googling for things in the wiki next time they want to find something. Please let us know if it fails.
  1. GENIE v3.02.00
  • Are preparing to integrate GENIE v3.02.00 into LArSoft, which will involve a test release. Expect this work to be completed in the next three weeks or so.
  • Note that there is already a v3.02.000 test release – v09_55_01_01 released on July 25. 
    • Have people looked at this?
  1. Spack update
  • From last time, discussed these steps / milestones for phase 2 of migration:
    1. The experiments must  convert all of their code to use Cetmodules and modern CMake best practices (a la LArSoft phase 1).
    2. The experiments must also  produce and/or verify Spack recipes for their own packages, and for all external dependencies not directly supported by SciSoft.
    3. The current LArSoft stack and its dependencies must be verified to be buildable by Spack. There have been many changed/added dependencies since the last time this was done, so this is not a trivial task.
    4. We must have a system usable by LArSoft and experimental release managers capable of building and releasing a fixed and reproducible distribution of their code and all dependencies via Spack for all supported platforms and compilers. These distributions must be installable on supported systems with maximum (re-)use of pre-built and cached binaries, and minimum rebuilding of packages unchanged from one release to the next.
    5. We must have a multi-package development system capable of using and producing Spack-built binary packages for distribution via BuildCache.
    6. Validate everything on the release current at this point, obtain sign-off from all experiments, then execute the migration.
  • Chris Green et al currently working on some combination of (c) through (e), which do not depend on experiment code, and can proceed in parallel with those items that do
  1. Geometry adaptations for pixel detectors
  • Have started a series of technical meetings to work through design and implementation details. Do not yet have a preliminary design, though conceptually, we know that we need to abstract and separate anode descriptions from the TPC volume descriptions. Anodes are not currently a concept in the geometry. Wire planes are, but they will not be an attribute of all TPCs. How to do that is the topic of the meetings.

8. LArSoft 2023 Work plan discussions will be starting.

Erica and Katherine will discuss priorities with each experiment in a series of meetings in October. The experiments should be prepared to detail their plans for the next year, the implied requirements for LArSoft, and how the LArSoft Project Team could help, as well as what the experiments might be able to contribute to LArSoft code. Based on those discussions, LArSoft proposes a plan of work for 2023 along with relative priorities of the various items. This plan is presented to the Steering Group for discussion and input.

Discussion on the project report:

Herb: Is there a problem if Lynn retires for real.

Erica: Yes. Patrick Gartung can do releases, but does not currently do everything that Lynn does due to commitments to other projects. I have worked on solving this problem, but does not appear to be a sufficiently critical issue for the division to warrant a solution at this time.

Herb: Back to multi-threading. Has anyone tried to run an art module where different modules have different threads?

Erica: I don’t think so. With different trigger paths, you could run maybe concurrently. But trigger paths define a sequence of operations. Would need strict adherence to ‘consumes’ and ‘produces’ at the very least to make this possible.

Round Robin: 

  1. ICARUS: Tracy Usher
    1. Standard statement on multi-threading. In principle, what is holding us up is our noise-removal code. It’s been on the list to fix. We know what we need to change, just haven’t gotten a person to do it. We recognize that it would be beneficial to us to do it. 
      • One other roadblock (excuse for us):  there are services in LArSoft where multi-threaded versions are on a branch, but haven’t been integrated. They are services that access databases 
      • Erica: This is the last thing that Saba was working on, using the art concurrent caching to ensure that the database was thread-safe.
      • Erica: Can you send me the specific services in question?  that you care about, that would be helpful.
      • Tracy:  Yes
    2. Tracy: Nominal start  of physic quality beam for Run 2 is October 15th. Our intention is to do a new production branch the week of Sept 19-23. That means we need a branch. Not a lot of time to shake out. 
    3. Production runs in three stages
      • stage 0:  signal processing, 
        • Ideally this will be frozen until next summer. This is where the huge amount of data is. Need this for keep-up processing. Want it ready the week before data taking begins
      • Stage 1:  the reconstruction side, starting from hits going through Pandora
        • Would not necessarily be frozen on that timescale. Sometime in December, we would try to freeze that. 
      • Stage 2:  Common Analysis  (CAF) output. The bulk of analysis would be there unless people wanted to look at event displays.
    4. Have to coordinate SBN-wide for data sets, which will be based on this format. Freeze dates will be tied to a joint SBN and ICARUS date.
    5. Erica:  Is there something you need that isn’t there for stage 0?
    6. Tracy:  No. The wirecell stuff has come in. Will use WireCell 2D simulation. Still debating about 2D deconvolution. Related to complications of runnnig WireCell things in the ICARUS environment. Intend to make this switch, but it is not clear whether that will happen this year. It is still the ultimate plan. 
    7. If we remove waveforms from stage 0, then the heaviest object left is reco-wire object. Those don’t compress nicely because they are float objects. We may substitute that with a ‘short int’ object. Simple to implement using code Gianluca developed. We won’t be able to use LArSoft Event Display with this, however.
    8. Don’t need to go back to floats since the information is stored in the hits. Have to use lighter-weight objects, smaller, and can use for hand scanning.

 

2. ArgoNeuT: Tingjun Yang

    1. Previous release  manager, Patrick Green, has graduated. Have a postdoc who took over, Wanwei Wu. 
      • The contact list has already been updated:

https://larsoft.github.io/LArSoftWiki/LArSoftInternals/Informal_list_of_experiment_contacts

3. Proto DUNE: Tingjun Yang

    1. Tom knows more about the general activities. Tom has been working on simulating events in HDF5 format. 
    2. Working on preparing ProtoDUNE horizontal drift, updating the geometry for Run 2. Pursuing similar activities for vertical drift (single-phase)

4. Near Detector DUNE: Tingjun Yang

        1. Able to run GENIEGen module with near detector geometry file. Gave a report last week at DUNE reco meeting saying it’s possible to do it using LArSoft.
        2. Tammy Walton is working on simulating near detector with edep-sim. Expressed interest / preference for doing this in LArSoft.
        3. Heidi Schelman expressed strong support for art-based simulation.
        4. People are aware that this is possible, and seems to be some traction for it.
        5. Geometry: There is a module 0. Took data at Bern. Is part of 2×2 prototype. Data is in HDF5 format. Trying to convert it to something LArSoft can understand. Need help in saving the location of each pixel.
          • Erica:  please set up a meeting to talk about it.
        6. Tracy: Have you been coordinating with the active machine learning group? They might be able to facilitate what you’re doing.
        7. Tingjun: I talked with many people in the machine learning group. Method is based on python, not based on LArSoft.
        8. Tracy: True, not based on LArSoft, but still impressive what they’re doing.
        9. Tingjun: Current focus is on the framework, not algorithms. Will need discussion in the future on the plan to put everything together.
        10. Tracy: There is an active machine learning group in ICARUS as well. Trying to do everything from 3D space point view. It’s not inside LArSoft, but they are trying to develop an interface between what they’re doing and LArSoft.

5. MicroBooNE: Herb Greenlee

          1. Trying to finish MCC9 analyses, so don’t need anything from LArSoft at the moment.
          2. Replying to Tracy:  SparseRawDigit is the MicroBooNE class in ubobj that has integer representation for waveforms. So not part of LArSoft, but could be moved into LArSoft.

Action Items

  1. ALL – provide feedback on Clang format and workshop writeup.
  2. ALL –  please let us know if searches of LArSoft information on github wiki fail. https://larsoft.github.io/LArSoftWiki/ 
  3. Tracy: List about the services in LArSoft where multi-threaded versions are needed. (May be implementations exist on a branch somewhere, but are not yet integrated into LArSoft)
  4. Tingjun:  schedule follow-up meeting for HDF5 discussion
  5. LArSoft: Update release manager for ArgoNeuT:  done.
  6. LArSoft: Check whether ‘consumes’ is enforced:  As noted above,  ‘produces’ has been enforced for some time. ‘consumes’ has been recommended for a long time, but is not enforced by default. One can enable it with a command line option:  

lar -c <my config> –errorOnMissingConsumes=true …

So MicroBooNE (or any experiment) could choose to enforce this independently of anyone else. 

There is also an option to find out what ‘consumes’ statements are missing:

lar -c <my configs> -M mt_diagnostics.txt …