All posts by klato

July 2021 Meeting Notes

Offline Leads meeting – July 15th, 2021

Attendees: Miquel Nebot-Guinot, Andrzej Szelc, Wesley Ketchum, Tom Junk,  Erica Snider, Katherine Lato

LArSoft:

  • Working on making services that access the database use the caching system. (What Kyle Knoepfel presented at the LArSoft Coordination Meeting in November, 2020.)
  • Have been working through issues related to art 3.09 migration.
    • Recently resolved two root issues. One still being tested. 
    • Takes time to iterate on issues in the product stack
    •  Expect to be completed soon.
  • First phase of SPACK migration requires art 3.09, and expect will follow relatively quickly after migration to art 3.09. This phase will be compatible with UPS and mrb, so will not require major changes in how we do things.

Round Robin:

  1. SBN Data/Infrastructure  (Wes, Miquel) 
    1. Need to think about the online systems for SBN. We use UPS local products, and run two environments — one on the DAQ side so more real-time/online, the other on data quality so more like offline. We need to get experience with this for the Spack transition, but nothing appears to be outside current methods. The way we do stuff in the online system mirrors what is done in the offline. 
    2. Looking to freeze the code and get things in order for the next few months. It is advantageous to us to have the new build as soon as possible. ICARUS major physics run in fall. Hoping to get frozen the pieces for the ICARUS data reconstruction. When we freeze the code, we’re going to want to optimize it. May reach out for help on profiling. Hopefully in 2-3 weeks, we’ll have code that does what we want and will have dedicated time for optimizing. This will be our general pattern moving forward:  freeze functional production code, dedicated time for optimization
  2. SBND: (Andrzej)  Getting ready to move to new Geant4 framework. Made a module to take the  CRT output from the new way (SimAuxDetHits) to the old (SimAuxDetChannels)? The module takes hits and packages them as channels. Two objects that are effectively identical. Is this something LArSoft would be interested in? 
    1. Erica:  Contributing that to LArSoft would be good. 
    2. Andrzej: Will let Ivan know to get in touch with LArSoft.
  3. DUNE: (Tom) 
    1. Been working on chopping DUNE TPC into pieces because the builds are slow. Chopped into smaller pieces by taking directories out and assigning them to UPS products. Not too different from how LArSoft arranges things. Wrote a script to do the chopping since code changes while working on the split. One issue, the FHiCL files don’t factor as easily as the code because they are included often and include many other files. The FHiCL files can depend on things not there in the code dependency tree, so if I put a file higher on the tree, it depends on things that aren’t there. Can get around this by setting up the whole tree, but then it’s just like dunetpc now.  For LArSoft, can people set up subsets or must they run the whole thing?
      1. Erica: LArSoft depends on having experiment code — no native detector, for instance, so can’t run anything outside the context of an experiment. Doing an integration type test therefore requires a lot of repositories. I would expect to set up everything to do integration tests. For unit tests, the repositories should be stand-alone if done correctly.. Mrb test runs unit tests at build time on one repository at a time. Historically, there were integration tests (so full art workflows) put in the unit test part. As an aside, would encourage DUNE to strip all that out, put all integration tests into CI workflows. Can define many workflows there if don’t want them all to be run automatically. Then make sure all unit tests are stand-alone, so testable one repository at a time with ‘mrb test’. 
    2. David Adams gave a talk yesterday. Again advocated structuring around art tools
      1. LArSoft MT work is de-servicing as much as we can, since at least some of the current services don’t need to be (e.g., (things where there is no need for global scope). Really just there to take advantage of art state transitions
      2. State transitions can be handled at module scope with tools in many cases. 
      3. Noted that ProtoDUNE pulls event data from a DB. Beam configuration is at the spill level (where a spill is ~15 sec long). Need to optimize DB access for these cases.
    3. Discussed FHiCL structures again w/in context of re-factoring repositories. Long discussion about trade-offs of aggregating configuration versus layering, shortcomings of the current scheme, other ways to organize the layering, the utility of base configurations,… Difficult to summarize, and no clear conclusions.
    4. Noted that since https access to Redmine repositories has been removed, many collaborations who want to develop code (international developers in particular) can no longer check out DUNE code.
      1. So want to deploy to GitHub. 
      2. Wes noted SBN was happy with move to GitHub and use of pull requests. Resulting in better and more stable code. Having pull request mechanism in place is helping to improve the quality and stability of the code. They are starting to do code reviews as the code comes in. Erica echoed similar situation for LArSoft. Particularly good that with pull requests, LArSoft is able to  test the code before merging. Tom not sure DUNE has the effort available. 
      3. Wes commented that there are instructions on Redmine for how to set up a mirror on GitHub, 
    5. Also a discussion of factoring along functional versus detector axes. DUNE has a lot of detectors, so much of the organization is along detector lines. A lot of the code is detector-specific. Can’t use ProtoDUNE code for DUNE FD.
      1. Wes offered to share examples of using two detectors — SBND and ICARUS with common SBN underneath. Driven by how people work. Have two collaborations working together, though, which may not fit the DUNE model as well.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

May 2021 Meeting Notes

This Offline Leads status update was handled via email and a google document.

LArSoft – Erica Snider

  • The previously proposed rollback of hdf5 package will not be necessary. We have the required e20 builds, which required patches to an externally supported package. Thank you to those who followed up with testing the rollback.
  • The migration to art 3.09 is in progress, and is expected to be completed by mid-May. This new version comes with three associated changes:  
    • e20 as the default build qualifier 
    •  new version of root that addresses a problem reading certain files (issue #25615). This version of art is also compatible with cet_modules, and will enable the first phase of migration to the Spack-based build system
    • Tensorflow v2.3
  • SBN previously requested assistance and possibly a tutorial on profiling tools and techniques. The SciSoft team is prepared to provide this assistance. SBN should make a specific request via the Offline Leads Meeting, or Redmine issue ticket
  • Update on the status of memory footprint increase reported by DUNE:  Tom Junk reported some progress on the DUNE side. There has been no further progress to report from the LArSoft side. Kyle Knoepfel is tasked with following up.
  • The project has no progress to report on geometry extensions for pixel detectors

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

dunepdsprce, dune-raw-data and dunetpc have been compiled and tested with e20.  It took a little maintenance as data read-in methods sometimes involved creating pointers to elements packed structures complaining about possible unaligned data; e20 emits a warning with these.  All fixed, though if someday in the future 32-bit objects get padded unless we say packed, we could be in for more maintenance.  Tom’s progress with the memory footprint issue consisted of identifying software components that take more memory in larsoft v09_16_00 as compared with v09_15_00, and a lot of it seems to be what ROOT loads with it.

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee

No Report

SBND – Andrzej Szelc

No Report

SBN Data/Infrastructure – Joseph Zennamo, Wesley Ketchum

From email: We have opened a request for a profiling tutorial for SBN developers:

https://cdcvs.fnal.gov/redmine/issues/25831

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

April 2021 Offline Leads Meeting Notes

Offline Leads Meeting – April 22, 2021

Attendees: Joseph Zennamo, Tingjun Yang, Erica Snider, Katherine Lato

1) We had a request to migrate “best effort” Ubuntu support from LTS 18 to LTS 20. This requires building under gcc v9 (e20). This now works, so will begin “best effort” for LTS 20. Proposing to move to e20. DUNE and uB have products that will need to be rebuilt with e20.

Discussion: Joseph asked about the impact of shifting to e20? 

Typically aren’t changes to interfaces, but compilers get better at enforcing the standard. Sometimes code that compiles in an earlier version of a compiler doesn’t compile because the code wasn’t compliant with the standard. 

Tingjun noted that they tried to move to e20 for ArgoNeut code. Has some issues with warnings in TenserFlow. Lynn provided a solution, they’re going to test that. May have similar issues with DUNE, should start testing it.

LArSoft will migrate once experiments give the all-clear.

2) There is a request to migrate to TensorFlow v2.3. The project is ready to do this, but we need people from the experiments to check that everything works as required under the new version. Only larrecodnn uses TensorFlow within core LArSoft.  Both argoneutcode and dunetpc use tensorflow.

Discussion: Have expanded the scope of this migration to include moving to the next version of TensorRT (now re-branded as Triton) at the same time. 

Leigh Whitehead said in email several weeks back that  DUNE is ready for TF v2.3. Tingjun noted that some things have changed, so they need to run some of the tests again.

3) Rollback of hdf5 v1_12 to hdf5 v1_10. (Noted that the older version builds with e20)

Discussion: SBN has no immediate use for this, but given their drive to use HPC resources, expect that HDF5 conversions will be a part of the workflow at some point. No opinion at this time.

Discussion at last LCM suggested DUNE is ok with a temporary rollback to hdf5 v1_10. Need to confirm.

4) Round table:

Tingjun: ArgoNeuT and DUNE issues for LArSoft

 

  1. A producer module crashes when reading older data. Submitted an art ticket for that. Kyle was consulting, but it’s been a while and it may impact us soon. Urgent. https://cdcvs.fnal.gov/redmine/issues/25615
  2. Since LArSoft moved to new ROOT version see a 20% increase in memory usage for DUNE production jobs. Reported this via LArSoft issue. Tom Junk is the contact. Not as urgent, but it has impact on our production since we have to request more slots. https://cdcvs.fnal.gov/redmine/issues/25512 
  3. Tom Junk additions after the meeting via email:
    1. dunetpc compiles (and links) with e20 but I have yet to run anything more than an event display with it.  There’s an e20 build of tensorflow v1_12_0d that is included in the dependency tree when I built dunetpc just now with e20.
    2. There were a couple of things in dune-raw-data and dunepdsprce that caused gcc v9_3_0 to emit new warnings but these have been straightened out.
    3. I have tested the rollback to hdf5 v1_10 with the raw data readin source we have in dune-raw-data and it works. There’s now a dune_raw_data v1_18_01 which builds with the older hdf5, ready to go when the rollback is deployed. DUNE also depends on hdf5 via hep_hpc, and there is a rolled-back version of that now (with e20 even.  Thanks, Lynn!) I am discussing with Kyle about how best to do delayed reading with HDF5.  This is important to keep memory consumption down for the DUNE far detector and even helps us with ProtoDUNE data, which I assume will be in HDF5 format moving forwards, if I read peoples’ slides right.  We had it working with ROOT, but it will take some design and coding to get it right with HDF5.
    4. Regarding the memory increase with larsoft v9_16_00 (ROOT v6_22), I ran valgrind and spotted a few things that were taking more memory.  I don’t have solutions, however.

SBN: 

  1. Just started a workflow group with SBN to digest how they’re going to do everything and think about it. There may be things that affect LArSoft in the future, but not right now.
  2. What kind of support on profiling SBN code is there?  For the first pass, is it possible to invite a profile expert to a SBN meeting to help developers and analyzers learn how to use these tools? At least for the simpler ones. Need a discussion suitable for analyzers.

Erica: SciSoft team can assist with profiling. The lab provides a set of profiling tools, though it changes with time. There is expertise within SCD in using these tools. Will try to find someone to provide the requested tutorial.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

February 2021 Offline Leads Meeting Notes

The February 2021 LArSoft Offline Leads status update were handled via email and a google document.

LArSoft – Erica Snider

LArSoft has migrated to art 3.06 as of LArSoft v09_16_00 released on Feb 4. The new version of art includes an update to root, v6_22_06a. Note that this version of root requires an additional set of ups qualifiers, e19:p383b:prof, etc.

In anticipation of the migration to the spack packaging system, we have begun work to migrate the LARSoft build to use cetmodules, which uses spack and CMake instead of cetbuildtools. cetmodules is backwards compatible with UPS, so can be used with either packaging system. There will be no impact on developers and end users when this change is introduced. More details will be provided soon once the full migration plan is completed.

There is a request to migrate to TensorFlow v2.3. The project is ready to do this, but we need people from the experiments to check that everything works as required under the new version. Only larrecodnn uses TensorFlow within core LArSoft. Both argoneutcode and dunetpc use tensorflow.

We also have a request to migrate “best effort” Ubuntu support from LTS 18 to LTS 20. This will probably require that we also migrate to gcc v9 (e20). Please send any comments you have regarding either of these changes. 

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

We upgraded DUNE’s stack to the new art and root with very few issues.  A development hdf5 reader module in dune-raw-data had used H5Cpp.h which is now removed and we are looking for alternative methods in hep_hepc but are having troubles finding a method to access file attributes. We are testing the tensorflow v2.3 product.

SBN Data/Infrastructure – Joseph Zennamo, Wesley Ketchum

We did a minor refactoring of some of our experiment code to ease further development and deployment: this has meant setting up an ‘sbnana’ that has minimal dependencies on art/LArSoft, helping us push forward on the CAFAna analysis framework development. We’re working on being ready for e20 releases ASAP, to help the Ubuntu migration to LTS20 go through faster. We’re also looking forward to the updates to the Event Reweighting presented by S. Gardiner, as that will be necessary for handing reweighting systematics in SBN sensitivity studies.

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee

No Report

SBND – Andrzej Szelc

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

January 2021 Offline Leads Meeting Notes

Offline Leads Meeting – Jan. 14th

Tom Junk, Tingjun Yang, Erica Snider, Katherine Lato

LArSoft Update:

  • Spack Update: SciSoft, after getting art to work with Spack, is now focusing on LArSoft. Kyle is working on a draft of the migration plan. 
    • Tom has looked at a few examples, trying things. Did not seem to be “easier” than current ‘system. The hope is that it at least solves some of the technical issues with portability, etc, and is no worse an experience for users. 
    • Two phases for the transition. In the first phase, allow Spack to operate with the underlying UPS. The second will do away with UPS.
  • Tickets for “numerous” items from work plan discussion were mainly ICARUS and SBND, who weren’t at the meeting. Documentation for running in containers was a DUNE request. No one present had tried them, so no feedback on this.
  • Documentation in general. Who do we target? New people, or experts? Is it complete? LArSoft asked for guidance on what to focus on and missing pieces. Also help with identifying things that are out of date.
  • Mu Wei presented Algorithms for calculating number of ions and photons at the January 12th LArSoft Coordination meeting, which led to a proposal to drop support for the Separate algorithm (where number of electrons and photons are uncorrelated). Discussed with those present, who agreed dropping Separate is reasonable.

Round Robin, DUNE:

  • Tom asked about Github going to two-factor authorization in August. Pull request was awkward. User side documentation isn’t as obvious as the administrative, but Tom did find it on the web. Might be helpful to add some pointers in the LArSoft wiki to the correct documentation in GitHub
  • Tingun mentioned a computing tutorial they’re having next Friday that will be going through a lot of LArSoft wiki pages. LArSoft asked to let us know if things are out of date, or need updating prior to that.
  • DUNE has a plan to split dunetpc but have been waiting, first on Spack migration. Have since decided they don’t need to wait on that. Discussed some of the technicalities, such as needing to do the build themselves piece by piece or getting an FWBuild-like recipe. Biggest work is probably breaking up into pieces, which they can do at any time. Will keep LArSoft advised.
  • Tingjun asked about status on updating TensorFlow. Erica noted that as of mid-December, there was a Tensorflow v2.3 build available.  Will inquire about plans for migrating LArSoft, and see about expediting them. 

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

November 2020 Offline Leads Meeting Notes

November Offline Leads Meeting, November 10, 2020.

Attendees: Tracy Usher, Wes Ketchum, Tingjun Yang, Herb Greenlee, Tom Junk, Joseph Zennamo, Erica Snider, Katherine Lato

Discussed the status of the 2021 LArSoft work plan after the series of meetings with each experiment. Some of the items now in the short term priority list were in the 2020 work plan as long term items, so part of the new plan is a reshuffling of priorities. A few items mentioned in the meetings are more appropriately handled as tickets, so will open those, get them assigned, and track the work. There were also requests for consultation, which we may also track via tickets or as a work item in the plan.

Question from Wes about Spack: What about packaging the number of things experiments need, but that are not part of LArSoft? 

Erica: Note, a lot of the products that we care about have already been packaged in Spack. One of the benefits of migrating. We envision that all the changes to LArSoft and experiment code would be made in advance of the  migration to Spack, in a fashion similar to other migrations we have done. That doesn’t include things outside LArSoft that experiments depend on, say for analysis, and we’ve so far not considered this issue. We’ll discuss the transition process in more detail once we get the new build system. There will be time to address those types of concerns at that time.

Wes: I assume the priority is high for LArSoft?

Yes.

Wes: This is more for Tom & Tingjun. I’m aware of efforts on HDF5 (Hierarchical Data Format) and wanting to write to HDF5 files from the DAQ. Not super immediate, but it might come up on the timescale of protoDUNE 2, which isn’t that far away. This may have impact on thinking about data formats and common analysis tools. NOvA did a fair bit of work to convert to HDF5. Is that something that needs to fit in the work plan somewhere? At a minimum doing data format conversation, or support for underlying things?

Tom: Kurt has supplied a basic data converter. Can’t read small parts of events, but that could be re-architected, so not a huge problem. There are generally more features in ROOT for reading files than for HDF5, including being able to make plots by clicking on a variable, etc. Also a lot of meta-data and headers we need to survive are missing in Kurt’s solution. It is a reinvention of the stuff we have in ROOT, but it’s a more modern solution to use HDF5.

Wes: We can talk about the end analysis part, but not sure if it’s needed.

Tingjun: Pandora needs planning meetings to incorporate machine learning. What’s the long-term plan for that?

Erica: That’s been on our list for some time. It came up in our meetings with the experiments. What’s been lacking is a clear project objective. Wasn’t much in general we could do except agree on an image format starting from wires or hits. There have been requests to include TensorFlow and PyTorch. Those can handled as tickets. Beyond this, it is unclear what is needed in terms of LArSoft infrastructure to better support machine learning. [Input on this is welcome.]

Still need to talk to people to turn some of the items in the plan into a detailed task list, which is true every year. Expect us to contact you for input to this part of the process.

Round robin:

  • SBN, Wes
    • In the process of getting production. It’s running now, and preparing for a much larger production in the new year. Some of the things we talked about (LArG4) are targeted for that. Want to close the loop in the next month. 
  • ICARUS, Tracy: 
    • The list you have is pretty complete. We’re just busy trying to get ourselves going. Multiple meetings within a few weeks of each other are taxing our resources.
  • DUNE, Tom: 
    • About pixels and the near detector. The software is already disjoint, with the near detector working in their own environment. That may be good because they’re in a hurry. At some point, they’ll need to be integrated.What’s in the plan is a good list.
  • ArgoNeuT, Tingjun: 
    • Several analyses still going on, and we appreciate the continuing support from the LArSoft team.
  • MicroBooNE, Herb: 
    • Nothing much to add. I heard you mention the things that I mentioned before.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

September 2020 Offline Leads Meeting Notes

LArSoft – Erica Snider

Every year we query the experiments about their plans for the upcoming year in preparation for the LArSoft work plan. These meetings between each experiment and Erica and Katherine will begin in September and continue into October with a draft 2021 LArSoft work plan being available in November and presented at the December Steering Group meeting.  We expect that the major focus will continue to be modernizing the code to be multi-threaded, or to run in multi-threaded environments. We will also continue to facilitate work toward using LArSoft on HPC resources and platforms as they become available. Expect that interest in using deep learning for LAr reconstruction will continue to grow. LArSoft will want to facilitate using these methodologies within art / LArSoft jobs. 

Pixel LArTPC readouts. 

  • Considering geometry service overhaul to accommodate pixels, where a complete re-factoring is indicated. While there, better factorize channel mapping initialization to simplify stand-alone construction and minimize dependencies.

Decoupling generators from LArSoft:  the key to solving this problem is to address how to initialize the geometry information needed by the generator without creating a compile-time dependence on a particular version of LArSoft.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby (from Alex Himel) 

Work continues on transitioning from the old to new LArG4, with good progress in protoDUNE simulation now being transferred to the Far Detector. We continue to take advantage of close collaboration with SBND to adopt improvements in photon simulation. At the most recent DUNE collaboration meeting, we held our first joint meeting between ND and FD Sim/Reco groups to start building towards more cohesive software across both detectors. 

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Andrzej Szelc

SBND is working towards preparing for a small MC production bound to happen in October in preparation for a larger-scale production in the fall. One of the things that is being worked on is an update/tweaking to the semi-analytic model of light simulation (Patrick Green). In parallel, there is an ongoing effort to get SBND running on the factorized version of LArG4. Preliminary tests look promising. In parallel work ongoing to implement a new way to use calorimetry with SCE corrections (G. Putnam, E. Tyley).

SBN Data/Infrastructure – Joseph Zennamo, Wesley Ketchum

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

August 2020 Offline Leads Meeting Notes

Attendees: Tom Junk (DUNE), Tracy Usher (ICARUS), Erica Snider, Katherine Lato

LArSoft

  • 2021 Workplan – we will be contacting each experiment about their plans for 2021
    • We expect that the major focus will continue to be modernizing the code to run multi-threaded, or to run in multi-threaded environments. Will also continue to facilitate work toward using LArSoft on HPC resources and platforms as they become available.
    • Deep learning
      • Expect that interest in using DL for LAr reconstruction will continue to grow. Already an effort in MicroBooNE to use DL to perform vertex reconstruction, a significant addition to the DL reconstruction chain.
      • LArSoft will want to facilitate using these methodologies within art / LArSoft jobs. At the very least, image translation for input to DL algorithms, and translation code to port the results back into art / LArSoft.
    • Ideas welcome.

DUNE

  • Asked DUNE whether the FD requires similar photon visibility features to that of ICARUS. They need to have visibility services understand that light can travel between TPCs, and through a semi-transparent cathode plane. Does DUNE have to deal with this as well?
    • Tom: If you want light to go from one TPC to the next, that is a huge volume. Therefore, a parameterized solution would probably work better that a voxelized lookup library in general.
    • The field cages have holes in them, so can admit light from outside the sensitive volume. Probably not a serious concern.
    • The photo-detectors are in the APAs, and the cathode planes are opaque and non-reflective, so even though APAs are transparent, light generated on one side will not be visible to other TPCs on the other side of the APA plane. Light can travel across TPC boundaries vertically, and horizontally within the confines of the enclosed APA and CPA planes.
    • Visibility parameterization will need to understand this detail of the geometry.
  • Discussed the need to integrate pixels and alter the geometry to accommodate them. This is relevant to the ND, and to some level, the somewhat strained way of handling the strip readout in dual-phase detectors.
    • Tom mentioned a student, or postdoc, who is working on their own Python ND simulation, doing diffusion and re-inventing other effects by themselves, giving the output for deep learning. Does not use Geant4, so is missing a lot of physics.
    • Also commented that several groups are working on alternatives to the gas detector. Everyone doing their own thing. 
      • Talking about using edep-sim to join the effort of several groups. This is an interface to Geant4 originally used by a non-liquid argon detector. It can handle the entire detector complex, in principle, so people have been making GDML to describe it all. At present, LArSoft can’t do that.
      • Not clear the experiment wants to build the GAr detector. May just get a stack of magnetized iron/scintillator. A MINERvA-like scintillator detector is also on the table. At the last review the MPD wasn’t even on the agenda. Only talked about an iron range-stack. Looking for international contributions for a GAr detector. Thinking about reconfiguring on day 1 so it is scintillator instead of gas to start, then change later. Things are up in the air. 
      • People are using simpler software chains for all of this work — not LArSoft, not art. They have little time and are afraid that the things they are working on might not pan out, so don’t want to make large investments (eg, in LArSoft). One drawback to this approach is that people may encounter a limitation that requires a huge effort that they aren’t able to tackle. Yet, this limitation may have already been addressed in LArSoft.
      • DUNE ND people include Clark McGrew who is the owner of edep-sim: https://github.com/ClarkMcGrew/edep-sim. The DUNE near detector software integration leader is Mat Muether (Wichita State). The DUNE ECAL/GEANT/Software expert is Eldwan Brianne (DESY). ND-LAr software person is  Andy Mastbaum.  Kazu Terao (SLAC) is on reconstruction.
    • DUNE still requests that pixels be integrated in LArSoft, since the pixel LArTPC is baselined for the ND. 
    • Noted that if you don’t have the gas in the detector,  not doing the physics that the gas lets us do.

ICARUS 

  • Two methods for doing light. 
    • Photon library. (Look up table). Effective, known solution, but large
    • Semi-analytic. Smaller, but then need to deal with details of handling optically non-trivial boundaries between TPCs, starting with the semi-transparent cathode plane
    • Still relying on photon library lookup method now. Gianluca has all the pieces to take into account the effects that weren’t in the first one (photons crossing the cathode plane). 
    • There is the SBN PDS group with coveners Diego Garcia and Alessandro Menenegolli. The group has as a goal to have the experiments use the same approach. Not only the cathode effect, but also being on the other side of the PMTs and outside the field cage, which they think will be an issue. These pose serious (but solvable)  problems for the semi-analytic approach. 
    • They are setting up to generate a new photon library. They are working on the semi-analytic approach, but not working full time on it. They are learning how to do this.
    • The experiment needs someone to generate the photon library with all the changes that Gianluca has made. Should happen in a month or so.
    • Diego is pushing that. Alexandra is co-creator of the group. They have a program that they laid out to compare the programs side-by-side. 
  • ICARUS started taking data. 
    • Already at full-field. Ramped up to 50% voltage a week ago. All with zero bias voltages, so the first induction plan is acting as a collection plane. On Tuesday, went up to full field. When they went to 50%, saw tracks. Had a few mapping issues, sorted them out before going to full field. 
    • The events are amazing to look at for having just turned on. Things are looking good. Will turn off on Friday since there is a power outage. Then they are going to turn on again next week. Will see first full events with bias voltages set. Happy to see how quickly it turned on and are producing. Work with MicroBooNE helped. 
  • One other thing that they haven’t been working on because of taking data is the database. 
    • Aside:  Tracy was initially unable to read from it using his laptop for first data. Turned out to be a slow internet connection. It was trying to push the data faster than he could take it, so it gave up. Fermilab fixed some parameters and he no longer has problems. They were very responsive to his initial report, and quickly fixed it.
    • Tracy anticipates problems with DB scaling on the grid, particularly when launching a lot of jobs at once.
    • MicroBooNE observed a DB scaling problem on the grid, particularly when launching a lot of jobs. They solved it by introducing an SQLite back end to the relevant services, and exporting those files with the grid jobs. That solved their problem. 
    • Tracy anticipates that ICARUS will have the same problem because they have 8x more channels. Wants to avoid hitting the same issue by adopting the MicroBooNE solution, written by Herb Greenlee.
    • Herb’s solution is sitting on a feature branch. ICARUS would like that integrated into larevt.
      • Erica will check on the status of this code and see if we can start the process of integrating into LArSoft.
      • Noted that this places an operational burden on the production staff, by requiring that the right files get transported with the jobs. Herb might (or might not) have helpful scripts. Will inquire.
  • Inquired as to status of v09. Need this to run their production in multi-threaded mode.
    • Plan was for this week. Right  now just working through merge conflicts in experiment code.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

July 2020 Offline Leads Meeting Notes

LArSoft – Erica Snider

  • LArSoft has immediate plans to migrate to Geant4 10.6.p01, as discussed at the June 30 LArSoft Coordination Meeting. All experiments should let us know whether they approve of this migration as soon as possible. If not prepared to sign off, let us know the timescale on which your experiment can make a decision, or the specific problem that is blocking approval.
  • Also as discussed at the June 30 LArSoft Coordination Meeting, the first LArSoft v09 series release will be ready shortly after the Geant4 migration (unless there is a significant delay for that). This will be the notional start of support for multi-threaded LArSoft releases, and the end of Mac OS builds in LArSoft distributions. The project team will assist experiments with making code currently in or destined for production workflows thread safe. Please open issue tickets to begin this process for any new requests.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby, Tom Junk

DUNE signs off on GEANT 4.10.6p01 for future LArSoft development.  DUNE requests Pandora LC Content be added to the pandora UPS product, for near detector development.  We are also interested in the bxdecay0 generator and would like to have that incorporated as a standard generator in LArSoft — other experiments may find it useful too. https://cdcvs.fnal.gov/redmine/issues/24614 

ICARUS – Daniele Gibin, Tracy Usher

Preparation for data taking: looking forward to v09 release of LArSoft for multi-threading capability. Also expecting data processing load to start to significantly ramp up on the time scale of weeks. Now working on running the conversion from daq to LArSoft format (the “decoder” job) in a nearline environment and make output available to the TITUS event display. 

Continuing tasks: Will be generating a new photon library in the coming weeks – anticipating 50,000 CPU hours necessary for this project. With data expect better understanding of noise, will be re-tuning signal processing/hit finding, probable impacts on work being done now to better tune Pandora track and shower finding. Have integrated the Wirecell drift simulation, work ongoing to complete integration of 2D deconvolution. 

Database: MicroBooNE had reported issues with data base access when running large production campaigns which was tracked to request conflicts with web database access. Herb Greenlee presented solution based on a file based solution, versus web based, which solved the problem. We are interested as well in this solution, it appears to require integration of a feature branch Herb had created into LArSoft so would like to see this happen. (Side note, issues seen with the channel mapping database, web based, were resolved with the database maintainers adjusting parameters on their side so this seems to be ok for us now, not clear if this was the same problem impacting uboone). 

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

SBN Data/Infrastructure – Joseph Zennamo, Wesley Ketchum

We’ve pushed SBND and ICARUS for signoff on the new G4: don’t anticipate any problems (and sorry for the delay in pushing). Both detectors are making progress on updating their geometries and being ready to use the new refactored LArG4. Once done, we are also interested in testing new recombination models that explicitly tie charge and light formation together, and so that will be a priority leading into the fall.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

May 2020 Offline Leads Meeting Notes

The May 2020 LArSoft Offline Leads status update was handled via email and a google document.  Past meeting notes are available at: https://larsoft.org/larsoft-offline-leads-meeting-notes/ Thank you.

LArSoft – Erica Snider

There have been issues with making the most recent version of Geant4, v10.6.p01, work with the legacy LArG4 infrastructure. At least one header that is used in code copied from Geant4 does not exist in this release. The build of the associated legacy LArG4 code was excluded from the recent LArSoft test release. Initial inspection suggests that the changes needed to fix the problem are isolated to the legacy LArG4, and Robert Hatcher is looking into making them, but he has limited time due to other pressing commitments. We cannot migrate to the new Geant4 until this problem is fixed.

  • Who is still using the legacy LArG4? For those who are, are there plans to migrate to the new system?

The legacy Redmine git repositories were made read-only as of April 24. 

The project continues to make significant efforts toward making important production workflow code for DUNE and SBN thread-safe, and in some cases, making it multi-threaded. This work must be closely coordinated with the experiments so that effort is not spent making unnecessary or low priority changes. Please let us know if you are not fully informed as to the status of on-going work for your experiment in these areas.

Please advise on how the new GitHub / pull-request system for LArSoft is working. Are we meeting the needs of your experiment? Has it raised or lowered barriers to introducing new code or changing existing code? What is the status of plans for your experiment with regard to using GitHub? (We have had extensive conversations and work related to SBN. What about the others?)

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

ProtoDUNE has moved to use refactored larg4 in the latest MC production. Also the EM shower daughters were saved in the files. The maximum memory consumption is above 4 GB on average per larg4 job. Studies showed that the biggest memory consumers are: photon library, simb::MCParticles and sim::SimEnergyDeposits. It was suggested to split the photon detector simulation from the rest of larg4 simulation. This reduced the memory consumption of both jobs to below 3 GB. 

A 1d convolutional neural network based signal ROI finder was developed and showed good performance on ProtoDUNE simulation and data. 

A full reconstruction chain was implemented to reconstruct Iceberg data. 

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

SBN Data/Infrastructure – Joseph Zennamo, Wesley Ketchum

Following up on LArG4: currently both ICARUS and SBND use the legacy larg4, and there may be some issues/guidance needed to update: the key thing right now is the updating the geometry. Given the statement above, we may look to prioritizing this effort.

Currently planning to transition to GitHub May 15. We’ve had some opinions about the extra hurdle on ability to collaborate without having direct access to push branches to the central repos, but some of this may still be growing pains, and some ideas to reduce this on experiment side. If this is an ongoing concern, may want to discuss ways to ease collaborative workflows.

Thanks for hosting the discussion on simulation size/memory at the previous coordination meeting. It will be good to continue to followup/keep track on that.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.