Offline Leads Meeting – Jan. 14th

Tom Junk, Tingjun Yang, Erica Snider, Katherine Lato

LArSoft Update:

• Spack Update: SciSoft, after getting art to work with Spack, is now focusing on LArSoft. Kyle is working on a draft of the migration plan. 
  • Tom has looked at a few examples, trying things. Did not seem to be “easier” than current ‘system. The hope is that it at least solves some of the technical issues with portability, etc, and is no worse an experience for users. 
  • Two phases for the transition. In the first phase, allow Spack to operate with the underlying UPS. The second will do away with UPS.
• Tickets for “numerous” items from work plan discussion were mainly ICARUS and SBND, who weren’t at the meeting. Documentation for running in containers was a DUNE request. No one present had tried them, so no feedback on this.
• Documentation in general. Who do we target? New people, or experts? Is it complete? LArSoft asked for guidance on what to focus on and missing pieces. Also help with identifying things that are out of date.
• Mu Wei presented Algorithms for calculating number of ions and photons at the January 12th LArSoft Coordination meeting, which led to a proposal to drop support for the Separate algorithm (where number of electrons and photons are uncorrelated). Discussed with those present, who agreed dropping Separate is reasonable.

Round Robin, DUNE:

• Tom asked about Github going to two-factor authorization in August. Pull request was awkward. User side documentation isn’t as obvious as the administrative, but Tom did find it on the web. Might be helpful to add some pointers in the LArSoft wiki to the correct documentation in GitHub
• Tingun mentioned a computing tutorial they’re having next Friday that will be going through a lot of LArSoft wiki pages. LArSoft asked to let us know if things are out of date, or need updating prior to that.
• DUNE has a plan to split dunetpc but have been waiting, first on Spack migration. Have since decided they don’t need to wait on that. Discussed some of the technicalities, such as needing to do the build themselves piece by piece or getting an FWBuild-like recipe. Biggest work is probably breaking up into pieces, which they can do at any time. Will keep LArSoft advised.
• Tingjun asked about status on updating TensorFlow. Erica noted that as of mid-December, there was a Tensorflow v2.3 build available.  Will inquire about plans for migrating LArSoft, and see about expediting them. 

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

November Offline Leads Meeting, November 10, 2020.

Attendees: Tracy Usher, Wes Ketchum, Tingjun Yang, Herb Greenlee, Tom Junk, Joseph Zennamo, Erica Snider, Katherine Lato

Discussed the status of the 2021 LArSoft work plan after the series of meetings with each experiment. Some of the items now in the short term priority list were in the 2020 work plan as long term items, so part of the new plan is a reshuffling of priorities. A few items mentioned in the meetings are more appropriately handled as tickets, so will open those, get them assigned, and track the work. There were also requests for consultation, which we may also track via tickets or as a work item in the plan.

Question from Wes about Spack: What about packaging the number of things experiments need, but that are not part of LArSoft? 

Erica: Note, a lot of the products that we care about have already been packaged in Spack. One of the benefits of migrating. We envision that all the changes to LArSoft and experiment code would be made in advance of the  migration to Spack, in a fashion similar to other migrations we have done. That doesn’t include things outside LArSoft that experiments depend on, say for analysis, and we’ve so far not considered this issue. We’ll discuss the transition process in more detail once we get the new build system. There will be time to address those types of concerns at that time.

Wes: I assume the priority is high for LArSoft?

Yes.

Wes: This is more for Tom & Tingjun. I’m aware of efforts on HDF5 (Hierarchical Data Format) and wanting to write to HDF5 files from the DAQ. Not super immediate, but it might come up on the timescale of protoDUNE 2, which isn’t that far away. This may have impact on thinking about data formats and common analysis tools. NOvA did a fair bit of work to convert to HDF5. Is that something that needs to fit in the work plan somewhere? At a minimum doing data format conversation, or support for underlying things?

Tom: Kurt has supplied a basic data converter. Can’t read small parts of events, but that could be re-architected, so not a huge problem. There are generally more features in ROOT for reading files than for HDF5, including being able to make plots by clicking on a variable, etc. Also a lot of meta-data and headers we need to survive are missing in Kurt’s solution. It is a reinvention of the stuff we have in ROOT, but it’s a more modern solution to use HDF5.

Wes: We can talk about the end analysis part, but not sure if it’s needed.

Tingjun: Pandora needs planning meetings to incorporate machine learning. What’s the long-term plan for that?

Erica: That’s been on our list for some time. It came up in our meetings with the experiments. What’s been lacking is a clear project objective. Wasn’t much in general we could do except agree on an image format starting from wires or hits. There have been requests to include TensorFlow and PyTorch. Those can handled as tickets. Beyond this, it is unclear what is needed in terms of LArSoft infrastructure to better support machine learning. [Input on this is welcome.]

Still need to talk to people to turn some of the items in the plan into a detailed task list, which is true every year. Expect us to contact you for input to this part of the process.

Round robin:

• SBN, Wes: 
  • In the process of getting production. It’s running now, and preparing for a much larger production in the new year. Some of the things we talked about (LArG4) are targeted for that. Want to close the loop in the next month. 
• ICARUS, Tracy: 
  • The list you have is pretty complete. We’re just busy trying to get ourselves going. Multiple meetings within a few weeks of each other are taxing our resources.
• DUNE, Tom: 
  • About pixels and the near detector. The software is already disjoint, with the near detector working in their own environment. That may be good because they’re in a hurry. At some point, they’ll need to be integrated.What’s in the plan is a good list.
• ArgoNeuT, Tingjun: 
  • Several analyses still going on, and we appreciate the continuing support from the LArSoft team.
• MicroBooNE, Herb: 
  • Nothing much to add. I heard you mention the things that I mentioned before.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

LArSoft – Erica Snider

Every year we query the experiments about their plans for the upcoming year in preparation for the LArSoft work plan. These meetings between each experiment and Erica and Katherine will begin in September and continue into October with a draft 2021 LArSoft work plan being available in November and presented at the December Steering Group meeting.  We expect that the major focus will continue to be modernizing the code to be multi-threaded, or to run in multi-threaded environments. We will also continue to facilitate work toward using LArSoft on HPC resources and platforms as they become available. Expect that interest in using deep learning for LAr reconstruction will continue to grow. LArSoft will want to facilitate using these methodologies within art / LArSoft jobs. 

Pixel LArTPC readouts. 

• Considering geometry service overhaul to accommodate pixels, where a complete re-factoring is indicated. While there, better factorize channel mapping initialization to simplify stand-alone construction and minimize dependencies.

Decoupling generators from LArSoft:  the key to solving this problem is to address how to initialize the geometry information needed by the generator without creating a compile-time dependence on a particular version of LArSoft.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby (from Alex Himel) 

Work continues on transitioning from the old to new LArG4, with good progress in protoDUNE simulation now being transferred to the Far Detector. We continue to take advantage of close collaboration with SBND to adopt improvements in photon simulation. At the most recent DUNE collaboration meeting, we held our first joint meeting between ND and FD Sim/Reco groups to start building towards more cohesive software across both detectors. 

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Andrzej Szelc

SBND is working towards preparing for a small MC production bound to happen in October in preparation for a larger-scale production in the fall. One of the things that is being worked on is an update/tweaking to the semi-analytic model of light simulation (Patrick Green). In parallel, there is an ongoing effort to get SBND running on the factorized version of LArG4. Preliminary tests look promising. In parallel work ongoing to implement a new way to use calorimetry with SCE corrections (G. Putnam, E. Tyley).

SBN Data/Infrastructure – Joseph Zennamo, Wesley Ketchum

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

Attendees: Tom Junk (DUNE), Tracy Usher (ICARUS), Erica Snider, Katherine Lato

LArSoft

• 2021 Workplan – we will be contacting each experiment about their plans for 2021
  • We expect that the major focus will continue to be modernizing the code to run multi-threaded, or to run in multi-threaded environments. Will also continue to facilitate work toward using LArSoft on HPC resources and platforms as they become available.
  • Deep learning
    • Expect that interest in using DL for LAr reconstruction will continue to grow. Already an effort in MicroBooNE to use DL to perform vertex reconstruction, a significant addition to the DL reconstruction chain.
    • LArSoft will want to facilitate using these methodologies within art / LArSoft jobs. At the very least, image translation for input to DL algorithms, and translation code to port the results back into art / LArSoft.
  • Ideas welcome.

DUNE

• Asked DUNE whether the FD requires similar photon visibility features to that of ICARUS. They need to have visibility services understand that light can travel between TPCs, and through a semi-transparent cathode plane. Does DUNE have to deal with this as well?
  • Tom: If you want light to go from one TPC to the next, that is a huge volume. Therefore, a parameterized solution would probably work better that a voxelized lookup library in general.
  • The field cages have holes in them, so can admit light from outside the sensitive volume. Probably not a serious concern.
  • The photo-detectors are in the APAs, and the cathode planes are opaque and non-reflective, so even though APAs are transparent, light generated on one side will not be visible to other TPCs on the other side of the APA plane. Light can travel across TPC boundaries vertically, and horizontally within the confines of the enclosed APA and CPA planes.
  • Visibility parameterization will need to understand this detail of the geometry.
• Discussed the need to integrate pixels and alter the geometry to accommodate them. This is relevant to the ND, and to some level, the somewhat strained way of handling the strip readout in dual-phase detectors.
  • Tom mentioned a student, or postdoc, who is working on their own Python ND simulation, doing diffusion and re-inventing other effects by themselves, giving the output for deep learning. Does not use Geant4, so is missing a lot of physics.
  • Also commented that several groups are working on alternatives to the gas detector. Everyone doing their own thing. 
    • Talking about using edep-sim to join the effort of several groups. This is an interface to Geant4 originally used by a non-liquid argon detector. It can handle the entire detector complex, in principle, so people have been making GDML to describe it all. At present, LArSoft can’t do that.
    • Not clear the experiment wants to build the GAr detector. May just get a stack of magnetized iron/scintillator. A MINERvA-like scintillator detector is also on the table. At the last review the MPD wasn’t even on the agenda. Only talked about an iron range-stack. Looking for international contributions for a GAr detector. Thinking about reconfiguring on day 1 so it is scintillator instead of gas to start, then change later. Things are up in the air. 
    • People are using simpler software chains for all of this work — not LArSoft, not art. They have little time and are afraid that the things they are working on might not pan out, so don’t want to make large investments (eg, in LArSoft). One drawback to this approach is that people may encounter a limitation that requires a huge effort that they aren’t able to tackle. Yet, this limitation may have already been addressed in LArSoft.
    • DUNE ND people include Clark McGrew who is the owner of edep-sim: https://github.com/ClarkMcGrew/edep-sim. The DUNE near detector software integration leader is Mat Muether (Wichita State). The DUNE ECAL/GEANT/Software expert is Eldwan Brianne (DESY). ND-LAr software person is  Andy Mastbaum.  Kazu Terao (SLAC) is on reconstruction.
  • DUNE still requests that pixels be integrated in LArSoft, since the pixel LArTPC is baselined for the ND. 
  • Noted that if you don’t have the gas in the detector,  not doing the physics that the gas lets us do.

ICARUS 

• Two methods for doing light. 
  • Photon library. (Look up table). Effective, known solution, but large
  • Semi-analytic. Smaller, but then need to deal with details of handling optically non-trivial boundaries between TPCs, starting with the semi-transparent cathode plane
  • Still relying on photon library lookup method now. Gianluca has all the pieces to take into account the effects that weren’t in the first one (photons crossing the cathode plane). 
  • There is the SBN PDS group with coveners Diego Garcia and Alessandro Menenegolli. The group has as a goal to have the experiments use the same approach. Not only the cathode effect, but also being on the other side of the PMTs and outside the field cage, which they think will be an issue. These pose serious (but solvable)  problems for the semi-analytic approach. 
  • They are setting up to generate a new photon library. They are working on the semi-analytic approach, but not working full time on it. They are learning how to do this.
  • The experiment needs someone to generate the photon library with all the changes that Gianluca has made. Should happen in a month or so.
  • Diego is pushing that. Alexandra is co-creator of the group. They have a program that they laid out to compare the programs side-by-side. 
• ICARUS started taking data. 
  • Already at full-field. Ramped up to 50% voltage a week ago. All with zero bias voltages, so the first induction plan is acting as a collection plane. On Tuesday, went up to full field. When they went to 50%, saw tracks. Had a few mapping issues, sorted them out before going to full field. 
  • The events are amazing to look at for having just turned on. Things are looking good. Will turn off on Friday since there is a power outage. Then they are going to turn on again next week. Will see first full events with bias voltages set. Happy to see how quickly it turned on and are producing. Work with MicroBooNE helped. 
• One other thing that they haven’t been working on because of taking data is the database. 
  • Aside:  Tracy was initially unable to read from it using his laptop for first data. Turned out to be a slow internet connection. It was trying to push the data faster than he could take it, so it gave up. Fermilab fixed some parameters and he no longer has problems. They were very responsive to his initial report, and quickly fixed it.
  • Tracy anticipates problems with DB scaling on the grid, particularly when launching a lot of jobs at once.
  • MicroBooNE observed a DB scaling problem on the grid, particularly when launching a lot of jobs. They solved it by introducing an SQLite back end to the relevant services, and exporting those files with the grid jobs. That solved their problem. 
  • Tracy anticipates that ICARUS will have the same problem because they have 8x more channels. Wants to avoid hitting the same issue by adopting the MicroBooNE solution, written by Herb Greenlee.
  • Herb’s solution is sitting on a feature branch. ICARUS would like that integrated into larevt.
    • Erica will check on the status of this code and see if we can start the process of integrating into LArSoft.
    • Noted that this places an operational burden on the production staff, by requiring that the right files get transported with the jobs. Herb might (or might not) have helpful scripts. Will inquire.
• Inquired as to status of v09. Need this to run their production in multi-threaded mode.
  • Plan was for this week. Right  now just working through merge conflicts in experiment code.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

LArSoft – Erica Snider

• LArSoft has immediate plans to migrate to Geant4 10.6.p01, as discussed at the June 30 LArSoft Coordination Meeting. All experiments should let us know whether they approve of this migration as soon as possible. If not prepared to sign off, let us know the timescale on which your experiment can make a decision, or the specific problem that is blocking approval.
• Also as discussed at the June 30 LArSoft Coordination Meeting, the first LArSoft v09 series release will be ready shortly after the Geant4 migration (unless there is a significant delay for that). This will be the notional start of support for multi-threaded LArSoft releases, and the end of Mac OS builds in LArSoft distributions. The project team will assist experiments with making code currently in or destined for production workflows thread safe. Please open issue tickets to begin this process for any new requests.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby, Tom Junk

DUNE signs off on GEANT 4.10.6p01 for future LArSoft development.  DUNE requests Pandora LC Content be added to the pandora UPS product, for near detector development.  We are also interested in the bxdecay0 generator and would like to have that incorporated as a standard generator in LArSoft — other experiments may find it useful too. https://cdcvs.fnal.gov/redmine/issues/24614 

ICARUS – Daniele Gibin, Tracy Usher

Preparation for data taking: looking forward to v09 release of LArSoft for multi-threading capability. Also expecting data processing load to start to significantly ramp up on the time scale of weeks. Now working on running the conversion from daq to LArSoft format (the “decoder” job) in a nearline environment and make output available to the TITUS event display. 

Continuing tasks: Will be generating a new photon library in the coming weeks – anticipating 50,000 CPU hours necessary for this project. With data expect better understanding of noise, will be re-tuning signal processing/hit finding, probable impacts on work being done now to better tune Pandora track and shower finding. Have integrated the Wirecell drift simulation, work ongoing to complete integration of 2D deconvolution. 

Database: MicroBooNE had reported issues with data base access when running large production campaigns which was tracked to request conflicts with web database access. Herb Greenlee presented solution based on a file based solution, versus web based, which solved the problem. We are interested as well in this solution, it appears to require integration of a feature branch Herb had created into LArSoft so would like to see this happen. (Side note, issues seen with the channel mapping database, web based, were resolved with the database maintainers adjusting parameters on their side so this seems to be ok for us now, not clear if this was the same problem impacting uboone). 

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

SBN Data/Infrastructure – Joseph Zennamo, Wesley Ketchum

We’ve pushed SBND and ICARUS for signoff on the new G4: don’t anticipate any problems (and sorry for the delay in pushing). Both detectors are making progress on updating their geometries and being ready to use the new refactored LArG4. Once done, we are also interested in testing new recombination models that explicitly tie charge and light formation together, and so that will be a priority leading into the fall.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The May 2020 LArSoft Offline Leads status update was handled via email and a google document.  Past meeting notes are available at: https://larsoft.org/larsoft-offline-leads-meeting-notes/ Thank you.

LArSoft – Erica Snider

There have been issues with making the most recent version of Geant4, v10.6.p01, work with the legacy LArG4 infrastructure. At least one header that is used in code copied from Geant4 does not exist in this release. The build of the associated legacy LArG4 code was excluded from the recent LArSoft test release. Initial inspection suggests that the changes needed to fix the problem are isolated to the legacy LArG4, and Robert Hatcher is looking into making them, but he has limited time due to other pressing commitments. We cannot migrate to the new Geant4 until this problem is fixed.

• Who is still using the legacy LArG4? For those who are, are there plans to migrate to the new system?

The legacy Redmine git repositories were made read-only as of April 24. 

The project continues to make significant efforts toward making important production workflow code for DUNE and SBN thread-safe, and in some cases, making it multi-threaded. This work must be closely coordinated with the experiments so that effort is not spent making unnecessary or low priority changes. Please let us know if you are not fully informed as to the status of on-going work for your experiment in these areas.

Please advise on how the new GitHub / pull-request system for LArSoft is working. Are we meeting the needs of your experiment? Has it raised or lowered barriers to introducing new code or changing existing code? What is the status of plans for your experiment with regard to using GitHub? (We have had extensive conversations and work related to SBN. What about the others?)

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

ProtoDUNE has moved to use refactored larg4 in the latest MC production. Also the EM shower daughters were saved in the files. The maximum memory consumption is above 4 GB on average per larg4 job. Studies showed that the biggest memory consumers are: photon library, simb::MCParticles and sim::SimEnergyDeposits. It was suggested to split the photon detector simulation from the rest of larg4 simulation. This reduced the memory consumption of both jobs to below 3 GB. 

A 1d convolutional neural network based signal ROI finder was developed and showed good performance on ProtoDUNE simulation and data. 

A full reconstruction chain was implemented to reconstruct Iceberg data. 

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

SBN Data/Infrastructure – Joseph Zennamo, Wesley Ketchum

Following up on LArG4: currently both ICARUS and SBND use the legacy larg4, and there may be some issues/guidance needed to update: the key thing right now is the updating the geometry. Given the statement above, we may look to prioritizing this effort.

Currently planning to transition to GitHub May 15. We’ve had some opinions about the extra hurdle on ability to collaborate without having direct access to push branches to the central repos, but some of this may still be growing pains, and some ideas to reduce this on experiment side. If this is an ongoing concern, may want to discuss ways to ease collaborative workflows.

Thanks for hosting the discussion on simulation size/memory at the previous coordination meeting. It will be good to continue to followup/keep track on that.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

Attendees: Tracy Usher, Tingun Yang, Erica Snider, Katherine Lato, Tom Junk (late)

LArSoft report

  Event Displays. Existing options 

1. LArSoft Event Display (ROOT based)
   1. In principle can display everything both in 2D and 3D 
   2. Very slow for large detectors.
   3. Works for all detectors. Can display multiple data items and labels at once
   4. Request from ProtoDUNE to improve speed and better support for multiple TPCs (they still believe it is useful, easy to modify, etc). But plan to look at TITUS
2. TITUS  (Qt based)
   1. Combination of python 3 and C++ and is based on the PyQT5 interface. 
   2. Updated version of the  event display supported by Marco Del Tutto and Gianluca Petrillo (originally authored by Corey Adams for MicroBooNE)
   3. Runs in gallery. Being used by ICARUS. Works for MicroBooNE, ArguNeuT and possibly DUNE. LArSoft would like to see this run in art environment also.
   4. Native 2D displays of low-level data
3. WebEVD (using three.js/WebGL for access to OpenGL) 
   1. Chris Backhouse developed 
   2. In use at ProtoDUNE. Has been updating based on feature requests. 
   3. Runs in LArSoft & Gallery. 
   4. 3D oriented. Doesn’t have a native 2D display.
   5. Request to be a UPS product distributed w LArSoft. Plan to do this. 
4. EVE based for SBND
   1. Umut Kose developed.
   2. 3D Oriented, 
   3. Study of using EVE for event display started after SBN workshop in 2019 at Fermilab. It aims to provide the functions of 3D and 2D projection views, animations, interactive display for users.
   4. Eve is a ROOT module based on experiment-independent part of the ALICE event display developed in cooperation between ALICE offline project and ROOT.
5. Bee event display (based on WebGL)
   1. Developed by Chao Zhang for use with wire-cell.
   2. 3D oriented.
   3. Uses three.js.
   4. Doesn’t run in LArSoft or Gallery,  uses wire-cell data format, so requires conversion step
   5. Authors aiming at PR-level displays

On-going work

• Supporting jenkins build of SBN online monitoring
• Supporting work on spack build of LArSoft / ICARUS code to run on Theta machine.
  • Running on Theta is part of a goal to run on HPC more generally
  • Also exercising the spack stuff
• Working on multi-threading for ProtoDUNE and ICARUS production workflows (data prep through hit-finding)
• LArSoft now using art 3.05, so have started the process of introducing thread-safe service implementations
  • Kyle working on DetectorClocks and DetectorProperties

Tingjun – ProtoDUNE

• Happy LArSoft is moving to TensorFlow 1.12 since it has made development work easier.
• ProtoDUNE is preparing for a new release for production. Uses the refactorized LArG4 simulation. 
• Things are going well. 

Tracy – ICARUS

• Good news on database — got libwda going right way. Now able to access the database and that work is making progress. Don’t need to include extra packages that were initially requested.  Noted that they expect to be able to use the standard LArSoft services for reading the database in most cases. The new code is just for filling the DBs
• There is a new SBN Data/Infrastructure group headed up by Wesley Ketchum and Joseph Zennamo that will be merging much of the SBND and ICARUS software efforts. Goal is to get everything into GitHub and adopt the same cmsbot scheme for pull requests and CI tests that LArSoft uses, etc. 
  • Suggested inviting Wes and Joseph to be part of this group since they are trying to make SBN to be like an experiment, rather than SBND and ICARUS alone. 

Tom Junk – DUNE

• Noted that the DUNE offline people don’t always get to the requests promptly. Now commits hang in level 2 review state longer than they should, so commits take longer than they did in the past. But things were committed before without review. 
  • Noted that the reviews were a good thing. Just need to balance level of review versus need to get the code merged.
  • Concerned about a flurry of work showing up that overwhelms level 2 reviewers. 
    • By design, the system scales by adding level 2 approvers within the experiment. CMS does this with domain-specific level 2 reviewers. Suggested DUNE pursue this if needed to distribute the 2 responsibilities.
• A 2×2 liquid argon pixel detector, ArgonCube/ProtoDUNE-ND, is coming online. That’s the only time pressure at present.
  • Discussed the software situation for that. Though the TDR work being carried out by LBL is in a different software framework, Tom believes the experiment will go with a more unified ND/FD software solution moving forward from there.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

These can be found in the notes for the March Steering Group meeting at:

https://indico.fnal.gov/event/23231/material/1/0.pdf

The February 2020  LArSoft Offline Leads status update was handled via email and a google document. 

LArSoft – Erica Snider

At the behest of SCD management, we have investigated a non-Fermilab partner to take the lead in providing a new LArSoft event display (ED). This effort has not panned out, so we are again in the position of having no effort to put into solving the general ED problem. We are now working on next steps, and are considering a completely different strategy to leverage the three most prevalent non-LArSoft EDs to provide a solution a LArSoft/gallery environment.

Licensing and copyright was discussed at the January 28th LArSoft Coordination Meeting. The proposal  is to have one file per repository that says, “Copyright 2020 Fermilab for the benefit of the LArSoft collaborating experiments.’, and license all but larpandoracontent and larpandora under the Apache License v2. The larpandora package will be licensed under GPL v3, as is required by the Pandora SDK license.

The project is seeking feedback on the question of whether to continue weekly releases. Based on comments received so far, it appears that the experiments might be open to a less frequent integration release schedule. Offline leads should please advise. Some background:

• The intent of weekly releases was to provide stability for people developing code. 
• Because the dependency versioning is tightly coupled to the source code, however, experiments are forced to move their repositories forward as integration releases appear, which in turn typically forces developers to move their working releases forward. 
• The weekly update therefore imposes a burden on developers, which at least partially obviates the hoped for stability frequent integration releases were intended to create.

There have been no significant issues reported related to the GitHub migration so far, only minor and easily fixed failures with workflow scripts. Some general guidance was requested for how to  collaborate on feature branches, which was provided at the Jan 28th LArSoft Coordination Meeting. We are now attempting to improve the usability of the system through improved tools. A first project in this direction is to provide users and L2 approvers a simple, visual way to assess the status of all pull requests within their projects, or that are otherwise of interest to a developer. The “project” view within GitHub, and OctoBox.com have been discussed as possible solutions. 

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

LArSoft released hep_hpc v0_12_01, which depends on hdf5 v1_10_5. The protodune analysis package/repository now depends on hep_hpc v0_12_01. 

Mike Wang implemented the tensorRT inference client in LArSoft/larreco, which allows larsoft job to evaluate convolutional neural networks on GPUs at a remote server. This has sped up significantly the inference of the networks in the ProtoDUNE reconstruction. 

Most of the scripts and code in the dune repositories have been modified to be python3 compatible. 

ICARUS – Daniele Gibin, Tracy Usher

Preparations for data taking continue, target date for turning on the High Voltage and seeing first field on data is approximately the beginning of May. The process for cooling and filling is underway now. 

Much focus has been on the initial stage which will translate from daq format to LArSoft format. In addition, the initial noise filtering – removing coherent noise – will be done followed by the 1D deconvolution and hit finding using the gaushit module. Effort of the past few months here has been focused on insuring all three steps are thread safe with the result that we have recently been able to demonstrate multi-threaded operation of this stage. We continue to refine this step. An issue is that the testing has been done today using feature branches of several LArSoft packages which are simply defining a few services as “SHARED” rather than “LEGACY”. As we approach real data taking it would be nice for these updates to be part of a LArSoft release so we can take advantage of multi-threaded operation. 

There will be strong need for an event display during the commissioning phase which is capable of displaying the full set of RawDigit data at full resolution. With 54,000 channels this is challenging for ICARUS,  the LArSoft event display is not capable of performing this task (in particular over a network connection). As such we have worked with Marco Del Tutto to update the event display originally authored by Corey Adams for MicroBooNE – now known as the TITUS event display. This is a combination of python (3) and C++ and is based on the PyQT5 interface. It works unbelievably well for displaying RawDigits at full resolution, in particular the zoom capability is excellent with no delay or redraw, etc. I personally highly recommend this as something that could be adopted by LArSoft. 

ICARUS needs to start facing up to logistic issues surrounding calibrations. In particular we need to be setting up several databases to handle calibration constants (e.g. preamplifier gains). The big issue is we have yet to identify a “database expert” on our side but also are not sure the process for interacting on the LArSoft side. 

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The January 2020 LArSoft Offline Leads status update was handled via a live meeting on January 22  with attendees: Tracy Usher, Andrzej Szelc, Herb Greenlee, Erica Snider, Katherine Lato

1. GitHub migration – 
   • Have started the migration. 
   • People who have said they will be Level 2 managers (from the experiments) will be in that role. The GitHub side of things is working properly with our scripts. 
   • We expect that there will be test failures. 
     • It’s up to the L2 managers to resolve these if related to problems with their experiment. If it’s some other experiment, everyone will be notified and can work with the author to try and resolve it. 
     • L2 managers should not approve changes that break other experiments before consulting with those experiments. LArSoft will mediate as best we can. We’ll be monitoring the system closely.
   • Feature branches cannot be pushed to the central repositories. They need to be shared out of someone’s private area  or from a GitHub organization if multiple people are using it.
   • Currently trailing white space causes a failure during “code checks”. 
     • We have a script that strips trailing white space that will fix this. Usually we recommend committing cosmetic changes separately from other changes.
     • The script is not context sensitive, so there is a chance that in some cases, using it will change the semantics
   • Herb Greenlee – we have production branches. Do we make our own fork? 
     • They will be migrated. All branches that have names that start with “v” have been migrated. But yes, then you need to make your own fork. 
   • Tracy – SBND and ICARUS have everything mirrored to GitHub already. Just waiting on LArSoft’s migration.
2. Licensing and copyright 
   • We’ve wanted to do this for some time, and with moving to GitHub it seems like this is a good time. 
   • We discussed licensing issues with Aaron Sauers of the Office of Partnership and Technology Transfer. He has advised that we can model a copyright on that used by CERN for CMS code, which states, “Copyright CERN for the benefit of the CMS Collaboration”, with appropriate references. He also confirmed that we can license a portion of LArSoft under GPL v3 (the Pandora interface packages that depend explicitly on GPL v3-licensed Pandora SDK), and the balance of LArSoft under Apache v2, our preferred choice, since the balance of the code does not depend in any way on the Pandora interface components. We plan to move forward with licensing under this guidance.  Wording will be ‘Copyright Fermilab for the benefit of experiments in the LArSoft Collaboration.’
   • Only Pandora was identified as a problem with licenses by the Technology Transfer people. 
   • Apache v2 is in a class that is least restrictive. 
   • GPL v3 forces you to make all the code that touches it to be GPL v3. There’s no reason why someone that touches LArSoft code should be forced to be GPL v3. 
3. Weekly releases vs some alternative 
   • • The intent of weekly releases was to provide stability for people developing code. Because the dependency versioning is tightly coupled to the source code, experiments are forced to move their repositories forward as integration releases appear.
     • Weekly releases creates a burden. (Not just on LArSoft, but on the experiments as well to keep up.) 
     • Are the weekly releases valuable? Or will some other model work as well or better? We’d like to have a discussion about this. 
       • Andrzej would be fine with them less often, but will check. 
       • Herb thinks could have less, but not just on-demand, since scheduled releases are good. 
       • Tracy agrees with Herb. Maybe go to having them every other week, the same week at the LArSoft Coordination meetings.
4. Round-table
   • • Andrzej – SBND
       • Doing work parallelizing. 
         • Getting code running to run on HPC computers, the Theta supercluster at Argonne. They have an allocation there.
         • Also running some Director’s Allocation for MicroBooNE stuff. Ran Genie and Corsika- had to change how random numbers are generated, per event, not per job. 
         • They can run LArSoft out of containers. Have run a reconstruction job. Ran into problem with database web servers. The problem might be on the Argonne side. They don’t allow enough connections to go out to connect to the web servers, not enough slots to go out? Tried to downgrade the number of connections. This caching feature of web servers doesn’t happen since everyone is providing timestamps in nanoseconds so each event has to get its own data rather than sharing. Tried various things, such as down-sampling timestamps. Got it working with a new version that reads a cache file locally, but the feature branch is now two versions higher than the production version.
     • Tracy – ICARUS
       • Still pushing forward to the idea that ICARUS will be taking data soon. Had a mini-collaboration meeting in mid-January. Expect the end of April before ready to turn on high voltage. Getting the  first data out of DAQ system the week of Jan 20th. Starting to show stress points, even reading only 4608 out of 53,000 channels. Bulk of focus is converting data to LArSoft format. Starting to see the first issues, so getting a good head start. 
       • LArSoft Event display isn’t going to be up to 53,000 channels in ICARUS. So far, they only write out (in MC) the channels that have signal. With larger sets, it’s been clear that running it over the network, it doesn’t work. People have given up on that. Over winter break, worked on Event Display based on QT with Gianluca Petrillo providing an interface to services within a gallery environment. So no longer need to hardcode the geometry. Also enables use with other detectors. Marco Del Tutto  has been working on that. Corey Adams and Marco have named their event display “TITUS”. They have released version 1.0 and it has been shown to work with all three of SBND, ICARUS and MicroBooNE detectors. It is not integrated into LArSoft/art proper but, rather, is integrated into gallery. Probably one could change this if desired. 
       •  Umut Kose has continued to develop an Eve based event display. This seems more oriented at the 3D side of things.
     • Herb – MicroBooNE
       • We are migrating to python 3. There is a “2to3” script that does most of it. Plan on running that on everything. It doesn’t make things backwards compatible — it’s just a straight conversion. He’s trying to make things compatible with 2 and 3 for some stuff. LArCV and LArLite need to work for both python 2 and 3 due to analyses working from existing production branches. This conversion is just for integration releases. 
       • Question:  Noted that python 2 won’t be supported after LArSoft migrates to art 3.05. Is it possible that a security issue with python 2 could arise that would make python 2 unusable, and that would not get patched (which would be a problem for the analyses noted above)? Yes, this is possible, but seems unlikely. Otherwise, things that require Python 2 don’t require a lot of maintenance.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.