Attendees: Tom Junk (DUNE), Tracy Usher (ICARUS), Erica Snider, Katherine Lato

LArSoft

• 2021 Workplan – we will be contacting each experiment about their plans for 2021
  • We expect that the major focus will continue to be modernizing the code to run multi-threaded, or to run in multi-threaded environments. Will also continue to facilitate work toward using LArSoft on HPC resources and platforms as they become available.
  • Deep learning
    • Expect that interest in using DL for LAr reconstruction will continue to grow. Already an effort in MicroBooNE to use DL to perform vertex reconstruction, a significant addition to the DL reconstruction chain.
    • LArSoft will want to facilitate using these methodologies within art / LArSoft jobs. At the very least, image translation for input to DL algorithms, and translation code to port the results back into art / LArSoft.
  • Ideas welcome.

DUNE

• Asked DUNE whether the FD requires similar photon visibility features to that of ICARUS. They need to have visibility services understand that light can travel between TPCs, and through a semi-transparent cathode plane. Does DUNE have to deal with this as well?
  • Tom: If you want light to go from one TPC to the next, that is a huge volume. Therefore, a parameterized solution would probably work better that a voxelized lookup library in general.
  • The field cages have holes in them, so can admit light from outside the sensitive volume. Probably not a serious concern.
  • The photo-detectors are in the APAs, and the cathode planes are opaque and non-reflective, so even though APAs are transparent, light generated on one side will not be visible to other TPCs on the other side of the APA plane. Light can travel across TPC boundaries vertically, and horizontally within the confines of the enclosed APA and CPA planes.
  • Visibility parameterization will need to understand this detail of the geometry.
• Discussed the need to integrate pixels and alter the geometry to accommodate them. This is relevant to the ND, and to some level, the somewhat strained way of handling the strip readout in dual-phase detectors.
  • Tom mentioned a student, or postdoc, who is working on their own Python ND simulation, doing diffusion and re-inventing other effects by themselves, giving the output for deep learning. Does not use Geant4, so is missing a lot of physics.
  • Also commented that several groups are working on alternatives to the gas detector. Everyone doing their own thing. 
    • Talking about using edep-sim to join the effort of several groups. This is an interface to Geant4 originally used by a non-liquid argon detector. It can handle the entire detector complex, in principle, so people have been making GDML to describe it all. At present, LArSoft can’t do that.
    • Not clear the experiment wants to build the GAr detector. May just get a stack of magnetized iron/scintillator. A MINERvA-like scintillator detector is also on the table. At the last review the MPD wasn’t even on the agenda. Only talked about an iron range-stack. Looking for international contributions for a GAr detector. Thinking about reconfiguring on day 1 so it is scintillator instead of gas to start, then change later. Things are up in the air. 
    • People are using simpler software chains for all of this work — not LArSoft, not art. They have little time and are afraid that the things they are working on might not pan out, so don’t want to make large investments (eg, in LArSoft). One drawback to this approach is that people may encounter a limitation that requires a huge effort that they aren’t able to tackle. Yet, this limitation may have already been addressed in LArSoft.
    • DUNE ND people include Clark McGrew who is the owner of edep-sim: https://github.com/ClarkMcGrew/edep-sim. The DUNE near detector software integration leader is Mat Muether (Wichita State). The DUNE ECAL/GEANT/Software expert is Eldwan Brianne (DESY). ND-LAr software person is  Andy Mastbaum.  Kazu Terao (SLAC) is on reconstruction.
  • DUNE still requests that pixels be integrated in LArSoft, since the pixel LArTPC is baselined for the ND. 
  • Noted that if you don’t have the gas in the detector,  not doing the physics that the gas lets us do.

ICARUS 

• Two methods for doing light. 
  • Photon library. (Look up table). Effective, known solution, but large
  • Semi-analytic. Smaller, but then need to deal with details of handling optically non-trivial boundaries between TPCs, starting with the semi-transparent cathode plane
  • Still relying on photon library lookup method now. Gianluca has all the pieces to take into account the effects that weren’t in the first one (photons crossing the cathode plane). 
  • There is the SBN PDS group with coveners Diego Garcia and Alessandro Menenegolli. The group has as a goal to have the experiments use the same approach. Not only the cathode effect, but also being on the other side of the PMTs and outside the field cage, which they think will be an issue. These pose serious (but solvable)  problems for the semi-analytic approach. 
  • They are setting up to generate a new photon library. They are working on the semi-analytic approach, but not working full time on it. They are learning how to do this.
  • The experiment needs someone to generate the photon library with all the changes that Gianluca has made. Should happen in a month or so.
  • Diego is pushing that. Alexandra is co-creator of the group. They have a program that they laid out to compare the programs side-by-side. 
• ICARUS started taking data. 
  • Already at full-field. Ramped up to 50% voltage a week ago. All with zero bias voltages, so the first induction plan is acting as a collection plane. On Tuesday, went up to full field. When they went to 50%, saw tracks. Had a few mapping issues, sorted them out before going to full field. 
  • The events are amazing to look at for having just turned on. Things are looking good. Will turn off on Friday since there is a power outage. Then they are going to turn on again next week. Will see first full events with bias voltages set. Happy to see how quickly it turned on and are producing. Work with MicroBooNE helped. 
• One other thing that they haven’t been working on because of taking data is the database. 
  • Aside:  Tracy was initially unable to read from it using his laptop for first data. Turned out to be a slow internet connection. It was trying to push the data faster than he could take it, so it gave up. Fermilab fixed some parameters and he no longer has problems. They were very responsive to his initial report, and quickly fixed it.
  • Tracy anticipates problems with DB scaling on the grid, particularly when launching a lot of jobs at once.
  • MicroBooNE observed a DB scaling problem on the grid, particularly when launching a lot of jobs. They solved it by introducing an SQLite back end to the relevant services, and exporting those files with the grid jobs. That solved their problem. 
  • Tracy anticipates that ICARUS will have the same problem because they have 8x more channels. Wants to avoid hitting the same issue by adopting the MicroBooNE solution, written by Herb Greenlee.
  • Herb’s solution is sitting on a feature branch. ICARUS would like that integrated into larevt.
    • Erica will check on the status of this code and see if we can start the process of integrating into LArSoft.
    • Noted that this places an operational burden on the production staff, by requiring that the right files get transported with the jobs. Herb might (or might not) have helpful scripts. Will inquire.
• Inquired as to status of v09. Need this to run their production in multi-threaded mode.
  • Plan was for this week. Right  now just working through merge conflicts in experiment code.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

LArSoft – Erica Snider

• LArSoft has immediate plans to migrate to Geant4 10.6.p01, as discussed at the June 30 LArSoft Coordination Meeting. All experiments should let us know whether they approve of this migration as soon as possible. If not prepared to sign off, let us know the timescale on which your experiment can make a decision, or the specific problem that is blocking approval.
• Also as discussed at the June 30 LArSoft Coordination Meeting, the first LArSoft v09 series release will be ready shortly after the Geant4 migration (unless there is a significant delay for that). This will be the notional start of support for multi-threaded LArSoft releases, and the end of Mac OS builds in LArSoft distributions. The project team will assist experiments with making code currently in or destined for production workflows thread safe. Please open issue tickets to begin this process for any new requests.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby, Tom Junk

DUNE signs off on GEANT 4.10.6p01 for future LArSoft development.  DUNE requests Pandora LC Content be added to the pandora UPS product, for near detector development.  We are also interested in the bxdecay0 generator and would like to have that incorporated as a standard generator in LArSoft — other experiments may find it useful too. https://cdcvs.fnal.gov/redmine/issues/24614 

ICARUS – Daniele Gibin, Tracy Usher

Preparation for data taking: looking forward to v09 release of LArSoft for multi-threading capability. Also expecting data processing load to start to significantly ramp up on the time scale of weeks. Now working on running the conversion from daq to LArSoft format (the “decoder” job) in a nearline environment and make output available to the TITUS event display. 

Continuing tasks: Will be generating a new photon library in the coming weeks – anticipating 50,000 CPU hours necessary for this project. With data expect better understanding of noise, will be re-tuning signal processing/hit finding, probable impacts on work being done now to better tune Pandora track and shower finding. Have integrated the Wirecell drift simulation, work ongoing to complete integration of 2D deconvolution. 

Database: MicroBooNE had reported issues with data base access when running large production campaigns which was tracked to request conflicts with web database access. Herb Greenlee presented solution based on a file based solution, versus web based, which solved the problem. We are interested as well in this solution, it appears to require integration of a feature branch Herb had created into LArSoft so would like to see this happen. (Side note, issues seen with the channel mapping database, web based, were resolved with the database maintainers adjusting parameters on their side so this seems to be ok for us now, not clear if this was the same problem impacting uboone). 

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

SBN Data/Infrastructure – Joseph Zennamo, Wesley Ketchum

We’ve pushed SBND and ICARUS for signoff on the new G4: don’t anticipate any problems (and sorry for the delay in pushing). Both detectors are making progress on updating their geometries and being ready to use the new refactored LArG4. Once done, we are also interested in testing new recombination models that explicitly tie charge and light formation together, and so that will be a priority leading into the fall.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The May 2020 LArSoft Offline Leads status update was handled via email and a google document.  Past meeting notes are available at: https://larsoft.org/larsoft-offline-leads-meeting-notes/ Thank you.

LArSoft – Erica Snider

There have been issues with making the most recent version of Geant4, v10.6.p01, work with the legacy LArG4 infrastructure. At least one header that is used in code copied from Geant4 does not exist in this release. The build of the associated legacy LArG4 code was excluded from the recent LArSoft test release. Initial inspection suggests that the changes needed to fix the problem are isolated to the legacy LArG4, and Robert Hatcher is looking into making them, but he has limited time due to other pressing commitments. We cannot migrate to the new Geant4 until this problem is fixed.

• Who is still using the legacy LArG4? For those who are, are there plans to migrate to the new system?

The legacy Redmine git repositories were made read-only as of April 24. 

The project continues to make significant efforts toward making important production workflow code for DUNE and SBN thread-safe, and in some cases, making it multi-threaded. This work must be closely coordinated with the experiments so that effort is not spent making unnecessary or low priority changes. Please let us know if you are not fully informed as to the status of on-going work for your experiment in these areas.

Please advise on how the new GitHub / pull-request system for LArSoft is working. Are we meeting the needs of your experiment? Has it raised or lowered barriers to introducing new code or changing existing code? What is the status of plans for your experiment with regard to using GitHub? (We have had extensive conversations and work related to SBN. What about the others?)

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

ProtoDUNE has moved to use refactored larg4 in the latest MC production. Also the EM shower daughters were saved in the files. The maximum memory consumption is above 4 GB on average per larg4 job. Studies showed that the biggest memory consumers are: photon library, simb::MCParticles and sim::SimEnergyDeposits. It was suggested to split the photon detector simulation from the rest of larg4 simulation. This reduced the memory consumption of both jobs to below 3 GB. 

A 1d convolutional neural network based signal ROI finder was developed and showed good performance on ProtoDUNE simulation and data. 

A full reconstruction chain was implemented to reconstruct Iceberg data. 

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

SBN Data/Infrastructure – Joseph Zennamo, Wesley Ketchum

Following up on LArG4: currently both ICARUS and SBND use the legacy larg4, and there may be some issues/guidance needed to update: the key thing right now is the updating the geometry. Given the statement above, we may look to prioritizing this effort.

Currently planning to transition to GitHub May 15. We’ve had some opinions about the extra hurdle on ability to collaborate without having direct access to push branches to the central repos, but some of this may still be growing pains, and some ideas to reduce this on experiment side. If this is an ongoing concern, may want to discuss ways to ease collaborative workflows.

Thanks for hosting the discussion on simulation size/memory at the previous coordination meeting. It will be good to continue to followup/keep track on that.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

Attendees: Tracy Usher, Tingun Yang, Erica Snider, Katherine Lato, Tom Junk (late)

LArSoft report

  Event Displays. Existing options 

1. LArSoft Event Display (ROOT based)
   1. In principle can display everything both in 2D and 3D 
   2. Very slow for large detectors.
   3. Works for all detectors. Can display multiple data items and labels at once
   4. Request from ProtoDUNE to improve speed and better support for multiple TPCs (they still believe it is useful, easy to modify, etc). But plan to look at TITUS
2. TITUS  (Qt based)
   1. Combination of python 3 and C++ and is based on the PyQT5 interface. 
   2. Updated version of the  event display supported by Marco Del Tutto and Gianluca Petrillo (originally authored by Corey Adams for MicroBooNE)
   3. Runs in gallery. Being used by ICARUS. Works for MicroBooNE, ArguNeuT and possibly DUNE. LArSoft would like to see this run in art environment also.
   4. Native 2D displays of low-level data
3. WebEVD (using three.js/WebGL for access to OpenGL) 
   1. Chris Backhouse developed 
   2. In use at ProtoDUNE. Has been updating based on feature requests. 
   3. Runs in LArSoft & Gallery. 
   4. 3D oriented. Doesn’t have a native 2D display.
   5. Request to be a UPS product distributed w LArSoft. Plan to do this. 
4. EVE based for SBND
   1. Umut Kose developed.
   2. 3D Oriented, 
   3. Study of using EVE for event display started after SBN workshop in 2019 at Fermilab. It aims to provide the functions of 3D and 2D projection views, animations, interactive display for users.
   4. Eve is a ROOT module based on experiment-independent part of the ALICE event display developed in cooperation between ALICE offline project and ROOT.
5. Bee event display (based on WebGL)
   1. Developed by Chao Zhang for use with wire-cell.
   2. 3D oriented.
   3. Uses three.js.
   4. Doesn’t run in LArSoft or Gallery,  uses wire-cell data format, so requires conversion step
   5. Authors aiming at PR-level displays

On-going work

• Supporting jenkins build of SBN online monitoring
• Supporting work on spack build of LArSoft / ICARUS code to run on Theta machine.
  • Running on Theta is part of a goal to run on HPC more generally
  • Also exercising the spack stuff
• Working on multi-threading for ProtoDUNE and ICARUS production workflows (data prep through hit-finding)
• LArSoft now using art 3.05, so have started the process of introducing thread-safe service implementations
  • Kyle working on DetectorClocks and DetectorProperties

Tingjun – ProtoDUNE

• Happy LArSoft is moving to TensorFlow 1.12 since it has made development work easier.
• ProtoDUNE is preparing for a new release for production. Uses the refactorized LArG4 simulation. 
• Things are going well. 

Tracy – ICARUS

• Good news on database — got libwda going right way. Now able to access the database and that work is making progress. Don’t need to include extra packages that were initially requested.  Noted that they expect to be able to use the standard LArSoft services for reading the database in most cases. The new code is just for filling the DBs
• There is a new SBN Data/Infrastructure group headed up by Wesley Ketchum and Joseph Zennamo that will be merging much of the SBND and ICARUS software efforts. Goal is to get everything into GitHub and adopt the same cmsbot scheme for pull requests and CI tests that LArSoft uses, etc. 
  • Suggested inviting Wes and Joseph to be part of this group since they are trying to make SBN to be like an experiment, rather than SBND and ICARUS alone. 

Tom Junk – DUNE

• Noted that the DUNE offline people don’t always get to the requests promptly. Now commits hang in level 2 review state longer than they should, so commits take longer than they did in the past. But things were committed before without review. 
  • Noted that the reviews were a good thing. Just need to balance level of review versus need to get the code merged.
  • Concerned about a flurry of work showing up that overwhelms level 2 reviewers. 
    • By design, the system scales by adding level 2 approvers within the experiment. CMS does this with domain-specific level 2 reviewers. Suggested DUNE pursue this if needed to distribute the 2 responsibilities.
• A 2×2 liquid argon pixel detector, ArgonCube/ProtoDUNE-ND, is coming online. That’s the only time pressure at present.
  • Discussed the software situation for that. Though the TDR work being carried out by LBL is in a different software framework, Tom believes the experiment will go with a more unified ND/FD software solution moving forward from there.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

These can be found in the notes for the March Steering Group meeting at:

https://indico.fnal.gov/event/23231/material/1/0.pdf

The February 2020  LArSoft Offline Leads status update was handled via email and a google document. 

LArSoft – Erica Snider

At the behest of SCD management, we have investigated a non-Fermilab partner to take the lead in providing a new LArSoft event display (ED). This effort has not panned out, so we are again in the position of having no effort to put into solving the general ED problem. We are now working on next steps, and are considering a completely different strategy to leverage the three most prevalent non-LArSoft EDs to provide a solution a LArSoft/gallery environment.

Licensing and copyright was discussed at the January 28th LArSoft Coordination Meeting. The proposal  is to have one file per repository that says, “Copyright 2020 Fermilab for the benefit of the LArSoft collaborating experiments.’, and license all but larpandoracontent and larpandora under the Apache License v2. The larpandora package will be licensed under GPL v3, as is required by the Pandora SDK license.

The project is seeking feedback on the question of whether to continue weekly releases. Based on comments received so far, it appears that the experiments might be open to a less frequent integration release schedule. Offline leads should please advise. Some background:

• The intent of weekly releases was to provide stability for people developing code. 
• Because the dependency versioning is tightly coupled to the source code, however, experiments are forced to move their repositories forward as integration releases appear, which in turn typically forces developers to move their working releases forward. 
• The weekly update therefore imposes a burden on developers, which at least partially obviates the hoped for stability frequent integration releases were intended to create.

There have been no significant issues reported related to the GitHub migration so far, only minor and easily fixed failures with workflow scripts. Some general guidance was requested for how to  collaborate on feature branches, which was provided at the Jan 28th LArSoft Coordination Meeting. We are now attempting to improve the usability of the system through improved tools. A first project in this direction is to provide users and L2 approvers a simple, visual way to assess the status of all pull requests within their projects, or that are otherwise of interest to a developer. The “project” view within GitHub, and OctoBox.com have been discussed as possible solutions. 

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

LArSoft released hep_hpc v0_12_01, which depends on hdf5 v1_10_5. The protodune analysis package/repository now depends on hep_hpc v0_12_01. 

Mike Wang implemented the tensorRT inference client in LArSoft/larreco, which allows larsoft job to evaluate convolutional neural networks on GPUs at a remote server. This has sped up significantly the inference of the networks in the ProtoDUNE reconstruction. 

Most of the scripts and code in the dune repositories have been modified to be python3 compatible. 

ICARUS – Daniele Gibin, Tracy Usher

Preparations for data taking continue, target date for turning on the High Voltage and seeing first field on data is approximately the beginning of May. The process for cooling and filling is underway now. 

Much focus has been on the initial stage which will translate from daq format to LArSoft format. In addition, the initial noise filtering – removing coherent noise – will be done followed by the 1D deconvolution and hit finding using the gaushit module. Effort of the past few months here has been focused on insuring all three steps are thread safe with the result that we have recently been able to demonstrate multi-threaded operation of this stage. We continue to refine this step. An issue is that the testing has been done today using feature branches of several LArSoft packages which are simply defining a few services as “SHARED” rather than “LEGACY”. As we approach real data taking it would be nice for these updates to be part of a LArSoft release so we can take advantage of multi-threaded operation. 

There will be strong need for an event display during the commissioning phase which is capable of displaying the full set of RawDigit data at full resolution. With 54,000 channels this is challenging for ICARUS,  the LArSoft event display is not capable of performing this task (in particular over a network connection). As such we have worked with Marco Del Tutto to update the event display originally authored by Corey Adams for MicroBooNE – now known as the TITUS event display. This is a combination of python (3) and C++ and is based on the PyQT5 interface. It works unbelievably well for displaying RawDigits at full resolution, in particular the zoom capability is excellent with no delay or redraw, etc. I personally highly recommend this as something that could be adopted by LArSoft. 

ICARUS needs to start facing up to logistic issues surrounding calibrations. In particular we need to be setting up several databases to handle calibration constants (e.g. preamplifier gains). The big issue is we have yet to identify a “database expert” on our side but also are not sure the process for interacting on the LArSoft side. 

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The January 2020 LArSoft Offline Leads status update was handled via a live meeting on January 22  with attendees: Tracy Usher, Andrzej Szelc, Herb Greenlee, Erica Snider, Katherine Lato

1. GitHub migration – 
   • Have started the migration. 
   • People who have said they will be Level 2 managers (from the experiments) will be in that role. The GitHub side of things is working properly with our scripts. 
   • We expect that there will be test failures. 
     • It’s up to the L2 managers to resolve these if related to problems with their experiment. If it’s some other experiment, everyone will be notified and can work with the author to try and resolve it. 
     • L2 managers should not approve changes that break other experiments before consulting with those experiments. LArSoft will mediate as best we can. We’ll be monitoring the system closely.
   • Feature branches cannot be pushed to the central repositories. They need to be shared out of someone’s private area  or from a GitHub organization if multiple people are using it.
   • Currently trailing white space causes a failure during “code checks”. 
     • We have a script that strips trailing white space that will fix this. Usually we recommend committing cosmetic changes separately from other changes.
     • The script is not context sensitive, so there is a chance that in some cases, using it will change the semantics
   • Herb Greenlee – we have production branches. Do we make our own fork? 
     • They will be migrated. All branches that have names that start with “v” have been migrated. But yes, then you need to make your own fork. 
   • Tracy – SBND and ICARUS have everything mirrored to GitHub already. Just waiting on LArSoft’s migration.
2. Licensing and copyright 
   • We’ve wanted to do this for some time, and with moving to GitHub it seems like this is a good time. 
   • We discussed licensing issues with Aaron Sauers of the Office of Partnership and Technology Transfer. He has advised that we can model a copyright on that used by CERN for CMS code, which states, “Copyright CERN for the benefit of the CMS Collaboration”, with appropriate references. He also confirmed that we can license a portion of LArSoft under GPL v3 (the Pandora interface packages that depend explicitly on GPL v3-licensed Pandora SDK), and the balance of LArSoft under Apache v2, our preferred choice, since the balance of the code does not depend in any way on the Pandora interface components. We plan to move forward with licensing under this guidance.  Wording will be ‘Copyright Fermilab for the benefit of experiments in the LArSoft Collaboration.’
   • Only Pandora was identified as a problem with licenses by the Technology Transfer people. 
   • Apache v2 is in a class that is least restrictive. 
   • GPL v3 forces you to make all the code that touches it to be GPL v3. There’s no reason why someone that touches LArSoft code should be forced to be GPL v3. 
3. Weekly releases vs some alternative 
   • • The intent of weekly releases was to provide stability for people developing code. Because the dependency versioning is tightly coupled to the source code, experiments are forced to move their repositories forward as integration releases appear.
     • Weekly releases creates a burden. (Not just on LArSoft, but on the experiments as well to keep up.) 
     • Are the weekly releases valuable? Or will some other model work as well or better? We’d like to have a discussion about this. 
       • Andrzej would be fine with them less often, but will check. 
       • Herb thinks could have less, but not just on-demand, since scheduled releases are good. 
       • Tracy agrees with Herb. Maybe go to having them every other week, the same week at the LArSoft Coordination meetings.
4. Round-table
   • • Andrzej – SBND
       • Doing work parallelizing. 
         • Getting code running to run on HPC computers, the Theta supercluster at Argonne. They have an allocation there.
         • Also running some Director’s Allocation for MicroBooNE stuff. Ran Genie and Corsika- had to change how random numbers are generated, per event, not per job. 
         • They can run LArSoft out of containers. Have run a reconstruction job. Ran into problem with database web servers. The problem might be on the Argonne side. They don’t allow enough connections to go out to connect to the web servers, not enough slots to go out? Tried to downgrade the number of connections. This caching feature of web servers doesn’t happen since everyone is providing timestamps in nanoseconds so each event has to get its own data rather than sharing. Tried various things, such as down-sampling timestamps. Got it working with a new version that reads a cache file locally, but the feature branch is now two versions higher than the production version.
     • Tracy – ICARUS
       • Still pushing forward to the idea that ICARUS will be taking data soon. Had a mini-collaboration meeting in mid-January. Expect the end of April before ready to turn on high voltage. Getting the  first data out of DAQ system the week of Jan 20th. Starting to show stress points, even reading only 4608 out of 53,000 channels. Bulk of focus is converting data to LArSoft format. Starting to see the first issues, so getting a good head start. 
       • LArSoft Event display isn’t going to be up to 53,000 channels in ICARUS. So far, they only write out (in MC) the channels that have signal. With larger sets, it’s been clear that running it over the network, it doesn’t work. People have given up on that. Over winter break, worked on Event Display based on QT with Gianluca Petrillo providing an interface to services within a gallery environment. So no longer need to hardcode the geometry. Also enables use with other detectors. Marco Del Tutto  has been working on that. Corey Adams and Marco have named their event display “TITUS”. They have released version 1.0 and it has been shown to work with all three of SBND, ICARUS and MicroBooNE detectors. It is not integrated into LArSoft/art proper but, rather, is integrated into gallery. Probably one could change this if desired. 
       •  Umut Kose has continued to develop an Eve based event display. This seems more oriented at the 3D side of things.
     • Herb – MicroBooNE
       • We are migrating to python 3. There is a “2to3” script that does most of it. Plan on running that on everything. It doesn’t make things backwards compatible — it’s just a straight conversion. He’s trying to make things compatible with 2 and 3 for some stuff. LArCV and LArLite need to work for both python 2 and 3 due to analyses working from existing production branches. This conversion is just for integration releases. 
       • Question:  Noted that python 2 won’t be supported after LArSoft migrates to art 3.05. Is it possible that a security issue with python 2 could arise that would make python 2 unusable, and that would not get patched (which would be a problem for the analyses noted above)? Yes, this is possible, but seems unlikely. Otherwise, things that require Python 2 don’t require a lot of maintenance.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The November 2019 LArSoft Offline Leads status update was handled via email and a google document. Past meeting notes are available at: https://larsoft.org/larsoft-offline-leads-meeting-notes/ 

LArSoft – Erica Snider

Presented the draft 2020 work plan to SCD management on Nov 7. The presentation noted that the work plan calls for 2 to 3 FTEs of development effort, depending primarily on the effort devoted to multi-threading the code, and the amount of work that the project needs to devote to toward various work items to which the experiments can contribute. The project has about 2 FTEs in the budget. No significant concerns were raised by management.

Testing of using LArSoft with GitHub and the pull request approval workflow has been opened to testing by the experiments. All experiment offline leads and release managers are encouraged to exercise the system, and send comments to the SciSoft team (at scisoft@fnal.gov). Instructions for using the system can be found on the LArSoft wiki under the “Working with GitHub” page (also linked from the “Quick Links”, Using LArSoft”, and “Developing with LArSoft” pages.) Any changes made to the repositories during testing will be deleted prior to the production migration. The current plan is to leave testing open for at least two weeks, but to complete the production migration in early December. 

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

ProtoDUNE developed a new way to handle raw digits in order to save memory usage. Tom Junk created a tool that decodes protoDUNE raw data and returns it to the caller in the form of vector<raw::RawDigit>. David Adams modified protoDUNE dataprep to use this tool instead of reading digits from the event data store. Other larsoft modules (such as GausHitFinder and various disambiguation algorithms) are modified accordingly to remove hit and raw digit associations. After all the changes, we see a ~10% reduction in memory usage in ProtoDUNE reconstruction job. This new tool allows us to decode raw data one APA at a time and drop raw digits once recob::Wires are made after deconvolution and is applicable to the DUNE far detector where there are 150 APAs per module. More details can be found in #23177.

DUNE made a patch release v08_27_01_01 with the help of Lynn Garren. This release includes artdaq_core bug fixes and a feature to limit tensorflow to only use one CPU to be more grid friendly. The dunetpc v08_27_01_01 is the current release for ProtoDUNE-SP keep-up production. 

A new package protoduneana is created. This new package depends on dunetpc and only contains ProtoDUNE analysis code. It is meant to speed up compiling time for people doing analysis. 

We are meeting regularly with Erica Snider, Saba Sehrish, and Kyle Knoepfel regarding a project to make the data preparation stages thread safe in order to improve our CPU utilization.

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

10/22/19 Attendees: Tingjun Yang, Thomas Junk, Mike Kirby, David Adams, Erica Snider, Katherine Lato

Agenda:  the 2020 work plan and the GitHub migration plan.

The  GitHub migration

• The project would like to complete the migration as soon as possible. The infrastructure is ready, but the documentation needs more work, and  is critical path for moving into community testing phase. 
• The exact procedures for Level 1 and Level 2 need to be worked out. 
• Need to know immediately who the level 2 managers are going to be. 
  • DUNE:
    • Level 2 managers:
      • Tom Junk
      • Tingjun Yang
      • Christoph Alt (has already been doing it for another project)
      • David Adams
    • How many Level 2 managers needed to approve a PR?
      • One. Expect that DUNE managers will approve only PRs from / for DUNE. (Exceptions ok if the manager is familiar with the work.)
  • ICARUS
    • Tracy Usher (agreed in email after the meeting)
  • Emailed other experiments requesting nominees
• LArSoft will clean the history during the migration. 
• Will experiments migrate to GitHub? 
  • DUNE yes probably yes, in part because Redmine is so slow. Improved performance of GitHub is one of the motivations for the LArSoft move. DUNE has no migration plan yet.
  • Noted that there is already a GitHub area, but it doesn’t have TPC code in it. DUNE will think about how to organize repositories
• Will open up the GitHub for testing once the documentation is in place. Would like the testing period to be as long as needed, but as short as possible.

Went through the draft 2020 work plan. 

• DUNE would like us to fix the existing event display in advance of having a new event display.
• SPACK.
  • Tom ran the MVP, it took all day, took all the memory, but it did work eventually. Not recommended in present state, though.
    • We believe this is related to the “concretization” stage, which is known to be slow. Chris  Green has been introducing significant optimizations, but having some difficulty getting SPACK owners to accept the changes.

• Pixels
  • Check with Mat Muether. Also, Gianluca Petrillo has been working on pixel-related Geometry.

• DUNE suggests we promote magnetic field one to a project.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The September 2019 LArSoft Offline Leads status update was handled via email and a google document.

LArSoft – Erica Snider

While we knew that Lynn Garren was retiring the end of January, 2020, we recently learned that Paul Russo is leaving the middle of September. Paul was providing customer support on LArSoft tickets and expertise on multi-threading for LArSoft. We do not yet know what the replacement strategy is. 

Due to the changes in available resources, the project needs to shift some responsibilities to the experiments, particularly in the area of code and release management. 

• MacOS support – LArSoft support for Mac builds is ending. Could either find someone else to support those builds, or have MacOS developers  use SLF containers. (Note that support for those containers will not come from the SciSoft team.)
• GitHub Migration – Need to identify people in experiments for manager roles defined in the pull request workflow. See slides from Patrick Gartung’s presentation Migration of LArSoft repos to GitHub. We will hold additional discussions and provide opportunities for beta testing prior to the migration to GitHub and deployment of the pull request system.

The project has begun the annual process of gathering input from the experiments to inform the 2020 LArSoft work plan. We have met with all the experiments. An initial draft work plan will be discussed at the October Offline Leads meeting.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

ProtoDUNE has started the second production campaign using larsoft/dunetpc v08_27_01. A ProtoDUNE-SP simulation task force is formed to improve detector simulation. The goal is to incorporate the refactorized larg4 and wirecell simulation. DUNE is in the process of transitioning to use tensorflow v1_12_0b with the help of larsoft team. More details in #22504.

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee

MicroBooNE is preparing a major production campaign (MCC 9.1) based on our MCC9 branch (based off larsoft v08_05_00_10).  No particular requests to larsoft.  

SBND – Roxanne Guenette, Andrzej Szelc

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.