The July & August 2019 LArSoft Offline Leads status update was handled via both a meeting (on July 30th) and a google document. People were asked to fill in the information in the document if they couldn’t make the July 30th meeting. Past meeting notes are available at: https://larsoft.org/larsoft-offline-leads-meeting-notes/ Thank you.

LArSoft – Erica Snider

The LArSoft project requests that the experiments provide feedback on the following two items:

• The LArSoft Spack MVP. See the instructions in the email “Announcing Spack / SpackDev MVP1a (“The LArSoft Edition”)” from Chris Green on June 20 to larsoft@fnal.gov.
  • This will be the build system for art / LArSoft and the replacement for UPS in the near future. Your feedback is critical to ensure that the final product is easy to use (ie, effective, efficient, intuitive, easy to learn, satisfying). This is our chance to change the outcome.
  •  This  PDF explains the MVP in detail, and how to set it up to evaluate the system. 
  • Please send any feedback or questions direct to spackdev-team@fnal.gov by July 20th.
• The migration to Genie v3
  • Although LArSoft is driving an effort to decouple the version of LArSoft from any specific version of Genie, this is not yet possible. Moreover, once it becomes possible, it is unlikely that any version of Genie v2 will be included.
  • We therefore need all experiments to sign off on the v3 migration, or to state their plans with respect to v3 (including any plan to stay at v2.x)
  • NOTE:  as of July 24, all experiments had signed off on the migration

The project is overseeing a considerable amount of work directed at making various parts of LArSoft thread safe, and in cases, parallelizing specific pieces of code. Some of this coding work will soon be integrated into LArSoft.  We expect that a number of services (including experiment-specific provider variants) and various pieces of common code may be affected. 

Work on the migration to GitHub is proceeding. More details of what this will look like to end users should be forthcoming over the next couple of months. We are expecting the main parts of the migration to be completed by mid-fall.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

We are working to make the software for the near detector complex work together.  This includes a pixel-based liquid-argon component, which will likely use LArSoft directly, a high-pressure gas TPC with an ECAL, called the Multi-Purpose Detector which has a software stack based on art and nutools and thus can run together with larsoft code, and a 3D Scintillator Tracker-Spectrometer.  We need to simulate events crossing from one component to another, so a unified GEANT4 simulation is required.  Common triggers and event builders will be needed to take data. A workshop was held on July 24 to discuss these issues.

We are also discussing the FD data model at a workshop at BNL  on August 14-16. We will discuss databases, data streams, DAQ interfaces, framework and memory management, and of course the event model.

DUNE has discussed the GENIE v3 upgrade.  While we are interested in continuing with GENIE v2, we can continue to work with older releases of larsoft and dunetpc.  If we absolutely must mix GENIE v2 with newer simulation and reconstruction code, either a patch release or careful accounting of the GENIE systematics weights needs to be done.

We tried the quick-start instructions for the Spack MVP, LArSoft edition and gave some feedback.

ICARUS – Daniele Gibin, (Tracy Usher)

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

Discussed what MVP meant, does enough to get a feel for it, but can still be changed if needed. 

Thinking about when to merge to develop. Things preventing:

• Genie v3 (which is now resolved)
• IO rule, Have a new pid data product which everyone should want. In order to use it, need the IO rule. Waiting for someone to make the change.

SBND – Roxanne Guenette, Andrzej Szelc

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

Because of the LArSoft Workshop, we didn’t have an Offline Leads meeting in June.

Notes from the workshop can be found at: https://larsoft.org/larsoft-workshop-june-2019/

The May 2019 LArSoft Offline Leads status update was handled via email and a google document.

LArSoft – Erica Snider

• Planning for the LArSoft Workshop 2019 is nearly complete. The agenda and speakers can be found on indico here:  https://indico.fnal.gov/event/20453/other-view?view=standard
• A  “minimally viable” Spack/SpaceDev-based build and development environment for LArSoft is under initial testing within the SciSoft team. The purpose of the current product is to provide sufficient functionality to allow meaningful testing and feedback on Spack and SpackDev. The plan is to distribute this version for testing and comment from the experiments once sufficient documentation and fixes for known bugs are available.
• LArSoft has been asked to upgrade to python 3 which requires dropping SLF6.  Seeking feedback from experiments and users.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

• No Report

ICARUS – Daniele Gibin, (Tracy Usher)

• Experience with running our last MC challenge exposed some issues with memory and cpu time for simulation and reconstruction of events. Primarily this is in the TPC reconstruction but there are also problems in the PMT simulation/reconstruction (primarily associated with the photon lookup library) and the CRT. Work has been underway to try to address issues with the TPC reconstruction first and this has led to a new workflow for both simulation and reconstruction where the signal processing is broken into groups by TPC (there are four TPCs in ICARUS) and then pattern recognition is done by Cryostat (there are two cryostats in ICARUS). The detector simulation then changes to output four collections of RawDigits, one per TPC and this is meant to emulate what the DAQ folks tell us they will do when we begin data taking.

This breakup should enable the ability for some amount of parallelization of the processing, also with each instance of a particular processing module using less memory.

In addition to this, there is also effort underway to reformat some of the signal processing modules to allow for vectorization techniques now available (or soon to be available?) in art. We hope to be in a position to test soon.

• Recent improvements to the photon lookup library have led to an explosion in the memory requirements once read back from a disk file, at last check taking some 6.5 GB of memory. This library is not only more detailed but extends now to both Cryostats. Gianluca Petrillo has implemented some code in LArSoft which will try to make use of symmetries in the library to reduce memory, but this needs to be tested. If this cannot make a significant reduction then it will be necessary to look at alternatives.
• The BNL team is starting preliminary effort to port the WireCell toolkit to ICARUS for simulation and noise filter/2D deconvolution. Currently targeting integration in time for commissioning.
• Finally… there is significant effort in developing machine learning techniques for 3D reconstruction using ICARUS as the current target platform. Toward that end the current LArSoft workflow produces 3D SpacePoints (using “Cluster3D”) which are then siphoned off into the deep learning effort’s workflows using a new version of the LArCV package (now LArCV3). It would be highly useful if this were an “official” LArSoft package, importantly including proper makefile set up to enable builds in the standard LArSoft framework (e.g. using mrb).

LArIAT – Jonathan Asaadi

• No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

• No Report

SBND – Roxanne Guenette, Andrzej Szelc

• No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

Attendees: Erica Snider, Herb Greenlee, Tracy Usher, Thomas Junk, Katherine Lato 

LArSoft Status

1) Discussion with Jim Amundson

• On the  Event Display project. We have a proposal for how to proceed with adopting a web-based event display and recruiting effort to support it. Erica is charged with following up and reporting back on progress. Tracy raised two points.
  • Running on laptops that are not connected to the internet
  • The problem of passing data about display objects over the network
    • The former problem is solved if the server can run locally (which is a requirement)
    • The latter is solved if there is a protocol for matching display data to the screen resolution. There are many successful examples of doing this in the industry.

• On Spack. Plan is to bring more resources to bear on this project, but not clear how or when this will happen.

2) Multi-threading, vectorization and other optimizations

• Tracy commented about talking with Mike and Giuseppe about optimized versions of deconvolution and noise suppression code for ICARUS, building on the multi-threading work Mike did for CalWire_DUNE.  
• ICARUS is willing to look at Dave Adams data-prep code as a possible common framework for performing this and similar calculations. Interfaces can be common if inputs are raw digits or wires, and outputs are wires. Provides flexibility to customize algorithms as experiments need.
• They also discussed 2D drift model that has been worked on a little. They demonstrated that it’s working and running 6 to 8 times slower than 1D, and are working to make it run faster. That could be useful in production if it could be made faster. That part of the code should lend itself to vectorization.

3) LArSoft Workshop planning

• Workshop planning and agenda – https://indico.fnal.gov/event/20453/
• The basic plan is a tutorial that starts on the afternoon of the 24th. Second day is multi-threading and vectorization in the morning followed by long-term vision of LArSoft discussion in the afternoon.
  • Could ICARUS contribute to long-term vision? Yes.
  • And the deep learning people could contribute.
  • ICARUS & DUNE should be talking to each other to build the event in a way that allows the possibility of sequentially processing blocks of low-level event data, such as raw digits, on the scale of a single TPC, while limiting the memory footprint to that block.

Round table

ICARUS – Tracy

• Have a lot of stuff to get done and not a lot of time left. Basic structure is in place. Need more people.
• Deep learning people at SLAC are using ProtoDUNE and ICARUS data to develop DL algorithms. In the next six months, there should be feedback from that effort that could be useful in long-term strategy.
• We are running simulations, Pandora, and light reconstruction, generally the full simulation and reconstruction workflow. The pathway works. Need to get serious about being ready for data taking. Have hooks in place for noise model. Doesn’t leave time for thinking about larger-scale problems. Optical reconstruction could be interesting. The one imported from MicroBooNE is basic, but the potential is there for something more sophisticated.

DUNE – Tom

• Personally working on GArSoft more than LArSoft last few months.
• ProtoDUNE analysis are coming along well. We don’t simulate electron diverters. The space charge parameterization is smooth, has an analytic smoothing function, doesn’t have sharp corners, as occurs near the diverters, so the edges are all wrong. But it’s probably on us to provide a map.
• The photon libraries were gigantic, Alex Himmel has a solution. A nearly fully parameterized solution.  (This was presented at a LCM.)
• Data chunking (processing low-level data one TPC at a time in order to minimize the memory footprint) is important for the future. Data preparation has to work on one piece of data at a time. Framework is a waterfall workflow, not designed to loop over pieces in place. Can code it outside of art, but would like to have the framework do the work.
  • Another complication is that due to various DAQ changes, it would be convenient to loop data products with different labels. But doing this in art causes all of the data to be read in, which make data chunking impossible.
• A DUNE problem, not LArSoft, is that the Geometry isn’t the best. Field cages are bigger than APAs, box bounded by field cage is bigger than the box bounded by the wires. We’re not simulating that stuff.

MicroBooNE – Herb

• Nothing to report.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The March 2019 LArSoft Offline Leads meeting was handled via email and a google document.

LArSoft – Erica Snider

• LArSoft Workshop Plans – targeting June 2019 — Planning for the summer LArSoft workshop has begun. We anticipate three sessions, the first of which will be an introduction to LArSoft that will include material from DUNE’s introductory TCP workshop series. This will be followed by a multi-threading and vectorization section targeting CPU-based grid processing, and discussion of HPC resource utilization. The third session will cover the long-term vision for LArSoft and will discuss how LArSoft should evolve as we move toward the era of DUNE data-taking.

Workshop Agenda:

• • LArSoft introduction / tutorial session
    • What should we include? Who is the target audience? What level are we aiming at?
    • Tools to assist in using FHICL?
  • Multi-threading, vectorization, HPC utilization
    • Introduction to multi-threading and vectorization
    • Multi-threading in art
    • Making code thread-safe
    • Experience of a beginner learning to make code thread-safe
    • Vectorizing code
    • HPC computing solutions for LArSoft
  • Long-term vision for LArSoft in the DUNE data-taking era
    • Discuss issues, seek input from experiments, SCD

Please comment and advise on this agenda. Received a suggestion to talk about the impending GitHub migration from the Steering Group. Should we talk about that?

• The LArTPC Data Overlay Workshop was held on March 4th with the goal of aligning experiments with a common infrastructure and approach to data overlays to be implemented within LArSoft. Key aspects included introducing the overlay method and interested parties, learning from past experience with discussion about adding simulated and data signals, calibrating simulation to match data, timing issues in simulation and data, and producing overlay samples. About 13 people participated in the workshop, including representatives from NOvA, CMS, all LAr experiments, and a theorist. The outcome of the workshop was a list of actions and developments needed for a common data overlay infrastructure, which can be found at the bottom of the meeting notes:  https://docs.google.com/document/d/1i2njli0lvTR7ZYnvlh-YIsVkFcgGpmdjJjwdIvVj73Q/edit?usp=sharing. The to-do list includes:
  • Port common waveform adding tools to LArSoft (Adi + LArSoft person)
    • Redmine tickets
  • Create art tool interfaces for the various mixers (Adi + DUNE person + LArSoft person)
    • OpDetWaveform
    • RawDigit
    • CRTHit
    • Redmine tickets
  • Investigate whether calibrations should be services or tools (Expts + LArSoft person)
    • The bigger question, which  corrections can be made more generic than they currently are
    • Already built on service interfaces
  • Figure out what to do with the art::Event::isRealData() method (Expts + art person)
    • Need a richer description for overlay data
    • Consider service, more methods, eg, hasRealData/hasMCData, etc
    • Something else
  • Are there general ways to think about timing that preserve the ability to do back-tracking in overlay data samples? (Expts)
  • Consider whether inline (eg, in-situ simulation) vs file-based primary / secondary overlay model is optimal
    • Trade-offs between the need to handle multiple input files vs controlling event re-use
  • Art EventMixing infrastructure needs to be more capable in order to be useful (Expts + art person + LArSoft person)
    • Particularly associations are currently not supported
  • Can we design a standardized way for grid jobs to deal with run/subrun/conditions alignment issues between input files  (FIFE (and Andrei Gaponenko) + ??)
• Ubuntu support. We wanted to remind offline leads that ubuntu is supported under an “on-demand” model.  It is important that offline leads and users know that they can request ubuntu releases as needed.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Tom Junk

We are starting to think about and work on ways to read in partial event data so that an entire Far Detector module’s worth of raw digits aren’t in memory at any given time.  ProtoDUNE-SP has already taken some test data that are broken into several smaller branches (with different module/instance names) which helps the input step. We propose that in the Far Detector, each APA gets a branch of fragments out of artdaq, so that GetByLabel and GetValidHandle don’t immediately reserve large amounts of memory just to get any data in.

ICARUS – Daniele Gibin, (Tracy Usher)

ICARUS is preparing to launch effectively MCC 2 which will provide simulated data for the upcoming SBN Workshop in Oxford the beginning of April. Much progress has been made since MCC 1, the signal processing (with a coherent noise model) is in good shape, we are running the latest pandora track/shower reconstruction plus downstream track fits, we now have the first attempt at an optical reconstruction and are in position to work on the CRT reconstruction. A goal for the workshop will be to try to do track/flash matching. There is still much to do… work is ongoing to extend the photon library to all four TPC’s, we still need to split the horizontal wires, etc.

I would guess that sometime in the Spring ICARUS should be ready to ask for help in doing a timing and memory usage review. Most of the current simulation/reconstruction jobs run fairly quickly but are only single particle or neutrino with no cosmic samples and here we suppress the channels with no MC signal. The CR sim/recon times jump dramatically, handling the >50,000 channels in the TPC alone is significant load. Also, the photon library consumes a fair bit of memory, as does the CRT simulation.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Andrzej Szelc

Main recent effort has been on optimizing the light simulation, towards trying to understand the impact of the beam rotation change in the BNB on physics analyses. We have implemented an improved optical library (needed adding stash-cache functionality following DUNE example), improved timing parametrization and semi-analytic light simulation (will be used soon). In parallel, we are starting to test the update to nutools implemented by Robert Hatcher, that allows tweaking the beam bucket parameters. Final step is to update the digitization simulation. Once that is done we plan to launch a relatively large data production in order to provide data samples for summer internships and understand the timing resolution.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

Attendees: Chris Jones, Erica Snider, Tom Junk, Herb Greenlee, Katherine Lato, Tracy Usher (in a follow on meeting)

Chris Jones shared information on GitHub – The current  way of operating is unsustainable as more experiments start taking data and need to closely manage their production LArSoft releases. So we asked Chris to share his experience in CMS with GitHub and the use of pull requests to manage code integration. His slides are available at CMS GitHub Usage for LArSoft. A zoom recording of his presentation is available upon private request.

The discussion included extensive details on the CMS-specific infrastructure used to support that system, and how elements could be adapted to the LArSoft community. Differences between how such a system might be used in single-experiment versus a multi-experiment collaboration were examined, along with various strategies for deploying the system. A significant finding was that the CMS system was initially deployed with relatively little supporting infrastructure. Despite this, Chris believed the system gained widespread approval quickly, due in part to the ease of using the tools provided by GitHub. This then led to a rapid and organic growth of tools that in time produced the current, highly sophisticated system. It is unclear how much, if any of the existing CMS code could be used or adapted to a non-CMS system.

In the CMS pull request system, there are three roles defined:  developers, reviewer and release managers. Most of the positions have at least two people in them. In CMS, people are adding comments because it’s easy. They didn’t expect this, but it happened. They built seven releases. Testing lots of things. ROOT, GEANT. They are a continuous integration machine. This presentation provided useful background.

• Update on LArSoft items
  • Progress on event display technology evaluation. Seeking input from community
    • Go over the four options + evaluation categories / criteria
    • Option A. Web browser-based system, three.js
    • Option B. Custom application
    • Option C. Vtk-based application
    • Option D. TEve-based application
    • Criteria — https://cdcvs.fnal.gov/redmine/issues/19037
      • Some of these solutions don’t directly talk to LArSoft. Instead, they read and interpret art-root files directly. We would prefer something that talks directly to LArSoft.
      • Lack of documentation on TEve will make it difficult to use.
  • Overlay workshop is on March 4th. Details are: here 
  • Experiments learning to build their own patch releases of LArSoft
    • Noted that MicroBooNE has asked for a new patch release, and is expected to ask for another by the end of the week
    • LArSoft team has limited capacity for creating releases, so tight production schedules will be difficult to manage, particularly with multiple experiments taking data. Experiments should expand their current release manager roles to include creation of LArSoft patch releases for production.
    • No major objections raised
  • Genie v3 status:  MicroBooNE has performed detailed validation, has patches for Genie, working to integrate it into MCC9 patch release
    • LArSoft team wants all experiments to be using v3 at the same time, if possible
    • Need input from DUNE on their plans asap
    • Will try to share the validation results from MicroBooNE, since nothing there was actually specific to MicroBooNE
• Round robin discussion
  • MicroBooNE
    • No issues to discuss
  • DUNE
    • The longer DUNE takes, the more prototypes we’ll have. ICEBERG is one of them.
    • Has been working mostly on GArSoft and ICEBERG recently
    • Happier with LArSoft than with dunetpc. Need to break that into smaller chunks

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The January 2019 LArSoft Offline Leads status update was handled via email and a google document.

LArSoft – Erica Snider

• The 2019 LArSoft workplan was updated after the steering group meeting in December and is available at: https://indico.fnal.gov/event/18678/material/2/0.pdf It is an ambitious plan and will require additional effort to complete.
• LArSoft will be working with Wes Ketchum to organize a mini-workshop on March 4 to discuss the problems, solutions and workflows for performing data overlays in LArSoft. These techniques will be used by all experiments, so should be of interest to the entire LArSoft community. Background can be found in Wes’s presentation Feature update to DetectorClocks service for data/simulation overlays.
• Steve Gardiner from ND/MicroBooNE created LArSoft v08_01_00_01, a test release based on Genie v3_00_02. Completing this integration has been a high priority, and we thank Steve for contributing this work. All experiments will need to sign-off on Genie v3 before it will be integrated into the develop branch, so experiments should perform whatever tests are needed using this release.
• MicroBooNE has declared LArSoft v08_00_00 as production for MCC9.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang

The first round of production of ProtoDUNE-SP data processed with LArSoft/dunetpc release v07_08_00 was very successful and we are using this pass to help us inform the TDR and to start analyses that we intend to publish.  We have tested the 2D deconvolution provided by the wire-cell team on ProtoDUNE data, which may be included in the second pass of production. We have moved our photon visibility libraries to StashCache. We would like to thank the art and LArSoft teams for helping us migrate to art v3.x.

ICARUS – Daniele Gibin, (Tracy Usher)

• Production running with POMS appears to be fairly turnkey now.
• A small group of analyzers have been busy uncovering issues with the TPC simulation and in the next icaruscode release should have the signal processing issues fairly well sorted.
• Work has begun on splitting the horizontal wires with the first test geometry made available recently. Gianluca Petrillo has been understanding impacts to the LArSoft code; hopefully this will be ready soon.
• ICARUS continues to work towards launch of the next MCC in early March with the goals of being able to do a demonstration of basic Cosmic Ray tagging and perhaps a basic physics analysis.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

MCC9 is progressing well with much good work on space charge and overlays with both highly relevant to future LArTPCs.

SBND – Roxanne Guenette, Andrzej Szelc

• SBND, thanks to work of Tom Brooks, has integrated the CRT simulation into our workflow.
• We are working to understand the timing resolution of the detector using LArSoft. This will require implementing the BnB bucket structure (investigated by A. Ezeribe, A. Scarrf, Sheffield), and new timing parametrizations to work with the optical library, as well as solutions to make it work faster (D. Garcia-Gamez, P. Green).
• We are making progress with the implementation of the Spacecharge simulation (A. Timilsina).
• We expect to produce large data samples with the improved parameters soon.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

Since this meeting was combined with the Steering Group meeting, the notes are available at:

https://indico.fnal.gov/event/18678/material/3/0.pdf

The November 2018 LArSoft Offline Leads status update was handled via email and a google document.

LArSoft – Erica Snider

LArSoft met with each experiment’s Spokespeople and Offline Leads to learn detailed plans for the next year, the implied requirements for LArSoft, and how LArSoft could help, as well as what the experiments might be able to contribute to LArSoft code. This input will be used to develop the 2019 LArSoft Work Plan.

Stage 1 of the LArG4 refactoring project will officially close with the acceptance of documentation, currently posted to https://cdcvs.fnal.gov/redmine/projects/larg4/wiki/Wiki. The offline leads are invited to review and comment on the documentation at any time.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Tom Junk

A recent shift away from using /grid/fermiapp for staging of built binaries made DUNE’s build script fail on mac.  CVMFS is unreliable and slow on mac, so we use the /grid/fermiapp mount. We see that larsoft and other experiments use buildFW in their Jenkins scripts and may want to consult with how to do that ourselves.   Keeping the latest release on /grid/fermiapp is the path of least resistance for us.

ICARUS – Daniele Gibin, (Tracy Usher)

ICARUS has launched MCC 1.1 which addressed some deficiencies in the initial 1.0 run but importantly also signifies the official switch to the POMS system. This week (final week of November) is being used to eliminate the last remaining issues and we expect to be generating moderate size (compared to MicroBooNE and ProtoDUNE) data sets in the next month.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.