All posts by klato

November 2019 Offline Leads Meeting Notes

The November 2019 LArSoft Offline Leads status update was handled via email and a google document. Past meeting notes are available at: 

LArSoft – Erica Snider

Presented the draft 2020 work plan to SCD management on Nov 7. The presentation noted that the work plan calls for 2 to 3 FTEs of development effort, depending primarily on the effort devoted to multi-threading the code, and the amount of work that the project needs to devote to toward various work items to which the experiments can contribute. The project has about 2 FTEs in the budget. No significant concerns were raised by management.

Testing of using LArSoft with GitHub and the pull request approval workflow has been opened to testing by the experiments. All experiment offline leads and release managers are encouraged to exercise the system, and send comments to the SciSoft team (at Instructions for using the system can be found on the LArSoft wiki under the “Working with GitHub” page (also linked from the “Quick Links”, Using LArSoft”, and “Developing with LArSoft” pages.) Any changes made to the repositories during testing will be deleted prior to the production migration. The current plan is to leave testing open for at least two weeks, but to complete the production migration in early December. 

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

ProtoDUNE developed a new way to handle raw digits in order to save memory usage. Tom Junk created a tool that decodes protoDUNE raw data and returns it to the caller in the form of vector<raw::RawDigit>. David Adams modified protoDUNE dataprep to use this tool instead of reading digits from the event data store. Other larsoft modules (such as GausHitFinder and various disambiguation algorithms) are modified accordingly to remove hit and raw digit associations. After all the changes, we see a ~10% reduction in memory usage in ProtoDUNE reconstruction job. This new tool allows us to decode raw data one APA at a time and drop raw digits once recob::Wires are made after deconvolution and is applicable to the DUNE far detector where there are 150 APAs per module. More details can be found in #23177.

DUNE made a patch release v08_27_01_01 with the help of Lynn Garren. This release includes artdaq_core bug fixes and a feature to limit tensorflow to only use one CPU to be more grid friendly. The dunetpc v08_27_01_01 is the current release for ProtoDUNE-SP keep-up production. 

A new package protoduneana is created. This new package depends on dunetpc and only contains ProtoDUNE analysis code. It is meant to speed up compiling time for people doing analysis. 

We are meeting regularly with Erica Snider, Saba Sehrish, and Kyle Knoepfel regarding a project to make the data preparation stages thread safe in order to improve our CPU utilization.

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

October 2019 Offline Leads Meeting Notes

10/22/19 Attendees: Tingjun Yang, Thomas Junk, Mike Kirby, David Adams, Erica Snider, Katherine Lato

Agenda:  the 2020 work plan and the GitHub migration plan.

The  GitHub migration

  • The project would like to complete the migration as soon as possible. The infrastructure is ready, but the documentation needs more work, and  is critical path for moving into community testing phase. 
  • The exact procedures for Level 1 and Level 2 need to be worked out. 
  • Need to know immediately who the level 2 managers are going to be. 
    • DUNE:
      • Level 2 managers:
        • Tom Junk
        • Tingjun Yang
        • Christoph Alt (has already been doing it for another project)
        • David Adams
      • How many Level 2 managers needed to approve a PR?
        • One. Expect that DUNE managers will approve only PRs from / for DUNE. (Exceptions ok if the manager is familiar with the work.)
    • ICARUS
      • Tracy Usher (agreed in email after the meeting)
    • Emailed other experiments requesting nominees
  • LArSoft will clean the history during the migration. 
  • Will experiments migrate to GitHub? 
    • DUNE yes probably yes, in part because Redmine is so slow. Improved performance of GitHub is one of the motivations for the LArSoft move. DUNE has no migration plan yet.
    • Noted that there is already a GitHub area, but it doesn’t have TPC code in it. DUNE will think about how to organize repositories
  • Will open up the GitHub for testing once the documentation is in place. Would like the testing period to be as long as needed, but as short as possible.

Went through the draft 2020 work plan. 

  • DUNE would like us to fix the existing event display in advance of having a new event display.
  • SPACK.
    • Tom ran the MVP, it took all day, took all the memory, but it did work eventually. Not recommended in present state, though.
      • We believe this is related to the “concretization” stage, which is known to be slow. Chris  Green has been introducing significant optimizations, but having some difficulty getting SPACK owners to accept the changes.
  • Pixels
    • Check with Mat Muether. Also, Gianluca Petrillo has been working on pixel-related Geometry.
  • DUNE suggests we promote magnetic field one to a project.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

September 2019 Offline Leads Meeting Notes

The September 2019 LArSoft Offline Leads status update was handled via email and a google document.

LArSoft – Erica Snider

While we knew that Lynn Garren was retiring the end of January, 2020, we recently learned that Paul Russo is leaving the middle of September. Paul was providing customer support on LArSoft tickets and expertise on multi-threading for LArSoft. We do not yet know what the replacement strategy is. 

Due to the changes in available resources, the project needs to shift some responsibilities to the experiments, particularly in the area of code and release management. 

  • MacOS support – LArSoft support for Mac builds is ending. Could either find someone else to support those builds, or have MacOS developers  use SLF containers. (Note that support for those containers will not come from the SciSoft team.)
  • GitHub Migration – Need to identify people in experiments for manager roles defined in the pull request workflow. See slides from Patrick Gartung’s presentation Migration of LArSoft repos to GitHub. We will hold additional discussions and provide opportunities for beta testing prior to the migration to GitHub and deployment of the pull request system.

The project has begun the annual process of gathering input from the experiments to inform the 2020 LArSoft work plan. We have met with all the experiments. An initial draft work plan will be discussed at the October Offline Leads meeting.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

ProtoDUNE has started the second production campaign using larsoft/dunetpc v08_27_01. A ProtoDUNE-SP simulation task force is formed to improve detector simulation. The goal is to incorporate the refactorized larg4 and wirecell simulation. DUNE is in the process of transitioning to use tensorflow v1_12_0b with the help of larsoft team. More details in #22504.

ICARUS – Daniele Gibin, Tracy Usher

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee

MicroBooNE is preparing a major production campaign (MCC 9.1) based on our MCC9 branch (based off larsoft v08_05_00_10).  No particular requests to larsoft.  

SBND – Roxanne Guenette, Andrzej Szelc

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

July-August 2019 Offline Leads Meeting Notes

The July & August 2019 LArSoft Offline Leads status update was handled via both a meeting (on July 30th) and a google document. People were asked to fill in the information in the document if they couldn’t make the July 30th meeting. Past meeting notes are available at: Thank you.

LArSoft – Erica Snider

The LArSoft project requests that the experiments provide feedback on the following two items:

  • The LArSoft Spack MVP. See the instructions in the email “Announcing Spack / SpackDev MVP1a (“The LArSoft Edition”)” from Chris Green on June 20 to
    • This will be the build system for art / LArSoft and the replacement for UPS in the near future. Your feedback is critical to ensure that the final product is easy to use (ie, effective, efficient, intuitive, easy to learn, satisfying). This is our chance to change the outcome.
    •  This  PDF explains the MVP in detail, and how to set it up to evaluate the system. 
    • Please send any feedback or questions direct to by July 20th.
  • The migration to Genie v3
    • Although LArSoft is driving an effort to decouple the version of LArSoft from any specific version of Genie, this is not yet possible. Moreover, once it becomes possible, it is unlikely that any version of Genie v2 will be included.
    • We therefore need all experiments to sign off on the v3 migration, or to state their plans with respect to v3 (including any plan to stay at v2.x)
    • NOTE:  as of July 24, all experiments had signed off on the migration

The project is overseeing a considerable amount of work directed at making various parts of LArSoft thread safe, and in cases, parallelizing specific pieces of code. Some of this coding work will soon be integrated into LArSoft.  We expect that a number of services (including experiment-specific provider variants) and various pieces of common code may be affected. 

Work on the migration to GitHub is proceeding. More details of what this will look like to end users should be forthcoming over the next couple of months. We are expecting the main parts of the migration to be completed by mid-fall.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

We are working to make the software for the near detector complex work together.  This includes a pixel-based liquid-argon component, which will likely use LArSoft directly, a high-pressure gas TPC with an ECAL, called the Multi-Purpose Detector which has a software stack based on art and nutools and thus can run together with larsoft code, and a 3D Scintillator Tracker-Spectrometer.  We need to simulate events crossing from one component to another, so a unified GEANT4 simulation is required.  Common triggers and event builders will be needed to take data. A workshop was held on July 24 to discuss these issues.

We are also discussing the FD data model at a workshop at BNL  on August 14-16. We will discuss databases, data streams, DAQ interfaces, framework and memory management, and of course the event model.

DUNE has discussed the GENIE v3 upgrade.  While we are interested in continuing with GENIE v2, we can continue to work with older releases of larsoft and dunetpc.  If we absolutely must mix GENIE v2 with newer simulation and reconstruction code, either a patch release or careful accounting of the GENIE systematics weights needs to be done.

We tried the quick-start instructions for the Spack MVP, LArSoft edition and gave some feedback.

ICARUS – Daniele Gibin, (Tracy Usher)

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

Discussed what MVP meant, does enough to get a feel for it, but can still be changed if needed. 

Thinking about when to merge to develop. Things preventing:

  • Genie v3 (which is now resolved)
  • IO rule, Have a new pid data product which everyone should want. In order to use it, need the IO rule. Waiting for someone to make the change.

SBND – Roxanne Guenette, Andrzej Szelc

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

LArSoft Workshop June 2019

 Workshop Overview

The annual LARSoft workshop was held on June 24 and 25 at Fermilab. There were three sessions:

  • Session 1:  LArSoft tutorial. 
    • Provide the basic knowledge and tools for navigating, using, writing and contributing LArSoft code.
  • Session 2:  Multi-threading and vectorization.
    • Multi-threading and vectorization targeting CPUs and grid processing, giving people the background and tools needed to approach the code and start thinking about making their code thread safe, trying to address memory issues, vectorizing, etc.
  • Session 3: Long-term vision for LArSoft.
    • To discuss ideas and concerns about how LArSoft should evolve with changes to the computing landscape as we move toward the era of DUNE data-taking.

The slides from the speakers can be found on indico. All sessions were recorded. They can be found at:

Session 1:  LArSoft tutorial

Overview and Introduction to LArSoft 

Erica Snider began the LArSoft workshop with an introduction to LArSoft.  The LArSoft collaboration consists of a group of experiments and software computing organizations contributing and sharing data simulation, reconstruction and analysis code for Liquid Argon TPC experiments. LArSoft also refers to the code that is shared amongst these experiments. Organizing principle for LArSoft based on a layering of functionality, dependencies.

LArSoft is not stand-alone code. It requires experiment / detector-specific configuration. The same basic design pertains to the experiment code. Nothing in core LArSoft code depends upon experiment code.




Technical details, code organization

Saba Sehrish covered repositories, UPS products, setting up and running LArSoft and contributing to LArSoft. There are 18 repositories containing the LArSoft code; each experiment has at least one code repository for detector-specific code.

Simplify your code

Kyle Knoepfel discussed how code becomes complex. Over time, code becomes larger and larger. Ways to combat this include:

  • remove files that you know are not needed
  • remove unnecessary header dependencies
  • remove unnecessary link-time dependencies
  • remove unnecessary functions
  • use modern C++ facilities to simplify your code
  • reduce coupling to art

Pandora tutorial

Andrew Smith discussed pattern recognition in LArTPC experiments, with Pandora being a general purpose, open-source framework for pattern recognition. It was initially used for future linear collider experiments, but now well established on many LArTPC experiments.

Useful source material on Pandora:

  1. Multi-day Pandora workshop in Cambridge, UK – 2016
    • Talks about how the algorithms work and step-by-step exercises about how you might develop a new algorithm using Pandora.
  2. LArSoft workshop in Manchester, UK – 2018
  3. Workshop on advanced computing & machine learning, Paraguay – 2018
    • Talks and exercises about running and using Pandora within LArSoft, including tutorials on using Pandora’s custom event display
  4. Experiment specific resources:

Practical guide to getting started in LArSoft

Tingjun Yang presented a practical guide to getting started in LArSoft. He used  ProtoDUNE examples that apply to most LArTPC experiments. A lot can be learn from existing code, talking to people and asking for help on SLACK.

How to tag and build a LArSoft patch release

Lynn Garren presented on how to tag and build a patch release by an experiment. MicroBooNE is already doing this. LArSoft provides tools, instructions, and consultation.  Up-to-date instructions are available at: How to tag and build a LArSoft patch release.

Session 2:  Multi-threading and vectorization

Introduction to multi-threading and vectorization

Matti Kortelainen discussed the motivation and the practical aspects of both multi-threading and vectorization.

Two models of parallelism: 1) Data parallelism: distribute data across “nodes”, which then operate on the data in parallel  2) Task parallelism: distribute tasks across “nodes”, which then run the tasks in parallel.

Two threads may “race” to read and write. There are many variations on what can happen.

A software thread  is the “Smallest sequence of programmed instructions that can be managed independently by a scheduler.” [Wikipedia]

Vectorization works well for math-heavy problems with large arrays/matrices/tensors of data. It doesn’t work so well for arbitrary data and algorithms.

Making code thread-safe

Kyle Knoepfel discussed how to make code thread-safe. The difficulty of this task depends on the context.

Multi-threaded art

Kyle Knoepfel described multi-threaded art. Modules on one trigger path may not consume products created by modules that are not on that same path. The design is largely based off of CMSSW’s design.

Experience learning to make code thread-safe

Mike Wang described his experience with making LArSoft code thread-safe. Except for the most trivial cases, do not expect to be able to hone in on a piece of LArSoft code (such as a particular service) and work on it in isolation in attempting to make it thread-safe. You are dealing with an intricate web of interconnecting and interacting pieces. Understanding how the code works and what it does, tedious as it may seem, goes a long way in facilitating the process of making the code thread-safe, helping avoid errors that will be very difficult to debug.

Long-term vision for LArSoft Overview

Adam Lyon noted that computing is changing (and the change has changed – GPUs over KNLs.) Future: multi-core, limited power/core, limited memory/core, memory bandwidth increasingly limiting. The DOE is spending $2B on new “Exascale” machines.

The Fermilab Scientific Computing Division is committed to LArSoft for current and future liquid argon experiments:

  • Fermilab SCD developers will continue to focus on infrastructure and software engineering
  • Continue to rely on developers from experiments
  • Continue to interface to neutrino toolkits like Pandora
  • Need to confront the HPC evolution
  • Reduce dependency on the framework

Computing in the time of DUNE; HPC computing solutions for LArSoft

As Giuseppe Cerati noted, technology is in rapid evolution. We can no longer rely on frequency (CPU clock speed) to keep growing exponentially. Must exploit parallelization to avoid sacrificing on physics performance.

Emerging architectures are about power efficiency. Technology driven by Machine Learning applications.

Many workflows of LArTPC experiments could exploit HPC resources – simulation, reconstruction (signal processing), deep learning (training and inference), analysis.

Data management and workflow solutions needed

Mike Kirby discussed data management and workflow solutions needed in the long-term based mainly on DUNE and MicroBooNE. “Event” volumes for DUNE are an order of magnitude beyond collider events. Already reducing the data volume from raw to just hits.

LArSoft framework works wonderfully for processing artroot files – there is a lack of a “framework” for processing non-artroot files (plain ntuples, etc) and this gap could be a problem – CAFAna is actively in use for DUNE and NOvA, but not a fully supported analysis framework.

With multiple Far Detector Modules and more than 100 Anode Plane Arrays possibly readout in a trigger record, the ability to distribute event “chunks” to multiple LArSoft processes/threads/jobs/etc and reassembly into reconstructed events should be explored.

DUNE perspective  on long-term vision

Tom Junk started by discussing the near detector for DUNE. Individual photon ray tracing is time consuming. They are studying a solution using a photon library for ArgonCUBE 2×2 prototype. LArSoft assumes “wire” in the core of Geometry design & APIs to query geometry information. This needs to be changed. Gianluca Petrillo at SLAC designed a generic “charge-sensitive element” to replace the current implementation in a non-distruptive manner. The goal is to run largeant for wire & pixel geometry.

We have some concerns about external source control support. There’s a lot of open-source code out there. Do we have to maintain every piece a DUNE collaborator wants to use?

ICARUS perspective on long-term vision

Tracy Usher pointed out some of the areas where ICARUS is already stressing the “standard” implementation of LArSoft based simulation and reconstruction. ICARUS stands for Imaging Cosmic And Rare Underground Signals. It is the result of some 20+ years of development of Liquid Argon TPCs as high resolution particle imaging detectors from ideas first presented by Carlo Rubbia in 1977.

ICARUS has more sense wires than SBND or MicroBooNE, and has 4 TPCs compared to 2 for SBND and 1 for MicroBooNE. ICARUS has horizontal wires, not vertical and they are split. It was originally optimized for detector Cosmic Rays.


Slides are available at There were a variety of speakers and topics, from introductory to advanced HPC techniques, enabling people to attend the sections of most interest to them. The quality of the talks was quite high with a lot of new, interesting content which we can now use to update the documentation on It will also serve as an excellent basis for discussion moving forward. 

Thank you to all who presented and/or participated!

May 2019 Offline Leads Meeting Notes

The May 2019 LArSoft Offline Leads status update was handled via email and a google document.

LArSoft – Erica Snider

  • Planning for the LArSoft Workshop 2019 is nearly complete. The agenda and speakers can be found on indico here:
  • A  “minimally viable” Spack/SpaceDev-based build and development environment for LArSoft is under initial testing within the SciSoft team. The purpose of the current product is to provide sufficient functionality to allow meaningful testing and feedback on Spack and SpackDev. The plan is to distribute this version for testing and comment from the experiments once sufficient documentation and fixes for known bugs are available.
  • LArSoft has been asked to upgrade to python 3 which requires dropping SLF6.  Seeking feedback from experiments and users.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Michael Kirby

  • No Report

ICARUS – Daniele Gibin, (Tracy Usher)

  • Experience with running our last MC challenge exposed some issues with memory and cpu time for simulation and reconstruction of events. Primarily this is in the TPC reconstruction but there are also problems in the PMT simulation/reconstruction (primarily associated with the photon lookup library) and the CRT. Work has been underway to try to address issues with the TPC reconstruction first and this has led to a new workflow for both simulation and reconstruction where the signal processing is broken into groups by TPC (there are four TPCs in ICARUS) and then pattern recognition is done by Cryostat (there are two cryostats in ICARUS). The detector simulation then changes to output four collections of RawDigits, one per TPC and this is meant to emulate what the DAQ folks tell us they will do when we begin data taking.

This breakup should enable the ability for some amount of parallelization of the processing, also with each instance of a particular processing module using less memory.

In addition to this, there is also effort underway to reformat some of the signal processing modules to allow for vectorization techniques now available (or soon to be available?) in art. We hope to be in a position to test soon.

  • Recent improvements to the photon lookup library have led to an explosion in the memory requirements once read back from a disk file, at last check taking some 6.5 GB of memory. This library is not only more detailed but extends now to both Cryostats. Gianluca Petrillo has implemented some code in LArSoft which will try to make use of symmetries in the library to reduce memory, but this needs to be tested. If this cannot make a significant reduction then it will be necessary to look at alternatives.
  • The BNL team is starting preliminary effort to port the WireCell toolkit to ICARUS for simulation and noise filter/2D deconvolution. Currently targeting integration in time for commissioning.
  • Finally… there is significant effort in developing machine learning techniques for 3D reconstruction using ICARUS as the current target platform. Toward that end the current LArSoft workflow produces 3D SpacePoints (using “Cluster3D”) which are then siphoned off into the deep learning effort’s workflows using a new version of the LArCV package (now LArCV3). It would be highly useful if this were an “official” LArSoft package, importantly including proper makefile set up to enable builds in the standard LArSoft framework (e.g. using mrb).

LArIAT – Jonathan Asaadi

  • No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

  • No Report

SBND – Roxanne Guenette, Andrzej Szelc

  • No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

April 2019 Offline Leads Meeting Notes

Attendees: Erica Snider, Herb Greenlee, Tracy Usher, Thomas Junk, Katherine Lato 

LArSoft Status

1) Discussion with Jim Amundson

  • On the  Event Display project. We have a proposal for how to proceed with adopting a web-based event display and recruiting effort to support it. Erica is charged with following up and reporting back on progress. Tracy raised two points.
    • Running on laptops that are not connected to the internet
    • The problem of passing data about display objects over the network
      • The former problem is solved if the server can run locally (which is a requirement)
      • The latter is solved if there is a protocol for matching display data to the screen resolution. There are many successful examples of doing this in the industry.
  • On Spack. Plan is to bring more resources to bear on this project, but not clear how or when this will happen.

2) Multi-threading, vectorization and other optimizations

  • Tracy commented about talking with Mike and Giuseppe about optimized versions of deconvolution and noise suppression code for ICARUS, building on the multi-threading work Mike did for CalWire_DUNE.  
  • ICARUS is willing to look at Dave Adams data-prep code as a possible common framework for performing this and similar calculations. Interfaces can be common if inputs are raw digits or wires, and outputs are wires. Provides flexibility to customize algorithms as experiments need.
  • They also discussed 2D drift model that has been worked on a little. They demonstrated that it’s working and running 6 to 8 times slower than 1D, and are working to make it run faster. That could be useful in production if it could be made faster. That part of the code should lend itself to vectorization.

3) LArSoft Workshop planning

  • Workshop planning and agenda –
  • The basic plan is a tutorial that starts on the afternoon of the 24th. Second day is multi-threading and vectorization in the morning followed by long-term vision of LArSoft discussion in the afternoon.
    • Could ICARUS contribute to long-term vision? Yes.
    • And the deep learning people could contribute.
    • ICARUS & DUNE should be talking to each other to build the event in a way that allows the possibility of sequentially processing blocks of low-level event data, such as raw digits, on the scale of a single TPC, while limiting the memory footprint to that block.

Round table

ICARUS – Tracy

  • Have a lot of stuff to get done and not a lot of time left. Basic structure is in place. Need more people.
  • Deep learning people at SLAC are using ProtoDUNE and ICARUS data to develop DL algorithms. In the next six months, there should be feedback from that effort that could be useful in long-term strategy.
  • We are running simulations, Pandora, and light reconstruction, generally the full simulation and reconstruction workflow. The pathway works. Need to get serious about being ready for data taking. Have hooks in place for noise model. Doesn’t leave time for thinking about larger-scale problems. Optical reconstruction could be interesting. The one imported from MicroBooNE is basic, but the potential is there for something more sophisticated.

DUNE – Tom

  • Personally working on GArSoft more than LArSoft last few months.
  • ProtoDUNE analysis are coming along well. We don’t simulate electron diverters. The space charge parameterization is smooth, has an analytic smoothing function, doesn’t have sharp corners, as occurs near the diverters, so the edges are all wrong. But it’s probably on us to provide a map.
  • The photon libraries were gigantic, Alex Himmel has a solution. A nearly fully parameterized solution.  (This was presented at a LCM.)
  • Data chunking (processing low-level data one TPC at a time in order to minimize the memory footprint) is important for the future. Data preparation has to work on one piece of data at a time. Framework is a waterfall workflow, not designed to loop over pieces in place. Can code it outside of art, but would like to have the framework do the work.
    • Another complication is that due to various DAQ changes, it would be convenient to loop data products with different labels. But doing this in art causes all of the data to be read in, which make data chunking impossible.
  • A DUNE problem, not LArSoft, is that the Geometry isn’t the best. Field cages are bigger than APAs, box bounded by field cage is bigger than the box bounded by the wires. We’re not simulating that stuff.

MicroBooNE – Herb

  • Nothing to report.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

March 2019 Offline Leads Meeting Notes

The March 2019 LArSoft Offline Leads meeting was handled via email and a google document.

LArSoft – Erica Snider

  • LArSoft Workshop Plans – targeting June 2019 — Planning for the summer LArSoft workshop has begun. We anticipate three sessions, the first of which will be an introduction to LArSoft that will include material from DUNE’s introductory TCP workshop series. This will be followed by a multi-threading and vectorization section targeting CPU-based grid processing, and discussion of HPC resource utilization. The third session will cover the long-term vision for LArSoft and will discuss how LArSoft should evolve as we move toward the era of DUNE data-taking.

Workshop Agenda:

    • LArSoft introduction / tutorial session
      • What should we include? Who is the target audience? What level are we aiming at?
      • Tools to assist in using FHICL?
    • Multi-threading, vectorization, HPC utilization
      • Introduction to multi-threading and vectorization
      • Multi-threading in art
      • Making code thread-safe
      • Experience of a beginner learning to make code thread-safe
      • Vectorizing code
      • HPC computing solutions for LArSoft
    • Long-term vision for LArSoft in the DUNE data-taking era
      • Discuss issues, seek input from experiments, SCD

Please comment and advise on this agenda. Received a suggestion to talk about the impending GitHub migration from the Steering Group. Should we talk about that?

  • The LArTPC Data Overlay Workshop was held on March 4th with the goal of aligning experiments with a common infrastructure and approach to data overlays to be implemented within LArSoft. Key aspects included introducing the overlay method and interested parties, learning from past experience with discussion about adding simulated and data signals, calibrating simulation to match data, timing issues in simulation and data, and producing overlay samples. About 13 people participated in the workshop, including representatives from NOvA, CMS, all LAr experiments, and a theorist. The outcome of the workshop was a list of actions and developments needed for a common data overlay infrastructure, which can be found at the bottom of the meeting notes: The to-do list includes:
    • Port common waveform adding tools to LArSoft (Adi + LArSoft person)
      • Redmine tickets
    • Create art tool interfaces for the various mixers (Adi + DUNE person + LArSoft person)
      • OpDetWaveform
      • RawDigit
      • CRTHit
      • Redmine tickets
    • Investigate whether calibrations should be services or tools (Expts + LArSoft person)
      • The bigger question, which  corrections can be made more generic than they currently are
      • Already built on service interfaces
    • Figure out what to do with the art::Event::isRealData() method (Expts + art person)
      • Need a richer description for overlay data
      • Consider service, more methods, eg, hasRealData/hasMCData, etc
      • Something else
    • Are there general ways to think about timing that preserve the ability to do back-tracking in overlay data samples? (Expts)
    • Consider whether inline (eg, in-situ simulation) vs file-based primary / secondary overlay model is optimal
      • Trade-offs between the need to handle multiple input files vs controlling event re-use
    • Art EventMixing infrastructure needs to be more capable in order to be useful (Expts + art person + LArSoft person)
      • Particularly associations are currently not supported
    • Can we design a standardized way for grid jobs to deal with run/subrun/conditions alignment issues between input files  (FIFE (and Andrei Gaponenko) + ??)
  • Ubuntu support. We wanted to remind offline leads that ubuntu is supported under an “on-demand” model.  It is important that offline leads and users know that they can request ubuntu releases as needed.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Tom Junk

We are starting to think about and work on ways to read in partial event data so that an entire Far Detector module’s worth of raw digits aren’t in memory at any given time.  ProtoDUNE-SP has already taken some test data that are broken into several smaller branches (with different module/instance names) which helps the input step. We propose that in the Far Detector, each APA gets a branch of fragments out of artdaq, so that GetByLabel and GetValidHandle don’t immediately reserve large amounts of memory just to get any data in.

ICARUS – Daniele Gibin, (Tracy Usher)

ICARUS is preparing to launch effectively MCC 2 which will provide simulated data for the upcoming SBN Workshop in Oxford the beginning of April. Much progress has been made since MCC 1, the signal processing (with a coherent noise model) is in good shape, we are running the latest pandora track/shower reconstruction plus downstream track fits, we now have the first attempt at an optical reconstruction and are in position to work on the CRT reconstruction. A goal for the workshop will be to try to do track/flash matching. There is still much to do… work is ongoing to extend the photon library to all four TPC’s, we still need to split the horizontal wires, etc.

I would guess that sometime in the Spring ICARUS should be ready to ask for help in doing a timing and memory usage review. Most of the current simulation/reconstruction jobs run fairly quickly but are only single particle or neutrino with no cosmic samples and here we suppress the channels with no MC signal. The CR sim/recon times jump dramatically, handling the >50,000 channels in the TPC alone is significant load. Also, the photon library consumes a fair bit of memory, as does the CRT simulation.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Andrzej Szelc

Main recent effort has been on optimizing the light simulation, towards trying to understand the impact of the beam rotation change in the BNB on physics analyses. We have implemented an improved optical library (needed adding stash-cache functionality following DUNE example), improved timing parametrization and semi-analytic light simulation (will be used soon). In parallel, we are starting to test the update to nutools implemented by Robert Hatcher, that allows tweaking the beam bucket parameters. Final step is to update the digitization simulation. Once that is done we plan to launch a relatively large data production in order to provide data samples for summer internships and understand the timing resolution.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

February 2019 Offline Leads Meeting Notes

Attendees: Chris Jones, Erica Snider, Tom Junk, Herb Greenlee, Katherine Lato, Tracy Usher (in a follow on meeting)

Chris Jones shared information on GitHub – The current  way of operating is unsustainable as more experiments start taking data and need to closely manage their production LArSoft releases. So we asked Chris to share his experience in CMS with GitHub and the use of pull requests to manage code integration. His slides are available at CMS GitHub Usage for LArSoft. A zoom recording of his presentation is available upon private request.

The discussion included extensive details on the CMS-specific infrastructure used to support that system, and how elements could be adapted to the LArSoft community. Differences between how such a system might be used in single-experiment versus a multi-experiment collaboration were examined, along with various strategies for deploying the system. A significant finding was that the CMS system was initially deployed with relatively little supporting infrastructure. Despite this, Chris believed the system gained widespread approval quickly, due in part to the ease of using the tools provided by GitHub. This then led to a rapid and organic growth of tools that in time produced the current, highly sophisticated system. It is unclear how much, if any of the existing CMS code could be used or adapted to a non-CMS system.

In the CMS pull request system, there are three roles defined:  developers, reviewer and release managers. Most of the positions have at least two people in them. In CMS, people are adding comments because it’s easy. They didn’t expect this, but it happened. They built seven releases. Testing lots of things. ROOT, GEANT. They are a continuous integration machine. This presentation provided useful background.

  • Update on LArSoft items
    • Progress on event display technology evaluation. Seeking input from community
      • Go over the four options + evaluation categories / criteria
      • Option A. Web browser-based system, three.js
      • Option B. Custom application
      • Option C. Vtk-based application
      • Option D. TEve-based application
      • Criteria —
        • Some of these solutions don’t directly talk to LArSoft. Instead, they read and interpret art-root files directly. We would prefer something that talks directly to LArSoft.
        • Lack of documentation on TEve will make it difficult to use.
    • Overlay workshop is on March 4th. Details are: here 
    • Experiments learning to build their own patch releases of LArSoft
      • Noted that MicroBooNE has asked for a new patch release, and is expected to ask for another by the end of the week
      • LArSoft team has limited capacity for creating releases, so tight production schedules will be difficult to manage, particularly with multiple experiments taking data. Experiments should expand their current release manager roles to include creation of LArSoft patch releases for production.
      • No major objections raised
    • Genie v3 status:  MicroBooNE has performed detailed validation, has patches for Genie, working to integrate it into MCC9 patch release
      • LArSoft team wants all experiments to be using v3 at the same time, if possible
      • Need input from DUNE on their plans asap
      • Will try to share the validation results from MicroBooNE, since nothing there was actually specific to MicroBooNE
  • Round robin discussion
    • MicroBooNE
      • No issues to discuss
    • DUNE
      • The longer DUNE takes, the more prototypes we’ll have. ICEBERG is one of them.
      • Has been working mostly on GArSoft and ICEBERG recently
      • Happier with LArSoft than with dunetpc. Need to break that into smaller chunks

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.