The March 2018 Offline Leads status update was handled via email and a google document. 

LArSoft – Erica Snider

The code integration model for LArSoft will become more complicated as more experiments approach data taking. Already an issue for MicroBooNE / Pandora. We will continue to explore this issue with the experiments over the coming weeks.

Have collected input on requirements for the LArSoft event display. This is the first step of investigating a long-term solution for a common LArSoft event display framework to replace the existing display.  The plan for this investigation is documented and tracked in LArSoft issue 19034, and is summarized below:

1. Develop a set of requirements, ranked according to the following categories:  essential, strongly-recommended, desired, and possibly-useful.
2. Get community input on the ranking and final list of requirements at two LArSoft Coordination meetings to ensure that people can give opinions.
3. Establish criteria to judge between the technical options, and review these with the community.
4. Identify a list of possible technologies such as Root, Venu, ParaView, QT and other potential technologies and how applicable they are.
5. Compare technologies. Ideally, write a toy display that demonstrates the requirements, or sufficient subset of critical requirements. Code it up on each platform
6. Evaluate based on established criteria in task 19037
7. Take the results to SCD management for review, along with an initial deployment proposal and a request for funding.

Gianluca is leaving soon. Working on replacement issues.

DUNE – Andrew John Norman, Heidi Schellman

We’re still gearing up for DC2 in April and ProtoDUNE data in the Fall.  One question relevant to the event display, since it is root based, is whether we could take advantage of the ROOT web interfaces instead of the xvnc to view.  https://swanserver.web.cern.ch/swanserver/rootdemo/  and https://root.cern/js/  describe the interface.  

ICARUS – Daniele Gibin

Progress is continuing with development of simulation and reconstruction in the LArSoft framework. ICARUS now has a light simulation and is able to produce optical hits. The full reconstruction chain has been implemented using TrajCluster, we are not yet able to use Pandora due to an artificial requirement that one of the wire planes in the TPC must have vertical wires (ICARUS has horizontal wires). The Pandorans are aiming to fix this soon.

The LArG4 refactoring by Wes Ketchum is needed/desired before making further improvements to the ICARUS detector simulation.

Due to the LArSoft event display being broken with the introduction of the new version of art in LarSoft v06_70_00, ICARUS is currently basing its work on LArSoft v06_69_01.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

MCC8 is in the analysis phase with final tweaks for special data sets.

Planning for MCC9 is essentially complete with an aggressive timeline to achieve improvements aimed at results for Summer 2019.

In both cases the version of LArG4 refactoring provided by Wes Ketchum will be necessary, it would be good to include this code on the LArSoft development branch as soon as practicable.

SBND – Roxanne Guenette, Andrzej Szelc

We have run into problems running the optical simulation. Up to now we have been using simphotons, but this has turned out to cause problems in the newest software release version because of memory. LitePhotons were used instead, but most of our infrastructure for timing and reflected light does not work with those.

We are now trying to figure out how to solve this problem. Potentially Wes’s refactorization may help, although we need to look closer into that.

We have first results of matching TPC tracks with Cosmic Ray Tracker ones, which will be useful in future analyses.

We have setup File transfer and SAM definitions (thanks to help from SCD) and are beginning to use that in running our jobs.

Open issues:

1. Restructuring of LArG4 – Hans Wenzel and William Seligman continue to work on this. https://cdcvs.fnal.gov/redmine/issues/14454
2. Can NumPy and SciPy have standard, LArSoft compatible distributions made available?
3. Selective loading of detector data. For DUNE FD want a functionality to load into memory data products only from selected part of the detector, e.g. selected TPCs; this may be a feature request to art however. Could be a usage pattern in LArSoft, or could be in art. After a meeting with Kyle in February, concluded that to deal with reading in part of the detector at one time we need to create data product instances that are distinguished via their tags, and hide the complexity from users. We’re tracking this in: https://cdcvs.fnal.gov/redmine/issues/19286

Offline Leads meeting on February 14, 2018

Attendees: Heidi Schellman, Tracy Usher, Erica Snider, Katherine Lato

A) Discussed Supporting LArSoft on Mac OSX via Docker  

1. There was a demo at the 1/30/18 Coordination Meeting of using Docker along with other VM technologies to build LArSoft code under SLF6 under Mac OSX. Members of DUNE have proposed this as a way to support LArSoft development under Mac OSX.
2. Discussed how to proceed. All agree on the importance of supporting development on the Mac. Should we continue investigating this with possible implementation?
   • Q:  What are the problems with cvmfs?
     • A:  There can be huge latencies while waiting for things to cache, even with simple commands. Tuning the cache might alleviate some of this, but if there are frequent release updates (which there are), then it might still be disruptively slow (since pre-caching is not an option). Smaller repositories would help too, particularly for DUNE, but that is not a general solution.
   • Q: do you use local installs or CVMFS for developing on Mac
     • A:  Most seem to do local installs. Debugging capabilities are more difficult if not built locally.
   • Q: How often do you update the release you’re developing against?
     • A:  People seem to do this often. If the code isn’t stale, there’s less pain with merging in changes. Especially when things are complicated. When updating weekly, it just takes 20 minutes to build in the background while you’re doing something else. This way, it does not interfere with development work, which is in contrast to cvmfs cache updates, which occur on demand during development work.
   • All agreed that one of the major concerns for those developing on Mac OSX is access to Mac-based IDEs and other development tools. Docker interferes with xcode, for instance, so that could not be used. The Docker proponents have been asked to investigate this issue. Even if they can be run, they will be less capable than with closer-to-native builds under Mac OSX. Native clang builds would be best. Erica and Tracy both think that supporting native Clang would be nice. Ensuring that the product stack was consistently built is an issue, though more so for production releases than for development. Native clang support is not a priority at the moment.
   • Encouraging native Mac development as a whole is a good thing. Things that run in the production environment need certainty. Things like the event display should be run locally.
3. Consensus from DUNE and MicroBooNE:  Docker could be used to run LArSoft in non-standard environments, or to gain access to resources that are not otherwise available. It should not, however, be used as a way to support development on Mac OSX.

B) ProtoDUNE support. LArSoft wants to ensure that we are doing what is needed in the short term and consistent with long-term plans. LArSoft has changed the geometry to support different drift directions and changes to Global Wire Coordinates, other changes had to be on the protoDUNE side. With Robert Sulej’s departure, aren’t sure if things are acceptable or if people are complaining about things needed in LArSoft.

1. Heidi- we ran on MonteCarlo and people were pretty happy. First actual data has come out of the cold box. At the next collaboration meeting in two months, there will be more discussion.
2. LArSoft encourages people to talk to Erica, Gianluca, or put in a ticket if there are any issues or potential issues. We want to keep the engagement and make sure people have a positive experience with LArSoft.
3. Heidi thought that giving experiment service credit for working on LArSoft items was mentioned as a good idea.

C) LArG4 update. Met with Hans last week. There are two big pieces–changes to Geant4 side and on the experiment side. Changes to Geant4 side mean that LArG4 doesn’t need to have a copy of the Geant4 code that was made several years ago. Get away from having a custom physics list in order to take advantage of those in Geant4. Tracy recently talked to Bill Seligman, who is working on the experiment-side component, and Hans, who is doing the Geant4 work. Bill is waiting on the Geant changes, but is otherwise ready to test his code. The re-factoring is blocking Wirecell testing. MicroBooNE may need a Plan B.

D) Update on 3D reconstruction workshop.

1. 3D pattern recognition workshop – Proposed scope will cover all existing work to produce 3D space points, plus algorithms to perform pattern recognition using space points as input. Will also work toward agreements on data products, interfaces between various workflow stages so that we can continue to share algorithms.
2. Alan Bross (who is part of Argon Cube) is interested in having the ALICE  pixel people there. The Alice people won’t be in the US, however, so want it to be at CERN. The US people want it to be here.
3. At a minimum, want all parties to agree on the interface layer between the low-level 3D data and the 3D pattern recognition. This preserves the possibility of sharing downstream pattern recognition code.  The plane readout people have to get to this point by merging 2D data. The pixel people start from this point. The common part of the workshop is then talking about the downstream piece.
4. Maybe have a joint early morning meeting in the US with remote CERN participation, then have the rest of the workshop in the US without them?  
   • Will pitch this.

E) Profiling – LArSoft team continued examining profiling results for ProtoDUNE, 35t and FD production workflow chains compiled by Soon Jun Yun. Asking DUNE to provide a person to review the results in more detail with LArSoft team and identify actionable bottlenecks. Sent email to Heidi and Tom.

F) Round Robin sharing.

• Talked about Event Display requirements. Tracy thinks Cory Adam’s model is good, but he has hard-coded in geometries.
• Fyi – new Macs may not be Intel based. May be issues with that.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The January Offline Leads ‘meeting’ was handled via email and a google document.

LArSoft – Erica Snider

1) LArSoft presented the plan for 2018 at the December 13th Steering Group meeting, going through the proposed short-term projects and long-term priorities.

• There was a discussion about having the experiments provide service credit to their respective experiments for work in support of LArSoft community solutions. Experiments often find this difficult to do because they have other types of indirect work that is not currently counted as service credit. We discussed carefully matching the project to the experiment that most needs it so it could be counted.
• Two items proposed by LArSoft will be added to the plan:
  • Re-architecture of art services in LArSoft to ensure thread safety
  • Support for transitioning code to art-independent repositories, making them available to run in external frameworks such as gallery.
• The LArSoft Work Plan Wrapup for 2017 was also made available to the Steering Group.
• There were no suggested changes proposed by the experiments, and the plan was approved by the two experiment representatives present (DUNE, MicroBooNE).

2) Event Display

 We are gathering requirements for an event display. Have requested feedback from the Offline leads on the draft.

3) Pixel detectors

 Erica is working with Alan Bross about having a 3D reconstruction workshop. The two primary goals of the workshop are to share ideas and on-going work on 3D pattern recognition algorithms, and to ensure that 3D pattern recognition code can be shared between experiments within LArSoft. Due to a number of key people who cannot attend the DUNE meeting, we decided to drop the idea of appending it to the end of the DUNE collaboration meeting at CERN. Will pursue alternatives that would simultaneously allow ALICE and LArSoft communities to attend. In the meantime, we will try to organize a video meeting in an attempt to develop an accord on coding standards. The purpose of those standards will be to establish a set of rules that will maintain the ability to merge coding efforts at a future time as the various stakeholders move ahead independently with code development.

4) A request from MicroBooNE was to be able to run LArSoft on NERSC. This seems to be coupled to item (7) in 2018 work plan, Architecture-dependent libraries. We have made note of it there.

5) LArSoft developers plan to meet with the art team in late January or early February to discuss the issue of selective loading of data based on geometric location, per a request from DUNE.

6) The LArSoft project is assessing how to respond to recent changes in the personnel available to work on LArSoft issues moving forward, particularly in the area of support for the ProtoDUNEs.

DUNE – Andrew John Norman, Heidi Schellman, Robert Sulej (ProtoDUNE)

DUNE is getting ready for ProtoDUNE running, building interfaces to reading raw data and preparing online monitoring and nearline monitoring payloads.  We are working on designing an event display that can be produced non-interactively in the nearline processing but displayed quickly, with buffers of precomputed event display images available for display over the web.  This allows for noise filtering and stuck code mitigation.

DUNE and ProtoDUNE management are assessing how to fill in the responsibilities that Robert Sulej and Dorota Stefan have carried so well for us these last few years.

ICARUS – Daniele Gibin

Progress has been made in the simulation of photons for the PMT system, also the CRT geometry has been finalized. Studies are ongoing in understanding the TPC charge resolution using the LArSoft simulation and reconstruction. ICARUS, partnering with SBND, has begun setting up the simulation and reconstruction production system (using POMS). Focus is now turning towards preparations for the upcoming SBN workshop in late March.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

Primary focus is on MCC 8 production which is progressing well.

Planning for MCC9 will start with a kick off meeting at MicroBooNE’s retreat the second full week of February.

SBND – Roxanne Guenette, Andrzej Szelc

No Report

Open issues:

1. Restructuring of LArG4 – Hans Wenzel and William Seligman continue to work on this. https://cdcvs.fnal.gov/redmine/issues/14454
2. Can NumPy and SciPy have standard, LArSoft compatible distributions made available?
3. Selective loading of detector data. For DUNE FD want a functionality to load into memory data products only from selected part of the detector, e.g. selected TPCs; this may be a feature request to art however. Could be a usage pattern in LArSoft, or could be in art. Will set up a meeting.
4. There was a question from DUNE about documentation for front-end interfaces to DB. Further discussion centered on access to beamline DB information, documenting how to access and populate conditions databases, etc., all of which are outside the scope of LArSoft. LArSoft documents guidelines that specify the structure of art services (or art tools) used to interface database information to detector-independent algorithm code, but does not specify how the data presented through that interface is obtained or collected. The LArSoft-facing interfaces need to be well documented as required by LArSoft coding guidelines. The LArSoft team can help provide guidance, but this isn’t an open issue for LArSoft.

There wasn’t an Offline Leads meeting in December, but the information from the Steering Group Meeting On December 13th may be of interest.

The November Offline Leads ‘meeting’ was handled via email and a google document. We received reports from: Erica Snider, Robert Sulej, Heidi Schellman, Andrzej Szelc and Tracy Usher. We did not hear from LArIAT.

We asked each experiment to provide input detailing their plans for the experiment for the next year, the implied requirements for LArSoft, and how LArSoft can help.

LArSoft – Erica Snider

• Soon Jun is working on profiling and optimization to identify problem areas and solutions in existing workflows. There will be a number of subtasks to accomplish this work in phases. The primary platform for the profiling task will be phi nodes on the Wilson cluster at FNAL where cvmfs access is already available for LArSoft/DUNE: /cvmfs/dune.opensciencegrid.org Contacted DUNE offline leaders and identified two workflows of LArSoft/Dune simulation and reconstruction for 1) single particle events and 2) cosmic events for ProtoDUNE and DUNE-FD. More workflows with different configurations may be added later if necessary. https://cdcvs.fnal.gov/redmine/issues/17921
• Giulherme Lima is working on vectorizing optimizations to identify and vectorize appropriate targets across LArSoft to take advantage of resources within our existing computing infrastructure that have not been utilized.
  There will be a number of subtasks to accomplish this work in phases.  https://cdcvs.fnal.gov/redmine/issues/17920

DUNE and ProtoDUNE – Andrew John Norman, Heidi Schellman (was Tom Junk), Robert Sulej (ProtoDUNE)

1. Introducing support for global wire coordinates is important and is documented in https://cdcvs.fnal.gov/redmine/issues/11522. At present, DUNE has implemented an ad hoc solution to provide global wire coordinate functionality and are looking for LArSoft to provide a native LArSoft solution.
2. Architecture-dependent libraries – avx2.
   • The standard setup procedure needs to support a generic library as well as one built with avx2.
   • At present, DUNE is building code with the lowest common denominator, which is sub-optimal. The setup tool (currently ups) should instead determine the best one to set up based on the platform.
   • There are a lot of areas that will have these types of issues. In general, want to take advantage of resources when they’re there. The solution might also allow use of GPU back-ends when available.
3. Event display – DUNE would like to see a common event display. Some important features:
   • It would be nice to be able to navigate through a giant event in a multi-TPC detector in a painless way.
   • There is a lot of raw data. It doesn’t fit on the screen. We should be able to zoom and pan with a data density match to the screen resolution.
4. DUNE  would profit from an option to load into memory only a selected fraction of the detector.
5. Review documentation on guidelines for the front-end interfaces used to access databases to determine how the code will interface with beamline and SAM databases.

ICARUS – Daniele Gibin ( text from Tracy)

• ICARUS has just seen its first reconstructed tracks from simulation. It has a basic geometry for the TPC readout (PMT’s and CRT in progress), basic simulation tools (including interface to genie, CORSIKA, etc.), a working detector simulation and then uses the LArSoft reconstruction chain (hit finding, track finding/fitting, etc.). All of this needs much more development. In addition, high priority tasks including simulating and then developing reconstruction for both the PMT and readout systems. As data time approaches, preparations will be needed to insure noise filtering is in place, etc.
• Following up on the above, the SBN project as a whole could benefit from more shared code in LArSoft, particularly in the signal processing area.
• An enhanced event display integrated into LArSoft is really quite critical to development of reconstruction code.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher (was Wesley Robert Ketchum)

1. Long-term – running LArSoft on NERSC.
2. Could NumPy and SciPy toolkits have a standard distribution? A lot of people use them, could they be distributed so they’re compatible with LArSoft version of python so that people don’t have to download them off the internet?
3. Event display – would like to use Qt to rewrite Event Display. Cory has an Event Display running off of Qt, but not within LArSoft. Note this needs a Qt build on osx for development purposes. Editorial note from Tracy – this should have some priority within LArSoft, at least to provide a framework (Qt distribution, basic interface to art run control, interface to geometry). There are a core group of event display users who could be pressed to develop the drawing routines.

SBND – Roxanne Guenette, Andrzej Szelc

We have recently been able to get Gallery (Gianluca, Wes) and Pandora (John Marshall) working in SBNDcode. In parallel, we are finalizing geometry, with all potential light detection elements. We used sbndcode for a LArSoft workshop in Manchester (UK and Swiss students).

Future:

1. Event Display – it would be great to have a relatively quick event display, that allows zooming through the whole detector (as per DUNE request).
2. Geant4 separation. Given that we have potentially 3 different light collection devices + the amount of data, the ability to split the LArG4 will be essential.
3. We are starting to look at SAM and databases. Not sure if this is very LArSoft specific, but we will likely be using LArSoft modules to communicate with that.
4. Anything that could speed up optical simulation would be extremely helpful.

Open issues from previous meetings:

1. From 3/22/17 meeting: Since SBND has been trying to include files inside GDML, have run into problems. Gianluca has been helping debug this. New version of ROOT may be a solution.

• • 4/18/17 update: The version of root used by LArSoft has changed a few times over the last few weeks.  Andrzej Szelc said they haven’t looked into this because they were focusing on getting the basic geometry in. Once that is done, will look at this and hope the new ROOT version may have fixed it. So, no ticket has been written yet (to ROOT or LArSoft) about this as waiting on whether it is still an issue.
  • 5/25/17 update – still waiting.
  • 6/14/17 update – Once SBND gets the latest version of ROOT in, they’ll see if this fixes the issue.
  • 7/27/17 update – SBND is still finalizing the new Geometry idea – expert moving. Once this is done, we have a student starting up that could follow up on whether the bugs are gone in the newest version of ROOT. In the meantime Gianluca with Vito’s help is putting sbndcode in the continuous integration framework – this will be very useful.
  • 8/16/17 – have basic geometry working. Still debugging before mounting full scale production.
  • 9/28/17 – SBND said the geometry is working and they’re using it, but they may go back and test if there is still an issue.
  • 10/16/17 – no changes
  • 11/20/17 – from SBND- no changes- will update if we ever get back to it. Ok to close.
  • Closed.

2. From May: Feedback at a recent SBN analysis meeting on proposed restructuring of G4 simulation step was that these were potentially very important changes. Is it possible to see all of this in place this summer for ICARUS large-scale processing and potentially MicroBooNE processing campaign?  Along with the work itself, how one updates/does the backtracking remains an unanswered question: not technically difficult, but could imply significant changes in downstream code based on how it is done. This may take another or a few more people working on it to really see it through.

• • The project and schedule are being tracked in issue 14454,   https://cdcvs.fnal.gov/redmine/issues/14454
  • 8/16/17 update – Up until June, thought could get it done by mid-summer. The original plan was to incrementally replace parts of the LArG4 code with the updated Geant4 functionality and interfaces. But it looks like it is easier to do a full replacement. Since we changed the scope, we’re awaiting a good estimate on completion.
    • A question about the design arose, which was whether the ionization and scintillation modeling is downstream of the energy deposition, or embedded in the Geant4 stepping. ICARUS (and presumably other experiments) would like to be able to run ionization and scintillation, and all other downstream simulation in a separate job from the energy deposition simulation. It is not clear whether this is possible if the ionization and scintillation is done by Geant4. We (LArSoft) believe this will be possible with the re-factored code, but should be verified with Hans.  Will set up a meeting with Hans, Wes and Erica when he’s back to clarify design details and get a reasonable end-date or if additional resources are needed.
  • 9/15/17 update – Hans delivered a draft of the data product which we passed along to Offline Leads on September 8. Requesting feedback.
  • 9/28/17 update – MicroBooNE feedback: Hans data product looks good, with just one concern that the number of photons may be misleading should it be updated later. Perhaps “initial_photons” or something like that? Not sure.  Also trust that “G4double” is ok, or will be changed to C++ “double”.
  • 10/16/17 – no change
  • 11/15/17 – Hans and William Seligman continue to work on this. William gave a presentation at the 11/21/17 LArSoft Coordination Meeting.

October 16, 2017 Notes from Offline Leads’ Meeting with secondary meeting on Oct 18th

Attendance:

• Herb Greenlee, Tracy Usher, Tom Junk, Erica Snider, Katherine Lato at the first meeting.
• Heidi Schellman, Tom Junk, Robert Sulej, Erica Snider, Katherine Lato at the second meeting

LArSoft

• Discussed with the Offline Leads about their doing more of the CI operations. Experiments like the way it currently works, but LArSoft asked them to consider having someone who has a role to look for problems in CI test results
• LArSoft is requesting  feedback on moving to a GitHub-hosted pull-request system, where request approvers would be selected from the experiments. Also discussed possible code re-organizations and introduction of sparse checkouts that might accompany the GitHub migration.
• Event display – experiments would like to use Qt (or another framework) to rewrite Event Display. If there was a package or framework to build off of, it wouldn’t be too much effort for experiments to rewrite the Event Display.
  • MicroBooNE has investigated using TEve, but can’t replicate the existing 2D display with that tool, so ROOT isn’t a viable option.  
  • Ideally, LArSoft would provide a sensible framework that could be extended as needed to provide features needed.
  • Cory has an Event Display running off of Qt, but not within LArSoft.
• Two new sub-projects are being  defined in LArSoft to take advantage of the availability of experts in these areas for the next few months

1. • Profiling and optimization work  to identify problem areas and solutions in existing workflows. Will be carried out by Soon Yung Jun https://cdcvs.fnal.gov/redmine/issues/17921
   • Vectorizing optimizations to identify and vectorize appropriate targets across LArSoft to take advantage of resources within our existing computing infrastructure that has not been utilized. Will be carried out by Giulherme Lima https://cdcvs.fnal.gov/redmine/issues/17920

• Pursuing a date and location for the 3D pattern recognition workshop, the proposal is to couple it to the DUNE Collaboration Meeting at CERN Jan 29 – Feb 1, 2018, so that would be Friday, Feb. 2, 2018. Proposed scope will cover all existing work to produce 3D space points, plus algorithms to perform pattern recognition using space points as input. Will also work toward agreements on data products, interfaces between various workflow stages so that we can continue to share algorithms.
• LArSoft asked for requirements for LArSoft for 2018, including looking at redmine issues that are accepted redmine issues, but not assigned for things that may be of priority to the experiments.

DUNE

Wirecell people are interested in knowing the data product that will be used as input to LArG4. Note, LArSOft sent email on September 8th asking for feedback.

Introducing support for global wire coordinates is important and is documented in https://cdcvs.fnal.gov/redmine/issues/11522.  Discussed some of the technical requirements.

• For pattern recognition, if TPCs are aligned near enough, the global wire coordinates make sense.
  • Detector construction alignment tolerances is therefore a critical issue, particularly TPC to TPC rotations.
  • Details are not known yet, but expect mis-alignments to be small compared to wire spacing. What does a global wire coordinate mean when you’re trying to get to 3D via 2D images? If rotations cannot be ignored at pattern recognition phase, then a global wire coordinate is not helpful, since you effectively change coordinate systems as you cross boundaries.
• Global wire coordinates may have implications for the event display.
• At present, DUNE has implemented an ad hoc solution to provide global wire coordinate functionality

LArSoft sent DUNE email requesting they identify a major DUNE production workflow that will be the target of the profiling work, and a person in DUNE who will work with Soon Yung Jun, who will provide the resources and connections within DUNE needed.

Architecture-dependent libraries – avx2.

• Need to support a generic library as well as one built with avx2.
• TensorFlow may have run into this issue. At present, DUNE is building code with the lowest common denominator, which is sub-optimal
• There are a lot of areas that will have these types of issues. In general, want to take advantage of resources when they’re there. The solution might also allow use of GPU back-ends when available.

Event display – DUNE would like to see a common event display. It would be nice to be able to navigate through a giant event in a painless way. There is a lot of raw data. It doesn’t fit on the screen. Have to be able to zoom and pan. Very interested in a common event display across LArSoft.

• LArSoft would like to support development of a new framework for the event display, one that will run in both art and gallery. At present, we do not have effort available for this.

Also mentioned at the meeting:

• Databases have come up a lot. LArSoft has guidelines for the front-end interfaces used to access databases. LArSoft also recommends using one of the two lab supported back-end packages whenever possible. (They are not part of LArSoft)
• They are thinking about multi-threading with new code.
• Discussion with CERN DAC people, how access to SAM meta-data works? Is this an art issue?

MicroBooNE

Have heard that ArgonCube believes they can’t use LArSoft because it can’t handle a pixel detector.

• LArSoft noted that with the LArG4 redesign, the anode region simulation is abstracted, which allows separate solutions for single phase, dual phase and pixel-based detectors. Erica talked at a pixel meeting and mentioned that LArSoft is restructuring the simulation in this way and stated that it would be possible to handle pixel readouts, though not immediately.
• They also want to be able to run LArSoft on NERSC, which is not currently possible
  • This might become easier once we have Spack build system, since that is a tool from the supercomputing world

Mentioned Bill Seligman’s work on LArG4.

• LArSoft is hoping that what he produces will be sufficiently easy to use extensible enough to be something people will want to use.
• Tracy mentioned that SLAC has people in their Geant4 group who are actively looking for projects. Might want to look into whether we can recruit one of them to work on LArG4 re-factoring. Depends on what Bill is doing, so talk to him first.

Event display – would like to use Qt to rewrite Event Display. Cory has an Event Display running off of Qt, but not within LArSoft.

Could NumPy and SciPy toolkits have a standard distribution? A lot of people use them, could they be distributed so they’re compatible with LArSoft version of python so that people don’t have to download them off the internet?

• LArSoft will look into developing a process for nominating tools to be included

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The September Offline Leads ‘meeting’ was handled via email and a google document. We received reports from: Erica Snider, Thomas Junk, Robert Sulej, Andrzej Szelc and Wes Ketchum. We did not hear from: ICARUS or LArIAT.

LArSoft Report – Erica Snider

1. Status of LArG4 – Hans has delivered a draft data product that will store energy depositions created by Geant4 that can then be used as input for ionization / scintillation modeling, electron and photon transport modeling, etc.  We are requesting your feedback on the item sent by Erica via email on September 8th.  Once we have agreement on this data product, we can use it as the basis for extracting the various LArTPC simulation effects into pluggable components to be used in the replacement of LArG4. Some coordination on the relevant interfaces will be needed, so please talk to us (Hans, Gianluca and Erica) before making changes.
2. Status of SPACK migration. Jim Amundson has turned development over to Chris Jones and discussed the details on September 14th.
3. Phase II track work – Phase 1 changes to recob::Track completed in June included added track fit information, introduced policies for trajectory points, etc. This provides the basis for a uniform definition of tracks from different algorithms, treatment of fts, etc. Phase 2 changes will introduce helpers to write tracks with all requires associations, a track “proxy” (previously called “facade”) that hides complications of finding associated hits and clusters. This will be extended to include a number of other associated data structures in Phase 3. Specification of the interfaces are being finalized and will be presented at a LArSoft Coordination meeting.
4. A recob::SpacePoint re-design effort has been launched, motivated by the several active 3D reconstruction efforts. Goals:
   • Provide  a uniform interface between 3D SpacePoint finding and downstream PR
     • Include all information needed to store results of / drive 3D pattern recognition
     • Complements recent discussion with pixel-based L/GAr TPC efforts
       • Discussing how to integrate pixel reconstruction into LArSoft
       • Enabled by 3D efforts!  
     • Include charge (new!), covariance matrix, chisq etc, by implicit association
       • Via parallel vectors
       • just need to know the index to navigate to associated elements
     • Plan to use “proxy” concept to simplify finding associated data
     • Straw proposal under discussion at recent architecture meeting. A number of issues raised in that discussion related to these choices are under review (e.g. track fitting)
     • Will get to recob::Vertex soon

When setting priorities, would like to hear opinions on items that are in the accepted redmine issues, not assigned to see if they are a priority for next year, or if any of them should be rejected because the need is no longer there.

DUNE & ProtoDUNE – Thomas Junk, Robert Sulej, Andrew John Norman

Answer to documentation question: We will distribute the source code documentation guideline wiki page to our developers (both DUNE & ProtoDUNE) via email.

Current issue:  Robert Sulej is interested in using TensorFlow with his CNN in the LArSoft environment.  Robert is working with Lynn, Patrick, and Erica to get TensorFlow as a UPS product.  The problem is that TensorFlow is its own little world and has its own build system. Tensorflow based code for the CNN inference mode is up and running. The EM/track/Michel selection model is ready for it. Tensorflow UPS is still in preparations (the library size needs to be addressed by Lynn, need to turn on “core2” optimizations). Hope to have the full machinery ready by 1-2 weeks

Hans’s new data product for persisting G4 steps will help in the WireCell integration as well as the FLUKA interface.

Hans’s data product (actually refactorization of the G4 step) enables the factorization of the dual-phase detector simulation. However, due to the DUNE TDR timeline, we proceed with the simulations performed with the modified current codes (on a branch not planned to merge with LArSoft develop).

The feature https://cdcvs.fnal.gov/redmine/issues/11522 ‘Global wire’ coordinate within LArSoft geometry, is important for both ProtoDUNE and DUNE. For the purpose of neutrino selection for DUNE TDR people are developing their own ways of imaging the events (on the hit level) contained in multiple TPC. We will need the same for ProtoDUNE at the level of ADC. The feature would really facilitate reconstruction, analysis codes and also event displays.

ICARUS – Daniele Gibin

No Report

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Wesley Robert Ketchum

Answer to documentation question: We will inform developers via email and meetings on the quidelines, and encourage that use for new code. We can start to work through modules in use in production to make sure it meets the guidelines. New modules going into production we can check to make sure this is included.

Hans data product looks good, with just one concern that the number of photons may be misleading should it be updated later. Perhaps “initial_photons” or something like that? Not sure.  Also trust that “G4double” is ok, or will be changed to C++ “double”.  We’ve identified Bill S. as someone from MicroBooNE who may be able to help implement further downstream changes working with the project. Encourage LArSoft project to reach out to him and formulate a plan.

MicroBooNE has recently tagged its reconstruction software release for the near-term, with a potential update planned for late fall/early winter that would be used for summer conferences. It’s currently anticipated we will need continued long-term support for the v06_26_01 branch of LArSoft, likely with potential changes to LArSoft releases near the end of the calendar year, and potentially urgent bug fixes after that point.

Tracy Usher is the new MicroBooNE Analysis Tools Co-convener, replacing Wes.

SBND – Roxanne Guenette, Andrzej Szelc

We have finally updated the geometry into a working state, and used it to launch a larger scale data production. Depending on the availability of Gustavo V., who first observed this problem, we might go back and test whether the issue is still there. We have a student who wants to start with the Geometry, this might be a good starting project for her as well.

We are now trying to understand storage to tape, given the size of our production and perhaps any data size reducing measures that MicroBooNE has developed.

Garcia-Gamez and V. Basque are testing the optical light simulation that allows using optical foils in the newest versions of LArSoft. In particular, they are testing the effects of changing the Rayleigh scattering length on the timing parametrizations that were developed – this is a work in progress currently.

Per Documentation we will send it out the SBND-software mailing list and discuss it at one of our Software/Physics meetings.

They are hosting a LArSoft tutorial for UK (and maybe European students) in Manchester a month from now similar to a self-organized one they did two years ago. Information is at: http://indico.hep.manchester.ac.uk/conferenceDisplay.py?ovw=True&confId=5237 

Open issues from previous meetings:

1. From 3/22/17 meeting: Since SBND has been trying to include files inside GDML, have run into problems. Gianluca has been helping debug this. New version of ROOT may be a solution.

• • 4/18/17 update: The version of root used by LArSoft has changed a few times over the last few weeks.  Andrzej Szelc said they haven’t looked into this because they were focusing on getting the basic geometry in. Once that is done, will look at this and hope the new ROOT version may have fixed it. So, no ticket has been written yet (to ROOT or LArSoft) about this as waiting on whether it is still an issue.
  • 5/25/17 update – still waiting.
  • 6/14/17 update – Once SBND gets the latest version of ROOT in, they’ll see if this fixes the issue.
  • 7/27/17 update – SBND is still finalizing the new Geometry idea – expert moving. Once this is done, we have a student starting up that could follow up on whether the bugs are gone in the newest version of ROOT. In the meantime Gianluca with Vito’s help is putting sbndcode in the continuous integration framework – this will be very useful.
  • 8/16/17 – have basic geometry working. Still debugging before mounting full scale production.
  • 9/28/17 – SBND said the geometry is working and they’re using it, but they may go back and test if there is still an issue.

2. From May: Feedback at a recent SBN analysis meeting on proposed restructuring of G4 simulation step was that these were potentially very important changes. Is it possible to see all of this in place this summer for ICARUS large-scale processing and potentially MicroBooNE processing campaign?  Along with the work itself, how one updates/does the backtracking remains an unanswered question: not technically difficult, but could imply significant changes in downstream code based on how it is done. This may take another or a few more people working on it to really see it through.

• • The project and schedule are being tracked in issue 14454,   https://cdcvs.fnal.gov/redmine/issues/14454
  • 8/16/17 update – Up until June, thought could get it done by mid-summer. The original plan was to incrementally replace parts of the LArG4 code with the updated Geant4 functionality and interfaces. But it looks like it is easier to do a full replacement. Since we changed the scope, we’re awaiting a good estimate on completion.
    • A question about the design arose, which was whether the ionization and scintillation modeling is downstream of the energy deposition, or embedded in the Geant4 stepping. ICARUS (and presumably other experiments) would like to be able to run ionization and scintillation, and all other downstream simulation in a separate job from the energy deposition simulation. It is not clear whether this is possible if the ionization and scintillation is done by Geant4. We (LArSoft) believe this will be possible with the re-factored code, but should be verified with Hans.  Will set up a meeting with Hans, Wes and Erica when he’s back to clarify design details and get a reasonable end-date or if additional resources are needed.
  • 9/15/17 update – Hans delivered a draft of the data product which we passed along to Offline Leads on September 8. Requesting feedback.
    • 9/28/17 update – MicroBooNE feedback: Hans data product looks good, with just one concern that the number of photons may be misleading should it be updated later. Perhaps “initial_photons” or something like that? Not sure.  Also trust that “G4double” is ok, or will be changed to C++ “double”.

3. From 6/14/17 meeting: MicroBooNE asked about how to give credit to authors, experiments and institutions for LArSoft code written.

• 7/27/17 update
  1. Have updated the recommendation on documenting code at http://larsoft.org/important-concepts-in-larsoft/design/.
  2. We committed to presenting a proposed documentation template for header files at a LArSoft Coordination Meeting.  (This was done on 6/27/17).
  3. LArSoft has a place to give author credit for algorithms and services on larsoft.org – as well as in the code itself. This is not being done enough, so suggest a multi-prong education campaign to encourage people to sign their name on the code they write and to contribute to the http://larsoft.org/doc-algorithms/ page. This will be highlighted in a LArSoft Coordinate Meeting with follow up but this needs to be done by all experiments, not just LArSoft.
  4. Presented proposal at June 27 Coordination Meeting. The action items from that discussion
     1. Add institution and experiment to the template
     2. Publish the template on a web page
     3. What are the plans from the experiments to push this practice out to the code?
        1. Recommend adding template documentation whenever code is updated?
        2. Make it the policy of the experiments
• 8/10/17 – update – have institution and author in the template and published it at: https://cdcvs.fnal.gov/redmine/projects/larsoft/wiki/Code_documentation_requirements_and_guidelines
  • Need to provide an environment variable so that the template to ctskelgen points to a file with the above field. https://cdcvs.fnal.gov/redmine/issues/17426 – redmine issue for that work assigned to Erica.
• 9/14/17 update – asking each experiment to report on their plan for pushing this practice out to the code. Closing this issue.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

August 16, 2017  Offline Leads Meeting  

Attendance: Erica Snider, Katherine Lato, Wes Ketchum, Thomas Junk, Herb Greenlee, Robert Sulej

Round Robin – we went over the open issues (see end of notes), then went around the room.

LArSoft  – Erica Snider

• DIscussed Geant4 work and how it has been rescoped. (see issue 2 below)
• Don’t know the target for new build system. Isn’t on the critical path for any experiments at this time.
• Floating Point Exceptions – agreed to fix these. They are almost always bugs and should be fixed. Often the first one fixed reveals more.
• Documentation standard – please remind developers to follow them. Should ctskelgen also work for tools? (Feature request?)
• Tensor Flow is high priority – had meetings on 8/16 and 8/17 to deal with issues involved.
• Gianluca is leaving in September of 2018. Not much news on replacement except that a dedicated developer isn’t likely. Feedback from experiments — having a dedicated developer like Gianluca has been invaluable. Someone thinking fulltime about things has been useful for collaborators for things they aren’t familiar with, developing best practices, code maintenance, etc. Having a developer go through common code to ensure the code is done, and done better. ICARUS relied on a small group of experts who aren’t tasked with doing that, so having a developer who can do the technical work relieves the load on the experiment and collaboration. When the solution is useful for more than one experiment, it’s good to have that solution be professional. It would be useful to have a team of developers work on the needs of the collaboration.

DUNE – Thomas Junk

Dune TPC is getting big and the build is slow, not a LArSoft issue, but Tom mentioned it.

ICARUS – Daniele Gibin not available but Wes Ketchum reported.

Could use support in getting things going on the grid and build. Lynn has given guidance, including looking at other experiment’s. Note, Patrick might be able to help with this. And Vito.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Wesley Robert Ketchum

 Working on getting openCV, LArCV as a UPS product. Don’t know if they need assistance from LArSoft since they’re working on integration.

ProtoDUNE – Robert Sulej

• Just finished a meeting about Tensor Flow, that is high priority.
• Interested in the material we already covered in this meeting, Geant 4. They met with Hans and he said he would try to develop a data product they could use in dual phase. It is also high priority for them. They need the data product before they need the rest of the factorization.

SBND – Andrzej Szelc

Report via email:

• We finally got our basic geometry working. We have launched our first larger scale production using it (this is still single file .gdml, not the advanced version mentioned below) and will be debugging any problems that arise before we mount full scale production in next weeks.
• In parallel, we are working toward testing that our optical simulation is working (also needed for full scale production).
• Next month we should have validation results of tests + some idea of how that all works with large scale production.
• We have had some first problems with running reco jobs – which might be due to disk space on grid nodes. We’re trying to understand what happened there.
• Finally, we have identified a person that will take over geometry management, so might come back to the multiple file design.

Open issues from previous meetings:

1. From 3/22/17 meeting: Since SBND has been trying to include files inside GDML, have run into problems. Gianluca has been helping debug this. New version of ROOT may be a solution.

• • 4/18/17 update: The version of root used by LArSoft has changed a few times over the last few weeks.  Andrzej Szelc said they haven’t looked into this because they were focusing on getting the basic geometry in. Once that is done, will look at this and hope the new ROOT version may have fixed it. So, no ticket has been written yet (to ROOT or LArSoft) about this as waiting on whether it is still an issue.
  • 5/25/17 update – still waiting.
  • 6/14/17 update – Once SBND gets the latest version of ROOT in, they’ll see if this fixes the issue.
  • 7/27/17 update – SBND is still finalizing the new Geometry idea – expert moving. Once this is done, we have a student starting up that could follow up on whether the bugs are gone in the newest version of ROOT. In the meantime Gianluca with Vito’s help is putting sbndcode in the continuous integration framework – this will be very useful.
  • 8/16/17 – have basic geometry working. Still debugging before mounting full scale production.

2. From May: Feedback at a recent SBN analysis meeting on proposed restructuring of G4 simulation step was that these were potentially very important changes. Is it possible to see all of this in place this summer for ICARUS large-scale processing and potentially MicroBooNE processing campaign?  Along with the work itself, how one updates/does the backtracking remains an unanswered question: not technically difficult, but could imply significant changes in downstream code based on how it is done. This may take another or a few more people working on it to really see it through.

• • The project and schedule are being tracked in issue 14454,   https://cdcvs.fnal.gov/redmine/issues/14454
  • 8/16/17 update – Up until June, thought could get it done by mid-summer. The original plan was to incrementally replace parts of the LArG4 code with the updated Geant4 functionality and interfaces. But it looks like it is easier to do a full replacement. Since we changed the scope, we’re awaiting a good estimate on completion.
    • A question about the design arose, which was whether the ionization and scintillation modeling is downstream of the energy deposition, or embedded in the Geant4 stepping. ICARUS (and presumably other experiments) would like to be able to run ionization and scintillation, and all other downstream simulation in a separate job from the energy deposition simulation. It is not clear whether this is possible if the ionization and scintillation is done by Geant4. We (LArSoft) believe this will be possible with the re-factored code, but should be verified with Hans.  Will set up a meeting with Hans, Wes and Erica when he’s back to clarify design details and get a reasonable end-date or if additional resources are needed.

3. From 6/14/17 meeting: MicroBooNE asked about how to give credit to authors, experiments and institutions for LArSoft code written.

• 7/27/17 update
  1. Have updated the recommendation on documenting code at http://larsoft.org/important-concepts-in-larsoft/design/.
  2. We committed to presenting a proposed documentation template for header files at a LArSoft Coordination Meeting.  (This was done on 6/27/17).
  3. LArSoft has a place to give author credit for algorithms and services on larsoft.org – as well as in the code itself. This is not being done enough, so suggest a multi-prong education campaign to encourage people to sign their name on the code they write and to contribute to the http://larsoft.org/doc-algorithms/ page. This will be highlighted in a LArSoft Coordinate Meeting with follow up but this needs to be done by all experiments, not just LArSoft.
  4. Presented proposal at June 27 Coordination Meeting. The action items from that discussion
     1. Add institution and experiment to the template
     2. Publish the template on a web page
     3. What are the plans from the experiments to push this practice out to the code?

1. 1. 1. 1. Recommend adding template documentation whenever code is updated?
         2. Make it the policy of the experiments

8/10/17 – update – have institution and author in the template and published it at: https://cdcvs.fnal.gov/redmine/projects/larsoft/wiki/Code_documentation_requirements_and_guidelines

Need to provide an environment variable so that the template to ctskelgen points to a file with the above field. https://cdcvs.fnal.gov/redmine/issues/17426 – redmine issue for that work assigned to Erica.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The July Offline Leads ‘meeting’ was handled via email and a google document. We heard from over half the experiments, not bad for a July. Next meeting is live.

LArSoft Report – Erica Snider

Floating Point Exceptions policy decision. After a discussion at the 6/27/17 LArSoft Coordination meeting led by Herb Greenlee, we concluded we don’t want to ignore FPEs. Thomas Junk has been tracking down where exceptions occur under the umbrella milestone of https://cdcvs.fnal.gov/redmine/issues/17047. Summary of the discussion at the June 27 Coordination Meeting: Continue reading July 2017 LArSoft Offline Leads Meeting Notes →