November 2017 Offline Leads meeting

leads

The November Offline Leads ‘meeting’ was handled via email and a google document. We received reports from: Erica Snider, Robert Sulej, Heidi Schellman, Andrzej Szelc and Tracy Usher. We did not hear from LArIAT.

We asked each experiment to provide input detailing their plans for the experiment for the next year, the implied requirements for LArSoft, and how LArSoft can help.

LArSoft – Erica Snider

  • Soon Jun is working on profiling and optimization to identify problem areas and solutions in existing workflows. There will be a number of subtasks to accomplish this work in phases. The primary platform for the profiling task will be phi nodes on the Wilson cluster at FNAL where cvmfs access is already available for LArSoft/DUNE: /cvmfs/dune.opensciencegrid.org Contacted DUNE offline leaders and identified two workflows of LArSoft/Dune simulation and reconstruction for 1) single particle events and 2) cosmic events for ProtoDUNE and DUNE-FD. More workflows with different configurations may be added later if necessary. https://cdcvs.fnal.gov/redmine/issues/17921
  • Giulherme Lima is working on vectorizing optimizations to identify and vectorize appropriate targets across LArSoft to take advantage of resources within our existing computing infrastructure that have not been utilized.
    There will be a number of subtasks to accomplish this work in phases.  https://cdcvs.fnal.gov/redmine/issues/17920

DUNE and ProtoDUNE – Andrew John Norman, Heidi Schellman (was Tom Junk), Robert Sulej (ProtoDUNE)

  1. Introducing support for global wire coordinates is important and is documented in https://cdcvs.fnal.gov/redmine/issues/11522. At present, DUNE has implemented an ad hoc solution to provide global wire coordinate functionality and are looking for LArSoft to provide a native LArSoft solution.
  2. Architecture-dependent libraries – avx2.
    • The standard setup procedure needs to support a generic library as well as one built with avx2.
    • At present, DUNE is building code with the lowest common denominator, which is sub-optimal. The setup tool (currently ups) should instead determine the best one to set up based on the platform.
    • There are a lot of areas that will have these types of issues. In general, want to take advantage of resources when they’re there. The solution might also allow use of GPU back-ends when available.
  3. Event display – DUNE would like to see a common event display. Some important features:
    • It would be nice to be able to navigate through a giant event in a multi-TPC detector in a painless way.
    • There is a lot of raw data. It doesn’t fit on the screen. We should be able to zoom and pan with a data density match to the screen resolution.
  4. DUNE  would profit from an option to load into memory only a selected fraction of the detector.
  5. Review documentation on guidelines for the front-end interfaces used to access databases to determine how the code will interface with beamline and SAM databases.

ICARUS – Daniele Gibin ( text from Tracy)

  • ICARUS has just seen its first reconstructed tracks from simulation. It has a basic geometry for the TPC readout (PMT’s and CRT in progress), basic simulation tools (including interface to genie, CORSIKA, etc.), a working detector simulation and then uses the LArSoft reconstruction chain (hit finding, track finding/fitting, etc.). All of this needs much more development. In addition, high priority tasks including simulating and then developing reconstruction for both the PMT and readout systems. As data time approaches, preparations will be needed to insure noise filtering is in place, etc.
  • Following up on the above, the SBN project as a whole could benefit from more shared code in LArSoft, particularly in the signal processing area.
  • An enhanced event display integrated into LArSoft is really quite critical to development of reconstruction code.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher (was Wesley Robert Ketchum)

  1. Long-term – running LArSoft on NERSC.
  2. Could NumPy and SciPy toolkits have a standard distribution? A lot of people use them, could they be distributed so they’re compatible with LArSoft version of python so that people don’t have to download them off the internet?
  3. Event display – would like to use Qt to rewrite Event Display. Cory has an Event Display running off of Qt, but not within LArSoft. Note this needs a Qt build on osx for development purposes. Editorial note from Tracy – this should have some priority within LArSoft, at least to provide a framework (Qt distribution, basic interface to art run control, interface to geometry). There are a core group of event display users who could be pressed to develop the drawing routines.

SBND – Roxanne Guenette, Andrzej Szelc

We have recently been able to get Gallery (Gianluca, Wes) and Pandora (John Marshall) working in SBNDcode. In parallel, we are finalizing geometry, with all potential light detection elements. We used sbndcode for a LArSoft workshop in Manchester (UK and Swiss students).

Future:

  1. Event Display – it would be great to have a relatively quick event display, that allows zooming through the whole detector (as per DUNE request).
  2. Geant4 separation. Given that we have potentially 3 different light collection devices + the amount of data, the ability to split the LArG4 will be essential.
  3. We are starting to look at SAM and databases. Not sure if this is very LArSoft specific, but we will likely be using LArSoft modules to communicate with that.
  4. Anything that could speed up optical simulation would be extremely helpful.

Open issues from previous meetings:

  1. From 3/22/17 meeting: Since SBND has been trying to include files inside GDML, have run into problems. Gianluca has been helping debug this. New version of ROOT may be a solution.
    • 4/18/17 update: The version of root used by LArSoft has changed a few times over the last few weeks.  Andrzej Szelc said they haven’t looked into this because they were focusing on getting the basic geometry in. Once that is done, will look at this and hope the new ROOT version may have fixed it. So, no ticket has been written yet (to ROOT or LArSoft) about this as waiting on whether it is still an issue.
    • 5/25/17 update – still waiting.
    • 6/14/17 update – Once SBND gets the latest version of ROOT in, they’ll see if this fixes the issue.
    • 7/27/17 update – SBND is still finalizing the new Geometry idea – expert moving. Once this is done, we have a student starting up that could follow up on whether the bugs are gone in the newest version of ROOT. In the meantime Gianluca with Vito’s help is putting sbndcode in the continuous integration framework – this will be very useful.
    • 8/16/17 – have basic geometry working. Still debugging before mounting full scale production.
    • 9/28/17 – SBND said the geometry is working and they’re using it, but they may go back and test if there is still an issue.
    • 10/16/17 – no changes
    • 11/20/17 – from SBND- no changes- will update if we ever get back to it. Ok to close.
    • Closed.

2. From May: Feedback at a recent SBN analysis meeting on proposed restructuring of G4 simulation step was that these were potentially very important changes. Is it possible to see all of this in place this summer for ICARUS large-scale processing and potentially MicroBooNE processing campaign?  Along with the work itself, how one updates/does the backtracking remains an unanswered question: not technically difficult, but could imply significant changes in downstream code based on how it is done. This may take another or a few more people working on it to really see it through.

    • The project and schedule are being tracked in issue 14454,   https://cdcvs.fnal.gov/redmine/issues/14454
    • 8/16/17 update – Up until June, thought could get it done by mid-summer. The original plan was to incrementally replace parts of the LArG4 code with the updated Geant4 functionality and interfaces. But it looks like it is easier to do a full replacement. Since we changed the scope, we’re awaiting a good estimate on completion.
      • A question about the design arose, which was whether the ionization and scintillation modeling is downstream of the energy deposition, or embedded in the Geant4 stepping. ICARUS (and presumably other experiments) would like to be able to run ionization and scintillation, and all other downstream simulation in a separate job from the energy deposition simulation. It is not clear whether this is possible if the ionization and scintillation is done by Geant4. We (LArSoft) believe this will be possible with the re-factored code, but should be verified with Hans.  Will set up a meeting with Hans, Wes and Erica when he’s back to clarify design details and get a reasonable end-date or if additional resources are needed.
    • 9/15/17 update – Hans delivered a draft of the data product which we passed along to Offline Leads on September 8. Requesting feedback.
    • 9/28/17 update – MicroBooNE feedback: Hans data product looks good, with just one concern that the number of photons may be misleading should it be updated later. Perhaps “initial_photons” or something like that? Not sure.  Also trust that “G4double” is ok, or will be changed to C++ “double”.
    • 10/16/17 – no change
    • 11/15/17 – Hans and William Seligman continue to work on this. William gave a presentation at the 11/21/17 LArSoft Coordination Meeting.