The November 2018 LArSoft Offline Leads status update was handled via email and a google document.

LArSoft – Erica Snider

LArSoft met with each experiment’s Spokespeople and Offline Leads to learn detailed plans for the next year, the implied requirements for LArSoft, and how LArSoft could help, as well as what the experiments might be able to contribute to LArSoft code. This input will be used to develop the 2019 LArSoft Work Plan.

Stage 1 of the LArG4 refactoring project will officially close with the acceptance of documentation, currently posted to https://cdcvs.fnal.gov/redmine/projects/larg4/wiki/Wiki. The offline leads are invited to review and comment on the documentation at any time.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang, Tom Junk

A recent shift away from using /grid/fermiapp for staging of built binaries made DUNE’s build script fail on mac.  CVMFS is unreliable and slow on mac, so we use the /grid/fermiapp mount. We see that larsoft and other experiments use buildFW in their Jenkins scripts and may want to consult with how to do that ourselves.   Keeping the latest release on /grid/fermiapp is the path of least resistance for us.

ICARUS – Daniele Gibin, (Tracy Usher)

ICARUS has launched MCC 1.1 which addressed some deficiencies in the initial 1.0 run but importantly also signifies the official switch to the POMS system. This week (final week of November) is being used to eliminate the last remaining issues and we expect to be generating moderate size (compared to MicroBooNE and ProtoDUNE) data sets in the next month.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

No Report

SBND – Roxanne Guenette, Andrzej Szelc

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

Attendance: Erica Snider, Tom Junk, Katherine Lato

1. LArSoft Status including open issues.
   • Comment on GitHub migration:  no progress during past weeks. SCD planning a meeting with CMS to discuss this migration with several experiments.
   • 3D workshop status
     • Planning a follow-up meeting to discuss common data layer concepts needed to interface arbitrary 3D PR algorithms and 3D imaging algorithms https://docs.google.com/document/d/1rDsoGQhJgSMhxwP2dH0U2rV5sCWFgU3hII9d-n4gMVE/edit
   • Update on event display
     • Paul Russo will be looking into some of the options for the technology to use to replace the LArSoft event display.   
     • Tom noted that there are lots of little things that are bothersome in the current Event Display.
   • LArG4 update
     • Documentation – Hans is providing examples of fhicl files, gdml files
     • AuxDet Module hasn’t been put into the event stream yet, but shouldn’t take long
     • He plans on presenting at the Oct. 23rd LArSoft Coordination meeting.
     • Need to make decision on how to re-architect the handling of LAr material properties (which has major implications for LArProperties and Geometry services). Note, this is part of phase 3.
2. Round Robin sharing.
   • DUNE –
     • art 3 is of interest. LArSoft has a list of changes for art 3, under breaking changes. https://cdcvs.fnal.gov/redmine/projects/larsoft/wiki/Update_from_art_v2_to_art_v3
     • Two scenarios involve taking data with long waveforms. (1) SN data stream, where data blocks can last ~30 sec. (2) Low frequency noise studies where data blocks are ~100k ticks long. This is an issue because raw digits use a short to define the number of ticks in the waveform. Both of these scenarios exceed the 65k limit. Case (2) can be solved using DAQ data structures, since this is a limited study that is not likely to be repeated (Tom believes. Dave Evans is the requester here). Case (1) will require some mechanism for triggering on the streaming data, then defining events that are ~30 sec long, then micro-triggering to produce events that fit within the existing raw digit format. Also discussed a third case, that of proton decay searches, but those will define three or four drift window events from the continuous stream. Conclusion is that no request for a change is requested, so this is not yet a LArSoft issue.
   • From Andrzej (last month’s meeting notes): UK groups will run a LArSoft workshop in Manchester, 10/31-11/2, which is geared up to get people started in LArSoft and basic reconstruction. http://indico.hep.manchester.ac.uk/conferenceDisplay.py?confId=5372
3. Input on priorities for 2019 LArSoft workplan.
   • Please let us know by mid-November if you have any specific items you’d like LArSoft to consider for 2019 workplan.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The September 2018 LArSoft Offline Leads status update was handled via email and a google document.

LArSoft – Erica Snider

• Working to develop a plan for migration to GitHub and pull requests. Will be seeking input from CMS on their experience with this exercise, and will continue discussing ongoing plans and concerns with the offline leads. The timescale for the migration is open for the time being.
• The LArSoft team has developed a draft interim error handling policy based on the use of the art::Exception class. The current draft is interim pending an update to the art exception wiki page.
• LArSoft met with various 3D pattern recognition stakeholders to discuss how to proceed with creating a common interface layer into 3D pattern recognition algorithms, and other topics of common interest to 3D reconstruction in LAr and GAr TPCs. A working outline agenda for a workshop can be found in https://docs.google.com/document/d/1rDsoGQhJgSMhxwP2dH0U2rV5sCWFgU3hII9d-n4gMVE/edit?usp=sharing
• The LArSoft team has started a survey of products and technologies that are candidates to replace the event display.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang

ProtoDUNE has started analyzing real data from the detector. The noise level is very low (4-5 ADCs) which leads to very high signal-to-noise ratio. Clean tracks are visible and successfully reconstructed by the Pandora and PMA algorithms. At this moment, the electron lifetime is around 100 us and increasing. The current focus on reconstruction is noise filtering and signal processing.  

ICARUS – Daniele Gibin, Tracy Usher

ICARUS has successfully launched its first Monte Carlo Challenge (dubbed MCC1) this past month and is beginning to digest the output. This is the first production run by ICARUS that includes TPC, PMT and CRT simulation and reconstruction. The production running is still working with project.py but we are hoping to be converted to the full POMS system within a few weeks (initial tests start this week). The hope is to get converted to POMS, digest the current production running with the eye to understanding the primary issues and to work to get those fixed through the fall and early winter, then to ramp up for the next challenge in the early winter which will have the goal of being “physics capable”.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

MCC9 is currently in development, Large data and MC production planned to start around December of 2018, with smaller test sample production before then. There is no final decision yet whether MicroBooNE will switch to the new style LArG4 refactoring. Before that can happen, there has to be a way to simulate CRT aux detectors.  There is no MCC9 production branch yet, and we haven’t decided when or whether to create an MCC9 branch off of develop. A major breaking change, such as migration to art 3.x could trigger the creation of a production branch.  MicroBooNE would prefer that such a major breaking change not happen soon.

SBND – Roxanne Guenette, Andrzej Szelc

SBND has, as of last week implemented the Space Charge simulation (thanks Arbin Timilsina), first tests are ongoing. We plan to transition our production system to POMS, and launch a quick production in mid October to test that infrastructure. We are also looking into plugging in analytical method of predicting light signals, which could speed up the simulation and lower the need for lookup libraries.

Non-SBND: UK groups will run a LArSoft workshop in Manchester, 10/31-11/2, which is geared up to get people started in LArSoft and basic reconstruction. http://indico.hep.manchester.ac.uk/conferenceDisplay.py?confId=5372

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

Attendance: Tom Junk, Herb Greenlee, Erica Snider, Katherine Lato

I. LArSoft Status including open issues.

1. • LArG4 status –
     • The first stage of Phase I was deployed in v07_03_00. Phase 1 remains incomplete due to missing documentation and the AuxDet module.
       • The short term need for the AuxDet subsystem will depend on the state of on-going developments in MicroBooNE. If not needed, then the first stage will meet the needs of MicroBooNE for MCC9
       • We do not know how long it will take to produce the documentation given the absence of Hans Wenzel, who is away through mid-September.
     • Work on Phase 2 is proceeding — abstracting the anode region simulation and handling of “S2” light generated in the dual-phase electron extraction and amplification region.
       • The S2 light work is almost completed.
       • Erica Snider met with Christoph Alt, Bea Oregui and Paul Russo to refine the design and interfaces for the anode region abstraction. The new design will accommodate dual-phase, single-phase and pixel detector workflows as configurable elements without encumbering any detector with features or elements not essential for that detector. Some interface work has already been completed.
       • Noted that the on-going Wire-Cell based electron drift and signal induction models will need to be adapted in the move to Phase II. No change is needed for Phase I.

1. • DUNE timestamp – closed
   • GitHub and licensing. Have redmine issue for licensing (so that’s closed)–  https://cdcvs.fnal.gov/redmine/issues/20370 [After the meeting:  For GitHub, migrating repositories, what about wiki pages and issues? May want a division level solution on this, not just LArSoft, so management will raise this including having a presentation from CMS about how they use GitHub. Need to understand the high level workflow. This issue is beyond just LArSoft, so we won’t be tracking it here.[] Closed.
   • Large detector strategy – asking DUNE to start thinking about this. Have ideas for solution, mainly dealing with small parts of the detector at a time. LArSoft wants to be included in ongoing discussions about this by DUNE, or other experiments.
   • 3D pattern recognition – had meeting. Working on a google document to outline agenda for a meeting https://docs.google.com/document/d/1rDsoGQhJgSMhxwP2dH0U2rV5sCWFgU3hII9d-n4gMVE/edit?usp=sharing
   • S2 light simulation –
     • Paul Russo provided update at SciSoft meeting on Monday. Bea is working on some photon library issues (among them, a bug introduced by MicroBooNE). Christoph Alt also working on it. All immediate questions are answered, so just working on code.

II. Discussion of larsim to lardetsim sign-off – no issues raised at the meeting. If people don’t check in private code, they’ll have to run scripts.

III. We asked for comments on:

1. New policy for merging code & migration to pull-request system – Want to move to pull-requests as part of migrating to GitHub with the experiments being responsible for their own pull-requests. This provides administrative control when someone makes a pull-request. If the CI test fails, they have to deal with it before it is merged. Each experiment needs to have a set of people who handle the pull requests. Asking each experiment to discuss this and let us know if there are any objections.
2. Plan of each experiment for monitoring the CI system for error reports, and responding to those errors with corrective actions. DUNE has a person. MicroBooNE is thinking about it.

IV. Round Robin sharing, including input to 2019 priorities

1. ProtoDUNE may soon have stuff they need.
2. MicroBooNE is interested in maintaining stability to process large volumes of data without infrastructure or interface changes.
   • Question about multi-threading. Plan for 2018 is to make our services thread safe and demonstrate scaling in a test job. Mike Wang is working on this.
3. ICARUS
   • Priority for 2019 — The build system issues on OSX need to be solved. A critical issue for code development at ICARUS, and has caused several people to stop developing. In advance of a full Spack/SpackDev-based solution, strongly recommends that we address the problem with the install and cmake stages. If not possible, then we should raise the priority of Spack/SpackDev development.
   • A new event display is also a high priority

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The July 2018 LArSoft Offline Leads status update was handled via email and a google document.

LArSoft – Erica Snider

1. LArG4 re-factoring update — The current plan as of 7/20/18 is to stage the LArG4 re-factoring in two major steps, targeting the first to be released in a v07 release candidate during the week of July 30. This release will have all major features except for AuxDet support. The second major candidate release update will include support for AuxDet’s as well. The first v07 release will be cut after appropriate testing, and completion of necessary documentation on a schedule to be determined later. This will complete phase I of the LArG4 re-factoring project. Notably, the handling of S2 light, which is formally part of phase II, may be included in the first v07 release.
2. DUNE timestamp request from Tom Junk https://cdcvs.fnal.gov/redmine/issues/20160 – There were no objections in principle at the last Offline Leads meeting.  Tom Junk presented Timestamp Data Product Proposal at the July 3rd Coordination meeting. LArSoft Collaboration approved addition of new data product for experiment-defined time-stamps, Branch: lardataobj feature/trj_rawdata_timestamp_june2018
3. GitHub and licensing — discussed at the June Offline Leads meeting.
   • GitHub:  The project has started to investigate the cost / benefit of migrating to GitHub, and will decide whether to make a proposal in the coming months.
   • Licensing:  LArSoft will propose putting the core LArSoft code (everything in the lar* repositories) under the Apache License V2.
4. Strategy for dealing with large detectors – deferred last month since DUNE wasn’t at the meeting.
5. 3D pattern recognition.
   • LArSoft would like agreement on a common representation for pixel data versus reconstructed 3D space-points from wires. The project is organizing a meeting in early August with interested parties to discuss whether and how to move forward with collaborating on a common foundation for direct sharing of 3D algorithms between pixel and detectors with wire readouts.
6. S2 light simulation
   • The project met on June 19th with members of DUNE, Geant4 and LArSoft teams (Jose alfonso Soto, Christoph Alt, Michel Sorel,  Beatriz Tapia Oregui, Paul Russo, Hans-Joachim Wenzel and Erica Snider) to discuss how to deal with the S2 light generated in the gaseous volume in dual phase TPCs.
   • DUNE noted that although there is more than an order of magnitude more S2 light than S1 light, the modeling need not be highly detailed.
   • The generation of this light is entangled with the electron drift model. We agreed on a basic workflow design that accommodates SP and DP detectors, avoids the use case of single photon propagation for S2 light, provides for detector-specific anode simulations, and minimizes the amount of new code that needs to be written. We further agreed on a careful review of the necessary interfaces before finalizing the design. The timescale for this work is still under discussion, but has started.

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang

ProtoDUNE has switched to use tool-based dataprep code (issue #20303).

TrajCluster is being restructured to run 3D slicing first and then run cluster reconstruction on each slice. This helps improve the speed to reconstruct events with lots of cosmic ray muons (ProtoDUNE and MicroBooNE). There was a presentation at the July 31 LArsoft coordination meeting on this.

ICARUS – Daniele Gibin, (Tracy Usher)

In preparation for MCC1 which hopes to launch the first full week of August. Toward that end have integrated pandora, have generated a new photon library (with updated geometry), have included the CRT into the simulation, have developed a coherent noise model which should represent what has been observed with tests of the new (warm) electronics at CERN, etc. We seem to still be not fully operational with POMS and current plans are to use project.py for MCC1 unless a solution to what appear to be ICARUS only problems can be solved. MCC1 will consist not only of single particle samples similar to what has been generated previously but also aims to generate of order 100k BNB + cosmic events. This will probably tax our current resources (jobs, disk space, tape, etc.) but need to find out sooner than later!

Looking forward, the next big task for the geometry will be the splitting of the horizontal wires into two halves, we are not sure what impact this has on LArSoft but expect things to break when this happens.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

All updates from MicroBooNE’s MCC8 campaign have been merged to the develop branch (both larsoft proper as well as uboonecode).

The current focus of MicroBooNE’s development on the develop branch is integrating code for a future MCC9 production campaign.  This will most likely make use of the the “MicroBooNE LArG4 refactoring” (as opposed to the official larsoft LArG4 refactoring).  A major subfocus is integrating wire cell simulation, which replaces the electron drift model of the old LArG4 simulation. In principle, this impacts larsoft via the larwirecell package.  We know for sure that there will need to be a new version (v0_8) of the wirecell external product (current version v0_7_0a).

The monolithic uboonecode repository was recently split into ten repositories (the “uboone suite” now consists of 13 packages/repositories).  This impacts larsoft minimally only to the extent that uboonecode is included in larsoft test builds.

MicroBooNE is beginning to think about SLF7 migration.  We have a ticket open requesting SLF7 build and login gpvm nodes.

SBND – Roxanne Guenette, Andrzej Szelc

No Report

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

LArSoft Offline Leads Meeting – 6/27/18
Attendees: Erica Snider, Tracy Usher, Herb Greenlee, Andrzej Szelc, Katherine Lato

LArSoft Status

• LArG4 restructuring update –
  1. Work that remains in phase one is a translation from MCParticle into the inputs that Geant uses. We believe testing of phase 1 will occur in early July.
  2. There are things in phase two that are critical for DUNE and ICARUS, and may benefit everyone else who needs various detector-specific customizations while using common bits of everything else. We don’t have a timescale for phase two.

• Requested comments on:
  1. DUNE timestamp request – https://cdcvs.fnal.gov/redmine/issues/20160 there’s a feature in the protoDUNE DAQ that has some ambiguity about the time various blocks of raw digits are read out. They’re proposing a timestamp per channel. Questions:
     • How does that affect things written to disk already? It adds 64 bits per channel. Backward compatibility problem if you need the timestamp, but not if you don’t. No one except DUNE currently needs it.
     • Had hoped there would be a DUNE representative to help discuss this at the meeting, but there wasn’t, but experiments represented had no objections in principle.
  2. GitHub and licensing – Apache license. Can use the code however you want, can repackage it, but can’t take it out of this license. While not opposed to migrating to GitHub, SBND wanted to know why? GitHub has a richer set of collaboration tools than redmine. It enables pull requests. We’d probably start by mirroring from redmine and eventually switch to mirroring from github. People could work with either one.
     • How do you control authorization and access? Public means public for everyone to access, not everyone to commit code. We’d like to structure access with groups, like we have now.
     • No objections in principle. “All the cool kids are doing this.”
  3. Strategy for dealing with large detectors – deferred since DUNE wasn’t at the meeting.

Round Robin:

ICARUS

1. Discussion in Europe about 3D pattern recognition.
   • LArSoft would like agreement on common representation for pixel data versus reconstructed 3D space-points from wires. The idea of having a workshop around a DUNE collaboration meeting with participation from ALICE didn’t gain traction. Could start with a short video-call meeting to discuss if people are interested in collaborating and figure out the next step from there.
2. Pandora fixed their problem with handling horizontal wires. ICARUS has a lot of questions surrounding geometry, adding more stuff to the detector geometry model. Have to deal with split wires in the simulation and reconstruction, where the split does not provide a logical TPC boundary.
   • LArSoft suggested that the simulation case could be handled locally with a detector-specific anode region simulation, which becomes possible in phase 2 of the LArG4 re-factoring. This would mean that ICARUS won’t have to rely on geometry system to give all the answers. Going this route, however,  risks issues on the reconstruction side, where algorithms rely on answers from the geometry system.
   • Analogy with DUNE raised. But ICARUS has two channels servicing one wire. DUNE has the inverse problem, a single channel servicing two wires. ICARUS should let LArSoft know if they run into things that the geometry isn’t addressing.
3. Question: Where are things on Event Display?
   • Erica is hoping to devote time in early July to taking the first steps in the evaluation with a goal to get it done by the end of the summer.

SBND

Working on reconstruction chain.

Update on earlier items:

• effects of misalignment in APAs. They haven’t had time to finish the code. Misaligning the APA changes the electric field, most important for a surface detector.
• Ran into problems running the optical simulation. This is a fixable problem. SBND talked to Wes, and understand what to do.

MicroBooNE

• Regarding LArG4 refactorization, will the experiment be tied to the fast optical simulation?
  • Yes when run in a production mode (eg, with full TPC simulation). Doing otherwise would require going back into Geant4. There is no provision for that, even after the phase one release, though it is not in principle precluded.
  • Will need a dedicated job with a different workflow to generate a photon library.
• Cherenkov light will be an issue for MicroBooNE and ICARUS.
  • Believe Geant4 can handle this. The only question is whether downstream optical detector simulation code looks for the result.
  • We might also want a look-up library for Cherenkov light
• The warning message about deprecation should be fixed. As requested, MicroBooNE opened a ticket for this:. https://cdcvs.fnal.gov/redmine/issues/20233

July meeting is via a google doc.

Any questions or comments, please email Erica or Katherine

Erica & Katherine

The May 2018 Offline Leads status update was handled via email and a google document.

Note, item 4 under LArSoft requests a response from each experiment. Please email Katherine Lato and Erica Snider.

LArSoft – Erica Snider

1. Event display:  As reported at the last LArSoft Coordination Meeting, the initial set of requirements for a new common event display framework have been approved. The next step, to be completed before the summer, is to define evaluation criteria and identify potential technical solutions. The plan will be to attempt to perform the evaluation over the summer.
2. LArG4 re-factoring:  Initial testing of a fully re-factored LArG4 and Geant4 within a stand-alone testing environment has been completed. Further testing requires integrating the re-factored LArG4 into LArSoft. This work is now under way. Two of the three downstream modules needed to complete the re-factoring have also been completed, an electron drift module written by Bill Seligman, and a fast photon propagation module, written by Wes Ketchum. Some work may be required to ensure that the necessary workflows are easily adapted to the needs of all experiments. The next phase of this project will address light from the Large Electron Multiplier in dual-phase LArTPCs. A later phase will address the issue of integrating multiple external detector types into the simulation, as will be required for DUNE ND. We  note that the current re-factoring introduces some of the necessary concepts into the geometry.
3. Re-architecting services for thread safety:  While Mike Wang was pulled away until mid-July, Paul Russo may be able to start work on this once multi-threaded art is released which is expected soon.
4. Heidi Schellman has raised the question of the long term vision for LArSoft projecting out to the 2025 era. The project believes this would be a useful discussion, and invites suggestions from the offline leads as to how we might approach the question. One could imagine holding a half-day workshop, for instance, to discuss ideas and issues. Such a discussion is timely given the current work exploring how the HEP computing ecosystem will evolve driven primarily by forces from HL-LHC, along with the continuing evolution of computing architectures. 
5. Regarding the issue of can NumPy and SciPy have standard, LArSoft compatible distributions made available, Mu2e has written a request for this,  19944. In response, python_tools v1_02_00, which includes numpy v_14_3 and scipy v1_1_0, is available in the following builds:
   1. e15 for SLF6 and SLF7
   2. e17 for SLF6 and SLF7
   3. c2 for SLF6, SLF7, Sierra, and High Sierra

DUNE – Andrew John Norman, Heidi Schellman, Tingjun Yang

• We discovered a problem with the simulation of atmospheric neutrinos in the DUNE far detector. This was fixed by Robert Hatcher in nutools v2_21_04 which is incorporated into larsoft v06_78_00. We still need to update the atmospheric neutrino flux driver to account for the orientation of the DUNE far detector. This is work in progress: 20034.
• We are testing a new mode in PMA track reconstruction using the newly available SpacePointSolver. This mode improves the speed of PMA reconstruction by a factor of 2 for ProtoDUNE. The initial test looks promising. More details in 19937.

ICARUS – Daniele Gibin

• No Report

LArIAT – Jonathan Asaadi

• No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

• A partial and somewhat MicroBooNE-specific, but backward compatible, LArG4 refactoring was included in larsoft v06_76_00.  We expect that this refactoring will be superseded by the eventual full refactoring mentioned by Erica above. The important thing is to maintain compatibility with the newly introduced SimEnergyDeposit data product.  For MCC9, the modeling of electron drift, including the effect of induced charge, will be handled outside of larsoft, by the wire cell package.
• The MicroBooNE offline coordinators are not many getting requests for supporting external products.  We know that some people are using numpy and scipy, so MicroBooNE would be supportive of including these in the standard larsoft distribution.  We believe that people who currently use these have figured out how to obtain them outside of larsoft.
• MicroBooNE also has a python/gallery event display that some people use, which is dependent on the external products Qt and PyQt to provide the gui.  The MicroBooNE collaborator who has been providing unofficial support for these external products has stopped doing that, so it will be difficult for users to keep using this event display.

SBND – Roxanne Guenette, Andrzej Szelc

• We have not yet updated the simulations wrt to the timing simulation problem reported two months ago, as we’re doing offline studies to generally improve it, and once that is done – hopefully in a few weeks we can start to implement the fixed version all together. We have been contacted by Wes about refactorization and its connection to light simulation but have not been able to meet yet.
• In parallel, we have been working on studies that look at effects of misalignment in APAs. Currently this is tailored for SBND geometry as is, and is somewhat hacky, but I have talked to Tingjun who expressed interest on behalf of DUNE in this – we will try to make it more generic and applicable to DUNE soon.
• Ad. NumPy/SciPy  – haven’t had explicit requests for this, but have seen persons using them. So have nothing against it, but not really required yet.
• Ad. Event display, we have so far been using the standard event display with its standard woes. Speeding it up would be extremely welcome.

Open issues:

1. Restructuring of LArG4 – Hans Wenzel and William Seligman continue to work on this. It is being tracked in https://cdcvs.fnal.gov/redmine/issues/14454 We are reporting on this every month and also regularly at LArSoft Coordination meetings.
2. Can NumPy and SciPy have standard, LArSoft compatible distributions made available? See discussion in LArSoft section, and issue 19944. This issue is now closed.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

April 18, 2018 Offline Leads Meeting Notes

Attendees: Herbert Greenlee, Erica Snider,  Saba Sehrish, Andrezej Szelc, Tracy Usher

A) Status of LArG4 refactoring:  The work is largely complete, except for two modules that have not yet been written:  the fast photon propagation / optical digitization module, and the response and digitization module for AuxDet detector elements (outside the TPC). The latter module is experiment / detector dependent, so needs to be provided by the experiments by re-factoring existing detsim modules into the new structure.

LArSoft realizes that refactored LArG4 is critical for MCC9. Since there have been issues with meeting deadlines in this work, we propose that we integrate Wes’s changes for the time being, and replace it with the factorized code once it is ready. Wes will present his changes at a LArSoft coordination meeting. These code bases can co-exist.

B) LArSoft v07 release candidates

There are three release candidates coming up for the LArSoft v07, listed in priority order:
— Geant4 update, a patch release to address the physics bug reported
— LArG4 refactoring
— _art_2.11 migration that requires migration to C++17

Herbert Greenlee mentioned they are already dealing with some of the issues with C++ 17 migration because of the new c2 (clang-based) builds, but moving to C++17 is otherwise not a high priority.

C) Code Integration Model:
— Herbert Greenlee mentioned a concern regarding having to rebuild the products whenever there is a change in LArG4, Geant. The LArSoft proposal is to decouple or abstract away from the Geant4 and Genie products so that experiments can choose what version they need. To achieve that, LArSoft will have to not depend on Geant4. There will need to be  a new interface product and LArSoft will need to provide a module that will interact with Geant4 output data products.
— More generally, we will likely need to discuss how to deal with different versions of reconstruction among the experiments. How can we continue to develop shared code if each experiment needs to create their own forks? We need a consensus answer to this question.
— Comments / feedback about the need of a new code integration model should be send to the larsoft-team.

D) Event Display: 
Division agrees it is a high priority to get a better event display. We have developed a set of draft requirements based on input from a number of event display users and developers, and are proceeding with the following plan:
— upgrade the requirements with the feedback we have received so far
— conduct a survey to identify some technical solutions,
— identify some core requirements for evaluation
— implement the core requirements in each solution.  A summer student has expressed interest in pursuing this project. If we don’t get that, we can ask SCD for more help.
— From Heidi:  all the effort should go to the new display but the new project will not be a fast process
— Push from ICARUS to make improvements in the display

E) Building with clang:

Building with clang on the Mac is slow. We have prelim results that suggest that re-enabling SIP improves the build time by 10 times. The install is also slow. This is caused by a cmake issue that results in all libraries being re-installed, regardless of whether changes were made to them. The problem is that cmake strips rpaths from libraries, then interprets the result as a changed library, even if the library was not rebuilt.

F) Multi-threading
We are starting a project to re-architect the way that some services work in order to be thread safe. This is only  necessary for services that change state on an event by event basis, such as those that deal with detector conditional, etc. None of those are currently thread safe.

We are also working to vectorize code.

A separate problem to deal with is managing how we match vectorized libraries to the hardware in an environment where platform capabilities are not uniform (e.g., for grid processing, or cloud processing with GPUs, etc.)

G) Transitioning Gianluca’s work:

Gianluca will be leaving on May 4th.

–thanks to Saba Sehrish for taking notes during the meeting.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.

The March 2018 Offline Leads status update was handled via email and a google document. 

LArSoft – Erica Snider

The code integration model for LArSoft will become more complicated as more experiments approach data taking. Already an issue for MicroBooNE / Pandora. We will continue to explore this issue with the experiments over the coming weeks.

Have collected input on requirements for the LArSoft event display. This is the first step of investigating a long-term solution for a common LArSoft event display framework to replace the existing display.  The plan for this investigation is documented and tracked in LArSoft issue 19034, and is summarized below:

1. Develop a set of requirements, ranked according to the following categories:  essential, strongly-recommended, desired, and possibly-useful.
2. Get community input on the ranking and final list of requirements at two LArSoft Coordination meetings to ensure that people can give opinions.
3. Establish criteria to judge between the technical options, and review these with the community.
4. Identify a list of possible technologies such as Root, Venu, ParaView, QT and other potential technologies and how applicable they are.
5. Compare technologies. Ideally, write a toy display that demonstrates the requirements, or sufficient subset of critical requirements. Code it up on each platform
6. Evaluate based on established criteria in task 19037
7. Take the results to SCD management for review, along with an initial deployment proposal and a request for funding.

Gianluca is leaving soon. Working on replacement issues.

DUNE – Andrew John Norman, Heidi Schellman

We’re still gearing up for DC2 in April and ProtoDUNE data in the Fall.  One question relevant to the event display, since it is root based, is whether we could take advantage of the ROOT web interfaces instead of the xvnc to view.  https://swanserver.web.cern.ch/swanserver/rootdemo/  and https://root.cern/js/  describe the interface.  

ICARUS – Daniele Gibin

Progress is continuing with development of simulation and reconstruction in the LArSoft framework. ICARUS now has a light simulation and is able to produce optical hits. The full reconstruction chain has been implemented using TrajCluster, we are not yet able to use Pandora due to an artificial requirement that one of the wire planes in the TPC must have vertical wires (ICARUS has horizontal wires). The Pandorans are aiming to fix this soon.

The LArG4 refactoring by Wes Ketchum is needed/desired before making further improvements to the ICARUS detector simulation.

Due to the LArSoft event display being broken with the introduction of the new version of art in LarSoft v06_70_00, ICARUS is currently basing its work on LArSoft v06_69_01.

LArIAT – Jonathan Asaadi

No Report

MicroBooNE – Herbert Greenlee, Tracy Usher

MCC8 is in the analysis phase with final tweaks for special data sets.

Planning for MCC9 is essentially complete with an aggressive timeline to achieve improvements aimed at results for Summer 2019.

In both cases the version of LArG4 refactoring provided by Wes Ketchum will be necessary, it would be good to include this code on the LArSoft development branch as soon as practicable.

SBND – Roxanne Guenette, Andrzej Szelc

We have run into problems running the optical simulation. Up to now we have been using simphotons, but this has turned out to cause problems in the newest software release version because of memory. LitePhotons were used instead, but most of our infrastructure for timing and reflected light does not work with those.

We are now trying to figure out how to solve this problem. Potentially Wes’s refactorization may help, although we need to look closer into that.

We have first results of matching TPC tracks with Cosmic Ray Tracker ones, which will be useful in future analyses.

We have setup File transfer and SAM definitions (thanks to help from SCD) and are beginning to use that in running our jobs.

Open issues:

1. Restructuring of LArG4 – Hans Wenzel and William Seligman continue to work on this. https://cdcvs.fnal.gov/redmine/issues/14454
2. Can NumPy and SciPy have standard, LArSoft compatible distributions made available?
3. Selective loading of detector data. For DUNE FD want a functionality to load into memory data products only from selected part of the detector, e.g. selected TPCs; this may be a feature request to art however. Could be a usage pattern in LArSoft, or could be in art. After a meeting with Kyle in February, concluded that to deal with reading in part of the detector at one time we need to create data product instances that are distinguished via their tags, and hide the complexity from users. We’re tracking this in: https://cdcvs.fnal.gov/redmine/issues/19286

Offline Leads meeting on February 14, 2018

Attendees: Heidi Schellman, Tracy Usher, Erica Snider, Katherine Lato

A) Discussed Supporting LArSoft on Mac OSX via Docker  

1. There was a demo at the 1/30/18 Coordination Meeting of using Docker along with other VM technologies to build LArSoft code under SLF6 under Mac OSX. Members of DUNE have proposed this as a way to support LArSoft development under Mac OSX.
2. Discussed how to proceed. All agree on the importance of supporting development on the Mac. Should we continue investigating this with possible implementation?
   • Q:  What are the problems with cvmfs?
     • A:  There can be huge latencies while waiting for things to cache, even with simple commands. Tuning the cache might alleviate some of this, but if there are frequent release updates (which there are), then it might still be disruptively slow (since pre-caching is not an option). Smaller repositories would help too, particularly for DUNE, but that is not a general solution.
   • Q: do you use local installs or CVMFS for developing on Mac
     • A:  Most seem to do local installs. Debugging capabilities are more difficult if not built locally.
   • Q: How often do you update the release you’re developing against?
     • A:  People seem to do this often. If the code isn’t stale, there’s less pain with merging in changes. Especially when things are complicated. When updating weekly, it just takes 20 minutes to build in the background while you’re doing something else. This way, it does not interfere with development work, which is in contrast to cvmfs cache updates, which occur on demand during development work.
   • All agreed that one of the major concerns for those developing on Mac OSX is access to Mac-based IDEs and other development tools. Docker interferes with xcode, for instance, so that could not be used. The Docker proponents have been asked to investigate this issue. Even if they can be run, they will be less capable than with closer-to-native builds under Mac OSX. Native clang builds would be best. Erica and Tracy both think that supporting native Clang would be nice. Ensuring that the product stack was consistently built is an issue, though more so for production releases than for development. Native clang support is not a priority at the moment.
   • Encouraging native Mac development as a whole is a good thing. Things that run in the production environment need certainty. Things like the event display should be run locally.
3. Consensus from DUNE and MicroBooNE:  Docker could be used to run LArSoft in non-standard environments, or to gain access to resources that are not otherwise available. It should not, however, be used as a way to support development on Mac OSX.

B) ProtoDUNE support. LArSoft wants to ensure that we are doing what is needed in the short term and consistent with long-term plans. LArSoft has changed the geometry to support different drift directions and changes to Global Wire Coordinates, other changes had to be on the protoDUNE side. With Robert Sulej’s departure, aren’t sure if things are acceptable or if people are complaining about things needed in LArSoft.

1. Heidi- we ran on MonteCarlo and people were pretty happy. First actual data has come out of the cold box. At the next collaboration meeting in two months, there will be more discussion.
2. LArSoft encourages people to talk to Erica, Gianluca, or put in a ticket if there are any issues or potential issues. We want to keep the engagement and make sure people have a positive experience with LArSoft.
3. Heidi thought that giving experiment service credit for working on LArSoft items was mentioned as a good idea.

C) LArG4 update. Met with Hans last week. There are two big pieces–changes to Geant4 side and on the experiment side. Changes to Geant4 side mean that LArG4 doesn’t need to have a copy of the Geant4 code that was made several years ago. Get away from having a custom physics list in order to take advantage of those in Geant4. Tracy recently talked to Bill Seligman, who is working on the experiment-side component, and Hans, who is doing the Geant4 work. Bill is waiting on the Geant changes, but is otherwise ready to test his code. The re-factoring is blocking Wirecell testing. MicroBooNE may need a Plan B.

D) Update on 3D reconstruction workshop.

1. 3D pattern recognition workshop – Proposed scope will cover all existing work to produce 3D space points, plus algorithms to perform pattern recognition using space points as input. Will also work toward agreements on data products, interfaces between various workflow stages so that we can continue to share algorithms.
2. Alan Bross (who is part of Argon Cube) is interested in having the ALICE  pixel people there. The Alice people won’t be in the US, however, so want it to be at CERN. The US people want it to be here.
3. At a minimum, want all parties to agree on the interface layer between the low-level 3D data and the 3D pattern recognition. This preserves the possibility of sharing downstream pattern recognition code.  The plane readout people have to get to this point by merging 2D data. The pixel people start from this point. The common part of the workshop is then talking about the downstream piece.
4. Maybe have a joint early morning meeting in the US with remote CERN participation, then have the rest of the workshop in the US without them?  
   • Will pitch this.

E) Profiling – LArSoft team continued examining profiling results for ProtoDUNE, 35t and FD production workflow chains compiled by Soon Jun Yun. Asking DUNE to provide a person to review the results in more detail with LArSoft team and identify actionable bottlenecks. Sent email to Heidi and Tom.

F) Round Robin sharing.

• Talked about Event Display requirements. Tracy thinks Cory Adam’s model is good, but he has hard-coded in geometries.
• Fyi – new Macs may not be Intel based. May be issues with that.

Please email Katherine Lato or Erica Snider for any corrections or additions to these notes.