Pipeline steps

This page describes what each step of the Simflow pipeline does and how it uses the metadata and configuration described in Data names and paths.

vtx — event vertex generation

Todo

Document the vertex generation step.

stp — particle simulation

Todo

Document the remage simulation step.

par — parameter extraction

The par step extracts per-detector observable models for each data taking run before the hit tier is built. The results are written to per-run YAML files on disk so that build_tier_hit does not need to load the (potentially large) data production parameter database at processing time.

HPGe energy resolution (extract_hpge_observables_models)

The extract_hpge_observables_models rule produces a per-detector YAML file mapping each HPGe detector to its energy resolution model. It is a collection step: it gathers what it can from the available sources and writes the result to disk. Completeness is validated downstream in build_tier_hit (see HPGe observable validation (build_tier_hit)).

The exact sources used depend on what is configured:

  1. l200data only, no HPGe-specific overrides — everything available in l200data is collected, independent of detector status.

  2. l200data + HPGe-specific overrides in Energy resolution model defaults, no default key — everything available in l200data is collected as in case 1; the explicitly listed detectors are then overridden.

  3. l200data + HPGe-specific overrides with a default key — the metadata takes over entirely: every HPGe detector in the channel map is expanded from the default (with optional per-detector overrides). l200data is not consulted. All detectors, including non-ON ones, are present in the output.

  4. HPGe-specific overrides with a default key, no l200data — same as case 3. l200data is not required.

HPGe A/E resolution (extract_hpge_observables_models)

The same rule produces a per-detector YAML file for the A/E resolution model, following the same four-case logic as HPGe energy resolution (extract_hpge_observables_models). The metadata directory is described in A/E resolution model defaults.

HPGe PSD cuts (extract_hpge_observables_models)

The same rule produces a per-detector YAML file for the PSD cut values, following the same four-case logic as HPGe energy resolution (extract_hpge_observables_models). The metadata directory is described in PSD cut defaults.

HPGe current pulse model (extract_current_pulse_model)

The extract_current_pulse_model rule produces one YAML file per (runid, hpge_detector) pair holding the current pulse model parameters for that detector in that run. Unlike the eresmod/aoeresmod/psdcuts extractions (which are per-run), this step has per-detector granularity. It follows the same four-case logic as HPGe energy resolution (extract_hpge_observables_models), but cases 3 and 4 bypass waveform fitting entirely and do not need l200data. The metadata directory is described in Current pulse model defaults.

Manual HPGe skip-list

In addition to the automatic exclusions performed by gen_list_of_hpges_valid_for_modeling (usability on + valid crystal metadata required), individual detectors can be removed from the modelable HPGe list for a given set of runs via the Manual HPGe skip-list directory. A WARNING is emitted for each skipped detector, including the reason string and the run identifier.

opt — optical hit building

Todo

Document the optical hit building step.

hit — hit tier building

HPGe observable validation (build_tier_hit)

build_tier_hit reads the per-detector YAML files produced in the par step and validates that every simulated detector has the parameters it needs. The hard-error vs. fallback policy differs slightly per observable:

Observable

Hard error

Fallback (+ warning)

Fallback key

Energy resolution

ON detector missing entry

off/ac detector

eresmod_default

A/E resolution

ON detector with psd_usability ≠ "missing" and PSD modelable (dtmap + currmod present) missing entry

all other cases

aoeresmod_default

PSD cuts

ON detector with psd_usability ≠ "missing" and PSD modelable (dtmap + currmod present) missing entry

all other cases

psdcuts_default

The fallback keys are read from hit tier. They are never triggered when a default key is present in the corresponding metadata (cases 3 and 4 of the extraction steps above), because all detectors are already covered in that case.

Note

An ON detector with psd_usability = "missing" explicitly signals that PSD data are unavailable for that detector (e.g. a known hardware issue). Similarly, a detector missing a drift-time map or current pulse model cannot have its PSD response simulated at all — the PSD output columns are filled with NaN in that case. Both situations fall back to the default A/E resolution and PSD cuts rather than raising a hard error. A detector listed in the Manual HPGe skip-list directory is treated identically: it has no drift-time map or current pulse model and therefore follows the same NaN PSD output / fallback A/E resolution and PSD cuts path, with no hard error.

evt — event building

Todo

Document the event building step.

simprod/config/tier/evt/{experiment}/settings.yaml — evt tier settings

A static YAML file with evt-tier-specific settings that apply to all simulations for a given experiment configuration.

simprod/config/tier/evt/{experiment}/settings.yaml
add_random_coincidences: false
geds_energy_thr_kev: 25
spms_energy_thr_pe: 0
buffer_len: "50*MB"
skip_opt: false
skip_hit: false

  • add_random_coincidences (bool) — when true, random-coincidence (RC) SiPM data is mixed in during event building.

  • geds_energy_thr_kev (int) — HPGe hit energy threshold in keV; hits below this value are discarded.

  • spms_energy_thr_pe (int) — SiPM hit threshold in photoelectrons; hits below this value are discarded.

  • buffer_len (str) — LH5 read chunk size (e.g. "50*MB"). Controls memory usage during processing; does not affect the output.

  • skip_opt (bool, default false) — when true, the opt (SiPM/LAr) tier is skipped: the opt Snakemake rule is not run and the evt output contains only HPGe data (no spms or coincident/spms tables).

  • skip_hit (bool, default false) — when true, the hit (HPGe) tier is skipped: the hit Snakemake rule is not run and the evt output contains only SiPM data (no geds or coincident/geds tables).

Note

Setting both skip_opt and skip_hit to true simultaneously is an error.

cvt — event concatenation

Todo

Document the event concatenation step.

pdf — PDF generation

Todo

Document the PDF generation step.