The Experiment Controls and Data Curation group (ECDC) and the Scientific Information Management Systems group (SIMS), along with their colleagues at the Data Management and Software Centre (DMSC), have demonstrated the full ESS experiment pipeline.
The ESS architecture for experiment control, data acquisition, and data curation requires orchestration of many software components. "Project CODA" (Continuous Data) has recently been completed by ECDC at ESS, testing the whole data pipeline for multiple ESS instruments. Using a mixture of real data and previously recorded experiment data, continuous data collection has been run on ODIN, NMX, BIFROST, and the Test Beamline.
The intention is to keep Project CODA running in preparation for commissioning activities and the user programme.
The ESS data pipeline explained
The pipeline starts with the User at a NICOS workstation for an instrument, controlling the experiment. Event Formation Units receive detector data while data from instrument devices is obtained via EPICS. All information is streamed to Kafka which acts as a data aggregator.
The experiment data is collected by the Filewriter which writes it into a self-contained Nexus file. The file is stored on a high-performance network storage system in Lund before being copied to the DMSC for permanent storage. The data files are ingested into the SciCat data catalogue alongside the experiment metadata.