Skip to main content

What are Data Loaders?

A lot of the data a trading business relies on arrives as files and API answers: a broker's end of day settlements as a CSV file, a wind farm's readings from a JSON API, an exchange's settlement workbook, a terminal's send-out as an XML document. Getting each one into the platform used to mean writing an ODSL script by hand. Data Loaders lets you build that loader step by step, on screen, and answers questions like:

  • Can I load this broker file every evening without writing a script?
  • What is actually in the file, and which columns are numbers, dates or text?
  • How do I turn its columns into the fields of an event type, with the right date format and decimal comma?
  • Which rows should be left out, and what happens to a row that cannot be read?
  • Where do the events go, and how do I stop a rerun from loading them twice?
  • Did last night's run work, and how many events did it load?

The Integration extension sends data out to downstream systems; Data Loaders is the other direction, bringing data in as events.

Throughout these pages we follow Northgate Energy, a fictional gas and power trader. Every evening its two brokers send a semicolon separated file of end of day gas settlements for TTF, THE, NBP and PEG. Priya Shah, a market data analyst, builds a data loader for it that saves the settlements as BrokerSettlement events in NGE.BROKERS:SETTLEMENTS, every weekday at 18:30.

The Data Loaders screen with four loaders, their source, event type, event list, schedule and last run

Key ideas​

IdeaWhat it means
Data loaderA design that says where the data comes from, how its rows become events and where the events go. It has a name and a code, such as BROKER_GAS_SETTLEMENTS.
SourceA CSV or text file, a JSON API, an XML document, an Excel workbook or a table on a web page, read from a URL, or your own ODSL code. The URL can contain the date being loaded, variables and secrets.
SampleThe first 50 rows of the source, read by the platform exactly as a run reads them. You design the loader with it.
Event typeThe structure of the events, like a type for master data. Its properties are the fields you map the columns to. You can choose one, define a new one from the columns, or use none.
MappingWhich column, fixed value or template fills each field, with its data type, date format, decimal separator, empty values and value map.
Event idBuilt from the event time and the columns that make a row unique, so loading a row again replaces its event instead of adding a copy.
DestinationThe master data record and the event list on it, for example NGE.BROKERS:SETTLEMENTS, and an optional dataset id (_dsid).
Curves and seriesEvent timeseries, event curves and event options made once from the events, kept on the same master data record and built by the platform whenever they are read.
DeployingSaves a workflow made of the standard ETL actions and a process that runs it, which you can run now or on a schedule.

What you get​

ScreenWhat it is for
Operations, Data LoadersThe list of data loaders with their schedule and last run, and the step by step editor: Extract, Examine, Event type, Map, Destination, Deploy and run, and the optional Curves and series step.
Insights tabThe Data Loader Runs insight: every run in a period, by loader and by day, with the failures.
Workflows (Workflow Manager)Each deployed loader's workflow, etl_<code>, shown as a diagram.
Home page tasksLoad data from a file or API, Check data loaders that failed (with a badge for open alerts) and Review data loader runs.

Where next​