Skip to main content

Reference


Where everything is kept​

WhatService and sourceId
A loader's design (with its sample)object, custom collection etl_loadersthe code, e.g. BROKER_GAS_SETTLEMENTS
Workflowworkflow, privateetl_<code>
ProcessprocessETL_<CODE> (service ETL)
Run report configurationreportconfig, privateETL_<CODE> (Fixed, category ETL)
Run reportsreport, privateETL_<CODE>:<date>
Custom extract actionaction, privateetl_extract_<code>
Standard actionsaction, privateetl_extract, etl_transform, etl_load, etl_report
Preview scriptscript, privateetl\sampler, made on first use
Secret valuessecretetl_<code>_<name> by default
Event timeseries, curves and optionsdata, private (on the record)RECORD:ID, e.g. NGE.BROKERS:SETTLE_TTF_FRONT
EventseventRECORD:LIST:EVENTID
Alertsalertkey etl/<code>, category etl

The settings the process passes to the workflow​

The process input is loader (and ondate for a run for a date). The broker settlements loader, shortened:

{
"code": "BROKER_GAS_SETTLEMENTS",
"name": "Broker gas settlements",
"source": {"kind": "csv", "url": "https://files.northgate-energy.example/brokers/settlements/{date:yyyyMMdd}.csv",
"separator": ";", "headerpos": "1", "datapos": "0",
"dates": [{"token": "{date:yyyyMMdd}", "which": "date", "format": "yyyyMMdd"}]},
"columns": ["TradeDate", "Hub", "Product", "Contract", "Bid", "Ask", "Settle", "VolumeMWh", "Currency", "Broker"],
"filters": [{"column": "Hub", "op": "notequals", "value": "TOTAL"}],
"mapping": [
{"target": "eventtime", "kind": "column", "column": "TradeDate", "dataType": "date", "format": "dd/MM/yyyy", "required": true},
{"target": "contract", "kind": "column", "column": "Contract", "dataType": "text", "values": [{"from": "Dec-26", "to": "2026M12"}]},
{"target": "settle", "kind": "column", "column": "Settle", "dataType": "number", "decimal": "comma", "nulls": "n/a"}
],
"event": {"type": "BrokerSettlement", "timezone": "UTC"},
"eventid": "{eventtime}_{Hub}_{Contract}_{Broker}",
"dsid": "NGE.BROKER.{Hub}",
"target": {"object": "NGE.BROKERS", "list": "SETTLEMENTS"},
"alerts": {"raise": true, "severity": "high"},
"failOnProblems": false
}
FieldNotes
sourcekind csv, json, xml, excel or odsl; url; csv separator (a regular expression for comma, semicolon, tab, pipe or spaces), headerpos, datapos; json and xml path; csv, json, xml and html headers; excel sheet, headerRow; html selector, headerRow; dates, the URL's placeholders.
source.variablesnames and sets (a list of values for each set, as text).
source.secretsname and secret (the secret record). Never the value.
columnsEvery column the mapping, conditions and templates use.
filterscolumn, op (notempty, empty, equals, notequals, contains, notcontains), value.
mappingtarget, kind (column, constant, template, ondate), column, value, dataType (text, number, integer, boolean, date, datetime), format, decimal, scale, nulls, required, values.
eventid, dsidTemplates with column names in braces; the event id also takes {eventtime} (yyyyMMddHHmmss), {date} and {row}.

Run report fields​

Each stored report has loader, name, target, last (the latest run) and runs. A run has key (its time, HHmmss), status, started, finished, ondate, url, extracted, events, saved, skipped, problemCount, problems, stage (extract, transform or load) and message.


Extension contents​

KindId
Extensionodsl.etl (Data Loaders)
ViewOperations, Data Loaders (feature odsl.etl.manage, policy extension.odsl.etl)
Insightodsl.etl.runs (Data Loader Runs)
Actionsetl_extract, etl_transform, etl_load, etl_report
Home page tasksodsl.etl.new-loader, odsl.etl.check-loaders, odsl.etl.runs

Troubleshooting​

What you seeWhat to do
The ETL actions ... are not in this environmentInstall the extension again, or run its upload-actions.odsl script in the environment.
Could not read ... FileNotFoundExceptionNo file at that URL for that date: check the date placeholders and the Date to load.
The source has no rowsCheck the separator, header line, records path or sheet.
A column is text when it should be a numberSet the decimal separator, and list texts such as n/a as empty values.
is not a date in the formatChange the field's date format; for 2026-11-06T17:30:00Z use ISO.
ids are made by more than one sample rowAdd columns to the event id, or the last of those rows replaces the others.
The secret ... could not be readSet its value on the Extract step (Set the value), or check the secret record name.
uses ..., which is not a variable or a secretAdd the variable or secret, or correct the name in the URL or headers.
No table rows found at ...The page has changed: choose the table again.
This platform does not have the ETL service (etl/v1) deployed yetThe platform version in use has no etl service route, so the page cannot be shown. Type the table selector, or ask your administrator to update the platform.
changed in the listThe loader was changed after it was deployed: select Deploy again.