Skip to main content

Datasets and data quality


Dataset​

A dataset is a set of data you expect to arrive regularly, for example one exchange product's settlement prices. The platform monitors each day's delivery against it. Service: Dataset (dataset/v1/info).

FieldTypeRequiredWhat it is
_idStringYesprovider.feed.product, for example ICE.NDEX.TFM
_typeStringNoDataset
dsidStringNoThe same as _id
nameStringNoThe name
provider, feed, productStringNoThe parts of the id
sourceStringNoprivate, common or public
qualityGroupStringNoThe quality group whose checks run on each delivery
expectedObjectNoHow many tenors of each kind to expect: numbers by period type, and * for the total
checksArrayNoQuality checks: each with type (function or group), name, and expression (for a function) or group
expectedChecksArrayNoCompleteness checks, in the same form as checks
{
"_id": "ICE.NDEX.TFM",
"_type": "Dataset",
"dsid": "ICE.NDEX.TFM",
"provider": "ICE",
"feed": "NDEX",
"product": "TFM",
"name": "ICE Endex Dutch TTF Gas Futures",
"qualityGroup": "default",
"expected": {"*": 82, "Month": 59, "Quarter": 19, "Year": 4},
"source": "private"
}

Dataset feed​

A feed groups the datasets of one provider that arrive together. Service: dataset/v1/feed.

FieldTypeWhat it is
_idStringprovider.feed, for example ICE.NDEX
_typeStringDatasetFeed
name, provider, feed, sourceStringAs for a dataset
calendarStringThe holiday calendar: the days the feed is delivered
timeStringWhen the feed should arrive, HH:mm and a timezone code, for example 18:30 EU1
lateStringWhen it is late, in the same form
smartFeedBooleanTrue to load the feed with the smart loader
loadersArrayFor a smart feed: each loader's type (standard, auto or manual), id, process, calendar, start and retryStrategy (strategy and retryMinutes)
{
"_id": "NGE.ETRM",
"_type": "DatasetFeed",
"provider": "NGE",
"feed": "ETRM",
"name": "Northgate ETRM end of day",
"calendar": "BUSINESS",
"source": "private",
"time": "19:00 EU1",
"late": "21:00 EU1"
}

Dataset delivery​

The record of a dataset's delivery for one date, written by the platform as data arrives. Services: Dataset Delivery (dataset_delivery/v1/{source}) and dataset/v1/delivery.

FieldTypeWhat it is
_idStringdatasetid:date, for example ICE.NDEX.TFM:2026-11-18
dsidStringThe dataset id
ondateDateThe delivery date
statusStringwaiting, active, partial, loaded, late, missing or holiday
statusinfoStringMore about the status
completeBooleanTrue when everything expected has arrived
score, scoreinfo4 no issues, 3 late by under an hour, 2 late by under four hours, 1 later, 0 holiday
expected, actualObjectThe tenors expected and delivered
timingsObjectexpected, late, loaded and late_delta (minutes late)
deliveriesObjectEach delivery: id, timestamp, reason, origin, tenors
checks, expectedChecksObjectThe results of the quality and completeness checks
timelineString[]Everything that happened, one message per line: time, level, origin, phase and message
issuesString[]What stops the delivery being complete
initialised, timestamp, loadedTimeUTCDateTimeWhen the record was started, last changed and when the data was loaded

See Dataset delivery configuration for each part in detail and a full example.


Correction​

A correction is a value that changed in a later delivery for the same date. Service: Correction (correction/v1/{source}), read-only.

FieldTypeWhat it is
dsidStringThe dataset id
ondateDateThe delivery date
propertyStringThe property that changed
tenorStringThe tenor that changed, for curve data, as a period code
previous, newAnyThe value before and after
lastDateTimeWhen the previous value was delivered
timestampDateTimeWhen the correction was delivered
originStringWhere the correction came from
invocationIdStringThe request that delivered it

Validation configuration​

The checks to run on one curve or timeseries. Service: Validation Config (validationconfig/v1), versioned. The _id is the data id.

FieldTypeRequiredWhat it is
_idStringYesThe data id, for example NGE.GAS.TTF:SETTLE
kindStringYescurve or timeseries
name, description, groupStringNoDescriptive fields
tagsString[]NoTags
enabledBooleanNoFalse to stop the checks
runOnUpdateBooleanNoRun the checks when the data is updated
scriptsString[]NoScripts whose functions the checks use
checksArrayNoUp to 50 checks: id, name, expression (one line of ODSL), scope (tenor, curve or point), severity (warning, failed or severe), enabled
optionsObjectNolookback, curveHistory, writeStatus, maxPoints
{
"_id": "NGE.GAS.TTF:SETTLE",
"kind": "curve",
"name": "TTF settlement",
"group": "Gas",
"enabled": true,
"runOnUpdate": true,
"scripts": ["#ValidationLibrary"],
"checks": [
{"id": "missing", "name": "Missing", "expression": "checkMissing()", "scope": "tenor", "severity": "failed", "enabled": true},
{"id": "gaps", "name": "Gaps", "expression": "checkCurveGaps()", "scope": "curve", "severity": "failed", "enabled": true}
],
"options": {"lookback": 30, "curveHistory": 1, "writeStatus": true, "maxPoints": 250}
}

See A validation configuration for each option.

Validation record​

The result of the checks for one data point, written by the platform. Service: Validation (validation/v1). The main fields:

FieldWhat it is
_idDATA_ID:TIME for a timeseries, DATA_ID:ONDATE for a curve
target, name, kind, group, tagsThe data item and its configuration
point, pointDate, valueThe data point checked
statusvalid, warning, failed, severe, error, skipped or overridden
checksThe result of each check
tenors, failedTenors, warningTenors, tenorCountsFor curves: the result for each contract
overridereason, user, timestamp and the status overridden
timelineThe last 100 events

See A validation record for every field.


Alert​

An alert is a problem with your data that stays open until someone closes it. Service: Alert (alert/v1/{source}). The main fields:

FieldTypeWhat it is
_idStringALR- and the year and a number, for example ALR-2026-000401; ALC- for common and ALP- for public alerts
title, message, descriptionStringWhat is wrong
categoryStringA grouping, such as dataset, curve or system
issueStringdelivery, completeness, quality, metadata, corrections, build, export or another
severityStringcritical, high, medium or low
statusStringopen or closed
agentStringWho has it: client, opendatadsl or a data vendor
subjectObjectWhat it is about: service, id and ondate
keyStringThe id of the problem, used to count repeats
occurrences, lastOccurredHow often and when it was last raised
assignee, resolution, tagsWho is working on it, how it was resolved, your labels
created, updated, closedObjectat and by
timelineArrayEverything that happened: each entry has id, type, at and by
_versionIntegerSend it back with a change: the change is refused if the alert changed meanwhile
{
"_id": "ALR-2026-000401",
"title": "ICE.NDEX.TFM is late",
"message": "The delivery for 2026-11-18 had not arrived by 19:30 EU1",
"category": "dataset",
"issue": "delivery",
"severity": "high",
"status": "open",
"agent": "opendatadsl",
"subject": {"service": "dataset", "id": "ICE.NDEX.TFM", "ondate": "2026-11-18"},
"key": "dataset/ICE.NDEX.TFM/2026-11-18/delivery",
"occurrences": 1,
"tags": [],
"created": {"at": "2026-11-18T19:30:02", "by": "system"},
"timeline": [
{"id": 1, "type": "created", "at": "2026-11-18T19:30:02", "by": "system", "status": "open", "agent": "opendatadsl", "severity": "high"}
],
"_version": 1
}

See Alert fields for every field, including root causes, and the timeline entry types.