Skip to main content

Dataset Monitoring

Dataset monitoring gives you visibility into whether your data is arriving on time, in full, and to the quality you expect. It tracks every data load against a configurable set of completeness and quality rules, scores each day's delivery, and raises alerts when things go wrong.

Core Concepts​

The monitoring system is built around four entities that nest inside each other:

EntityDescriptionExample ID
Dataset FeedA configuration defining the expected time window for a specific feed from a single providerICE.IFEU
DatasetAn individual product within a feed — defines what quantity of data is expected dailyICE.IFEU.B
Dataset DeliveryA daily record of what actually arrived for a dataset, including timing, completeness, and quality scoresICE.IFEU.B:2024-06-12
Smart LoaderAn optional wrapper that re-runs the load process automatically until completeness criteria are met—

Delivery Lifecycle​

Each day, a new delivery record is initialised for every monitored dataset. The record tracks the dataset through its lifecycle:

  1. Initialised — the delivery record is created at the start of the day with status waiting and a starting score of 4
  2. Data loaded — when data arrives tagged with a _dsid, the platform checks for corrections, calculates metrics, assesses completeness, and updates the delivery status to partial or loaded
  3. Quality checks run — any configured quality checks are applied to the full dataset after each load
  4. Lateness check — a background process checks whether datasets have arrived within their configured time window and reduces the score if they are late or missing

Delivery Scoring​

Each delivery is scored daily based on how promptly the data arrived:

ScoreMeaning
4Data arrived on time
3Data was up to 1 hour late
2Data was 1–4 hours late
1Data was more than 4 hours late
0No data expected (e.g. holiday)

Setting Up a Private Dataset Feed​

For data you load yourself, you define the feed, dataset, and checks. The dataset ID always follows the format PROVIDER.FEED.PRODUCT — use a short form of your company name as the provider.

Create a dataset feed​

dsf = object as DatasetFeed
    dsid = "ACME.PRICES"
    name = "ACME Daily Prices"
    calendar = "BUSINESS"
    timezone = "Europe/London"
    time = "18:00 EU1"
    late = "20:00 EU1"
end
save dsf

Create a dataset with expected tenor counts​

ds = object as Dataset
    dsid = "ACME.PRICES.NBP"
    name = "ACME NBP Daily Prices"
    expected = SimpleObject()
end
ds.expected.set("*", 12)       // 12 tenors total expected
ds.expected.set("Month", 12)   // all 12 should be monthly tenors
save ds

Add quality checks​

Quality checks are ODSL functions that run after each load to verify values are within acceptable bounds:

ds = object as Dataset
    dsid = "ACME.PRICES.NBP"
end

// Check that SETTLEMENT_PRICE is never zero
check1 = SimpleObject()
check1.type = "function"
check1.script = "#QualityCheckDatasets"
check1.name = "Check for zero settlement prices"
check1.expression = "zeroCheck('SETTLEMENT_PRICE')"

ds.checks = [check1]
save ds
tip

Reusable quality checks can be grouped into a Quality Group and applied to multiple datasets at once. This avoids repeating the same check configuration across every dataset.

Monitoring Public and Common Datasets​

For datasets loaded by OpenDataDSL (public) or shared feeds (common), you can add your own quality checks and categorisation without modifying the core configuration:

// Add a reference to a common dataset with your own quality check
ds = object as Dataset
    dsid = "ICE.NDEX.NLB"
    source = "common"
end
save ds

Querying Delivery Records​

// All deliveries for today
find ${dataset:"delivery"}

// All deliveries for a specific date
find ${dataset:"delivery/2024-06-25"}

Alerts​

When a dataset is late, incomplete or fails a quality check, the platform raises an alert for it, with the dataset id and the delivery date. An alert stays open until the problem is solved; some close themselves, for example when the missing data arrives. Your team sees them, with the delivery behind each one, in the Alerts extension, and can comment on them or hand them to OpenDataDSL.

To be told when an alert is raised for a dataset, create an automation on the alert service, with the dataset id:

POST https://api.opendatadsl.com/api/automation/v1
Authorization: Bearer {{token}}

{
  "_type": "VarAutomation",
  "target": "email",
  "enabled": true,
  "properties": {
    "to": "ops@company.com",
    "subject": "ALERT: ICE.NDEX.NLB"
  },
  "conditions": [{
    "source": "private",
    "service": "alert",
    "id": "ICE.NDEX.NLB",
    "action": "raised"
  }]
}

Further Reading​