Skip to main content

Loading Data

OpenDataDSL supports several patterns for getting data into the platform. This page introduces each approach so you can choose what fits your use case.

Overview​

MethodBest for
ETL WorkflowsScheduled collection from external sources — websites, APIs, FTP
EventsTransactional or tick data where each record has its own timestamp and properties
Queue ingestionPushing data from external systems into the platform asynchronously
Direct API / SDKWriting data programmatically from your own application

ETL Workflows​

The most common pattern for loading public or vendor data is an ETL (Extract, Transform, Load) workflow. You build a workflow from reusable actions, schedule it as a process, and the platform runs it automatically.

The typical steps are:

  1. Extract — pull raw data from an HTTP URL, FTP server, file store, or API using a built-in extractor action
  2. Transform — map the raw data into your declared types using a transformer
  3. Load — save the resulting objects and timeseries/curve data to the platform

Example — loading ECB FX data daily​

// Create a process that runs the public ECB FX workflow every weekday at 17:00
process = Process()
process.service = "ETL"
process.name = "ECB_FX_WORKFLOW"
process.description = "Daily ETL for ECB foreign exchange rates"
process.enabled = true
process.workflow = "#xml_data_loader"
process.input.url = "https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml"
process.input.tx = "ECB_FX"
process.cron = "0 17 ? * MON,TUE,WED,THU,FRI *"
save ${process:process}
tip

The public action library includes ready-made extractors for XML, JSON, CSV, HTML, FTP, and more. Start from a public workflow and customise it for your source.

Further reading: Workflows and Processes

Events​

Events are used for transactional or tick-level data where each record carries its own timestamp, start/end time, and a set of typed properties. They are well suited to trade data, exchange settlement prices, and REMIT market messages.

Events are grouped by a shared ID (_dsid) that identifies the provider, feed, and product — for example ICE.IFEU.B. This ID is also used by the dataset monitoring system to track completeness and quality.

Creating an event type​

TradeEvent = type
    PRICE as number()
    VOLUME as number()
end
TradeEvent.event = true
save TradeEvent

Further reading: Loading Data using Events

Queue Ingestion​

If you have an external system that generates data — a trading platform, risk system, or custom application — you can push that data into OpenDataDSL via a message queue. The platform listens on the queue and triggers a script or workflow to process and save each message.

Setup steps​

1. Create a trigger queue

inbound = Queue("inbound")
inbound.handler = "trigger"
save inbound

2. Create an automation to route queue messages to a processing script

POST https://api.opendatadsl.com/api/automation/v1
Authorization: Bearer {{token}}

{
  "_type": "VarAutomation",
  "target": "odsl.script",
  "enabled": true,
  "properties": {
    "script": "#my-load-script"
  },
  "conditions": [{
    "source": "private",
    "service": "queue",
    "id": "inbound",
    "action": "message"
  }]
}

3. Send data to the queue

Any application can publish a message to the queue. Include an origin property in the message so the platform knows the source queue.

note

Queue ingestion decouples the data producer from the platform — your external system does not need to know anything about OpenDataDSL's API, only the queue endpoint.

Further reading: Loading data from a message queue · Automation REST service

Direct API / SDK​

For applications that need to write data directly, OpenDataDSL provides a REST API and SDKs for Python, Java, .NET, MATLAB, and JavaScript.

Example — saving a timeseries via Python SDK​

from odsl import OpenDataDSL

odsl = OpenDataDSL()

ts = odsl.TimeSeries("BUSINESS")
ts.add("2024-01-02", 101.5)
ts.add("2024-01-03", 102.1)

obj = odsl.Object()
obj["MY_PRICE"] = ts
odsl.save("object", "MY_OBJECT", obj)

Example — saving an object via REST API​

POST https://api.opendatadsl.com/api/object/v1
Authorization: Bearer {{token}}

{
  "_id": "MY_OBJECT",
  "_type": "MyType",
  "name": "My Object"
}

Further reading: Python SDK · Java SDK · REST API Guide

Choosing an approach​

You want to…Use…
Collect data from a public website or vendor API on a scheduleETL workflow + process
Load tick-by-tick or transactional data with rich per-record metadataEvents
Push data from an existing internal system without API integrationQueue ingestion
Write data from your own application using a typed SDKDirect API / SDK
Load data once or interactively during developmentDirect API / ODSL script

Next Steps​