Skip to main content

Subscribing to Data-Feeds - Part 1

Introduction​

A "Getting Started" guide to OpenDataDSL - subscribe to our FREE public data feeds.

This is the first of the "Getting Started" series of tutorials which take you through real-world examples of how to work with OpenDataDSL.

In this tutorial, we take you step-by-step through the process of subscribing to a FREE data feed and take you through various options for feeding that data into databases and applications within your own enterprise.

The diagram below explains the areas that this tutorial covers.

We are going to feed the data file to your choice of 3 different consumers:

  • A C# application which will feed the data into a local SQL Server Database
  • A Java application that uses Camel Enterprise Integration Pattern (EIP) which will feed into a local Kafka topic
  • A Cloud BI Reporting tool
info

In our tutorial, only one consumer will work at a time. You can implement all the consumers if you like by configuring addition queue targets to send the subscriptions to.

Step 1 - Subscribing to a data feed​

In this tutorial, we are going to subscribe to a daily feed of FX rates as provided by the European Central Bank. The dataset code is ECB_FX, there are 32 exchange rates against the Euro from various other currencies.

Create a queue​

In this step we will create a queue that will be used with the subscription that we create.

  • Select Queues
  • Click the New button to open up the queue creation screen.
  • Give the queue the name tutorial
  • Ignore the handler as we are going to handle the queue consumption ourself
  • After creation, you will see the queue details on the screen, copy the connection string, you will need this for later on

Subscribe to data​

We now need to create a subscription that will feed some FX TimeSeries into our newly created queue.

  • Select Subscriptions
  • Click the New button to open up the subscription creation screen
  • Give the subscription the name sub_ecb_fx
  • Click on the Add Data button
  • Add a Filter by clicking the + button and select Dataset
  • Find ECB_FX in the list
  • Click the check-box next to a few of the data items
  • Click OK
  • Click on the Add Target button
  • Select Queue Target from the dropdown and then select the tutorial queue that we created in the previous step
  • Click OK to save our subscription

Trigger the subscription​

We now need to trigger the subscription to feed some data into the queue so that we can test. You can do this as many times as you need when you are testing our subscription consumer.

  • Select Subscriptions
  • Click on the sub_ecb_fx subscription.
  • Click the trigger item
  • Select a date that you know there is some data (e.g. 4th October 2021) for and click ok

This will send a subscription record to the queue, which will look something like this:

{
  "subscription": "61602bc1b5a8223b48998ce6",
  "name": "sub_ecb_fx",
  "date": "Mon, 04 Oct 2021 00:00:00 GMT",
  "status": {
    "GBP": {
      "_type": "#Delta",
      "id": "#ECB_FX.EURGBP:SPOT",
      "action": "create",
      "time": "Mon, 04 Oct 2021 00:00:00 GMT",
      "tenors": [
        {
          "_type": "#DeltaTenor",
          "time": "2021-10-04T00:00:00.000Z",
          "tenor": "SPOT",
          "value": 0.8553
        }
      ]
    },
    "JPY": {
      "_type": "#Delta",
      "id": "#ECB_FX.EURJPY:SPOT",
      "action": "create",
      "time": "Mon, 04 Oct 2021 00:00:00 GMT",
      "tenors": [
        {
          "_type": "#DeltaTenor",
          "time": "2021-10-04T00:00:00.000Z",
          "tenor": "SPOT",
          "value": 129.21
        }
      ]
    },
    "USD": {
      "_type": "#Delta",
      "id": "#ECB_FX.EURUSD:SPOT",
      "action": "create",
      "time": "Mon, 04 Oct 2021 00:00:00 GMT",
      "tenors": [
        {
          "_type": "#DeltaTenor",
          "time": "2021-10-04T00:00:00.000Z",
          "tenor": "SPOT",
          "value": 1.1636
        }
      ]
    }
  }
}

Step 2 - Create our consumer​

Here we need to decide which consumer(s) we want to create for our subscription