Module 08: Queue Management

Message queues are the glue between the platform and your own systems: processes send batches to the data loader, external applications send data in, and automations send curves out to an ETRM. In this module you will create and configure queues, connect them to handlers and automations, and monitor what passes through them.

Learning Objectives​

By the end of this module you will be able to:

• Explain the default queue, named queues and handlers
• Create, configure, list and delete queues
• Set up a trigger queue that loads data through an automation
• Send data to a queue from ODSL
• Read the queue log to see what was sent and whether it worked
• Monitor message counts, including dead-lettered messages, with metrics

Step 1: Queues and Handlers​

You have one default queue, which is not handled by anything: data sent there sits for the retention time and is then removed. You can create named queues up to your quota, each with its own handler:

Step 2: Create a Queue​

//#region Create a queue
q = Queue("etrm-curves")
q.description = "Forward curves for the ETRM"
q.retention = "P7D"
q.timeout = "PT5M"
save q
print q
//#endregion

Queue configurations are versioned, so ${queue:"etrm-curves":*} lists the changes.

Step 3: List and Inspect Queues​

//#region List queues
queues = find ${queue}
print queues

etrm = ${queue:"etrm-curves"}
print etrm
//#endregion

Reading a queue also returns its current state:

Step 4: Set Up a Trigger Queue​

To load data that arrives on a queue:

1. Create the queue with the trigger handler.
2. Add an automation on the message event of the queue service, with the queue name as the id, targeting the script or process that transforms and loads the data.

//#region Create a trigger queue
inbound = Queue("supplier-inbound")
inbound.description = "Supplier prices pushed to us"
inbound.handler = "trigger"
save inbound
//#endregion

See Creating Automations for building the automation, and Loading data from a message queue for the full example. External senders must add an origin property to each message so the trigger knows the source queue.

Step 5: Send Data to a Queue​

The send command sends a batch to a queue. Without a queue name it goes to the default queue:

//#region Send a batch
batch = PROCESS.createBatch()
obj1 = Object()
obj1.name = "Hello"
batch.add(obj1)

send batch to "supplier-inbound"

// Back-fill without triggering downstream processes and exports
send batch -noevent
//#endregion

Use size to change how many items are sent per message, for example send batch size 10.

Step 6: Read the Queue Log​

Every message sent to your queues is logged in the queue log:

//#region Messages sent to a queue
log = find ${queuelog} where queue = "etrm-curves"
for entry in log
    print entry.timestamp + " " + entry.status + " " + entry.statusMessage
next
//#endregion

For automations that send to a queue, the automation log and the Automations extension's Activity tab show each run and the target's message (see Module 09).

Step 7: Monitor with Metrics​

The AzureServiceBus metric provider gives queue metrics such as ActiveMessages, DeadletteredMessages, IncomingMessages and OutgoingMessages, filtered by queue:

//#region Average active messages per hour
cal = ${calendar:"#PT1H"}
ActiveMessages = MetricTimeSeries("AzureServiceBus", "ActiveMessages")
ActiveMessages.calendar = cal
ActiveMessages.filter = ?queue='etrm-curves'
ActiveMessages.observed = "averaged"
//#endregion

A rising active count means the consumer is not keeping up, or has stopped. A non-zero dead-letter count means messages could not be delivered. You will store and chart metrics like this in Module 10.

Step 8: Delete a Queue​

//#region Delete a queue
delete ${queue:"supplier-inbound"}
//#endregion

Check first that no automation or external application still uses it.

Exercise​

1. Create a queue called training-queue with a retention of P1D and a timeout of PT5M.
2. Read it back and note its message counts.
3. Send a small batch to it, then find the entry in the queue log.
4. Delete the queue.

Summary​