Skip to main content

Processes and scripts


Process​

A process runs a script or a workflow, on demand or on a schedule. Service: Process (process/v1), versioned.

FieldTypeRequiredWhat it is
_typeStringYesVarProcess
nameStringYesThe process name. It is also the _id
serviceStringNoA grouping for the process, for example ETL, defaults to default
descriptionStringNoA description
scriptStringOne ofThe script the process runs
workflowStringOne ofThe workflow the process runs
cronStringNoThe schedule, see Cron
enabledBooleanNoFalse to stop the schedule
inputObjectNoValues passed to the script or workflow
propertiesObjectNoYour own information about the process
settingsObjectNoHow and where the process runs, see below
tagsString[]NoTags used to find the process
dataPackageBooleanOutputTrue for a data package process
processTypeStringOutputRistretto
runAsObjectPlatformThe person whose access the process runs with: the person who saved it
environmentStringPlatformThe data environment it was saved in

Settings​

FieldTypeDefaultWhat it is
jobStringYour tenant's jobThe Azure Batch job to run the process in
batchAccountStringOpenDataDSLA client batch account registered for your tenant. Needs image
imageStringA container image to run instead of the standard runtime
commandlineStringRuns the ODSL process runnerThe command line; ${taskid} is replaced with the execution id
maxTimeStringPT1HThe longest the process can run, as a duration
retentionStringPT10MHow long the task files are kept after it ends
retriesInteger0Retries of a failed task
{
"_type": "VarProcess",
"name": "NGE_TTF_SETTLEMENT_LOADER",
"service": "ETL",
"description": "Loads the ICE Endex TTF settlement prices",
"script": "northgate\\loaders\\ttf-settlement",
"cron": "30 18 * * 1-5 * Europe/London",
"enabled": true,
"input": {"product": "TFM", "dataset": "ICE.NDEX.TFM"},
"settings": {"maxTime": "PT30M", "retries": 1},
"tags": ["gas", "ice"]
}

Process execution​

Each run of a process is recorded in the Process execution service (process-exec/v1). To run a process, POST {"name": "NGE_TTF_SETTLEMENT_LOADER"}, with values for the run in body (or input).

FieldTypeWhat it is
_idStringThe execution id
nameStringThe process name
serviceStringThe process service
environmentStringThe data environment
triggerStringscheduled, manual or automation
reasonStringWhy it ran
runAsStringWho it ran as
start, endDateTimeWhen it started and ended
statusStringtriggered, start, running, waiting, retrying, reschedule, failed, success or warning
messageStringThe latest message
phasesArrayFor a workflow: each phase with name, start, end, status, message and log
loaderMessageStringFor a loader: the load summary
dataObjectFor a loader: the data updated, by date, in curves, series, events and matrix
{
"_id": "05f1ec1a-f2a1-4cba-87d4-b46e0f8e40b6",
"service": "ETL",
"name": "NGE_TTF_SETTLEMENT_LOADER",
"environment": "production",
"trigger": "scheduled",
"status": "success",
"message": "Completed Successfully",
"start": "2026-11-18T18:30:07.858",
"end": "2026-11-18T18:30:24.445",
"runAs": "priya.shah@northgate-energy.com",
"loaderMessage": "Updated 1 curve for 2026-11-18",
"data": {
"curves": {"2026-11-18": ["NGE.GAS.TTF:SETTLE"]},
"series": {}
}
}

Script​

Service: Script (script/v1/{source}), versioned.

FieldTypeRequiredWhat it is
_idStringYesThe script id. Private ids often have folders separated by backslashes, for example northgate\curves\build-spark-spreads
_typeStringNoVarScript
scriptStringYesThe code, Base64 encoded
scriptTypeStringNoThe language: odsl (the default), mustache, html, mermaid, python, javascript and so on
categoryStringNoA category. For ODSL, an @category tag in the first comment sets it
descriptionStringNoA description. For ODSL, taken from the first comment
nameStringNoFor html, mustache and mermaid scripts, from an @name line in the first comment
outputFormatStringNoFor html, mustache and mermaid scripts, from an @content line: the content type the template makes
{
"_id": "northgate\\curves\\build-spark-spreads",
"_type": "VarScript",
"scriptType": "odsl",
"category": "curves",
"description": "Builds the German clean spark spread",
"script": "LyoqIEJ1aWxkcyB0aGUgR2VybWFuIGNsZWFuIHNwYXJrIHNwcmVhZCBAY2F0ZWdvcnkgY3VydmVzICovCg=="
}

Action, workflow, transformer and extractor​

These are compiled from ODSL scripts. Each has the source in script, Base64 encoded, plus:

_typeServiceOther fields
VarActionActioncategory, description, inputs, outputs, exits
VarWorkflowWorkflowcategory, description, phases, inputs, outputs, exits
VarTransformerTransformercategory, description, transformer (the compiled transformer)
VarExtractorExtractorcategory, description, extractor (the compiled extractor)

inputs and outputs are lists of arguments:

FieldTypeWhat it is
_idStringThe argument name
_typeStringVarArg
argTypeStringThe variable type of the argument
optionalBooleanTrue if the argument can be left out
descriptionStringA description

exits is a list of the names of the action's or workflow's exit transitions, for example ["ok", "fail"], and phases the names of the workflow's phases.


Batch​

A batch saves many objects in one request. Service: Batch (batch/v1).

FieldTypeRequiredWhat it is
_typeStringYesVarBatch
_idStringNoAn id for the batch
elementsArrayYesThe objects to save, each with its data
nameStringNoThe name of the loader, shown in the summary
serviceStringNoThe service of the loader
loaderStringNoThe loader
correlationIdStringNoAn id to match the batch with your own records
historyBooleanNoLoad as history: faster, and no automations run
fastBooleanNoFast mode
{
"_type": "VarBatch",
"name": "NGE_TTF_SETTLEMENT_LOADER",
"elements": [
{"_id": "NGE.GAS.TTF", "SETTLE": {"_type": "VarCurve", "ondate": {"curveDate": "2026-11-18", "expiryCalendar": "#REOMHENG"}, "contracts": [{"tenor": "M01", "value": 42.15}]}},
{"_id": "NGE.GAS.NBP", "SETTLE": {"_type": "VarCurve", "ondate": {"curveDate": "2026-11-18", "expiryCalendar": "#REOMHENG"}, "contracts": [{"tenor": "M01", "value": 104.30}]}}
]
}

Reading the Batch service returns batch summaries:

FieldTypeWhat it is
_idStringThe batch id
userStringWho sent it
environmentStringThe data environment
timestampDateTimeWhen it was loaded
service, nameStringThe loader
reasonStringWhy it was loaded
sourceStringThe source it was loaded to
status, messageStringThe result
summaryObjectWhat was loaded, by date: curves, series, events, matrix and failures

Task tile​

A task tile is a task people start their day from, shown on the portal home page by the Home extension. Service: Task (task/v1). Everyone in your company can read tiles; changing them needs a tenant admin.

FieldTypeRequiredWhat it is
_idStringYesThe tile id, with at least one dot, for example nge.home.check-overnight-loads
_typeStringNoVarTaskTile
nameStringYesThe task, up to 120 characters
descriptionStringNoMore about it, up to 400 characters
groupStringNocheck, find, build, deliver or another short name, defaults to other
iconStringNoA Bootstrap icon name, defaults to check2-square
priorityIntegerNo0 to 100, higher shows first, defaults to 50
featuredBooleanNoFalse to show it only under See all tasks
enabledBooleanNoFalse to hide it
keywordsString[]NoExtra words the task search matches
extensionStringNoThe extension that added the tile
requiresString[]NoOnly show the tile when these extensions are installed
rolesString[]NoThe roles the tile is suggested to
targetObjectNoWhat the tile opens: {type: "view", extension, view, properties}, {type: "url", url, newTab} or {type: "portal", section, name, tabname}
alertsObjectNoWhich open alerts badge the tile: filter (stored as JSON text), sources, followedOnly, label, okLabel
propertiesObjectNoAnything else to keep with the tile
created, createdBy, updated, updatedByPlatformWhen and by whom the tile was made and changed
{
"_id": "nge.home.check-overnight-loads",
"_type": "VarTaskTile",
"name": "Check for overnight production issues",
"description": "Failed, late or partial loads since the last close",
"group": "check",
"icon": "exclamation-triangle",
"priority": 80,
"featured": true,
"enabled": true,
"keywords": ["load", "failed", "late"],
"roles": ["operations"],
"target": {"type": "view", "extension": "odsl.alerts", "view": "Manage"},
"alerts": {"filter": "{\"issue\": {\"$in\": [\"delivery\", \"completeness\"]}}", "sources": ["private"], "label": "issue", "okLabel": "All on time"}
}
note

The Task service used to keep personal to-do tasks (VarTask). Those are no longer listed by the service.


Queue​

A message queue. Service: Queue (queue/v1), versioned.

FieldTypeRequiredWhat it is
queueStringYesThe queue name: letters, numbers, -, . and _, starting and ending with a letter or number
descriptionStringNoA description
retentionStringNoHow long messages are kept, as a duration, defaults to P14D
timeoutStringNoHow long a reader has to process a message, defaults to PT1M, at most PT5M
handlerStringNoWhat handles the messages
_id, tid, tenantStringPlatformThe full queue id and your tenant
{
"queue": "nge-trades",
"description": "Trades from the Northgate ETRM",
"retention": "P7D",
"timeout": "PT2M"
}

Logs​

These records are written by the platform and are read-only.

Script log​

Script Log service (scriptlog/v1): one record per run of a script.

FieldTypeWhat it is
script, scriptVersion, scriptTypeThe script that ran
function, region, regionLineStringThe function or region run, if any
statusStringsuccess, warning or failed
messageStringA summary, or the error
triggerStringWhat ran it
logString[]The log messages
timestampDateTimeWhen it ran
invocationId, rootInvocationIdStringThe request that ran it, and the one that started the chain

Queue log​

Queue Log service (queuelog/v1): one record per message sent to a queue, with timestamp, queue, queueid, messageId, subject, size, contentType, contentId, trigger, origin, status, statusMessage, reason and invocationId.

Schedules and schedule log​

Crontab service (crontab/v1): the schedules that run in your data environment.

FieldTypeWhat it is
nameStringWhat is scheduled: a process name, playbook: and a task id, an agent id, or AlertCheck- and a frequency
typeStringprocess, curve, playbook, agent or alert
minute, hour, dom, month, dow, yearArrayThe parsed schedule
tzStringThe timezone, defaults to UTC
environmentStringThe data environment
jobStringThe job that runs it, if not the default

Crontab Log service (crontablog/v1): one record each time a schedule fires, with name, event (the schedule id), trigger (the time it fired for), timestamp, environment and invocationId.