Skip to main content

Data Management

These commands help you manage the resources in your data

delete​

Used to rollback versions or delete items from the database

Syntax​

Standard form:

delete activevar (log string)?

Bulk delete form:

delete activevar (log string)? (where condition)?

Description​

In the standard form, it rolls back the version of an item from the database, the activevar item can be one of:

  • action
  • calendar
  • critical (future use)
  • curve
  • dashboard
  • data (curve, timeseries, matrix)
  • dataset (dataset, dataset_feed, dataset_delivery)
  • environment
  • event
  • expiry
  • extractor
  • index
  • object
  • policy
  • process
  • queue
  • reportconfig
  • report
  • script
  • secret
  • subscription
  • task
  • transformer
  • type
  • user
  • workflow

In bulk delete form, only the following services are supported:

  • object

You can specify the version number to delete with the activevar syntax: ${service:"resource":version}

The version can also be *, in which case it will fully delete the item from the database

The optional log option allows you to specify a reason for the deletion which is added to the audit log

Example​

// To rollback to the previous version
delete ${object:"TEST"}

// To rollback to the previous version, specifying an audit log message 
delete ${script:"Test Script"} log "Removed TEST object"

// To delete a specific version
delete ${object:"TEST":1}

// To delete all versions
delete ${object:"TEST":*}

find​

Used to search the database

Syntax​

varname = find 
  (top n)?
  (unique field from | profile profilename (for range)? from)?
  (activevar|avservice)
  (where (condition)+)?
  (end)?
Layout​

The find command can be used in a single line or multi-line format for better readability, e.g.

// Single line search of private audit records
records = find ${audit} where timestamp > ${date:"today"} and timestamp < ${date:"tomorrow"}

// Multi-line version of the same search
records = find ${audit} where 
    timestamp > ${date:"today"} 
    and timestamp < ${date:"tomorrow"}
end
Options​
top​

The top option allows you to find a smaller sample of data, e.g.

// Retrieve the first 15 objects
objects = find top 15 ${object} where dataset="ARGUS_DEL"
unique​

The unique option allows you to get a list of unique values for a specific field in a resource

profile​

The profile option is only used when searching through objects. It allows you to search objects, but return a list of data objects linked to the object where the profile name matches the passed in profilename. This option also allows you to specify a range, which can be a single date or a range of dates in the following syntax:

  • Single date as a Date or a String, e.g. “2021-07-16”
  • Single date using the date active variable service, e.g. ${date:”yesterday”}
  • From a date, e.g. from(“2021-01-01”)
  • The last number of days, e.g. last(3)
  • A range of dates using between, e.g. between(“2021-01-01”,${date:”yesterday”})
condition​

The condition option is used to filter the results of the find command. The syntax of the conditions is as follows:

expression (<|<=|>|>=|=|==|!=|like|intersects|within) expression
    | condition (and|or) condition
    | ( condition )

####### Examples of conditions

category = "extractors"
timestamp > "2020-11-03T12:23:40"
timestamp > ${date:"today"} and timestamp < ${date:"tomorrow"}
name like "ch"
location within Sphere(\[ 51.72961, 0.47612 \], 20 / 3963.2)

Description​

The find command is a powerful way of searching a resource for the data you require. It returns a virtual list of items that match the specified conditions.

Examples​

Find all​

In its simplest form, the find command can be used to list all items from a service, e.g.

// Get a list of all public calendars
calendars = find ${calendar:public}
Simple filtering​
// Get a list of public actions for a specific category
pactions = find ${action:public} where category = "extractors"
Filtering by date​
// Get a list of private audit records for today
records = find ${audit} where timestamp > ${date:"today"} and timestamp < ${date:"tomorrow"}
A list of data using the object profile option​
data = find profile SPOT for yesterday from ${object:public} where dataset == "ECB_FX"
Various date range queries​
// Data in get query
data = ${data:"#ECB_FX.EURZAR:SPOT"} for ${date:"yesterday"}
data = ${data:"#ECB_FX.EURZAR:SPOT"} for yesterday
data = ${data:"#ECB_FX.EURZAR:SPOT"} for last(3)
data = ${data:"#ECB_FX.EURZAR:SPOT"} for from(yesterday)
data = ${data:"#ECB_FX.EURZAR:SPOT"} for between("2021-05-01",yesterday)
Geospatial​
// Define a polygon and search for objects within it
london = Polygon(\[\[51.5386, -0.4956\],\[51.6445, -0.0753\],\[51.5205, 0.1753\],\[51.3479, -0.1163\],\[51.5386, -0.4956\]\])
items = find ${object:"TestGeometry"} where location within london

rename​

Used to rename a resource in the database

Syntax​

rename ${service:"resource"} as newname (log "reason")?

Description​

The rename command is used to change the unique id of a specific resource in a service.

Examples​

// Rename a privat eobject
rename ${object:"OBJ1"} as UK_OBJ

// Rename a calendar id and add an audit log message
rename ${calendar:"mycal"} as UKHOURLY log "Changed to correct id"

save​

Used to save items to the database

Syntax​

save varname   (log reason) (replace)
save activevar (log reason) (replace)

Description​

The save command saves an item to the database.

There are 2 forms:

  • Simply save using the variable name and let OpenDataDSL determine the service to use, e.g. save x
  • Explicitly state the service using an activevar, e.g. save ${object:x}
Log​

The optional log reason is to give a human-readable reason why this item is being saved and this is stored in the audit log.

Replace​

The optional replace statement is used with the object service to signify that you want to replace the stored object with the passed in object. This overrides the default action which is to merge the contents of the supplied object with the stored object.

Services​

The following active variable services can be used with the save command:

  • action
  • calendar
  • curve
  • environment
  • expiry
  • extractor
  • index
  • object
  • process
  • queue
  • script
  • subscription
  • task
  • transformer
  • type
  • workflow

Example​

// Create private type
TypeExample = type
    name as String() default "test"
end

// Save the type
save TypeExample

// Create a private object
ObjExample = object as TypeExample
    name = "Hello"
end

// Save the object
save ObjExample log "Created the new object"

// Change the object
ObjExample.name = "Hello World"

// Save the object using replace option to replace the stored object
save ObjExample log "Changed the object" replace

tag​

Tags a version of an item with a name

Syntax​

tag var AS name

Description​

The tag command is used to name a specific version of an item so that it can be read using the supplied name, e.g. you can tag versions of your actions with PROD or TEST to ensure that you use a specific version of an action in a workflow.

Examples​

tag ${action:"ReadStuff":1} as PROD

prodaction = ${action:"ReadStuff":PROD}

update​

Used to bulk update items in the database

Syntax​

update avservice (where condition)? (upsert)?
    (log string)?
    ( (updateoperator|comment)* | (pipelineOperator|comment)* )
end

Description​

Only the following services are supported:

  • object

The optional where condition limits the documents that are updated, if ommitted, all documents are updated

The optional upsert option will insert a document if there is no matching document

The optional log option allows you to specify a reason for the deletion which is added to the audit log

Examples​

Example using update operators

update ${object:"bulk"}
    set(name="test",type="real")
    set(description=Hello)
    inc(count=1,test=5)    
    rename(temp="real")
    unset(test)    
end

Example using pipeline operators

update ${object:"bulk"}
    addFields status="Modified", comments=["$misc1", "$misc2"]
    project misc1, misc2
end

Example using upsert

update ${object:"bulk"} where _id=1 upsert
    set(item="apple")
    setOnInsert(defaultQty=100)
end

Example using a log message

update ${object:"bulk"} where name="b4"
    log "Audit record 123456"
    set(test="Hello Again")
    inc(count=1)
end