Skip to main content

Data Management

These commands help you manage the resources in your data

delete

Used to rollback versions or delete items from the database

Syntax

Standard form:

delete activevar (log string)?

Bulk delete form:

delete activevar (log string)? (where condition)?

Description

In the standard form, it rolls back the version of an item from the database, the activevar item can be one of:

  • action
  • calendar
  • critical (future use)
  • curve
  • dashboard
  • data (curve, timeseries, matrix)
  • dataset (dataset, dataset_feed, dataset_delivery)
  • environment
  • event
  • expiry
  • extractor
  • index
  • object
  • policy
  • process
  • queue
  • reportconfig
  • report
  • script
  • secret
  • subscription
  • task
  • transformer
  • type
  • user
  • workflow

In bulk delete form, only the following services are supported:

  • object

You can specify the version number to delete with the activevar syntax: ${service:"resource":version}

The version can also be *, in which case it will fully delete the item from the database

The optional log option allows you to specify a reason for the deletion which is added to the audit log

Example

// To rollback to the previous version
delete ${object:"TEST"}

// To rollback to the previous version, specifying an audit log message
delete ${script:"Test Script"} log "Removed TEST object"

// To delete a specific version
delete ${object:"TEST":1}

// To delete all versions
delete ${object:"TEST":*}

find

Used to search the database

Syntax

varname = find 
(top n)?
(unique field from | profile profilename (for range)? from)?
(activevar|avservice)
(where (condition)+)?
(end)?
Layout

The find command can be used in a single line or multi-line format for better readability, e.g.

// Single line search of private audit records
records = find ${audit} where timestamp > ${date:"today"} and timestamp < ${date:"tomorrow"}

// Multi-line version of the same search
records = find ${audit} where
timestamp > ${date:"today"}
and timestamp < ${date:"tomorrow"}
end
Options
top

The top option allows you to find a smaller sample of data, e.g.

// Retrieve the first 15 objects
objects = find top 15 ${object} where dataset="ARGUS_DEL"
unique

The unique option allows you to get a list of unique values for a specific field in a resource

profile

The profile option is only used when searching through objects. It allows you to search objects, but return a list of data objects linked to the object where the profile name matches the passed in profilename. This option also allows you to specify a range, which can be a single date or a range of dates in the following syntax:

  • Single date as a Date or a String, e.g. “2021-07-16”
  • Single date using the date active variable service, e.g. ${date:”yesterday”}
  • From a date, e.g. from(“2021-01-01”)
  • The last number of days, e.g. last(3)
  • A range of dates using between, e.g. between(“2021-01-01”,${date:”yesterday”})
condition

The condition option is used to filter the results of the find command. The syntax of the conditions is as follows:

expression (<|<=|>|>=|=|==|!=|like|intersects|within) expression
| condition (and|or) condition
| ( condition )

####### Examples of conditions

category = "extractors"
timestamp > "2020-11-03T12:23:40"
timestamp > ${date:"today"} and timestamp < ${date:"tomorrow"}
name like "ch"
location within Sphere(\[ 51.72961, 0.47612 \], 20 / 3963.2)

Description

The find command is a powerful way of searching a resource for the data you require. It returns a virtual list of items that match the specified conditions.

Examples

Find all

In its simplest form, the find command can be used to list all items from a service, e.g.

// Get a list of all public calendars
calendars = find ${calendar:public}
Simple filtering
// Get a list of public actions for a specific category
pactions = find ${action:public} where category = "extractors"
Filtering by date
// Get a list of private audit records for today
records = find ${audit} where timestamp > ${date:"today"} and timestamp < ${date:"tomorrow"}
A list of data using the object profile option
data = find profile SPOT for yesterday from ${object:public} where dataset == "ECB_FX"
Various date range queries
// Data in get query
data = ${data:"#ECB_FX.EURZAR:SPOT"} for ${date:"yesterday"}
data = ${data:"#ECB_FX.EURZAR:SPOT"} for yesterday
data = ${data:"#ECB_FX.EURZAR:SPOT"} for last(3)
data = ${data:"#ECB_FX.EURZAR:SPOT"} for from(yesterday)
data = ${data:"#ECB_FX.EURZAR:SPOT"} for between("2021-05-01",yesterday)
Geospatial
// Define a polygon and search for objects within it
london = Polygon(\[\[51.5386, -0.4956\],\[51.6445, -0.0753\],\[51.5205, 0.1753\],\[51.3479, -0.1163\],\[51.5386, -0.4956\]\])
items = find ${object:"TestGeometry"} where location within london

rename

Used to rename a resource in the database

Syntax

rename ${service:"resource"} as newname (log "reason")?

Description

The rename command is used to change the unique id of a specific resource in a service.

Examples

// Rename a privat eobject
rename ${object:"OBJ1"} as UK_OBJ

// Rename a calendar id and add an audit log message
rename ${calendar:"mycal"} as UKHOURLY log "Changed to correct id"

save

Used to save items to the database

Syntax

save varname   (log reason) (replace)
save activevar (log reason) (replace)

Description

The save command saves an item to the database.

There are 2 forms:

  • Simply save using the variable name and let OpenDataDSL determine the service to use, e.g. save x
  • Explicitly state the service using an activevar, e.g. save ${object:x}
Log

The optional log reason is to give a human-readable reason why this item is being saved and this is stored in the audit log.

Replace

The optional replace statement is used with the object service to signify that you want to replace the stored object with the passed in object. This overrides the default action which is to merge the contents of the supplied object with the stored object.

Services

The following active variable services can be used with the save command:

  • action
  • calendar
  • curve
  • environment
  • expiry
  • extractor
  • index
  • object
  • process
  • queue
  • script
  • subscription
  • task
  • transformer
  • type
  • workflow

Example

// Create private type
TypeExample = type
name as String() default "test"
end

// Save the type
save TypeExample

// Create a private object
ObjExample = object as TypeExample
name = "Hello"
end

// Save the object
save ObjExample log "Created the new object"

// Change the object
ObjExample.name = "Hello World"

// Save the object using replace option to replace the stored object
save ObjExample log "Changed the object" replace

tag

Tags a version of an item with a name

Syntax

tag var AS name

Description

The tag command is used to name a specific version of an item so that it can be read using the supplied name, e.g. you can tag versions of your actions with PROD or TEST to ensure that you use a specific version of an action in a workflow.

Examples

tag ${action:"ReadStuff":1} as PROD

prodaction = ${action:"ReadStuff":PROD}

update

Used to bulk update items in the database

Syntax

update avservice (where condition)? (upsert)?
(log string)?
( (updateoperator|comment)* | (pipelineOperator|comment)* )
end

Description

Only the following services are supported:

  • object

The optional where condition limits the documents that are updated, if ommitted, all documents are updated

The optional upsert option will insert a document if there is no matching document

The optional log option allows you to specify a reason for the deletion which is added to the audit log

Examples

Example using update operators

update ${object:"bulk"}
set(name="test",type="real")
set(description=Hello)
inc(count=1,test=5)
rename(temp="real")
unset(test)
end

Example using pipeline operators

update ${object:"bulk"}
addFields status="Modified", comments=["$misc1", "$misc2"]
project misc1, misc2
end

Example using upsert

update ${object:"bulk"} where _id=1 upsert
set(item="apple")
setOnInsert(defaultQty=100)
end

Example using a log message

update ${object:"bulk"} where name="b4"
log "Audit record 123456"
set(test="Hello Again")
inc(count=1)
end