Writing Clarive Rules in HCL

Clarive rules have always been built as trees. A pipeline is a set of steps dragged from a palette in the Rule Designer, each one configured in a dialog and nested inside the stage or condition it belongs to. On screen the tree is easy to follow. It has been hard to do anything else with it. Comparing two versions meant opening both, and finding every step that ran on one server meant clicking through every rule that might have one.

In Clarive 7.22 every rule can also be written as HCL, the text format introduced earlier in this series. Nothing about how rules run has changed. The tree now has a text form as well, and the text can be edited, reviewed in Git and imported into another installation.

Here is the example the documentation opens with:

pipeline "deploy-foo" {
  name = "Deploy Foo"
  desc = "Deploys foo"
  when = "promote"

  step "PRE" {
    sh "Restart app" {
      host      = generic_server.alpha
      run       = "systemctl restart app"
      capture   = "restart_out"
      timeout   = 300
      semaphore = "lock-a"
      parallel  = "fork"
      on_error  = "continue"
      note      = "be careful"
    }
  }

  if "is prod" {
    cond = "bl eq 'PROD'"

    log "say it" {
      msg   = "hello"
      level = "warn"
    }
  }

  else {
    perl "run perl" {
      lang       = "perl"
      code       = <<-EOT
        my $x = 1;
        return $x;
      EOT
      disabled   = true
      never_runs = true
    }
  }

  wait "join" {}
}

One block per step

A rule is one block, named by its type. pipeline, workflow and event are the ones most people meet first. Forms, dashboards, web services (ws) and independent rules (rule) have keywords of their own. The label is the slug of the rule's name, so this one's address is pipeline.deploy-foo, and that is how a call step in another rule refers to it.

Each node of the tree is a block nested inside the one that holds it. The keyword names the operation, and the label is the step's own name, free text kept exactly as the author typed it. The operations people write most often have short keywords chosen by hand, 39 of them. sh runs a script, ship copies files to a server and fetch brings them back. let sets a variable. Every other operation gets a keyword built from its registry name, so service.genexus.buildRevision becomes genexus_buildRevision. The documentation counts 319 registered operations, and each one reads back as the operation it was written from.

sh is the one keyword that stands for two operations. With a host it runs on that server, and without one it runs locally. A few parameters also get friendlier names than the keys stored in the database, so sh takes run and host where the stored keys are path and server.

Inside a step, everything is one flat list. The operation's own parameters come first, in the order its dialog lays them out, so a file reads top to bottom the way the dialog does. The settings from the dialog's Options tab follow, and the text from its Note tab comes last. In the example, host and run belong to sh, while capture, timeout, semaphore, parallel, on_error and note belong to the step. A setting that tunes something appears only next to what it tunes. A retry count is written only beside an on_error, because the rule compiler reads it only there:

sh "deploy" {
  run                 = "make deploy"
  on_error            = "trap"
  trap_timeout        = 600
  trap_timeout_action = "retry"
  trap_max_retry      = 3
  trap_rollback       = false
}

A step that is switched off carries disabled = true. forward_only, rollback_only and never_runs cover the other run states. A step that runs normally in both directions carries none of them.

Conditions and code

Conditions have a short form. cond = "bl eq 'PROD'" is a single test, and all, any and none each take a list of them. An else block sits right after its if, as a sibling instead of inside it, because that is how the tree stores it. An else anywhere else fails to compile.

Perl and shell code go in heredocs, and the indentation of their lines relative to one another is kept. HCL interprets nothing inside a string. A ${job_dir} in a script reaches the rule as those exact characters, the same text the step's dialog would have stored.

What gets left out

The Rule Designer saves every field of a step's dialog, whether or not anyone touched it. The export leaves out each value that matches what the dialog would have filled in by itself, so a step in a file shows only what somebody chose. A value that looks empty can still be a choice. The documentation's example is a ship step whose backup_mode = "none" stays in the file, because the dialog's own default is "backup".

Import puts the omitted values back from the same table of defaults, so the short form and the long one describe the same step. The table belongs to a release, and a later release may change a default. For archives, and for files headed to an installation on another release, cla export --all-attributes writes out every parameter of every step. It applies to rules only, not to resources. The cla export page has the ship step both ways.

Checking before saving

A rule that reads correctly can still fail to build, so the import checks. A changed rule goes through the same save the Rule Designer uses, and its steps are compiled first. A rule that doesn't compile is refused. Nothing is half-saved. A step naming an operation this installation doesn't have, or pointing at an object that isn't there, is reported with its file and line, and the import stops. A rule that hasn't changed keeps its version number. The steps of a changed rule are given new internal ids on the way in, so their job history and profile start again.

In the browser, errors come back on the HCL line they belong to. Embedded Perl is compiled block by block, and the compiler's own message appears beside the line that holds it. A rule that parses but doesn't compile as a whole gets its errors on the step each one came from, in Perl's words, because Perl on its own would count lines in code generated from the entire rule and shown nowhere on screen.

Before a team starts editing an installation's rules as text, it's worth running cla rule-check. It writes every rule as HCL, reads it back and compiles both versions, then reports any rule that wouldn't come back the same. Nothing is written.

ok   rule 24 "Deploy foo" (pipeline)
FAIL rule 2 "Releases" (dashboard)
     error lost at /1 dashlet.topic.number_of_topics "Foo" data.categories: [7,44,45] -> (absent)

2 rule(s) checked: 1 come back the same, 1 would be changed by a save from the HCL view

It exits with 0 when every rule survives and 1 when any wouldn't. The usual cause of a failure is a value holding a database id that nothing can name, often a reference to something deleted since. The export won't write an id, so the text has nothing in that place. --id 24,31 checks only those two rules, and --notes also lists what changes only in notation.

Why text helps

A change to a rule can now be reviewed like any other change. The rule's file goes into Git, and a pull request shows exactly which lines of a pipeline changed. Whoever approves a change to a production deploy reads the one step that is different, with the lines around it, instead of opening two versions of the tree side by side and comparing them dialog by dialog.

Rules can be searched. Each rule is its own file, named by its address, so finding every pipeline with a step on one server takes a single command:

grep -l 'generic_server.web01' cla-objects/prod/pipeline.*.hcl

Rules also travel. A step that ran on generic_server.web01 on TEST runs on whatever web01 is on PROD, whatever id it carries there, because the address is looked up on the installation that imports it.

Event rules need one check for now. The filter that decides which events an event rule answers to isn't written to the file, and importing the rule clears the filter on the installation receiving it. After importing an event rule, look at its scopes before it next fires.

The rule reference, with every operation keyword and the defaults behind it, is in the documentation. The next post covers the HCL views in the Rule Designer and the rest of the interface. If you'd like to see your own pipelines as text, get in touch.