dougal-software/etc/qc

QC tests

Introduction

QC tests are defined and parametrised out of source in YAML files. In the project definition file, the qc.definitions and qc.parameters keys point to, respectively, the definition and parametrisation files for the QCs to be applied to a given project.

Different QCs may be defined for different projects by saving them to separate definition files; conversely, the same QCs may be reused across projects.

The parameters for each QC test are saved to a separate file. This is to allow QCs to be reused across projects, possibly with different parameters.

Running

For all projects that have QCs defined, the tests can be run by calling the lib/www/server/lib/qc.js script. This can be done from a cronjob, e.g.,:

# max-old-space-size increases the memory available to Node.js, to deal with complicated tests or large projects.
*/5 * * * * NODE_OPTIONS="--max-old-space-size=4096" node $HOME/software/lib/www/server/lib/qc.js >/dev/null

QC definition file

The QC definition YAML file should consist of a list of tests. These may be organised hierarchically if the user wishes to do so.

A QC definition consists of the following attributes:

Attribute	Description
name	A short name for the test or hierarchical group.
description	A more detailed description. Markdown is accepted.
disabled	If true, the test or branch will not be run.
iterate	What to iterate over. It can take one of three values: shots, sequences or lines. If not present it defaults to shots, which means the script will be called once per every shot in the prospect.
labels	Array of labels to apply to the test or to the branch.
children	Any element having a children attribute becomes a branch. The contents of this attribute are a list of tests or branches, same as the top-level list.
check	A script consisting of JavaScript code which defines the test that is to be run. Tests that pass should return true whereas failing tests should return a string with a message describing the failure. Note that it is valid to have both check and children in the same item.

Test definitions

Tests can be defined as arbitrary JavaScript, which will be run in a sandbox. The sandboxed code does not have access to the filesystem or the console object.

The result of the test is the last expression evaluated by the script. This should be the primitive true if the test is successful, or a string describing the failure if it is not.

The script has access to the following variables:

parameters

An object containing all the parameter definitions for the project, for instance:

{
  gunDepth: 6,
  gunDepthTolerance: 0.5,
  gunTiming: 0.9999,
  gunTimingSubarrayAverage: 0.5,
  gunPressureNominal: 2000,
  gunPressureToleranceRatio: 0.025,
  crosslineError: 12,
  crosslineErrorAverage: 9,
  inlineErrorRunningAverageValue: 2,
  inlineErrorRunningAverageShots: 40
}

currentItem

The item being iterated over. This can be of type Shot, Sequence, or Preplot depending on the value of this test's iterate attribute.

shots

An array of Shot objects.

Example:

{
  type: 'final',
  _id: [ 7, 1764 ],
  sequence: 7,
  line: 5500,
  point: 1764,
  objref: 3,
  tstamp: "2020-09-07T04:10:13.680Z",
  hash: '2173892:1599458941.3070147:1599458941.3070147:80621034',
  geometry: '{"type":"Point","coordinates":[2.471571453,59.169725413]}',
  error_i: 2.7102108739654795,
  error_j: -0.06460360411324473,
  preplot_geometry: '{"type":"Point","crs":{"type":"name","properties":{"name":"EPSG:23031"}},"coordinates":[469883.4,6559284.9]}',
  raw_geometry: '{"type":"Point","crs":{"type":"name","properties":{"name":"EPSG:23031"}},"coordinates":[469881.87,6559285.76]}',
  final_geometry: '{"type":"Point","crs":{"type":"name","properties":{"name":"EPSG:23031"}},"coordinates":[469881.09,6559286.22]}',
  pp_meta: {},
  raw_meta: {
    smsrc: {
      guns: [Array],
      line: '1054980007S00000',
      mask: 38,
      shot: 1764,
      time: "b'20/09/07:04:10:13'",
      spare: '',
      header: '*SMSRC',
      spread: 3,
      volume: 3050,
      blk_siz: 2282,
      manifold: 2027,
      num_auto: 0,
      num_guns: 52,
      trg_mode: 'E',
      avg_delta: 0,
      num_delta: 0,
      std_delta: 0.073,
      baro_press: null,
      num_active: 52,
      num_nofire: 0,
      src_number: 2,
      num_subarray: 6
    }
  },
  final_meta: {},
  _: [Function]
}

sequences

An array of Sequence objects.

Example:

{
    "_id": 9,
    "sequence": 9,
    "line": 5562,
    "fsp": 2580,
    "lsp": 996,
    "fsp_final": 2548,
    "lsp_final": 1000,
    "ts0": "2020-09-07T08:34:13.112Z",
    "ts1": "2020-09-07T10:47:49.116Z",
    "ts0_final": "2020-09-07T08:36:58.984Z",
    "ts1_final": "2020-09-07T10:47:28.608Z",
    "duration": {
        "hours": 2,
        "minutes": 13,
        "seconds": 36,
        "milliseconds": 4
    },
    "duration_final": {
        "hours": 2,
        "minutes": 10,
        "seconds": 29,
        "milliseconds": 624
    },
    "num_preplots": 775,
    "num_points": 775,
    "missing_shots": 0,
    "length": 19350.1845360761,
    "azimuth": 26.443105805883572,
    "remarks": "",
    "remarks_final": "",
    "status": "final"
    _: [Function]
}

preplots

An array of Preplot objects.

Example:

{
  _id: [ null, 2348, 5130 ],
  line: 5130,
  point: 2348,
  class: 'V',
  ntba: false,
  geometry: '{"type":"Point","crs":{"type":"name","properties":{"name":"EPSG:23031"}},"coordinates":[470769.8,6550688.8]}',
  meta: {},
  count: 0
}

The _ function

Each of the above objects has a function named _. This is a helper to quickly access own nested attributes without having to check if the attribute or one of its parents exist. For instance, on a Shot item, currentItem._('raw_meta.smsrc') will return the SmartSource gun data if it exists, or undefined if either smsrc or raw_meta are not defined.