dougal-software/etc/qc/README.md

# QC tests

## Introduction

QC tests are defined and parametrised out of source in YAML files. In the project definition file, the `qc.definitions` and `qc.parameters` keys point to, respectively, the definition and parametrisation files for the QCs to be applied to a given project.

Different QCs may be defined for different projects by saving them to separate definition files; conversely, the same QCs may be reused across projects.

The parameters for each QC test are saved to a separate file. This is to allow QCs to be reused across projects, possibly with different parameters.

## Running

For all projects that have QCs defined, the tests can be run by calling the [`lib/www/server/lib/qc.js`](/lib/www/server/lib/qc.js) script. This can be done from a cronjob, e.g.,:

```cron
# max-old-space-size increases the memory available to Node.js, to deal with complicated tests or large projects.
*/5 * * * * NODE_OPTIONS="--max-old-space-size=4096" node $HOME/software/lib/www/server/lib/qc.js >/dev/null

```

## QC definition file

The QC definition YAML file should consist of a list of tests. These may be organised hierarchically if the user wishes to do so.

A QC definition consists of the following attributes:

Attribute     | Description
--------------|-------------
`name`        | A short name for the test or hierarchical group.
`description` | A more detailed description. Markdown is accepted.
`disabled`    | If `true`, the test or branch will not be run.
`iterate`     | What to iterate over. It can take one of three values: `shots`, `sequences` or `lines`. If not present it defaults to `shots`, which means the script will be called once per every shot in the prospect.
`labels`      | Array of labels to apply to the test or to the branch.
`children`    | Any element having a `children` attribute becomes a branch. The contents of this attribute are a list of tests or branches, same as the top-level list.
`check`       | A script consisting of JavaScript code which defines the test that is to be run. Tests that pass should return **`true`** whereas failing tests should return a string with a message describing the failure. Note that it is valid to have both `check` and `children` in the same item.

### Test definitions

Tests can be defined as arbitrary JavaScript, which will be run in a sandbox. The sandboxed code does not have access to the filesystem or the `console` object.

The result of the test is the last expression evaluated by the script. This should be the primitive **`true`** if the test is successful, or a string describing the failure if it is not.

The script has access to the following variables:

#### `parameters`

An object containing all the parameter definitions for the project, for instance:

```javascript
{
  gunDepth: 6,
  gunDepthTolerance: 0.5,
  gunTiming: 0.9999,
  gunTimingSubarrayAverage: 0.5,
  gunPressureNominal: 2000,
  gunPressureToleranceRatio: 0.025,
  crosslineError: 12,
  crosslineErrorAverage: 9,
  inlineErrorRunningAverageValue: 2,
  inlineErrorRunningAverageShots: 40
}
```

#### `currentItem`

The item being iterated over. This can be of type `Shot`, `Sequence`, or `Preplot` depending on the value of this test's `iterate` attribute.

#### `shots`

An array of `Shot` objects.

Example:

```javascript
{
  type: 'final',
  _id: [ 7, 1764 ],
  sequence: 7,
  line: 5500,
  point: 1764,
  objref: 3,
  tstamp: "2020-09-07T04:10:13.680Z",
  hash: '2173892:1599458941.3070147:1599458941.3070147:80621034',
  geometry: '{"type":"Point","coordinates":[2.471571453,59.169725413]}',
  error_i: 2.7102108739654795,
  error_j: -0.06460360411324473,
  preplot_geometry: '{"type":"Point","crs":{"type":"name","properties":{"name":"EPSG:23031"}},"coordinates":[469883.4,6559284.9]}',
  raw_geometry: '{"type":"Point","crs":{"type":"name","properties":{"name":"EPSG:23031"}},"coordinates":[469881.87,6559285.76]}',
  final_geometry: '{"type":"Point","crs":{"type":"name","properties":{"name":"EPSG:23031"}},"coordinates":[469881.09,6559286.22]}',
  pp_meta: {},
  raw_meta: {
    smsrc: {
      guns: [Array],
      line: '1054980007S00000',
      mask: 38,
      shot: 1764,
      time: "b'20/09/07:04:10:13'",
      spare: '',
      header: '*SMSRC',
      spread: 3,
      volume: 3050,
      blk_siz: 2282,
      manifold: 2027,
      num_auto: 0,
      num_guns: 52,
      trg_mode: 'E',
      avg_delta: 0,
      num_delta: 0,
      std_delta: 0.073,
      baro_press: null,
      num_active: 52,
      num_nofire: 0,
      src_number: 2,
      num_subarray: 6
    }
  },
  final_meta: {},
  _: [Function]
}
```

#### `sequences`

An array of `Sequence` objects.

Example:

```javascript
{
    "_id": 9,
    "sequence": 9,
    "line": 5562,
    "fsp": 2580,
    "lsp": 996,
    "fsp_final": 2548,
    "lsp_final": 1000,
    "ts0": "2020-09-07T08:34:13.112Z",
    "ts1": "2020-09-07T10:47:49.116Z",
    "ts0_final": "2020-09-07T08:36:58.984Z",
    "ts1_final": "2020-09-07T10:47:28.608Z",
    "duration": {
        "hours": 2,
        "minutes": 13,
        "seconds": 36,
        "milliseconds": 4
    },
    "duration_final": {
        "hours": 2,
        "minutes": 10,
        "seconds": 29,
        "milliseconds": 624
    },
    "num_preplots": 775,
    "num_points": 775,
    "missing_shots": 0,
    "length": 19350.1845360761,
    "azimuth": 26.443105805883572,
    "remarks": "",
    "remarks_final": "",
    "status": "final"
    _: [Function]
}

```

#### `preplots`

An array of `Preplot` objects.

Example:

```javascript
{
  _id: [ null, 2348, 5130 ],
  line: 5130,
  point: 2348,
  class: 'V',
  ntba: false,
  geometry: '{"type":"Point","crs":{"type":"name","properties":{"name":"EPSG:23031"}},"coordinates":[470769.8,6550688.8]}',
  meta: {},
  count: 0
}
```

#### The `_` function

Each of the above objects has a function named `_`. This is a helper to quickly access own nested attributes without having to check if the attribute or one of its parents exist. For instance, on a `Shot` item, `currentItem._('raw_meta.smsrc')` will return the SmartSource gun data if it exists, or undefined if either `smsrc` or `raw_meta` are not defined.