Using timestamps does not work as we might be
importing files with timestamps older than the
last QC run. Those would not be detected by a
timestamp based method but would be by this
digest based approach.
There is a project-wide digest and per sequence
digests. The former takes the path and hashes of
all files known to Dougal for this project (the
`files` table), concatenantes them and computes
the MD5 checksum. Sequence digests do the same
but only including the files related to that
sequence.
This provides essentially the same user experience as the old
endpoint, with one exception as of this commit:
* The user is not able to “accept” or “unaccept” QC events.
We return the QC definitions tree structure, augmented with
a `sequences` attribute which contains `raw_lines` tuples
which are in turn augmented with a `shots` attribute
containing `event_log` tuples. The whole structure looks
something like:
qc_test:
qc_test:
sequences:
- sequence0:
shots: [sp0, sp1, …]
- sequence1:
shots: [sp0, sp1, …]
qc_test:
sequences:
- sequence0:
shots: [sp0, sp1, …]
…
The results are now saved as follows:
For shot QCs, failing tests result in an event being created in
the event_log table. The text of the event is the QC result message,
while the labels are as set in the QC definition. It is conventionally
expected that these include a `QC` label. The event `meta` contains a
`qc_id` attribute with the ID of the failing QC.
For sequences, failing tests result in a `meta` entry under `qc`, with
the QC ID as the key and the result message as the value.
Finally, the project's `info` table still has a `qc` key, but unlike
with the old code, which stored all the QC results in a huge object
under this key, now only the timestamp of the last time a QC was run on
this project is stored, as `{ "updatedOn": timestamp }`.
The QCs are launched by calling the main() function in /lib/qc/index.js.
This function will first check the timestamp of the files imported into
the project and only run QCs if any of the file timestamps are later
than `info.qc.updatedOn`. Likewise, for each sequence, the timestamp of
the files conforming that sequence is checked against
`info.qc.updatedOn` and only those which are newer are actually
processed. This cuts down the running time very considerably.
The logic now is much easier on memory too, as it doesn't load the
whole project at once into memory. Instead, shotpoint QCs are processed
first, and for this a cursor is used, fetching one shotpoint at a
time. Then the sequence QCs are run, also one sequence at a time
(fetched via an individual query touching the `sequences_summary` view,
rather than via a cursor; we reuse some of the lib/db functions here),
for each sequence all its shotpoints and a list of missing shots are
also fetched (via lib/db function reuse) and passed to the QC functions
as predefined variables.
The logic of the QC functions is also changed. Now they can return:
* If a QC passes, the function MUST return boolean `true`.
* If a QC fails, the function MAY return a string describing the nature
of the failure, or in the case of an `iterate: sequence` type test,
it may return an object with these attributes:
- `remarks`: a string describing the nature of the failure;
- `labels`: a set of labels to associate with this failure;
- `shots`: a object in which each attribute denotes a shotpoint number
and the value consists of either a string or an object with
`remarks` (string), `labels` (array of strings) attributes. This allows
us to add detail about which shotpoints exactly contribute to cause a
sequence-wide test failure (this may not be applicable to every
sequence-wide QC) and it's also a handy way to detect and insert events
for missing shots.
* For QCs which may give false positives, such as missing gun data, a
new QC definition attribute is introduced: if `ignoreAllFailed` is
boolean `true` and all shots fail the test for a sequence, or all
sequences fail the test for a prospect, the results of the QC will be
ignored, as if the test had passed. This is mostly to deal with gun or
any other data that may be temporarily missing.
The comparison is strict non-equality so a null role cannot
be forced via the API.
The need for this is so that we can reuse this function to
save QC results, which is something that does not take
place over the API.
It replaces the `Actions` column in the old table and provides
more actions.
The user can now edit not just the comments and labels but also
the timestamp / shotpoint as requested in #78 (closes#78).
Because events are grouped by timestamp / shotpoint (each row
represents a unique timestamp or shotpoint), the behaviour is
slightly different depending on whether the user clicks on a
row containing a single (editable) event, or on one of multiple
editable events in the same row. Also, rows containing only
read-only events are recognised and no edition actions are
provided for those.
It implements the same functionality as in other tabs
such as sequences, lines, etc., but it is disabled here
because in my opinion it doesn't look too nice.
It will probably be a matter of enabling it at some point
and asking for feedback on user preference.
Returns a list of labels used in the current view,
in order of popularity (most used first).
NOTE: this property is not actually used. It's
technically dead code.
This assumes that adding or removing labels is a relatively
common action to do on an event and provides a quicker
and simpler mechanism than bringing up the full event
dialogue.
This is meant to be invoked from a context menu action or
similar.
When an event has been modified, this control opens a dialogue
where the previous version of the event may be reviewed and if
necessary restored.
Technically, this was the quid of and closes#138.
