This event handler checks if there is an UTC date jump between
consecutive shots. If a jump is detected, it sends to new entries
to the event log, for the last shot and first shot of the previous
and current dates, respectively.
Fixes#223.
Events being created or edited via the API now call
replace_placeholders() making it possible to use
shortcuts to enter some event-related information.
See #229 for details.
It uses https://github.com/debug-js/debug but it is meant to be
called like this:
const debug = require("DOUGAL_ROOT/debug")(__filename);
That way the calling module's path is used as the debug namespace.
This can be used as a shortcut when requiring a module from deep
within the file hierarchy, e.g., instead of:
require("../../../../lib/db");
one can do:
require("DOUGAL_ROOT/lib/db");
Related to #206. If this is indeed what is causing the alerts,
we will change the logic so that it simply logs (or ignores)
invalid speeds rather than throwing.
When importing an old project, the first QC run could take a while
and cause a bit of backlog, but during normal shooting it is expected
that it will finish quite quickly (and this is monitored anyway).
Commands used:
find . -type f -name '*.js'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
find . -type f -name '*.vue'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
find . -type f -name '*.py'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
QCs may be re-run for specific sequences or for a whole
project by defining an environment variable, as follows:
For an entire project:
* DOUGAL_FORCE_QC="project-id"
For specific sequences:
* DOUGAL_FORCE_QC="project-id sequence1 sequence2 … sequenceN"
When saving shot-by-shot results for a sequence,
*all* existing QC events for that sequence will be
removed first.
We do this because otherwise we may end up with QC
data for shots that no longer exist. Also, in the
case that we have QCed based on raw data, QC results
for shots which are not in the final data would stay
around even though those shots are no longer valid.
Using timestamps does not work as we might be
importing files with timestamps older than the
last QC run. Those would not be detected by a
timestamp based method but would be by this
digest based approach.
There is a project-wide digest and per sequence
digests. The former takes the path and hashes of
all files known to Dougal for this project (the
`files` table), concatenantes them and computes
the MD5 checksum. Sequence digests do the same
but only including the files related to that
sequence.
We return the QC definitions tree structure, augmented with
a `sequences` attribute which contains `raw_lines` tuples
which are in turn augmented with a `shots` attribute
containing `event_log` tuples. The whole structure looks
something like:
qc_test:
qc_test:
sequences:
- sequence0:
shots: [sp0, sp1, …]
- sequence1:
shots: [sp0, sp1, …]
qc_test:
sequences:
- sequence0:
shots: [sp0, sp1, …]
…
The results are now saved as follows:
For shot QCs, failing tests result in an event being created in
the event_log table. The text of the event is the QC result message,
while the labels are as set in the QC definition. It is conventionally
expected that these include a `QC` label. The event `meta` contains a
`qc_id` attribute with the ID of the failing QC.
For sequences, failing tests result in a `meta` entry under `qc`, with
the QC ID as the key and the result message as the value.
Finally, the project's `info` table still has a `qc` key, but unlike
with the old code, which stored all the QC results in a huge object
under this key, now only the timestamp of the last time a QC was run on
this project is stored, as `{ "updatedOn": timestamp }`.
The QCs are launched by calling the main() function in /lib/qc/index.js.
This function will first check the timestamp of the files imported into
the project and only run QCs if any of the file timestamps are later
than `info.qc.updatedOn`. Likewise, for each sequence, the timestamp of
the files conforming that sequence is checked against
`info.qc.updatedOn` and only those which are newer are actually
processed. This cuts down the running time very considerably.
The logic now is much easier on memory too, as it doesn't load the
whole project at once into memory. Instead, shotpoint QCs are processed
first, and for this a cursor is used, fetching one shotpoint at a
time. Then the sequence QCs are run, also one sequence at a time
(fetched via an individual query touching the `sequences_summary` view,
rather than via a cursor; we reuse some of the lib/db functions here),
for each sequence all its shotpoints and a list of missing shots are
also fetched (via lib/db function reuse) and passed to the QC functions
as predefined variables.
The logic of the QC functions is also changed. Now they can return:
* If a QC passes, the function MUST return boolean `true`.
* If a QC fails, the function MAY return a string describing the nature
of the failure, or in the case of an `iterate: sequence` type test,
it may return an object with these attributes:
- `remarks`: a string describing the nature of the failure;
- `labels`: a set of labels to associate with this failure;
- `shots`: a object in which each attribute denotes a shotpoint number
and the value consists of either a string or an object with
`remarks` (string), `labels` (array of strings) attributes. This allows
us to add detail about which shotpoints exactly contribute to cause a
sequence-wide test failure (this may not be applicable to every
sequence-wide QC) and it's also a handy way to detect and insert events
for missing shots.
* For QCs which may give false positives, such as missing gun data, a
new QC definition attribute is introduced: if `ignoreAllFailed` is
boolean `true` and all shots fail the test for a sequence, or all
sequences fail the test for a prospect, the results of the QC will be
ignored, as if the test had passed. This is mostly to deal with gun or
any other data that may be temporarily missing.
The comparison is strict non-equality so a null role cannot
be forced via the API.
The need for this is so that we can reuse this function to
save QC results, which is something that does not take
place over the API.
