Commands used:
find . -type f -name '*.js'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
find . -type f -name '*.vue'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
find . -type f -name '*.py'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
QCs may be re-run for specific sequences or for a whole
project by defining an environment variable, as follows:
For an entire project:
* DOUGAL_FORCE_QC="project-id"
For specific sequences:
* DOUGAL_FORCE_QC="project-id sequence1 sequence2 … sequenceN"
When saving shot-by-shot results for a sequence,
*all* existing QC events for that sequence will be
removed first.
We do this because otherwise we may end up with QC
data for shots that no longer exist. Also, in the
case that we have QCed based on raw data, QC results
for shots which are not in the final data would stay
around even though those shots are no longer valid.
Using timestamps does not work as we might be
importing files with timestamps older than the
last QC run. Those would not be detected by a
timestamp based method but would be by this
digest based approach.
There is a project-wide digest and per sequence
digests. The former takes the path and hashes of
all files known to Dougal for this project (the
`files` table), concatenantes them and computes
the MD5 checksum. Sequence digests do the same
but only including the files related to that
sequence.
We return the QC definitions tree structure, augmented with
a `sequences` attribute which contains `raw_lines` tuples
which are in turn augmented with a `shots` attribute
containing `event_log` tuples. The whole structure looks
something like:
qc_test:
qc_test:
sequences:
- sequence0:
shots: [sp0, sp1, …]
- sequence1:
shots: [sp0, sp1, …]
qc_test:
sequences:
- sequence0:
shots: [sp0, sp1, …]
…
The results are now saved as follows:
For shot QCs, failing tests result in an event being created in
the event_log table. The text of the event is the QC result message,
while the labels are as set in the QC definition. It is conventionally
expected that these include a `QC` label. The event `meta` contains a
`qc_id` attribute with the ID of the failing QC.
For sequences, failing tests result in a `meta` entry under `qc`, with
the QC ID as the key and the result message as the value.
Finally, the project's `info` table still has a `qc` key, but unlike
with the old code, which stored all the QC results in a huge object
under this key, now only the timestamp of the last time a QC was run on
this project is stored, as `{ "updatedOn": timestamp }`.
The QCs are launched by calling the main() function in /lib/qc/index.js.
This function will first check the timestamp of the files imported into
the project and only run QCs if any of the file timestamps are later
than `info.qc.updatedOn`. Likewise, for each sequence, the timestamp of
the files conforming that sequence is checked against
`info.qc.updatedOn` and only those which are newer are actually
processed. This cuts down the running time very considerably.
The logic now is much easier on memory too, as it doesn't load the
whole project at once into memory. Instead, shotpoint QCs are processed
first, and for this a cursor is used, fetching one shotpoint at a
time. Then the sequence QCs are run, also one sequence at a time
(fetched via an individual query touching the `sequences_summary` view,
rather than via a cursor; we reuse some of the lib/db functions here),
for each sequence all its shotpoints and a list of missing shots are
also fetched (via lib/db function reuse) and passed to the QC functions
as predefined variables.
The logic of the QC functions is also changed. Now they can return:
* If a QC passes, the function MUST return boolean `true`.
* If a QC fails, the function MAY return a string describing the nature
of the failure, or in the case of an `iterate: sequence` type test,
it may return an object with these attributes:
- `remarks`: a string describing the nature of the failure;
- `labels`: a set of labels to associate with this failure;
- `shots`: a object in which each attribute denotes a shotpoint number
and the value consists of either a string or an object with
`remarks` (string), `labels` (array of strings) attributes. This allows
us to add detail about which shotpoints exactly contribute to cause a
sequence-wide test failure (this may not be applicable to every
sequence-wide QC) and it's also a handy way to detect and insert events
for missing shots.
* For QCs which may give false positives, such as missing gun data, a
new QC definition attribute is introduced: if `ignoreAllFailed` is
boolean `true` and all shots fail the test for a sequence, or all
sequences fail the test for a prospect, the results of the QC will be
ignored, as if the test had passed. This is mostly to deal with gun or
any other data that may be temporarily missing.
The comparison is strict non-equality so a null role cannot
be forced via the API.
The need for this is so that we can reuse this function to
save QC results, which is something that does not take
place over the API.
This reports the current server version, from Git by
default.
Also, and of more interest, it reports whether the
current database schema is compatible with the
server code.
This code implements the backend processing side
of the ASAQC queue, i.e., the bit that communicates
with the remote API.
Its expected use it to have it running at regular
intervals, e.g., via cron. The entry point is:
lib/www/server/queues/asaqc/index.js
That file is executable and can be run directly
from the shell or within a script. Read the comments
in that file for further instructions.
These functions, in general following the same HTTP-verb
approach as the rest of the database interface, are for
use with both the HTTP API and the queue processor.
