Commit Graph

26 Commits

Author SHA1 Message Date
D. Berge
f708410e47 Conditionally enable / disable system import / export 2020-09-02 15:05:43 +02:00
D. Berge
a8830c9a05 Add system data import module 2020-09-02 15:05:43 +02:00
D. Berge
cf477c921e Add data export task to bin/runner.sh 2020-09-02 09:08:10 +02:00
D. Berge
4c103fafca Add system data export module.
It exports the data that is entered directly into
Dougal as opposed to being read from an external
source.

As of this commit, not all direct data is exported.
Specifically, sequence comments (raw and final),
sequence and shot NTBA and shot NTBP statuses are not
exported.
2020-09-02 09:07:06 +02:00
D. Berge
1fc069a0bd Fix path to NTBP regexp pattern 2020-09-01 10:59:29 +02:00
D. Berge
c4c09f0a9d Delete online data before importing sequence.
The deferred import routing will delete any online
data for any raw sequences that it imports.
2020-08-31 13:09:57 +02:00
D. Berge
5363ce8bab Handle NTBP flags in raw data.
This works by matching a regular expression against
the full path name (not just the file name). If it
matches, the NTBP flag is set to true.
2020-08-31 13:04:46 +02:00
D. Berge
c32c408d12 Import .profile into runner.sh.
So that it has access to the right environment
when run by cron.
2020-08-31 12:57:55 +02:00
D. Berge
54a717dc91 Do not fail hard if preplots do not exist.
They might not have been done yet, and it will be obvious
to the user if they're not there.
2020-08-29 13:14:20 +02:00
D. Berge
71e0326141 Add missing import 2020-08-29 10:40:55 +02:00
D. Berge
91d4059cdf Make send_alert usable as a module 2020-08-28 17:33:32 +02:00
D. Berge
22f806e81e Warn on import problems.
If there is a problem with files matching the capture
globs but not matching the file name regexp patterns,
these routines will emit a message to stderr and skip
the non-matching file.
2020-08-28 17:31:25 +02:00
D. Berge
639ac237e3 Replace Python globbing library.
This was motivated by the need to recurse down paths.
2020-08-28 13:49:43 +02:00
D. Berge
0c7b3146ae Update label definitions when updating survey config.
It will not delete any labels that have been removed
from the configuration, as those may be used, but it
will add new labels and modify existing ones if they
changed.
2020-08-25 17:52:17 +02:00
D. Berge
3ce6f1dce6 Add tasks runner script.
This script runs the deferred imports. It is meant to
be called from a cronjob at regular intervals – every
one or two minutes is probably a good setting.

It checks if another instance is already running before
doing its thing.

If anything goes wrong (any of the called processes exits
with non-zero condition) it will send an alert to GitLab,
provided that the authorisation key is known.
2020-08-24 21:01:47 +02:00
D. Berge
b4b06c8744 Instrument alerts for deferred imports.
In case of errors (or anything else of note), send_alert.py
can be used to push information to a GitLab alerts endpoint.

It is generic enough that it can be used with anything else, though.
2020-08-24 20:02:12 +02:00
D. Berge
bc784989f6 Make P1 imports not fail if called needlessly.
If their respective configuration keys are not
defined in a survey configuration, the import
routines will print an informational message
and exit successfully.
2020-08-24 19:45:59 +02:00
D. Berge
f2373a0454 Use environment variables for DB connection if possible 2020-08-12 18:18:47 +02:00
D. Berge
b1a088da36 Prefer environment variables to connect to DB 2020-08-12 14:38:31 +02:00
D. Berge
9c58ec30b5 Add function to import final P1/11.
Based on the analogous import_final_p190.py code.
2020-08-10 22:58:31 +02:00
D. Berge
3755564414 Add function to import raw P1/11.
Based on the analogous import_raw_p190.py code.
2020-08-10 22:58:31 +02:00
D. Berge
4c54b7002a Add DB functions to save raw and final data from P1/11 2020-08-10 22:58:31 +02:00
D. Berge
b36c8b97d1 Add P1/11 parsing Python library.
This is a super-simple library that does the minimum required
to get things going for the specific operations where this
code is foreseen to be used in the immediate future. It is not
and it does not aim to be a complete, generic or universal P1/11
parsing solution.
2020-08-10 22:50:07 +02:00
D. Berge
a0faa51602 Fix PREFIX usage 2020-08-10 22:41:37 +02:00
D. Berge
ed6b051669 Add DOUGAL_ROOT environment variable support.
When searching for configuration files, we look in the
following places:

1. $DOUGAL_ROOT/etc/
2. $HOME/software/etc/
3. ./software/etc/
2020-08-09 16:43:51 +02:00
D. Berge
4c5d29494c Initial commit 2020-08-08 23:59:13 +02:00