As the survey definitions will no longer be stored in files
under etc/surveys/ but directly on the database, this
function replaces configuration.surveys()
The Dougal database will no longer store physical file paths
but rather logical ones, relative to (config.yaml).imports.paths.
These functions translate between physical and logical paths.
A configuration item `imports.mounts` is added to
`etc/config.yaml`. This should be a list of paths
which must be non-empty. If any of the paths in that
list is empty, runner.sh will abort.
Closes#200.
The script bin/daily_tasks.py is intended to be run shortly after
midnight every day (e.g., via cron).
At the moment it inserts any missing LDSP / FDSP events. It can
be extended with other tasks as needed either by expanding
Datastore.run_daily_tasks() or by adding to bin/daily_tasks.py.
Fixes#223.
This script now works with the new event log.
Fixes#234. Midnight positions can be added via a cronjob such
as:
$DOUGAL_ROOT/BIN/insert_event.py -t "$(date -I) 00:00:00Z" \
-l Daily -l Prod \
"Midnight position: @DMS@ (@POS@)"
When importing an old project, the first QC run could take a while
and cause a bit of backlog, but during normal shooting it is expected
that it will finish quite quickly (and this is monitored anyway).
Commands used:
find . -type f -name '*.js'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
find . -type f -name '*.vue'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
find . -type f -name '*.py'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
The INSERT INTO raw_lines / final_lines will not always be executed as
the lines may already exist (particularly in raw_lines because of
*online*), so whether it worked or not we merge the metadata immediately
afterwards (this may cause an extra notification to be fired).
The reason why need to do it like this instead of relying on a trigger
is because the entry in final_lines is created first and the final_shots
are populated. If we first the trigger on final_lines it is not going
to find any shots; if we fire it as a row trigger on final_shots it
would try to label every point in sequence as it is imported; finally if
we fire it as a statement trigger on final_shots we have no idea which
sequence was imported.
If a final sequence file or directory name matches a pattern
which is recognised to indicate a ‘pending acceptance’ status,
the final data (if any exists) for that sequence will be deleted
and a comment added to the effect that the sequence has been
marked as ‘pending’.
To accept the sequence, rename its final file or directory name
accordingly.
Note: it is the *final* data that is searched for a matching
pattern, not the raw.
Closes#91.
Even if it hasn't, the file gets imported anyway
(into the `files` table) but we exit early to avoid
an error when trying to determine shooting direction.
This check is not necessary for final P1/11 or gun data.
Fixes#104.
We now check that a file is at least a few seconds old
before attempting to import it.
The actual minimum age can be configured in etc/config.yaml or
else is defaults to 10 seconds.
The idea is that this should give the OS enough time to fully
write the file before we import it.
The timestamp being looked at is the modification time.
Fixes#92.
We do this so that we can look for the "saillineOffset"
parameter, which we expect to be present in source
preplot imports and allows us to correlate source
and sail lines.
The change to bin/sps.py is necessary to let the JSON
serialisation take place.
Doing otherwise will result in the gun data file
appearing has having been read, but no data will
have been saved as there was nowhere to save to.
Fixes#29.
Each of the save_* operations starts a transaction
(which is automatically commited if all goes well).
The main reason for this is to ensure that by the
time raw_lines and final_lines events fire, the
corresponding entries in raw_shots and final_shots
have already been populated.
Unlike system_imports.py and system_exports.py, which
deal with whole tables via COPY, this allows us to
export / import *either* whole tables or specific
columns only.
The data will be exported to text files containing
the selected columns + the primary key columns for
the table.
When importing, those tables for which a selection
of columns was exported must already be populated.
The import process will overwrite the data of the
non primary key columns it knows about. If whole
tables are exported, on the other hand, when
re-importing rows will be appended rather than
updated. It is the user's responsibility to make
sure that this will not cause any conflicts.
