Commit Graph

102 Commits

Author SHA1 Message Date
D. Berge
6162a5bdee Stop importing P1/90s until scripts are upgraded.
See #266.
2023-09-14 13:09:38 +02:00
D. Berge
ed91026319 Add tolltip and popup options to map layer configuration.
- `tooltip` takes the name of a GeoJSON property that will be
  shown in a tooltip when hovering the mouse over a feature.

- `popup` can take either the name of a property as above, or
  the boolean value `true`. In the latter case, a table of all
  the feature's properties will be shown when clicking on the
  feature. In the former case, only the value of the designated
  property will be shown.
2023-09-13 12:55:37 +02:00
D. Berge
441a4e296d Import map layers from the runner 2023-09-13 11:24:04 +02:00
D. Berge
f9a70e0145 Refactor map layer importer.
- Now a layer may consist of a path pointing to a directory plus a
  glob, or a path pointing directly to a single file.
- If a file already exists in the database, check if the layer
  name has changed and if so, update it.
- Do not import the actual file contents, as the path is enough
  (it can be retrieved via the /file/:path API endpoint).
2023-09-12 11:05:10 +02:00
D. Berge
b71489cee1 Add get_file_data() function to datastore 2023-09-12 11:04:37 +02:00
D. Berge
b2d1798338 Add map layer importer 2023-09-11 10:00:59 +02:00
D. Berge
5230ff63e3 Use new database API calls for configuration 2023-09-09 16:39:53 +02:00
D. Berge
2b364bbff7 Make bin script compatible with Python 3.6 2023-09-09 16:38:51 +02:00
D. Berge
53f2108e37 Adapt import functions to use logical paths 2023-08-30 14:56:09 +02:00
D. Berge
ccf4bbf547 Use logical paths rather than physical 2023-08-30 14:54:27 +02:00
D. Berge
c99a625b60 Add function to retrieve survey configurations from DB.
As the survey definitions will no longer be stored in files
under etc/surveys/ but directly on the database, this
function replaces configuration.surveys()
2023-08-30 14:27:15 +02:00
D. Berge
25ab623328 Add functions for translating paths.
The Dougal database will no longer store physical file paths
but rather logical ones, relative to (config.yaml).imports.paths.

These functions translate between physical and logical paths.
2023-08-30 14:17:47 +02:00
D. Berge
455888bdac Fix method signature 2023-08-30 14:16:08 +02:00
D. Berge
c3b3a4c70f Remove lock file if inhibiting tasks 2023-04-11 20:50:59 +02:00
D. Berge
8118641231 Do not run tasks if required mounts are not present.
A configuration item `imports.mounts` is added to
`etc/config.yaml`. This should be a list of paths
which must be non-empty. If any of the paths in that
list is empty, runner.sh will abort.

Closes #200.
2023-04-10 15:04:12 +02:00
D. Berge
04df9f41cc Add script for daily database housekeeping.
The script bin/daily_tasks.py is intended to be run shortly after
midnight every day (e.g., via cron).

At the moment it inserts any missing LDSP / FDSP events. It can
be extended with other tasks as needed either by expanding
Datastore.run_daily_tasks() or by adding to bin/daily_tasks.py.

Fixes #223.
2022-05-13 19:04:39 +02:00
D. Berge
632dd1ee75 Add placeholder replacement to log housekeeping tasks 2022-05-12 22:57:23 +02:00
D. Berge
bb5de9a00e Update insert_event.py.
This script now works with the new event log.

Fixes #234. Midnight positions can be added via a cronjob such
as:

$DOUGAL_ROOT/BIN/insert_event.py -t "$(date -I) 00:00:00Z" \
    -l Daily -l Prod \
    "Midnight position: @DMS@ (@POS@)"
2022-05-12 22:21:38 +02:00
D. Berge
f7b6ca3f79 Log runner output to syslog (if so configured).
The variable DOUGAL_LOG_FACILITY must be defined in the environment
(e.g., in ~/.dougalrc) for syslog to be enabled.
2022-05-08 15:30:05 +02:00
D. Berge
a7cce69c81 Add logging statements 2022-05-08 15:26:15 +02:00
D. Berge
2b20a5d69f Update line details on reimport conflict.
To deal with misnamed lines.

Fixes #240.
2022-05-08 15:25:11 +02:00
D. Berge
af6e419aab Run QCs from runner.
When importing an old project, the first QC run could take a while
and cause a bit of backlog, but during normal shooting it is expected
that it will finish quite quickly (and this is monitored anyway).
2022-05-01 21:26:10 +02:00
D. Berge
6516896bae Disable system imports in runner.
They're not really used. Will probably remove at a later date.
2022-05-01 21:24:56 +02:00
D. Berge
40d96230d2 Adjust planner times from runner.
Fixes #167.
2022-05-01 20:27:19 +02:00
D. Berge
fd41d2a6fa Launch database housekeeping tasks from runner 2022-05-01 20:10:27 +02:00
D. Berge
e3a3bdb153 Clean up whitespace.
Commands used:

find . -type f -name '*.js'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
find . -type f -name '*.vue'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
find . -type f -name '*.py'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
2022-04-29 14:48:21 +02:00
D. Berge
0e534b583c Do not assume that lines have remarks.
Fixes #202.
2022-04-29 14:32:46 +02:00
D. Berge
68322ef562 Fix misleading comment.
Use an EPSG code that is actually in the work area of the Dougal boats.
2022-04-26 17:36:48 +02:00
D. Berge
26033b2a37 Fix syntax error.
Introduced by commit ead938b40f.
2022-04-13 09:04:52 +02:00
D. Berge
ead938b40f Inhibit exports.
They don't seem to be used, and for backups it's better to
just back up the whole database instead, which is being done
remotely.
2022-03-18 13:32:43 +01:00
D. Berge
17bb88faf4 Cope with P1/11s with no S records 2022-03-07 21:08:22 +01:00
D. Berge
a52c7e91f5 Document in runner.sh how to run ASAQC in test mode 2022-03-07 21:07:20 +01:00
D. Berge
53ed096e1b Modify file hashing function.
We remove the inode from the hash as it is unstable when the
files are on an SMB filesystem, and replace it with an MD5
of the absolute file path.
2022-02-07 17:03:10 +01:00
D. Berge
51452c978a Add ASAQC task to runner 2021-10-04 21:26:13 +02:00
D. Berge
7aecb514db Clear QC metadata when importing gun data.
Fixes #118.
2021-05-26 00:30:58 +02:00
D. Berge
ad395aa6e4 Include the planned lines table in system dumps 2021-05-26 00:15:09 +02:00
D. Berge
523ec937dd Always merge metadata on import.
The INSERT INTO raw_lines / final_lines will not always be executed as
the lines may already exist (particularly in raw_lines because of
*online*), so whether it worked or not we merge the metadata immediately
afterwards (this may cause an extra notification to be fired).
2021-05-25 03:19:42 +02:00
D. Berge
bc5aef5144 Run post-import functions after final lines.
The reason why need to do it like this instead of relying on a trigger
is because the entry in final_lines is created first and the final_shots
are populated. If we first the trigger on final_lines it is not going
to find any shots; if we fire it as a row trigger on final_shots it
would try to label every point in sequence as it is imported; finally if
we fire it as a statement trigger on final_shots we have no idea which
sequence was imported.
2021-05-24 16:59:56 +02:00
D. Berge
9875ae86f3 Record P1/11 line name in database on import 2021-05-24 13:30:25 +02:00
D. Berge
67af85eca9 Recognise PENDING status in sequence imports.
If a final sequence file or directory name matches a pattern
which is recognised to indicate a ‘pending acceptance’ status,
the final data (if any exists) for that sequence will be deleted
and a comment added to the effect that the sequence has been
marked as ‘pending’.

To accept the sequence, rename its final file or directory name
accordingly.

Note: it is the *final* data that is searched for a matching
pattern, not the raw.

Closes #91.
2021-05-21 15:15:15 +02:00
D. Berge
779b28a331 Add info table to system dumps 2021-05-21 12:18:36 +02:00
D. Berge
b1e152179e Add new command: insert_event.py
Used to insert a timed event in the log.
2021-05-15 01:56:49 +02:00
D. Berge
838b45ef26 Do not fail if some data files are missing 2021-05-15 01:51:55 +02:00
D. Berge
f1cbbdb56b Check if raw P1/11 has records.
Even if it hasn't, the file gets imported anyway
(into the `files` table) but we exit early to avoid
an error when trying to determine shooting direction.

This check is not necessary for final P1/11 or gun data.

Fixes #104.
2021-05-12 20:35:51 +02:00
D. Berge
0fdb42c593 Do not import files that have just been modified.
We now check that a file is at least a few seconds old
before attempting to import it.

The actual minimum age can be configured in etc/config.yaml or
else is defaults to 10 seconds.

The idea is that this should give the OS enough time to fully
write the file before we import it.

The timestamp being looked at is the modification time.

Fixes #92.
2021-05-07 13:50:32 +02:00
D. Berge
83d966c4b7 Do not import gun data into online shots.
These shots will be deleted when the raw P1 is
imported deleting the gun data, but the gun data
file will show as having been imported.

(#29)
2020-10-06 18:33:59 +02:00
D. Berge
3ed5558490 Fix query to account for new column type.
The type of the file_data.data column was
changed from JSON to JSONB by commit
2d270cdef9.

Fixes #54.
2020-10-04 04:29:14 +02:00
D. Berge
35a6a9188a Add missing shots to structured sequence export files.
Closes #8, #9, #12.
2020-10-04 03:51:30 +02:00
D. Berge
700e683022 Import preplot file configuration into database.
We do this so that we can look for the "saillineOffset"
parameter, which we expect to be present in source
preplot imports and allows us to correlate source
and sail lines.

The change to bin/sps.py is necessary to let the JSON
serialisation take place.
2020-10-04 03:42:19 +02:00
D. Berge
0a684cd02a Add option to import file data for preplot files 2020-10-04 03:41:20 +02:00