Commit Graph

125 Commits

Author SHA1 Message Date
D. Berge
f6faad17db Fix Python's idiotic syntax 2025-08-07 17:17:43 +02:00
D. Berge
7f5f64acb1 Check for lineNameInfo when importing P1/11 2025-08-07 11:00:42 +02:00
D. Berge
8f87df1e2f Comment out debug output 2025-08-07 10:52:13 +02:00
D. Berge
cfa771a830 Skip info for saillines with no preplot.
It may happen that the sailline info file has data for more lines
than are actually in the preplot (e.g., if importing a reduced
preplot file). In this case, we don't want a constraint violation
error due to missing corresponding lines in `preplot_lines` so we
check for that and only import lines that do exist in `preplot_lines`
2025-07-19 11:31:52 +02:00
D. Berge
7a7106e735 Default to text if no field type is specified. 2024-08-22 18:44:24 +02:00
D. Berge
d5a10ca273 Allow also str as a field type specifier 2024-08-22 18:43:57 +02:00
D. Berge
bc5dfe9c2a Add fixed strings support to parse_line 2024-05-05 19:34:01 +02:00
D. Berge
524420d945 Support lineNameInfo in SmartSource header imports.
Closes #305.
2024-05-04 17:41:14 +02:00
D. Berge
e48c734ea9 Support lineNameInfo in final P1/11 imports 2024-05-04 17:35:05 +02:00
D. Berge
5aaad01238 Support lineNameInfo in raw P1/11 imports 2024-05-04 17:33:50 +02:00
D. Berge
90782c1b09 Support import of preplot lines ancillary information.
Closes #264.
2024-05-04 17:32:30 +02:00
D. Berge
995e0b9f81 Remove unused import 2024-05-03 11:46:16 +02:00
D. Berge
3488c8bf4d Support preplot imports in additional formats.
This adds support for SPS v1, SPS v2.1, custom fixed-width,
CSV and custom sailline info preplot imports.
2024-05-03 11:44:32 +02:00
D. Berge
7e1023f6e8 Support import of delimited formats.
This supports CSV and similar formats, as well as sailline
imports, which is a CSV file with a specific set of column
definitions.

Does not yet support P111 import (for which there is an
implementation already).
2024-05-03 11:42:20 +02:00
D. Berge
41e058ac64 Add TODO comment 2024-05-03 11:41:50 +02:00
D. Berge
2086133109 Fix bool casting.
A true value is any text that starts with `t` (case insensitive) or
any non-zero integer.
2024-05-03 11:40:53 +02:00
D. Berge
bb70cf1a3d Check enum keys against text instead of cast value 2024-05-03 11:40:21 +02:00
D. Berge
be0d7b269f Support import of various fixed-width formats.
This supports reading: SPSv1, SPSv2.1, P190 and custom
fixed-width formats. Supports skipping lines by startswith()
matching or by complete match (e.g., "EOF").

Closes #300 (SPS v1)
Closes #301 (SPS v2.1)
Closes #302 (P1/90)
2024-05-01 10:47:54 +02:00
D. Berge
934b921f69 Include schema in returned survey configuration object 2024-05-01 10:40:04 +02:00
D. Berge
8ec918bc7c Rename "SPS" preplots import to "legacy fixed width" 2024-05-01 10:40:04 +02:00
D. Berge
9388cd4861 Make daily_tasks work with new project configuration 2023-09-30 20:36:46 +02:00
D. Berge
947faf8c05 Provide default glob specification for map layer imports 2023-09-24 11:34:10 +02:00
D. Berge
a948556455 Fail gracefully if map layer data does not exist.
Fixes #272.
2023-09-24 11:33:32 +02:00
D. Berge
6162a5bdee Stop importing P1/90s until scripts are upgraded.
See #266.
2023-09-14 13:09:38 +02:00
D. Berge
ed91026319 Add tolltip and popup options to map layer configuration.
- `tooltip` takes the name of a GeoJSON property that will be
  shown in a tooltip when hovering the mouse over a feature.

- `popup` can take either the name of a property as above, or
  the boolean value `true`. In the latter case, a table of all
  the feature's properties will be shown when clicking on the
  feature. In the former case, only the value of the designated
  property will be shown.
2023-09-13 12:55:37 +02:00
D. Berge
441a4e296d Import map layers from the runner 2023-09-13 11:24:04 +02:00
D. Berge
f9a70e0145 Refactor map layer importer.
- Now a layer may consist of a path pointing to a directory plus a
  glob, or a path pointing directly to a single file.
- If a file already exists in the database, check if the layer
  name has changed and if so, update it.
- Do not import the actual file contents, as the path is enough
  (it can be retrieved via the /file/:path API endpoint).
2023-09-12 11:05:10 +02:00
D. Berge
b71489cee1 Add get_file_data() function to datastore 2023-09-12 11:04:37 +02:00
D. Berge
b2d1798338 Add map layer importer 2023-09-11 10:00:59 +02:00
D. Berge
5230ff63e3 Use new database API calls for configuration 2023-09-09 16:39:53 +02:00
D. Berge
2b364bbff7 Make bin script compatible with Python 3.6 2023-09-09 16:38:51 +02:00
D. Berge
53f2108e37 Adapt import functions to use logical paths 2023-08-30 14:56:09 +02:00
D. Berge
ccf4bbf547 Use logical paths rather than physical 2023-08-30 14:54:27 +02:00
D. Berge
c99a625b60 Add function to retrieve survey configurations from DB.
As the survey definitions will no longer be stored in files
under etc/surveys/ but directly on the database, this
function replaces configuration.surveys()
2023-08-30 14:27:15 +02:00
D. Berge
25ab623328 Add functions for translating paths.
The Dougal database will no longer store physical file paths
but rather logical ones, relative to (config.yaml).imports.paths.

These functions translate between physical and logical paths.
2023-08-30 14:17:47 +02:00
D. Berge
455888bdac Fix method signature 2023-08-30 14:16:08 +02:00
D. Berge
c3b3a4c70f Remove lock file if inhibiting tasks 2023-04-11 20:50:59 +02:00
D. Berge
8118641231 Do not run tasks if required mounts are not present.
A configuration item `imports.mounts` is added to
`etc/config.yaml`. This should be a list of paths
which must be non-empty. If any of the paths in that
list is empty, runner.sh will abort.

Closes #200.
2023-04-10 15:04:12 +02:00
D. Berge
04df9f41cc Add script for daily database housekeeping.
The script bin/daily_tasks.py is intended to be run shortly after
midnight every day (e.g., via cron).

At the moment it inserts any missing LDSP / FDSP events. It can
be extended with other tasks as needed either by expanding
Datastore.run_daily_tasks() or by adding to bin/daily_tasks.py.

Fixes #223.
2022-05-13 19:04:39 +02:00
D. Berge
632dd1ee75 Add placeholder replacement to log housekeeping tasks 2022-05-12 22:57:23 +02:00
D. Berge
bb5de9a00e Update insert_event.py.
This script now works with the new event log.

Fixes #234. Midnight positions can be added via a cronjob such
as:

$DOUGAL_ROOT/BIN/insert_event.py -t "$(date -I) 00:00:00Z" \
    -l Daily -l Prod \
    "Midnight position: @DMS@ (@POS@)"
2022-05-12 22:21:38 +02:00
D. Berge
f7b6ca3f79 Log runner output to syslog (if so configured).
The variable DOUGAL_LOG_FACILITY must be defined in the environment
(e.g., in ~/.dougalrc) for syslog to be enabled.
2022-05-08 15:30:05 +02:00
D. Berge
a7cce69c81 Add logging statements 2022-05-08 15:26:15 +02:00
D. Berge
2b20a5d69f Update line details on reimport conflict.
To deal with misnamed lines.

Fixes #240.
2022-05-08 15:25:11 +02:00
D. Berge
af6e419aab Run QCs from runner.
When importing an old project, the first QC run could take a while
and cause a bit of backlog, but during normal shooting it is expected
that it will finish quite quickly (and this is monitored anyway).
2022-05-01 21:26:10 +02:00
D. Berge
6516896bae Disable system imports in runner.
They're not really used. Will probably remove at a later date.
2022-05-01 21:24:56 +02:00
D. Berge
40d96230d2 Adjust planner times from runner.
Fixes #167.
2022-05-01 20:27:19 +02:00
D. Berge
fd41d2a6fa Launch database housekeeping tasks from runner 2022-05-01 20:10:27 +02:00
D. Berge
e3a3bdb153 Clean up whitespace.
Commands used:

find . -type f -name '*.js'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
find . -type f -name '*.vue'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
find . -type f -name '*.py'| while read FILE; do if echo $FILE |grep -qv node_modules; then sed -ri 's/^\s+$//' "$FILE"; fi; done
2022-04-29 14:48:21 +02:00
D. Berge
0e534b583c Do not assume that lines have remarks.
Fixes #202.
2022-04-29 14:32:46 +02:00