Commit Graph

183 Commits

Author SHA1 Message Date
D. Berge
ad3998d4c6 Add database upgrade file 2021-05-24 17:41:11 +02:00
D. Berge
8638f42e6d Add database upgrade files.
These files contain the sequence of SQL commands needed to bring
a database or project schema up to date with the latest template
database or project schema.

These files must be applied manually. Check the comments at the top of
the file for instructions.
2021-05-24 17:39:01 +02:00
D. Berge
4d97784829 Upgrade database project schema template.
Adds:

* label_in_sequence (_sequence integer, _label text):
  Returns events containing the specified label.

* handle_final_line_events (_seq integer, _label text, _column text):
  - If _label does not exist in the events for sequence _seq:
    it adds a new _label label at the shotpoint obtained from
    final_lines_summary[_column].
  - If _label does exist (and hasn't been auto-added by this function
    in a previous run), it will add information about it to the final
    line's metadata.

* final_line_post_import (_seq integer):
  Calls handle_final_line_events() on the given sequence to check
  for FSP, FGSP, LGSP and LSP labels.

* events_seq_labels_single ():
  Trigger function to ensure that labels that have the attribute
  `model.multiple` set to `false` occur at most only once per
  sequence. If a new instance is added to a sequence, the previous
  instance is deleted.

* Trigger on events_seq_labels that calls events_seq_labels_single().

* Trigger on events_timed_labels that calls events_seq_labels_single().
2021-05-24 16:49:39 +02:00
D. Berge
53f71f7005 Set primary key on events_seq_labels in schema template 2021-05-23 22:27:00 +02:00
D. Berge
5de64e6b45 Add meta column to events view in schema template 2021-05-23 22:26:00 +02:00
D. Berge
1649de6c68 Update default sequence HTML template 2021-05-20 20:37:37 +02:00
D. Berge
c832d8b107 Commit default template for sequences 2021-05-20 18:35:56 +02:00
D. Berge
7917eeeb0b Add table info to schema.
This one is independent of any projects so it goes
into `public`.
2021-05-20 18:07:05 +02:00
D. Berge
76c9c3ef2a Assign (some) offline navdata to a survey.
There is no concept of ‘current survey’ in Dougal, and
assigning navigation data to a particular survey is full
of edge cases but sometimes it is necessary or at least
convenient to do so.

This commit implements once such strategy, which consists
of checking the distance to the preplots of all active
surveys (well, those that do have preplots anyway) and
picking the nearest one.

To reduce load, we only do this every once in a while as
governed by the `offline_survey_detect_interval` option
in the configuration.

This strategy is only active if the configuration option
`offline_survey_heuristics == "nearest_preplot"` for the
corresponding navigation header.
2021-05-16 03:16:19 +02:00
D. Berge
df3a0b4c50 Be explicit about what type of data is being QC'ed.
The source deviation QCs now tell the user whether raw
or final data is being QC'ed.
2021-05-07 21:29:39 +02:00
D. Berge
f87aa08246 Check if gun data missing for entire line.
The `sequences` object now carries the attribute
`has_smsrc_data`, a boolean which is true iff
there is at least one `smsrc` record in the raw
shots metadata.

This is used by:

1. A new sequence-wise test which reports if gun
   data is missing for the entire sequence.

2. The individual `missing_gun_data` test which
   is inhibited if `has_smsrc_data` for the
   corresponding sequence is false.

Closes #93.
2021-05-07 14:04:48 +02:00
D. Berge
0fdb42c593 Do not import files that have just been modified.
We now check that a file is at least a few seconds old
before attempting to import it.

The actual minimum age can be configured in etc/config.yaml or
else is defaults to 10 seconds.

The idea is that this should give the OS enough time to fully
write the file before we import it.

The timestamp being looked at is the modification time.

Fixes #92.
2021-05-07 13:50:32 +02:00
D. Berge
250ffe243d Fix JWT token time to live.
Now half an hour.
2020-10-23 14:49:52 +02:00
D. Berge
ab8a66bdcf Set JWT default options 2020-10-11 17:58:41 +02:00
D. Berge
6e7ba82ed3 Add planner elements to DB schema 2020-10-09 13:54:45 +02:00
D. Berge
739cf4b9ec Add missing triggers in preplot, raw and final tables 2020-10-06 19:19:35 +02:00
D. Berge
afa34867f5 Modify functions accessing file_data to return jsonb.
Following schema change in 2d270cdef9.
2020-10-04 05:13:17 +02:00
D. Berge
3de7d5d334 Re-export schema template.
No changes, it merely shuffles the position of raw_lines
in the export.
2020-10-04 04:07:50 +02:00
D. Berge
3c215e9973 Fix error with definition of missing_final_points view 2020-10-04 04:06:15 +02:00
D. Berge
362d9dc1a5 Re-export schema template.
No changes, it merely shuffles the position of raw_lines
in the export.
2020-10-04 03:37:20 +02:00
D. Berge
1249c976ef Add views related to missing shots and outstanding production 2020-10-04 03:33:23 +02:00
D. Berge
2d270cdef9 Change file_data column type from JSON to JSONB 2020-10-04 01:22:42 +02:00
D. Berge
198d0072d4 Show sequence remarks + NTBP status on map.
Closes #50.
2020-10-02 17:58:44 +02:00
D. Berge
11a5020004 Remove old QCs when importing final shots.
When final shots are inserted, updated or
deleted, the corresponding QC info (which
is always held in raw_shots.meta->'qc')
is deleted.

If applicable, it will be recreated on the
next QC run.
2020-10-02 17:30:42 +02:00
D. Berge
c12c2a3861 Update events view.
Searches for timed events geometry within
the event's own metadata.
2020-09-30 22:49:01 +02:00
D. Berge
21439fdd3e Associate a position with timed events if possible.
When inserting or updating a timed event, a trigger
searches the real time events table for a position
close (within one minute) of the event time and
adds it to the event's metadata.
2020-09-30 22:45:50 +02:00
D. Berge
78adb2bef7 Make set_survey argument case insensitive 2020-09-26 22:53:48 +02:00
D. Berge
77aae68603 Include project ID in DB notifications 2020-09-25 18:27:59 +02:00
D. Berge
9470f41f4b Fix lookup of timed event labels.
Fixes #44.
2020-09-24 15:32:25 +02:00
D. Berge
4dc1c7df8e Re-export schema template.
No changes to the actual schema.
2020-09-24 15:31:53 +02:00
D. Berge
80324130f9 Return event geometries as GeoJSON 2020-09-24 15:24:47 +02:00
D. Berge
a4ed7f7b62 Return all QC events, not just those with labels 2020-09-23 20:19:13 +02:00
D. Berge
33c23c1239 Return event labels directly from events view.
This speeds up the query by orders of magnitude.
2020-09-23 16:31:55 +02:00
D. Berge
6bab21bce4 Fix QC test definition 2020-09-23 15:45:41 +02:00
D. Berge
ae0052de0c Add id to QC tests.
Each test has to have a unique id, in order that
we can associate results and labels with them and
cope with reorganisation of the QC tree.
2020-09-20 17:00:52 +02:00
D. Berge
94c3ed1584 Update schema.
* Report also schema in projects_summary
* Add notify triggers to events_timed, events_seq
2020-09-14 23:54:48 +02:00
D. Berge
1348eb9a8a Rename QCFail label to QC 2020-09-14 20:45:34 +02:00
D. Berge
c9b9a009af Speed up the events view.
We do this with the help of some denormalisation
and a handy trigger.

Closes #35.
2020-09-14 03:01:55 +02:00
D. Berge
b5b91d41c9 Reset event serial ids after re-import.
When the database is recreated, the sequences
used in the events_timed and events_seq tables
will be at their initial values, which will
almost certainly conflict with existing data
when it is imported via COPY.

With this commit, we set the current value for
those sequences to something usable.

Fixes #33.
2020-09-14 01:36:40 +02:00
D. Berge
da7a977c59 Change definition of Preplot QC objects 2020-09-12 22:25:45 +02:00
D. Berge
13d4771589 Add description of QC tests 2020-09-12 22:07:08 +02:00
D. Berge
52cdc8904b QC: Add gun deltas warning check 2020-09-12 21:22:34 +02:00
D. Berge
69f43c129b Remove spurious line from QC definition 2020-09-12 21:22:13 +02:00
D. Berge
8a5d103754 Fix typo 2020-09-12 20:59:20 +02:00
D. Berge
1d3d202d1f QC: change threshold of inline moving average check 2020-09-12 19:56:34 +02:00
D. Berge
cd76df9329 QC: check inline position shot by shot 2020-09-12 19:56:00 +02:00
D. Berge
5e130c3e42 QC: measure gun not manifold pressure 2020-09-12 19:54:01 +02:00
D. Berge
6d5167c052 Add more QCs 2020-09-12 19:17:14 +02:00
D. Berge
17d8041945 Update standard QC definitions + parameters 2020-09-10 23:36:13 +02:00
D. Berge
8841ffc10b Add QC definitions and parameters for Siddis surveys.
These are incomplete but meant to serve as a starting
point for other projects.
2020-09-10 23:20:58 +02:00