Class: DwcOccurrence

Inherits:
ApplicationRecord show all
Includes:
Housekeeping
Defined in:
app/models/dwc_occurrence.rb

Overview

A Darwin Core Record for the Occurrence core. Field generated from Ruby dwc-meta, which references the same spec that is used in the IPT, and the Dwc Assistant. Each record references a specific CollectionObject, AssertedDistribution, or FieldOccurrence.

Important: This is a cache/index, data here are periodically destroyed and regenerated from multiple tables in TW.

DWC attributes are camelCase to facilitate matching dwcClass is a replacement for the Rails reserved 'Class'

All DC attributes (attributes not in DwcOccurrence::TW_ATTRIBUTES) in this table are namespaced to dc ("http://purl.org/dc/terms/", "http://rs.tdwg.org/dwc/terms/")

README:

There is a two part strategy to building the index. 1) An individual record will rebuild on request with `parameter to collection_objects/123/dwc*?build=true`.
2) Wipe, and rebuild on some schedule. It would in theory be possible to track and rebuild when a class of every property was created (or updated), however
this is a lot of overhead to inject/code for a lot of models. It would inject latency at numerous stages that would perhaps impact UI performance.

Several terms are introduced in code:

  • ghost - A DwcOccurrence record whose dwc_occurrence object has been destroyed (i.e. an error in cleanup, should ideally never happen)
  • stale - an aproximation checking to see that the time of build of related records is older than the current index
  • flagged (for rebuild) - a record related to the dwc_occurrence_object(s) has been updated, triggering the need for re-indexing 1 or more records

TODO: The basisOfRecord CVTs are not super informative. We know collection object is definitely 1:1 with PreservedSpecimen, however AssertedDistribution could be HumanObservation (if source is person), or ... what? if its a published record. Seems we need a 'PublishedAssertation', just like we model the data.

Gotchas.

  • updated_at is set by touching the record, not via housekeeping.

Constant Summary collapse

DC_NAMESPACE = 
'http://rs.tdwg.org/dwc/terms/'.freeze
TW_ATTRIBUTES =

Not yet implemented, but likely needed (at an even higher level) ? :id

[
  :id,
  :project_id,
  :created_at,
  :updated_at,
  :created_by_id,
  :updated_by_id,
  :dwc_occurrence_object_type,
  :dwc_occurrence_object_id
].freeze
API_EXCLUDED_ATTRIBUTES = 
%w[
  created_by_id
  updated_by_id
].freeze
HEADER_CONVERTERS = 
{
  'dwcClass' => 'class',
}.freeze
NOMENCLATURE_RANKS =

Supported ranks (fields in db)

[
  :kingdom,
  :phylum,
  :dwcClass,
  :order,
  :superfamily,
  :family,
  :subfamily,
  :tribe,
  :subtribe,
  :genus,
  :subgenus,
  :specificEpithet
].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Housekeeping

#has_polymorphic_relationship?

Methods inherited from ApplicationRecord

transaction_with_retry

Instance Attribute Details

#occurrence_identifier ⇒ Object

Returns the value of attribute occurrence_identifier.



96
97
98
 
# File 'app/models/dwc_occurrence.rb', line 96

def occurrence_identifier
  @occurrence_identifier
end

Class Method Details

.annotates? ⇒ Boolean

Returns:

  • (Boolean) 


164
165
166
 
# File 'app/models/dwc_occurrence.rb', line 164

def self.annotates?
  false
end

.api_columns ⇒ Object 



129
130
131
 
# File 'app/models/dwc_occurrence.rb', line 129

def self.api_columns
  column_names - API_EXCLUDED_ATTRIBUTES
end

.by_collection_object_filter(filter_scope: nil, project_id: nil) ⇒ Object

TODO: use filters Return scopes by a collection object filter



193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
 
# File 'app/models/dwc_occurrence.rb', line 193

def self.by_collection_object_filter(filter_scope: nil, project_id: nil)
  return DwcOccurrence.none if project_id.nil? || filter_scope.nil?

  c = ::CollectionObject.arel_table
  d = arel_table

  # TODO: hackish
  k = ::CollectionObject.select('coscope.id').from( '(' + filter_scope.to_sql + ') as coscope ' )

  a = self.object_join('CollectionObject')
    .where('dwc_occurrences.project_id = ?', project_id)
    .where(dwc_occurrence_object_id: k)
    .select(::DwcOccurrence.target_occurrence_columns) # TODO !! Will have to change when AssertedDistribution and other types merge in
  a
end

.object_join(target) ⇒ Object 



168
169
170
171
172
173
174
 
# File 'app/models/dwc_occurrence.rb', line 168

def self.object_join(target)
  return DwcOccurrence.none unless ['CollectionObject', 'AssertedDistribution', 'FieldOccurrence'].include?(target)
  a = arel_table
  b = target.safe_constantize.arel_table # hmm - :: required
  j = a.join(b).on(a[:dwc_occurrence_object_type].eq(target).and(a[:dwc_occurrence_object_id].eq(b[:id])))
  joins(j.join_sources)
end

.scoped_by_otu(otu) ⇒ Scope

Returns all DwcOccurrences for the Otu

  • Includes synonymy (coordinate OTUs).

Returns:

  • (Scope) —

    all DwcOccurrences for the Otu

    • Includes synonymy (coordinate OTUs).


179
180
181
182
183
184
185
186
187
188
189
 
# File 'app/models/dwc_occurrence.rb', line 179

def self.scoped_by_otu(otu)
  if otu.taxon_name_id.present?
    ::Queries::DwcOccurrence::Filter.new({
      taxon_name_id: otu.taxon_name_id,
    }).all
  else
    ::Queries::DwcOccurrence::Filter.new({
      otu_id: otu.id,
    }).all
  end
end

.stale(kind = 'CollectionObject') ⇒ Object 



352
353
354
355
356
 
# File 'app/models/dwc_occurrence.rb', line 352

def self.stale(kind = 'CollectionObject')
  tbl = kind.tableize
  DwcOccurrence.joins("LEFT JOIN #{tbl} tbl on dwc_occurrences.dwc_occurrence_object_id = tbl.id")
    .where('tbl.id IS NULL and dwc_occurrences.dwc_occurrence_object_type = ?', kind )
end

.sweep ⇒ Object

Delete all DwcOccurrence records where object is missing.



345
346
347
348
349
350
 
# File 'app/models/dwc_occurrence.rb', line 345

def self.sweep
  %w{CollectionObject AssertedDistribution FieldOccurrence}.each do |k|
    stale(k).delete_all
  end
  true
end

.target_occurrence_columns ⇒ Array

!! TODO: When we come to adding AssertedDistributions, FieldOccurrnces, etc. we will have to make this more flexible

Returns:

  • (Array) —

    of symbols



213
214
215
216
217
218
219
220
221
222
223
224
225
 
# File 'app/models/dwc_occurrence.rb', line 213

def self.target_occurrence_columns
  # The final DwCA file *will* have an id column, as required for matching
  # with extensions, but its values will be copies of occurrenceID - we don't
  # want to send the ephemeral dwc_occurrence.id values to GBIF.
  # Order doesn't matter for archives, but users are used to this order so try
  # to preserve it (I guess).
  [:id,
   :basisOfRecord,
   :occurrenceID,
   :dwc_occurrence_object_id,   # !! We don't want this, but need it in joins, it is removed in trim via Export::Dwca::Occurrence::Data.excluded_occurrence_columns
   :dwc_occurrence_object_type, # !! ^
  ] + CollectionObject::DwcExtensions::DWC_OCCURRENCE_MAP.keys
end

Instance Method Details

#api_attributes ⇒ Object 



133
134
135
 
# File 'app/models/dwc_occurrence.rb', line 133

def api_attributes
  as_json.except(*API_EXCLUDED_ATTRIBUTES)
end

#as_json(options = {}) ⇒ Object

Strip nils when to_json used



112
113
114
 
# File 'app/models/dwc_occurrence.rb', line 112

def as_json(options = {})
  super(options.merge(except: attributes.keys.select{ |key| self[key].nil? }))
end

#asserted_distribution ⇒ Object 



141
142
143
 
# File 'app/models/dwc_occurrence.rb', line 141

def asserted_distribution
  dwc_occurrence_object_type == 'AssertedDistribution' ? dwc_occurrence_object : nil
end

#basis ⇒ Object 



227
228
229
230
231
232
233
 
# File 'app/models/dwc_occurrence.rb', line 227

def basis
  if dwc_occurrence_object&.respond_to?(:dwc_occurrence_basis)
    dwc_occurrence_object.dwc_occurrence_basis
  else
    'Undefined'
  end
end

#collecting_event ⇒ Object 



149
150
151
 
# File 'app/models/dwc_occurrence.rb', line 149

def collecting_event
  collection_object&.collecting_event || field_occurrence&.collecting_event
end

#collection_object ⇒ Object 



137
138
139
 
# File 'app/models/dwc_occurrence.rb', line 137

def collection_object
  dwc_occurrence_object_type == 'CollectionObject' ? dwc_occurrence_object : nil
end

#computed_occurrence_columns ⇒ Scope

Returns the columns inferred to have occurrence export data.

Returns:

  • (Scope) —

    the columns inferred to have occurrence export data



100
 
# File 'app/models/dwc_occurrence.rb', line 100

scope :computed_occurrence_columns, -> { select(target_occurrence_columns) }

#create_object_uuid ⇒ Object (protected) 



331
332
333
334
335
336
337
 
# File 'app/models/dwc_occurrence.rb', line 331

def create_object_uuid
  @occurrence_identifier = Identifier::Global::Uuid::TaxonworksDwcOccurrence.create!(
    identifier_object: dwc_occurrence_object,
    by: dwc_occurrence_object&.creator, # revisit, why required?
    project_id: dwc_occurrence_object&.project_id, # Current.project_id,  # revisit, why required?
    is_generated: true)
end

#dwc_json ⇒ Object

Returns Hash

  • Legally formatted DwC fields only, with things like dwcClass translated
  • Only fields with values returned
  • Keys are sorted.

Returns:

  • Hash

    • Legally formatted DwC fields only, with things like dwcClass translated
    • Only fields with values returned
    • Keys are sorted


121
122
123
124
125
126
127
 
# File 'app/models/dwc_occurrence.rb', line 121

def dwc_json
  a = as_json.reject!{|k,v| TW_ATTRIBUTES.include?(k.to_sym) || v.nil?}
  HEADER_CONVERTERS.keys.each do |k|
    a[ HEADER_CONVERTERS[k] ] = a.delete(k) if a[k]
  end
  a.sort.to_h
end

#field_occurrence ⇒ Object 



145
146
147
 
# File 'app/models/dwc_occurrence.rb', line 145

def field_occurrence
  dwc_occurrence_object_type == 'FieldOccurrence' ? dwc_occurrence_object : nil
end

#generate_uuid_if_required(force = false) ⇒ Object

TODO: quick check if occurrenceID exists in table?! <-> locking sync !?

Parameters:

  • force (Boolean) (defaults to: false) —

    true - only create identifier if identifier exists false - check if occurrenceID is present, if it is, assume identifier (still) exists



247
248
249
250
251
252
253
254
255
 
# File 'app/models/dwc_occurrence.rb', line 247

def generate_uuid_if_required(force = false)
  if force # really make sure there is an object to work with
    create_object_uuid if !occurrence_identifier && !dwc_occurrence_object.nil? # TODO: can be simplified when inverse_of/validation added to identifiers
  else # assume if occurrenceID is not blank identifier is present
    if occurrenceID.blank?
      create_object_uuid if !occurrence_identifier && !dwc_occurrence_object.nil? # TODO: can be simplified when inverse_of/validation added to identifiers
    end
  end
end

#is_stale? ⇒ Boolean

!! This a spot check, it's not (yet) coded to be comprehensive. !! You should request a full rebuild (rebuild=true) at display time !! to ensure an up-to-date individual record

Returns:

  • (Boolean) —

    By looking at the data, determine if a related record has been updated since this record ws updated at.



265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
 
# File 'app/models/dwc_occurrence.rb', line 265

def is_stale?
  case dwc_occurrence_object_type
  when 'CollectionObject'
    times = is_stale_metadata.values
    n = read_attribute(:updated_at)

    times.each do |v|
      return true if v > n
    end

    return false
  else # AssertedDistribution
    return  dwc_occurrence_object.updated_at > updated_at
  end
end

#is_stale_metadata ⇒ Object 



281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
 
# File 'app/models/dwc_occurrence.rb', line 281

def is_stale_metadata
  case dwc_occurrence_object_type
  when 'CollectionObject'

    o = CollectionObject.select(:id, :updated_at, :collecting_event_id).find_by(id: dwc_occurrence_object_id)
    ce = CollectingEvent.select(:id, :updated_at).find_by(id: o.collecting_event_id)

    td =  dwc_occurrence_object&.taxon_determinations.order(:position).first

    tdr = if td&.otu&.taxon_name&.cached_name_and_author_year != scientificName
            td.updated_at
          else
            nil
          end

    tc = if fieldNumber != o.dwc_field_number
           collecting_event.identifiers.where(type: 'Identifier::Local::FieldNumber').first.updated_at
         else
           nil
         end

    return {
      collection_object: o.updated_at, # Shouldn't be neccessary since on_save rebuilds, but cheap here
      collecting_event: ce&.updated_at,
      trip_code: tc,
      taxon_determination: dwc_occurrence_object.taxon_determinations.order(:position)&.first&.updated_at,
      taxon_determination_reorder: tdr,
      taxon_determination_roles: dwc_occurrence_object.taxon_determinations.order(:position)&.first&.updated_at,
      biocuration_classification: dwc_occurrence_object.biocuration_classifications.order(:updated_at).first&.updated_at,
      georeferences: dwc_occurrence_object.georeferences.order(:updated_at).first&.updated_at,

      data_attributes: dwc_occurrence_object.data_attributes.order(:updated_at).first&.updated_at,

      collection_object_roles: dwc_occurrence_object.roles.order(:updated_at).first&.updated_at,
      collecting_event_data_attributes: dwc_occurrence_object.collecting_event&.data_attributes&.order(:updated_at)&.first&.updated_at,
      collecting_event_roles: dwc_occurrence_object.collecting_event&.roles&.order(:updated_at)&.first&.updated_at
      # citations?
      # tags?!
    }.select{|k,v| !v.nil?}

  else # AssertedDistribution
    {
      asserted_distribution: dwc_occurrence_object.updated_at,
      # TODO: Citations
    }
  end
end

#otu ⇒ Object 



153
154
155
156
157
158
159
160
161
162
 
# File 'app/models/dwc_occurrence.rb', line 153

def otu
  case dwc_occurrence_object_type
  when 'AssertedDistribution'
    dwc_occurrence_object.otu
  when 'CollectionObject'
    collection_object.otu
  when 'FieldOccurrence'
    field_occurrence.otu
  end
end

#set_metadata_attributes ⇒ Object (protected) 



339
340
341
342
 
# File 'app/models/dwc_occurrence.rb', line 339

def set_metadata_attributes
  write_attribute( :basisOfRecord, basis)
  write_attribute( :occurrenceID, occurrence_identifier&.identifier)  # TODO: Slightly janky to touch this here, might not be needed with new hooks
end

#uuid_identifier_scope ⇒ Object 



235
236
237
 
# File 'app/models/dwc_occurrence.rb', line 235

def uuid_identifier_scope
  dwc_occurrence_object&.identifiers&.where('identifiers.type like ?', 'Identifier::Global::Uuid%')&.order(:position)
end