Class: DwcOccurrence

Inherits:
ApplicationRecord show all
Includes:
Housekeeping
Defined in:
app/models/dwc_occurrence.rb

Overview

A Darwin Core Record for the Occurrence core. Field generated from Ruby dwc-meta, which references the same spec that is used in the IPT, and the Dwc Assistant. Each record references a specific CollectionObject or AssertedDistribution.

Important: This is a cache/index, data here are periodically destroyed and regenerated from multiple tables in TW.

DWC attributes are camelCase to facilitate matching dwcClass is a replacement for the Rails reserved 'Class'

All DC attributes (attributes not in DwcOccurrence::TW_ATTRIBUTES) in this table are namespaced to dc (“purl.org/dc/terms/”, “rs.tdwg.org/dwc/terms/”)

README:

 There is a two part strategy to building the index. 1) An individual record will rebuild on request with `parameter to collection_objects/123/dwc*?build=true`.
 2) Wipe, and rebuild on some schedule. It would in theory be possible to track and rebuild when a class of every property was created (or updated), however
 this is a lot of overhead to inject/code for a lot of models. It would inject latency at numerous stages that would perhaps impact UI performance.

Notes -

This experiment isn't going to work at scale, too many preceding calls for big returns

after_find :refresh, if: -> { persisted? && is_stale? }

def refresh
  dwc_occurrence_object.set_dwc_occurrence
end

TODO: The basisOfRecord CVTs are not super informative.

We know collection object is definitely 1:1 with PreservedSpecimen, however
AssertedDistribution could be HumanObservation (if source is person), or ... what? if
its a published record.  Seems we need a 'PublishedAssertation', just like we model the data.

Gotchas.

* updated_at is set by touching the record, not via housekeeping.

Constant Summary collapse

DC_NAMESPACE = 
'http://rs.tdwg.org/dwc/terms/'.freeze
TW_ATTRIBUTES =

Not yet implemented, but likely needed (at an even higher level) ? :id

[
  :id,
  :project_id,
  :created_at,
  :updated_at,
  :created_by_id,
  :updated_by_id,
  :dwc_occurrence_object_type,
  :dwc_occurence_object_id
].freeze
HEADER_CONVERTERS = 
{
  'dwcClass' => 'class',
}.freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Housekeeping

#has_polymorphic_relationship?

Methods inherited from ApplicationRecord

transaction_with_retry

Instance Attribute Details

#occurrence_identifier ⇒ Object

Returns the value of attribute occurrence_identifier.



75
76
77
 
# File 'app/models/dwc_occurrence.rb', line 75

def occurrence_identifier
  @occurrence_identifier
end

Class Method Details

.annotates? ⇒ Boolean

Returns:

  • (Boolean) 


77
78
79
 
# File 'app/models/dwc_occurrence.rb', line 77

def self.annotates?
  false
end

.asserted_distributions_join ⇒ ActiveRecord::Relation

that matches, consider moving to Shared

Returns:

  • (ActiveRecord::Relation) 


95
96
97
98
99
100
 
# File 'app/models/dwc_occurrence.rb', line 95

def self.asserted_distributions_join
  a = arel_table
  b = ::AssertedDistribution.arel_table
  j = a.join(b).on(a[:dwc_occurrence_object_type].eq('AssertedDistribution').and(a[:dwc_occurrence_object_id].eq(b[:id])))
  joins(j.join_sources)
end

.by_collection_object_filter(filter_scope: nil, project_id: nil) ⇒ Object

Return scopes by a collection object filter



138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
 
# File 'app/models/dwc_occurrence.rb', line 138

def self.by_collection_object_filter(filter_scope: nil, project_id: nil)
  return DwcOccurrence.none if project_id.nil? || filter_scope.nil?

  c = ::CollectionObject.arel_table
  d = arel_table

  # TODO: hackish
  k = ::CollectionObject.select('coscope.id').from( '(' + filter_scope.to_sql + ') as coscope ' )

  a = self.collection_objects_join
    .where('dwc_occurrences.project_id = ?', project_id)
    .where(dwc_occurrence_object_id: k)
    .select(::DwcOccurrence.target_columns) # TODO !! Will have to change when AssertedDistribution and other types merge in
  a
end

.collection_objects_join ⇒ ActiveRecord::Relation

that matches, consider moving to Shared

Returns:

  • (ActiveRecord::Relation) 


86
87
88
89
90
91
 
# File 'app/models/dwc_occurrence.rb', line 86

def self.collection_objects_join
  a = arel_table
  b = ::CollectionObject.arel_table
  j = a.join(b).on(a[:dwc_occurrence_object_type].eq('CollectionObject').and(a[:dwc_occurrence_object_id].eq(b[:id])))
  joins(j.join_sources)
end

.computed_columns ⇒ Scope

Returns the columns inferred to have data.

Returns:

  • (Scope) —

    the columns inferred to have data



190
191
192
 
# File 'app/models/dwc_occurrence.rb', line 190

def self.computed_columns
  select(target_columns)
end

.empty_fields ⇒ Array

Returns of column names as symbols that are blank in ALL projects (not just this one).

Returns:

  • (Array) —

    of column names as symbols that are blank in ALL projects (not just this one)



156
157
158
159
160
161
162
163
164
165
166
167
 
# File 'app/models/dwc_occurrence.rb', line 156

def self.empty_fields
  empty_in_all_projects = ActiveRecord::Base.connection.execute("select attname
  from pg_stats
  where tablename = 'dwc_occurrences'
  and most_common_vals is null
  and most_common_freqs is null
  and histogram_bounds is null
  and correlation is null
  and null_frac = 1;").pluck('attname').map(&:to_sym)

  empty_in_all_projects #  - target_columns
end

.excluded_columns ⇒ Array

Returns of symbols.

Returns:

  • (Array) —

    of symbols



184
185
186
 
# File 'app/models/dwc_occurrence.rb', line 184

def self.excluded_columns
  ::DwcOccurrence.columns.collect{|c| c.name.to_sym} - (self.target_columns - [:dwc_occurrence_object_id, :dwc_occurrence_object_type])
end

.scoped_by_otu(otu) ⇒ Scope

Returns all DwcOccurrences for the Otu

  • Includes synonymy (coordinate OTUs).

Returns:

  • (Scope) —

    all DwcOccurrences for the Otu

    • Includes synonymy (coordinate OTUs).



107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
 
# File 'app/models/dwc_occurrence.rb', line 107

def self.scoped_by_otu(otu)
  a,b = nil, nil

  if otu.taxon_name_id.present?
    a = ::Queries::DwcOccurrence::Filter.new(
      asserted_distribution_query: {
        taxon_name_query: {
          taxon_name_id: otu.taxon_name_id,
          descendants: false,
          synonymify: true } })

    b = ::Queries::DwcOccurrence::Filter.new(
      collection_object_query: {
        taxon_name_query: {
          taxon_name_id: otu.taxon_name_id,
          descendants: false,
          synonymify: true } })
  else
    a = ::Queries::DwcOccurrence::Filter.new(
      asserted_distribution_query: {
        otu_id: otu.id})

    b = ::Queries::DwcOccurrence::Filter.new(
      collection_object_query: {
        otu_query: { otu_id: otu.id}})
  end

  from("((#{a.all.to_sql}) UNION (#{b.all.to_sql})) as dwc_occurrences")
end

.target_columns ⇒ Array

!! TODO: When we come to adding AssertedDistributions, FieldOccurrnces, etc. we will have to make this more flexible

Returns:

  • (Array) —

    of symbols



173
174
175
176
177
178
179
180
 
# File 'app/models/dwc_occurrence.rb', line 173

def self.target_columns
  [:id, # must be in position 0
   :occurrenceID,
   :basisOfRecord,
   :dwc_occurrence_object_id,   # !! We don't want this, but need it in joins, it is removed in trim via `.excluded_columns` below
   :dwc_occurrence_object_type, # !! ^
  ] + CollectionObject::DwcExtensions::DWC_OCCURRENCE_MAP.keys
end

Instance Method Details

#basis ⇒ Object 



194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
 
# File 'app/models/dwc_occurrence.rb', line 194

def basis
  case dwc_occurrence_object_type
  when 'CollectionObject'
    if dwc_occurrence_object.is_fossil?
      return 'FossilSpecimen'
    else
      return 'PreservedSpecimen'
    end
  when 'AssertedDistribution'
    # Used to fork b/b Source::Human and Source::Bibtex:
    case dwc_occurrence_object.source&.type || dwc_occurrence_object.sources.order(cached_nomenclature_date: :DESC).first.type
    when 'Source::Bibtex'
      return 'MaterialCitation'
    when 'Source::Human'
      return 'HumanObservation'
    else # Not recommended at this point
      return 'Occurrence'
    end
  end
  'Undefined'
end

#create_object_uuid ⇒ Object (protected) 



286
287
288
289
290
291
292
 
# File 'app/models/dwc_occurrence.rb', line 286

def create_object_uuid
  @occurrence_identifier = Identifier::Global::Uuid::TaxonworksDwcOccurrence.create!(
    identifier_object: dwc_occurrence_object,
    by: dwc_occurrence_object&.creator, # revisit, why required?
    project_id: dwc_occurrence_object&.project_id, # Current.project_id,  # revisit, why required?
    is_generated: true)
end

#generate_uuid_if_required(force = false) ⇒ Object

TODO: quick check if occurrenceID exists in table?! <-> locking sync !?

Parameters:

  • force (Boolean) (defaults to: false) —

    true - only create identifier if identifier exists false - check if occurrenceID is present, if it is, assume identifier (still) exists



228
229
230
231
232
233
234
235
236
 
# File 'app/models/dwc_occurrence.rb', line 228

def generate_uuid_if_required(force = false)
  if force # really make sure there is an object to work with
    create_object_uuid if !occurrence_identifier && !dwc_occurrence_object.nil? # TODO: can be simplified when inverse_of/validation added to identifiers
  else # assume if occurrenceID is not blank identifier is present
    if occurrenceID.blank?
      create_object_uuid if !occurrence_identifier && !dwc_occurrence_object.nil? # TODO: can be simplified when inverse_of/validation added to identifiers
    end
  end
end

#is_stale? ⇒ Boolean

!! This a spot check, it's not (yet) coded to be comprehensive. !! You should request a full rebuild (rebuild=true) at display time !! to ensure an up-to-date individual record

Returns:

  • (Boolean) —

    By looking at the data, determine if a related record has been updated since this record ws updated at.



246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
 
# File 'app/models/dwc_occurrence.rb', line 246

def is_stale?
  case dwc_occurrence_object_type
  when 'CollectionObject'

    # Checks made on values, not time (necessary to handle lists)
    return true if taxon_determination_is_stale?

    # Order these roughgly by most probable/frequent changes, and/or fewest db calls required
    times = [
      dwc_occurrence_object.updated_at, # Shouldn't be neccessary since on_save rebuilds, but cheap here
      dwc_occurrence_object.collecting_event&.updated_at,
      dwc_occurrence_object.taxon_determinations.order(:position)&.first&.updated_at,
      dwc_occurrence_object.biocuration_classifications.order(:updated_at).first&.updated_at,
      dwc_occurrence_object.georeferences.order(:updated_at).first&.updated_at,
      dwc_occurrence_object.data_attributes.order(:updated_at).first&.updated_at,

      dwc_occurrence_object.collecting_event&.data_attributes&.order(:updated_at)&.first&.updated_at,
    ]

    n = read_attribute(:updated_at) # || Time.current <- if this then never stale!

    times.each do |v|
      next if v.nil?
      return true if v > n
    end

    return false
  else # AssertedDistribution
    dwc_occurrence_object.updated_at > updated_at
  end
end

#set_metadata_attributes ⇒ Object (protected) 



294
295
296
297
 
# File 'app/models/dwc_occurrence.rb', line 294

def set_metadata_attributes
  write_attribute( :basisOfRecord, basis)
  write_attribute( :occurrenceID, occurrence_identifier&.identifier)
end

#taxon_determination_is_stale? ⇒ Boolean

Returns Boolean acts_as_list actions don't trigger updated_at changes, handle this by comparing current values.

Returns:

  • (Boolean) —

    Boolean acts_as_list actions don't trigger updated_at changes, handle this by comparing current values



280
281
282
 
# File 'app/models/dwc_occurrence.rb', line 280

def taxon_determination_is_stale?
  dwc_occurrence_object&.taxon_determinations.order(:position).first&.otu&.taxon_name&.cached_name_and_author_year != scientificName
end

#uuid_identifier_scope ⇒ Object 



216
217
218
 
# File 'app/models/dwc_occurrence.rb', line 216

def uuid_identifier_scope
  dwc_occurrence_object&.identifiers&.where('identifiers.type like ?', 'Identifier::Global::Uuid%')&.order(:position)
end