Module: Export::Dwca

Defined in:
lib/export/dwca.rb,
lib/export/dwca/data.rb

Defined Under Namespace

Modules: GbifProfile Classes: Data

Constant Summary collapse

INDEX_VERSION =

Version is a way to track dates where the indexing changed significantly such that all or most of the index should be regenerated. To add a version use `Time.now` via IRB

[
  '2021-10-12 17:00:00.000000 -0500',    # First major refactor
  '2021-10-15 17:00:00.000000 -0500'     # Minor  Excludes footprintWKT, and references to GeographicArea in gazetteer; new form of media links
]

Class Method Summary collapse

Class Method Details

.build_index_async(klass, record_scope) ⇒ Object

When we re-index a large set of data then we run it in the background. To determine when it is done we poll by the last record to be indexed.

Parameters:

  • klass (ActiveRecord class)

    e.g. CollectionObject

  • record_scope (An ActiveRecord scope)

Returns:

  • Hash total: total records to expect start_time: the time indexing started sample: Array of object global ids spread across 10 (or fewer) intervals of the recordset



52
53
54
55
56
# File 'lib/export/dwca.rb', line 52

def self.build_index_async(klass, record_scope)
  s = record_scope.order(:id)
  ::DwcaCreateIndexJob.perform_later(klass.to_s, sql_scope: s.to_sql)
  (klass, s)
end

.download_async(record_scope, request = nil) ⇒ Download

Returns the download object containing the archive.

Parameters:

  • record_scope (ActiveRecord::Relation)

    a relation that returns DwcOccurrence records

Returns:

  • (Download)

    the download object containing the archive



23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# File 'lib/export/dwca.rb', line 23

def self.download_async(record_scope, request = nil)
  name = "dwc-a_#{DateTime.now}.zip"

  download = ::Download::DwcArchive.create!(
    name: "DwC Archive generated at #{Time.now}.",
    description: 'A Darwin Core archive.',
    filename: name,
    request: request,
    expires: 2.days.from_now,
    total_records: record_scope.size # Was haveing problems with count() TODO: increment after when extensions are allowed.
  )

  # Note we pass a string with the record scope
  ::DwcaCreateDownloadJob.perform_later(download, core_scope: record_scope.to_sql)

  download
end

.index_metadata(klass, record_scope) ⇒ Object



58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
# File 'lib/export/dwca.rb', line 58

def self.(klass, record_scope)
  a = record_scope.first&.to_global_id&.to_s
  b = record_scope.last&.to_global_id&.to_s

  t = record_scope.size # was haveing problems with count

   = {
    total: t,
    start_time: Time.now,
    sample: [a, b].compact
  }

  if b && (t > 2)
    max = 9
    max = t if t < 9

    ids = klass
      .select('*')
      .from("(select id, type, ROW_NUMBER() OVER (ORDER BY id ASC) rn from (#{record_scope.to_sql}) b ) a")
      .where("a.rn % ((SELECT COUNT(*) FROM (#{record_scope.to_sql}) c) / #{max}) = 0")
      .limit(max)
      .collect{|o| o.to_global_id.to_s}

    [:sample].insert(1, *ids)
  end

  [:sample].uniq!
  
end