Module: Shared::Unify

Extended by:
ActiveSupport::Concern
Included in:
Topic
Defined in:
app/models/concerns/shared/unify.rb

Overview

A module to unify two objects into 1, or to move data between objects.

!! The module works on relations, not attributes, which are ignored and untouched (but see position, and is_original). !! For example if two objects have differing name fields this is ignored.

When :only or :except are provided, then the remove_object IS NOT DESTROYED, only data are moved between objects.555h

If they are not provided, we attempt to destroy the remove_object

  • If a related object is now a duplicate then its annotations are moved to the deduplicate object

  • If preview = true then rolls back all changes.

  • Annotation classes (e.g. Notes) can not be unified except through their relation to unified objects.

  • Users and projects can not be unified, though technically the approach should be be a hard/but robust approach to the problem, with some key exceptions (e.g. two root TaxonNames)

  • Classes that are exposed in the UI are defined at app/javascript/vue/tasks/unify/objects/constants/types.js.

  • Run rake tw:development:linting:inverse_of_preventing_unify judiciously when modifying models or this code. It will catch missing inverse_of parameters required to unify objects. Note that it will always report some missing relationships that do not matter.

Constant Summary collapse

EXCLUDE_RELATIONS =

Never auto-handle these, let the final destroy remove them. Housekeeping relations are not hit here, we don't merge users at the moment.

[
  :versions,             # Not picked up, but adding in case
  :dwc_occurrence,       # Will be destroyed on related objects destruction
  :pinboard_items,       # Technically not needed here
  :cached_map_register,  # Destroyed on merge of things like Georeferences and AssertedDistributions
  :cached_map_items,
  :cached_maps           # Destroy alternate,
]

Instance Method Summary collapse

Instance Method Details

#deduplicate_update_target(object) ⇒ Object (private)

Parameters:

  • object (ActiveRecord::Base) —

    see log_unify_result



304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
 
# File 'app/models/concerns/shared/unify.rb', line 304

def deduplicate_update_target(object)
  i = object.identical

  # There is exactly 1 match, merge is unambiguous
  if i.size == 1
    j = i.first
    # object's own FK is still dirty from the failed update attempt (it
    # points at self/KEEP, not its real enclosing remove_object); reload so
    # the nested unify below moves object's actual remaining relations, not
    # phantom ones based on that dirty state.
    j.unify(object.reload)
  else
    # Merge would be ambiguous, there are multiple matches
    return false
  end
end

#except_relations ⇒ Object

Per class, when merging skip these relations



38
39
40
 
# File 'app/models/concerns/shared/unify.rb', line 38

def except_relations
  []
end

#inferred_relations ⇒ Object

Our target is a list of relations that we can iterate through and, by inspection, update related records to point to self.

  • We don't want to target convienience relations as they are in essence alias of base-class relations and redundant
  • We don't want anything that relates to a calculated cached value
  • We do want to catch relations that are edges in which the same class of object is on both sides, these require an alias. We inspect for related_<name> as a pattern to select these.

TODO: Revist. depending on therelated_XXX naming pattern is brittle-ish, perhaps converge on using unife_relations` to force inclusion.

Note: class_name based exclusions prevent a lot of duplicated efforts, as much of their use is based on convienience relations on things like subclassed or scoped data.

Returns:

  • Array of ActiveRecord::Reflection



70
71
72
73
74
75
76
77
 
# File 'app/models/concerns/shared/unify.rb', line 70

def inferred_relations
  ( unify_relations +
   ::ApplicationEnumeration.klass_reflections(self.class, :has_many) +
   ::ApplicationEnumeration.klass_reflections(self.class, :has_one))
    .delete_if{|r| r.options[:foreign_key] =~ /cache/}
    .delete_if{|r| EXCLUDE_RELATIONS.include?(r.name.to_sym)}
    .delete_if{|r| !r.name.match(/related/) && ( r.options[:through].present? || r.options[:class_name].present? )}
end

#log_unify_result(object, relation, result) ⇒ Object (private)

During logging attempt to resolve duplicate objects issues by moving annotations from the would-be duplicate to an identical existing record.

Parameters:

  • object (ActiveRecord::Base) —

    a record in another table that has a FK pointing to the remove_object (DESTROY) being unified away; the unify loop just attempted to flip that FK to self (KEEP) and the attempted update is reflected in object's in-memory state even if the save failed

  • relation (ActiveRecord::Reflection) —

    the has_many/has_one reflection on self (KEEP) for the association being processed; relation.options is the belongs_to on object's class whose FK we tried to flip

  • result (Hash) —

    the running unify result accumulator



334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
 
# File 'app/models/concerns/shared/unify.rb', line 334

def log_unify_result(object, relation, result)
  n = relation.name.to_s.humanize

  # Handle an edge case, preserve Citations that
  # would only be invalid due to origin flag
  if object.class.name == 'Citation' && object.errors.key?(:is_original) && object.is_original
    object.is_original = false
    object.save
  end

  if object.errors.key?(:position)
    object.position = nil
    object.save
  end

  if object.errors.any?
    # If the attempted move failed, it may be because an identical record
    # already exists on self (a duplicate) - try to find and merge into it.
    # #identical won't match unless there's a genuine duplicate, so this is
    # safe to attempt for *any* validation failure, not just the ones that
    # look like a uniqueness conflict.
    dedup_result = deduplicate_update_target(object)
    if dedup_result && dedup_result[:result][:unified]
      result[:details][n][:deduplicated] += 1
    else
      result[:result][:unified] = false
      result[:details][n][:unmerged] += 1
      result[:details][n][:errors] ||= []
      result[:details][n][:errors].push( {id: object.id, message: object.errors.full_messages.join('; ')} )
    end
  else
    result[:details][n][:merged] += 1
  end

  result
end

#merge_relations(only: [], except: []) ⇒ Object

Perhaps used_inferred to hash

Returns:

  • Array of ActiveRecord::Reflection



44
45
46
47
48
49
50
51
52
 
# File 'app/models/concerns/shared/unify.rb', line 44

def merge_relations(only: [], except: [])
  o = (only_relations + [only&.map(&:to_sym)].flatten).uniq
  if o.any?
    used_inferred_relations.select{|a| o.include?(a.name)}
  else
    e = (except_relations + [except&.map(&:to_sym)].flatten).uniq
    used_inferred_relations.select{|a| !e.include?(a.name)}
  end
end

#only_relations ⇒ Object

Per class, Iterating through all of these



33
34
35
 
# File 'app/models/concerns/shared/unify.rb', line 33

def only_relations
  []
end

#pre_validate(remove_object, result) ⇒ Object (private) 



266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
 
# File 'app/models/concerns/shared/unify.rb', line 266

def pre_validate(remove_object, result)
  s = result

  if s[:result][:target_project_id].nil?
    if is_community?
      s[:result].merge!(
        unified: false,
        message: 'Can not merge community objects without project context.'
      )
    else
      s[:result][:target_project_id] = project_id
    end
  end

  if remove_object == self
    s[:result].merge!(
      unified: false,
      message: 'Can not unify the same objects.'
    )
  end

  if !is_community?
    if project_id != remove_object.project_id
      s[:result].merge!(
        unified: false,
        message: 'Danger, objects come from different projects.')
    end
  end

  if remove_object.class.name != self.class.name
    s[:result].merge!(
      unified: false,
      message: "Can not unify objects of different types (#{remove_object.class.name} and #{self.class.name}).")
  end
  s
end

#relation_label(relation) ⇒ Object 



235
236
237
 
# File 'app/models/concerns/shared/unify.rb', line 235

def relation_label(relation)
  relation.name.to_s.humanize
end

#resolve_acts_as_list_scope_column(klass, name) ⇒ Object (private)

acts_as_list scope: [...] entries may be either a real column name, or a belongs_to association name that acts_as_list itself resolves to that association's foreign_key (e.g. LoanItem's scope: [:loan, :project_id])

  • resolve the same way here so grouping in restore_list_order is keyed by the real column acts_as_list itself scopes by.


416
417
418
419
420
421
 
# File 'app/models/concerns/shared/unify.rb', line 416

def resolve_acts_as_list_scope_column(klass, name)
  return name.to_s if klass.column_names.include?(name.to_s)

  r = klass.reflect_on_association(name)
  r&.belongs_to? ? r.foreign_key : name.to_s
end

#restore_list_order(state) ⇒ Object (private)

Re-apply the pre-merge position order captured by snapshot_list_order, appending incoming records after the surviving object's existing records.



426
427
428
429
430
431
432
 
# File 'app/models/concerns/shared/unify.rb', line 426

def restore_list_order(state)
  (state[:existing] + state[:incoming])
    .group_by { |row| row[1..] } # each row's own scope column values
    .each_value do |rows|
      rows.each_with_index { |row, idx| state[:klass].where(id: row[0]).update_all(position: idx + 1) }
    end
end

#snapshot_list_order(relation, incoming) ⇒ Object (private)

Snapshot the current position order on both sides of a has_many before records are re-parented, so the restore below is independent of how acts_as_list repositions records during update. Returns nil when the association does not use acts_as_list.

Also captures each record's own acts_as_list scope column values (e.g. a TaxonDetermination's taxon_determination_object_id/_type), not just its id: existing (self's) and incoming (remove_object's) has_many members can straddle multiple different real acts_as_list scope groups at once - e.g. taxon determinations on different collection objects when unifying OTUs (see specs).



393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
 
# File 'app/models/concerns/shared/unify.rb', line 393

def snapshot_list_order(relation, incoming)
  related_class = incoming.klass
  return nil unless related_class.respond_to?(:acts_as_list_options)

  # reassigned_columns values are going to be set to self's values after a
  # successful relation move, so *don't* scope by them when grouping
  # acts_as_list groups (they're currently different than self! - see specs).
  reassigned_columns = [relation.foreign_key, relation.type].compact
  scope_columns = Array(related_class.acts_as_list_options[:scope])
    .map { |s| resolve_acts_as_list_scope_column(related_class, s) } - reassigned_columns

  {
    klass: related_class,
    existing: send(relation.name).order(:position).pluck(:id, *scope_columns),
    incoming: incoming.order(:position).pluck(:id, *scope_columns)
  }
end

#stub_unify_result(result, relation_name, attempted) ⇒ Object (private) 



371
372
373
374
375
376
377
378
379
380
 
# File 'app/models/concerns/shared/unify.rb', line 371

def stub_unify_result(result, relation_name, attempted)
  result[:details].merge!(
    relation_name => {
      attempted:,
      merged: 0,
      unmerged: 0,
      deduplicated: 0
    }
  )
end

#unify(remove_object, only: [], except: [], preview: false, cutoff: 250, target_project_id: nil) ⇒ Object

See header.

Parameters:

  • remove_object —

    this object will be destroyed if possible

  • only (Array of Symbols) (defaults to: []) —

    only operate on these relations, useful for partial merges/moving objects

  • except (Array of Symbols) (defaults to: []) —

    don't operate on these relations

  • preview (defaults to: false) —

    Boolean if true then roll back all operations

  • cutoff (defaults to: 250) —

    Integer if more than cutoff relations are observed then always rollback TODO: add delayed job handling

  • target_project_id (Integer) (defaults to: nil) —

    required when self is_community?, scopes operations to target project only

Returns:

  • Hash a result



114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
 
# File 'app/models/concerns/shared/unify.rb', line 114

def unify(remove_object, only: [], except: [], preview: false, cutoff: 250, target_project_id: nil)
  s = {
    result: { unified: nil, total_related: 0, target_project_id:},
    details: {},
  }

  pre_validate(remove_object, s)
  return s if s[:result][:unified] == false
  pid = s[:result][:target_project_id]

  # Whether this call intends to fully destroy remove_object (as opposed
  # to only:/except: moving some data between the two objects).
  will_destroy = only.empty? && except.empty?
  destroy_key = { id: remove_object.id, type: remove_object.class.base_class.name }

  self.class.transaction do
    # before_unify # potential hooks, appear not to be required

    # Record remove_object as committed-to-destruction *before* any related
    # records are touched below, so those related records can see, for the
    # whole duration of this call (including any nested unify a dedup
    # triggers), that it's already slated for destruction.
    if will_destroy
      UnifyDestroyContext.objects_in_destroy ||= Set.new
      UnifyDestroyContext.objects_in_destroy << destroy_key
    end

    begin
      merge_relations(only:, except:).each do |r|
        n = relation_label(r)

        case ::ApplicationEnumeration.relationship_type(r)

        when :has_many
          i = remove_object.send(r.name)

          unless ::ApplicationEnumeration.relation_targets_community?(r)
            i = i.where(project_id: pid)
          end

          next unless i.any?

          t = i.size
          stub_unify_result(s, n, t)

          s[:result][:total_related] += t
          next if s[:result][:total_related] > cutoff

          list_state = snapshot_list_order(r, i)

          i.find_each do |j|
            j.update(r.options[:inverse_of] => self)
            log_unify_result(j, r, s)
          end

          restore_list_order(list_state) if list_state

        when :has_one, :belongs_to
          i = remove_object.send(r.name)
          if !i.nil?
            stub_unify_result(s, n, 1)

            i.update(r.options[:inverse_of] => self)
            log_unify_result(i, r, s)
          end
        end
      end

      if cutoff_hit = s[:result][:total_related] > cutoff
        s[:result][:unified] = false
        s[:result][:message] = "Related cutoff threshold (> #{cutoff}) hit, unify is not yet allowed on these objects."
      elsif s[:result][:unified] != false

        begin
          remove_object.reload # reset all in-memory has_many caches that would prevent destroy

          remove_object.destroy! if will_destroy

        rescue ActiveRecord::InvalidForeignKey => e
          # InvalidForeignKey comes from the DB adapter, so e has no `.record`.
          s[:result][:unified] = false
          s[:details].merge!(
            Object: {
              errors: [
                {
                  id: remove_object.id,
                  exception: e.class.name,
                  message: e.message
                }
              ]
            }
          )
          raise ActiveRecord::Rollback
        rescue ActiveRecord::RecordNotDestroyed => e
          s[:result][:unified] = false
          s[:details].merge!(
            Object: {
              errors: [
                { id: e.record.id, message: e.record.errors.full_messages.join('; ') }
              ]
            }
          )

          raise ActiveRecord::Rollback
        end
      end
    ensure
      UnifyDestroyContext.objects_in_destroy&.delete(destroy_key) if will_destroy
    end

    # after_unify # potential hooks, appear not to be required

    if preview || cutoff_hit || s[:result][:unified] == false
      raise ActiveRecord::Rollback
    end
  end

  s[:result][:unified] = true unless s[:result][:unified] == false
  s
end

#unify_relations ⇒ Object

Override in instances methods, see Serial for eg



86
87
88
 
# File 'app/models/concerns/shared/unify.rb', line 86

def unify_relations
  []
end

#unify_relations_metadata(target_project_id: nil) ⇒ Object 



239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
 
# File 'app/models/concerns/shared/unify.rb', line 239

def unify_relations_metadata(target_project_id: nil)
  s = {}

  merge_relations.each do |r|
    name = relation_label(r)

    case ::ApplicationEnumeration.relationship_type(r)
    when :has_many
      if ::ApplicationEnumeration.relation_targets_community?(r)
        i = send(r.name)
      else
        i = send(r.name).where(project_id: target_project_id)
      end

      next unless i.count > 0
      s[r.name] = { total: i.count, name: }
    when :has_one
      if send(r.name).present?
        s[r.name] = { total: 1, name: }
      end
    end
  end
  s.sort.to_h
end

#used_inferred_relations ⇒ Object

Keep separated from inferred_relations so we can better audit all models in rake linting

Returns:

  • Array of ActiveRecord::Reflection



81
82
83
 
# File 'app/models/concerns/shared/unify.rb', line 81

def used_inferred_relations
  (inferred_relations.select{|r| !r.options[:inverse_of].nil?} + unify_relations).uniq
end