DetectDuplicate

DetectDuplicate 2.0.0

Bundle: org.apache.nifi | nifi-standard-nar
Description: Caches a value, computed from FlowFile attributes, for each incoming FlowFile and determines if the cached value has already been seen. If so, routes the FlowFile to 'duplicate' with an attribute named 'original.identifier' that specifies the original FlowFile's "description", which is specified in the <FlowFile Description> property. If the FlowFile is not determined to be a duplicate, the Processor routes the FlowFile to 'non-duplicate'
Tags: dedupe, dupe, duplicate, hash
Input Requirement: REQUIRED
Supports Sensitive Dynamic Properties: false

Properties

Age Off Duration
Time interval to age off cached FlowFiles

Display Name

Age Off Duration

Description

Time interval to age off cached FlowFiles

API Name

Age Off Duration

Expression Language Scope

Not Supported

Sensitive

false

Required

false
Cache Entry Identifier
A FlowFile attribute, or the results of an Attribute Expression Language statement, which will be evaluated against a FlowFile in order to determine the value used to identify duplicates; it is this value that is cached

Display Name

Cache Entry Identifier

Description

A FlowFile attribute, or the results of an Attribute Expression Language statement, which will be evaluated against a FlowFile in order to determine the value used to identify duplicates; it is this value that is cached

API Name

Cache Entry Identifier

Default Value

${hash.value}

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

true
Cache The Entry Identifier
When true this cause the processor to check for duplicates and cache the Entry Identifier. When false, the processor would only check for duplicates and not cache the Entry Identifier, requiring another processor to add identifiers to the distributed cache.
Display Name

Cache The Entry Identifier

Description

When true this cause the processor to check for duplicates and cache the Entry Identifier. When false, the processor would only check for duplicates and not cache the Entry Identifier, requiring another processor to add identifiers to the distributed cache.

API Name

Cache The Entry Identifier

Default Value

true

Allowable Values
- true
- false
Expression Language Scope

Not Supported

Sensitive

false

Required

false
Distributed Cache Service
The Controller Service that is used to cache unique identifiers, used to determine duplicates

Display Name

Distributed Cache Service

Description

The Controller Service that is used to cache unique identifiers, used to determine duplicates

API Name

Distributed Cache Service

Service Interface

org.apache.nifi.distributed.cache.client.DistributedMapCacheClient

Service Implementations

org.apache.nifi.hazelcast.services.cacheclient.HazelcastMapCacheClient

org.apache.nifi.distributed.cache.client.MapCacheClientService

org.apache.nifi.redis.service.RedisDistributedMapCacheClientService

org.apache.nifi.redis.service.SimpleRedisDistributedMapCacheClientService

Expression Language Scope

Not Supported

Sensitive

false

Required

true
FlowFile Description
When a FlowFile is added to the cache, this value is stored along with it so that if a duplicate is found, this description of the original FlowFile will be added to the duplicate's "original.flowfile.description" attribute

Display Name

FlowFile Description

Description

When a FlowFile is added to the cache, this value is stored along with it so that if a duplicate is found, this description of the original FlowFile will be added to the duplicate's "original.flowfile.description" attribute

API Name

FlowFile Description

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

true

Relationships

Name	Description
duplicate	If a FlowFile has been detected to be a duplicate, it will be routed to this relationship
non-duplicate	If a FlowFile's Cache Entry Identifier was not found in the cache, it will be routed to this relationship
failure	If unable to communicate with the cache, the FlowFile will be penalized and routed to this relationship

Writes Attributes

Name	Description
original.flowfile.description	All FlowFiles routed to the duplicate relationship will have an attribute added named original.flowfile.description. The value of this attribute is determined by the attributes of the original copy of the data and by the FlowFile Description property.