ExtractAvroMetadata

ExtractAvroMetadata 2.0.0

Bundle: org.apache.nifi | nifi-avro-nar
Description: Extracts metadata from the header of an Avro datafile.
Tags: avro, metadata, schema
Input Requirement: REQUIRED
Supports Sensitive Dynamic Properties: false

Properties

Count Items
If true the number of items in the datafile will be counted and stored in a FlowFile attribute 'item.count'. The counting is done by reading blocks and getting the number of items for each block, thus avoiding de-serializing. The items being counted will be the top-level items in the datafile. For example, with a schema of type record the items will be the records, and for a schema of type Array the items will be the arrays (not the number of entries in each array).
Display Name

Count Items

Description

If true the number of items in the datafile will be counted and stored in a FlowFile attribute 'item.count'. The counting is done by reading blocks and getting the number of items for each block, thus avoiding de-serializing. The items being counted will be the top-level items in the datafile. For example, with a schema of type record the items will be the records, and for a schema of type Array the items will be the arrays (not the number of entries in each array).

API Name

Count Items

Default Value

false

Allowable Values
- true
- false
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Fingerprint Algorithm
The algorithm used to generate the schema fingerprint. Available choices are based on the Avro recommended practices for fingerprint generation.
Display Name

Fingerprint Algorithm

Description

The algorithm used to generate the schema fingerprint. Available choices are based on the Avro recommended practices for fingerprint generation.

API Name

Fingerprint Algorithm

Default Value

CRC-64-AVRO

Allowable Values
- CRC-64-AVRO
- MD5
- SHA-256
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Metadata Keys
A comma-separated list of keys indicating key/value pairs to extract from the Avro file header. The key 'avro.schema' can be used to extract the full schema in JSON format, and 'avro.codec' can be used to extract the codec name if one exists.

Display Name

Metadata Keys

Description

A comma-separated list of keys indicating key/value pairs to extract from the Avro file header. The key 'avro.schema' can be used to extract the full schema in JSON format, and 'avro.codec' can be used to extract the codec name if one exists.

API Name

Metadata Keys

Expression Language Scope

Not Supported

Sensitive

false

Required

false

Relationships

Name	Description
success	A FlowFile is routed to this relationship after metadata has been extracted.
failure	A FlowFile is routed to this relationship if it cannot be parsed as Avro or metadata cannot be extracted for any reason

Writes Attributes

Name	Description
schema.type	The type of the schema (i.e. record, enum, etc.).
schema.name	Contains the name when the type is a record, enum or fixed, otherwise contains the name of the primitive type.
schema.fingerprint	The result of the Fingerprint Algorithm as a Hex string.
item.count	The total number of items in the datafile, only written if Count Items is set to true.