ConsumeElasticsearch

ConsumeElasticsearch 2.0.0

Bundle: org.apache.nifi | nifi-elasticsearch-restapi-nar
Description: A processor that repeatedly runs a paginated query against a field using a Range query to consume new Documents from an Elasticsearch index/query. The processor will retrieve multiple pages of results until either no more results are available or the Pagination Keep Alive expiration is reached, after which the Range query will automatically update the field constraint based on the last retrieved Document value.
Tags: elasticsearch, elasticsearch5, elasticsearch6, elasticsearch7, elasticsearch8, json, page, query, scroll, search
Input Requirement: FORBIDDEN
Supports Sensitive Dynamic Properties: false

Properties

Query Attribute
If set, the executed query will be set on each result flowfile in the specified attribute.

Display Name

Query Attribute

Description

If set, the executed query will be set on each result flowfile in the specified attribute.

API Name

el-query-attribute

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false
Client Service
An Elasticsearch client service to use for running queries.

Display Name

Client Service

Description

An Elasticsearch client service to use for running queries.

API Name

el-rest-client-service

Service Interface

org.apache.nifi.elasticsearch.ElasticSearchClientService

Service Implementations

org.apache.nifi.elasticsearch.ElasticSearchClientServiceImpl

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Index
The name of the index to use.

Display Name

Index

Description

The name of the index to use.

API Name

el-rest-fetch-index

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

true
Aggregation Results Format
Format of Aggregation output.
Display Name

Aggregation Results Format

Description

Format of Aggregation output.

API Name

el-rest-format-aggregations

Default Value

FULL

Allowable Values
- FULL
- BUCKETS_ONLY
- METADATA_ONLY
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Search Results Format
Format of Hits output.
Display Name

Search Results Format

Description

Format of Hits output.

API Name

el-rest-format-hits

Default Value

FULL

Allowable Values
- FULL
- SOURCE_ONLY
- METADATA_ONLY
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Output No Hits
Output a "hits" flowfile even if no hits found for query. If true, an empty "hits" flowfile will be output even if "aggregations" are output.
Display Name

Output No Hits

Description

Output a "hits" flowfile even if no hits found for query. If true, an empty "hits" flowfile will be output even if "aggregations" are output.

API Name

el-rest-output-no-hits

Default Value

false

Allowable Values
- true
- false
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Pagination Keep Alive
Pagination "keep_alive" period. Period Elasticsearch will keep the scroll/pit cursor alive in between requests (this is not the time expected for all pages to be returned, but the maximum allowed time for requests between page retrievals).

Display Name

Pagination Keep Alive

Description

Pagination "keep_alive" period. Period Elasticsearch will keep the scroll/pit cursor alive in between requests (this is not the time expected for all pages to be returned, but the maximum allowed time for requests between page retrievals).

API Name

el-rest-pagination-keep-alive

Default Value

10 mins

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Pagination Type
Pagination method to use. Not all types are available for all Elasticsearch versions, check the Elasticsearch docs to confirm which are applicable and recommended for your service.
Display Name

Pagination Type

Description

Pagination method to use. Not all types are available for all Elasticsearch versions, check the Elasticsearch docs to confirm which are applicable and recommended for your service.

API Name

el-rest-pagination-type

Default Value

pagination-scroll

Allowable Values
- SCROLL
- SEARCH_AFTER
- POINT_IN_TIME
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Aggregation Results Split
Output a flowfile containing all aggregations or one flowfile for each individual aggregation.
Display Name

Aggregation Results Split

Description

Output a flowfile containing all aggregations or one flowfile for each individual aggregation.

API Name

el-rest-split-up-aggregations

Default Value

splitUp-no

Allowable Values
- PER_HIT
- PER_RESPONSE
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Search Results Split
Output a flowfile containing all hits or one flowfile for each individual hit or one flowfile containing all hits from all paged responses.
Display Name

Search Results Split

Description

Output a flowfile containing all hits or one flowfile for each individual hit or one flowfile containing all hits from all paged responses.

API Name

el-rest-split-up-hits

Default Value

splitUp-no

Allowable Values
- PER_HIT
- PER_RESPONSE
- PER_QUERY
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Type
The type of this document (used by Elasticsearch for indexing and searching).

Display Name

Type

Description

The type of this document (used by Elasticsearch for indexing and searching).

API Name

el-rest-type

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false
Additional Filters
One or more query filters in JSON syntax, not Lucene syntax. Ex: [{"match":{"somefield":"somevalue"}}, {"match":{"anotherfield":"anothervalue"}}]. These filters wil be used as part of a Bool query's filter.

Display Name

Additional Filters

Description

One or more query filters in JSON syntax, not Lucene syntax. Ex: [{"match":{"somefield":"somevalue"}}, {"match":{"anotherfield":"anothervalue"}}]. These filters wil be used as part of a Bool query's filter.

API Name

es-rest-additional-filters

Expression Language Scope

Not Supported

Sensitive

false

Required

false
Aggregations
One or more query aggregations (or "aggs"), in JSON syntax. Ex: {"items": {"terms": {"field": "product", "size": 10}}}

Display Name

Aggregations

Description

One or more query aggregations (or "aggs"), in JSON syntax. Ex: {"items": {"terms": {"field": "product", "size": 10}}}

API Name

es-rest-query-aggs

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false
Fields
Fields of indexed documents to be retrieved, in JSON syntax. Ex: ["user.id", "http.response.*", {"field": "@timestamp", "format": "epoch_millis"}]

Display Name

Fields

Description

Fields of indexed documents to be retrieved, in JSON syntax. Ex: ["user.id", "http.response.*", {"field": "@timestamp", "format": "epoch_millis"}]

API Name

es-rest-query-fields

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false
Script Fields
Fields to created using script evaluation at query runtime, in JSON syntax. Ex: {"test1": {"script": {"lang": "painless", "source": "doc['price'].value * 2"}}, "test2": {"script": {"lang": "painless", "source": "doc['price'].value * params.factor", "params": {"factor": 2.0}}}}

Display Name

Script Fields

Description

Fields to created using script evaluation at query runtime, in JSON syntax. Ex: {"test1": {"script": {"lang": "painless", "source": "doc['price'].value * 2"}}, "test2": {"script": {"lang": "painless", "source": "doc['price'].value * params.factor", "params": {"factor": 2.0}}}}

API Name

es-rest-query-script-fields

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false
Sort
Sort results by one or more fields, in JSON syntax. Ex: [{"price" : {"order" : "asc", "mode" : "avg"}}, {"post_date" : {"format": "strict_date_optional_time_nanos"}}]

Display Name

Sort

Description

Sort results by one or more fields, in JSON syntax. Ex: [{"price" : {"order" : "asc", "mode" : "avg"}}, {"post_date" : {"format": "strict_date_optional_time_nanos"}}]

API Name

es-rest-query-sort

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false
Range Query Field
Field to be tracked as part of an Elasticsearch Range query using a "gt" bound match. This field must exist within the Elasticsearch document for it to be retrieved.

Display Name

Range Query Field

Description

Field to be tracked as part of an Elasticsearch Range query using a "gt" bound match. This field must exist within the Elasticsearch document for it to be retrieved.

API Name

es-rest-range-field

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Initial Value Date Format
If the "Range Query Field" is a Date field, convert the "Initial Value" to a date with this format. If not specified, Elasticsearch will use the date format provided by the "Range Query Field"'s mapping. For valid syntax, see https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html
Display Name

Initial Value Date Format

Description

If the "Range Query Field" is a Date field, convert the "Initial Value" to a date with this format. If not specified, Elasticsearch will use the date format provided by the "Range Query Field"'s mapping. For valid syntax, see https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html

API Name

es-rest-range-format

Expression Language Scope

Not Supported

Sensitive

false

Required

false

Dependencies
- Initial Value is set to any value specified
Initial Value
The initial value to use for the query if the processor has not run previously. If the processor has run previously and stored a value in its state, this property will be ignored. If no value is provided, and the processor has not previously run, no Range query bounds will be used, i.e. all documents will be retrieved in the specified "Sort Order".

Display Name

Initial Value

Description

The initial value to use for the query if the processor has not run previously. If the processor has run previously and stored a value in its state, this property will be ignored. If no value is provided, and the processor has not previously run, no Range query bounds will be used, i.e. all documents will be retrieved in the specified "Sort Order".

API Name

es-rest-range-initial-value

Expression Language Scope

Not Supported

Sensitive

false

Required

false
Initial Value Date Time Zone
If the "Range Query Field" is a Date field, convert the "Initial Value" to UTC with this time zone. Valid values are ISO 8601 UTC offsets, such as "+01:00" or "-08:00", and IANA time zone IDs, such as "Europe/London".
Display Name

Initial Value Date Time Zone

Description

If the "Range Query Field" is a Date field, convert the "Initial Value" to UTC with this time zone. Valid values are ISO 8601 UTC offsets, such as "+01:00" or "-08:00", and IANA time zone IDs, such as "Europe/London".

API Name

es-rest-range-time-zone

Expression Language Scope

Not Supported

Sensitive

false

Required

false

Dependencies
- Initial Value is set to any value specified
Size
The maximum number of documents to retrieve in the query. If the query is paginated, this "size" applies to each page of the query, not the "size" of the entire result set.

Display Name

Size

Description

The maximum number of documents to retrieve in the query. If the query is paginated, this "size" applies to each page of the query, not the "size" of the entire result set.

API Name

es-rest-size

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false
Sort Order
The order in which to sort the "Range Query Field". A "sort" clause for the "Range Query Field" field will be prepended to any provided "Sort" clauses. If a "sort" clause already exists for the "Range Query Field" field, it will not be updated.
Display Name

Sort Order

Description

The order in which to sort the "Range Query Field". A "sort" clause for the "Range Query Field" field will be prepended to any provided "Sort" clauses. If a "sort" clause already exists for the "Range Query Field" field, it will not be updated.

API Name

es-rest-sort-order

Default Value

asc

Allowable Values
- asc
- desc
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Max JSON Field String Length
The maximum allowed length of a string value when parsing a JSON document or attribute.

Display Name

Max JSON Field String Length

Description

The maximum allowed length of a string value when parsing a JSON document or attribute.

API Name

Max JSON Field String Length

Default Value

20 MB

Expression Language Scope

Not Supported

Sensitive

false

Required

true

Dynamic Properties

The name of a URL query parameter to add
Adds the specified property name/value as a query parameter in the Elasticsearch URL used for processing. These parameters will override any matching parameters in the query request body. For SCROLL type queries, these parameters are only used in the initial (first page) query as the Elasticsearch Scroll API does not support the same query parameters for subsequent pages of data.

Name

The name of a URL query parameter to add

Description

Adds the specified property name/value as a query parameter in the Elasticsearch URL used for processing. These parameters will override any matching parameters in the query request body. For SCROLL type queries, these parameters are only used in the initial (first page) query as the Elasticsearch Scroll API does not support the same query parameters for subsequent pages of data.

Value

The value of the URL query parameter

Expression Language Scope

ENVIRONMENT

State Management

Scopes	Description
CLUSTER	The pagination state (scrollId, searchAfter, pitId, hitCount, pageCount, pageExpirationTimestamp, trackingRangeValue) is retained in between invocations of this processor until the Scroll/PiT has expired (when the current time is later than the last query execution plus the Pagination Keep Alive interval).

System Resource Considerations

Resource	Description
MEMORY	Care should be taken on the size of each page because each response from Elasticsearch will be loaded into memory all at once and converted into the resulting flowfiles.

Relationships

Name	Description
aggregations	Aggregations are routed to this relationship.
hits	Search hits are routed to this relationship.

Writes Attributes

Name	Description
mime.type	application/json
page.number	The number of the page (request), starting from 1, in which the results were returned that are in the output flowfile
hit.count	The number of hits that are in the output flowfile
elasticsearch.query.error	The error message provided by Elasticsearch if there is an error querying the index.