JsonQueryElasticsearch

JsonQueryElasticsearch 2.0.0

Bundle: org.apache.nifi | nifi-elasticsearch-restapi-nar
Description: A processor that allows the user to run a query (with aggregations) written with the Elasticsearch JSON DSL. It does not automatically paginate queries for the user. If an incoming relationship is added to this processor, it will use the flowfile's content for the query. Care should be taken on the size of the query because the entire response from Elasticsearch will be loaded into memory all at once and converted into the resulting flowfiles.
Tags: elasticsearch, elasticsearch5, elasticsearch6, elasticsearch7, elasticsearch8, get, json, query, read
Input Requirement: ALLOWED
Supports Sensitive Dynamic Properties: false

Properties

Query Attribute
If set, the executed query will be set on each result flowfile in the specified attribute.

Display Name

Query Attribute

Description

If set, the executed query will be set on each result flowfile in the specified attribute.

API Name

el-query-attribute

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false
Client Service
An Elasticsearch client service to use for running queries.

Display Name

Client Service

Description

An Elasticsearch client service to use for running queries.

API Name

el-rest-client-service

Service Interface

org.apache.nifi.elasticsearch.ElasticSearchClientService

Service Implementations

org.apache.nifi.elasticsearch.ElasticSearchClientServiceImpl

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Index
The name of the index to use.

Display Name

Index

Description

The name of the index to use.

API Name

el-rest-fetch-index

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

true
Aggregation Results Format
Format of Aggregation output.
Display Name

Aggregation Results Format

Description

Format of Aggregation output.

API Name

el-rest-format-aggregations

Default Value

FULL

Allowable Values
- FULL
- BUCKETS_ONLY
- METADATA_ONLY
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Search Results Format
Format of Hits output.
Display Name

Search Results Format

Description

Format of Hits output.

API Name

el-rest-format-hits

Default Value

FULL

Allowable Values
- FULL
- SOURCE_ONLY
- METADATA_ONLY
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Output No Hits
Output a "hits" flowfile even if no hits found for query. If true, an empty "hits" flowfile will be output even if "aggregations" are output.
Display Name

Output No Hits

Description

Output a "hits" flowfile even if no hits found for query. If true, an empty "hits" flowfile will be output even if "aggregations" are output.

API Name

el-rest-output-no-hits

Default Value

false

Allowable Values
- true
- false
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Query
A query in JSON syntax, not Lucene syntax. Ex: {"query":{"match":{"somefield":"somevalue"}}}. If this parameter is not set, the query will be read from the flowfile content. If the query (property and flowfile content) is empty, a default empty JSON Object will be used, which will result in a "match_all" query in Elasticsearch.
Display Name

Query

Description

A query in JSON syntax, not Lucene syntax. Ex: {"query":{"match":{"somefield":"somevalue"}}}. If this parameter is not set, the query will be read from the flowfile content. If the query (property and flowfile content) is empty, a default empty JSON Object will be used, which will result in a "match_all" query in Elasticsearch.

API Name

el-rest-query

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies
- Query Definition Style is set to any of [full]
Query Clause
A "query" clause in JSON syntax, not Lucene syntax. Ex: {"match":{"somefield":"somevalue"}}. If the query is empty, a default JSON Object will be used, which will result in a "match_all" query in Elasticsearch.
Display Name

Query Clause

Description

A "query" clause in JSON syntax, not Lucene syntax. Ex: {"match":{"somefield":"somevalue"}}. If the query is empty, a default JSON Object will be used, which will result in a "match_all" query in Elasticsearch.

API Name

el-rest-query-clause

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies
- Query Definition Style is set to any of [build]
Query Definition Style
How the JSON Query will be defined for use by the processor.
Display Name

Query Definition Style

Description

How the JSON Query will be defined for use by the processor.

API Name

el-rest-query-definition-style

Default Value

full

Allowable Values
- FULL_QUERY
- BUILD_QUERY
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Aggregation Results Split
Output a flowfile containing all aggregations or one flowfile for each individual aggregation.
Display Name

Aggregation Results Split

Description

Output a flowfile containing all aggregations or one flowfile for each individual aggregation.

API Name

el-rest-split-up-aggregations

Default Value

splitUp-no

Allowable Values
- PER_HIT
- PER_RESPONSE
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Search Results Split
Output a flowfile containing all hits or one flowfile for each individual hit.
Display Name

Search Results Split

Description

Output a flowfile containing all hits or one flowfile for each individual hit.

API Name

el-rest-split-up-hits

Default Value

splitUp-no

Allowable Values
- PER_HIT
- PER_RESPONSE
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Type
The type of this document (used by Elasticsearch for indexing and searching).

Display Name

Type

Description

The type of this document (used by Elasticsearch for indexing and searching).

API Name

el-rest-type

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false
Aggregations
One or more query aggregations (or "aggs"), in JSON syntax. Ex: {"items": {"terms": {"field": "product", "size": 10}}}
Display Name

Aggregations

Description

One or more query aggregations (or "aggs"), in JSON syntax. Ex: {"items": {"terms": {"field": "product", "size": 10}}}

API Name

es-rest-query-aggs

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies
- Query Definition Style is set to any of [build]
Fields
Fields of indexed documents to be retrieved, in JSON syntax. Ex: ["user.id", "http.response.*", {"field": "@timestamp", "format": "epoch_millis"}]
Display Name

Fields

Description

Fields of indexed documents to be retrieved, in JSON syntax. Ex: ["user.id", "http.response.*", {"field": "@timestamp", "format": "epoch_millis"}]

API Name

es-rest-query-fields

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies
- Query Definition Style is set to any of [build]
Script Fields
Fields to created using script evaluation at query runtime, in JSON syntax. Ex: {"test1": {"script": {"lang": "painless", "source": "doc['price'].value * 2"}}, "test2": {"script": {"lang": "painless", "source": "doc['price'].value * params.factor", "params": {"factor": 2.0}}}}
Display Name

Script Fields

Description

Fields to created using script evaluation at query runtime, in JSON syntax. Ex: {"test1": {"script": {"lang": "painless", "source": "doc['price'].value * 2"}}, "test2": {"script": {"lang": "painless", "source": "doc['price'].value * params.factor", "params": {"factor": 2.0}}}}

API Name

es-rest-query-script-fields

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies
- Query Definition Style is set to any of [build]
Sort
Sort results by one or more fields, in JSON syntax. Ex: [{"price" : {"order" : "asc", "mode" : "avg"}}, {"post_date" : {"format": "strict_date_optional_time_nanos"}}]
Display Name

Sort

Description

Sort results by one or more fields, in JSON syntax. Ex: [{"price" : {"order" : "asc", "mode" : "avg"}}, {"post_date" : {"format": "strict_date_optional_time_nanos"}}]

API Name

es-rest-query-sort

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies
- Query Definition Style is set to any of [build]
Size
The maximum number of documents to retrieve in the query. If the query is paginated, this "size" applies to each page of the query, not the "size" of the entire result set.
Display Name

Size

Description

The maximum number of documents to retrieve in the query. If the query is paginated, this "size" applies to each page of the query, not the "size" of the entire result set.

API Name

es-rest-size

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies
- Query Definition Style is set to any of [build]
Max JSON Field String Length
The maximum allowed length of a string value when parsing a JSON document or attribute.

Display Name

Max JSON Field String Length

Description

The maximum allowed length of a string value when parsing a JSON document or attribute.

API Name

Max JSON Field String Length

Default Value

20 MB

Expression Language Scope

Not Supported

Sensitive

false

Required

true

Dynamic Properties

The name of a URL query parameter to add
Adds the specified property name/value as a query parameter in the Elasticsearch URL used for processing. These parameters will override any matching parameters in the query request body

Name

The name of a URL query parameter to add

Description

Adds the specified property name/value as a query parameter in the Elasticsearch URL used for processing. These parameters will override any matching parameters in the query request body

Value

The value of the URL query parameter

Expression Language Scope

FLOWFILE_ATTRIBUTES

Relationships

Name	Description
failure	All flowfiles that fail for reasons unrelated to server availability go to this relationship.
aggregations	Aggregations are routed to this relationship.
hits	Search hits are routed to this relationship.
original	All original flowfiles that don't cause an error to occur go to this relationship.

Writes Attributes

Name	Description
mime.type	application/json
aggregation.name	The name of the aggregation whose results are in the output flowfile
aggregation.number	The number of the aggregation whose results are in the output flowfile
hit.count	The number of hits that are in the output flowfile
elasticsearch.query.error	The error message provided by Elasticsearch if there is an error querying the index.