UpdateRecord 2.1.0

Bundle
org.apache.nifi | nifi-standard-nar
Description
Updates the contents of a FlowFile that contains Record-oriented data (i.e., data that can be read via a RecordReader and written by a RecordWriter). This Processor requires that at least one user-defined Property be added. The name of the Property should indicate a RecordPath that determines the field that should be updated. The value of the Property is either a replacement value (optionally making use of the Expression Language) or is itself a RecordPath that extracts a value from the Record. Whether the Property value is determined to be a RecordPath or a literal value depends on the configuration of the <Replacement Value Strategy> Property.
Tags
avro, csv, freeform, generic, json, log, logs, record, schema, text, update
Input Requirement
REQUIRED
Supports Sensitive Dynamic Properties
false
  • Additional Details for UpdateRecord 2.1.0

    UpdateRecord

    UpdateRecord makes use of the NiFi RecordPath Domain-Specific Language (DSL) to allow the user to indicate which field(s) in the Record should be updated. Users do this by adding a User-defined Property to the Processor’s configuration. The name of the User-defined Property must be the RecordPath text that should be evaluated against each Record. The value of the Property specifies what value should go into that selected Record field.

    When specifying the replacement value (the value of the User-defined Property), the user is able to specify a literal value such as the number 10; an Expression Language Expression to reference FlowFile attributes, such as${filename}; or another RecordPath path from which to retrieve the desired value from the Record itself. Whether the value entered should be interpreted as a literal or a RecordPath path is determined by the value of the Property.

    If a RecordPath is given and does not match any field in an input Record, that Property will be skipped and all other Properties will still be evaluated. If the RecordPath matches exactly one field, that field will be updated with the corresponding value. If multiple fields match the RecordPath, then all fields that match will be updated. If the replacement value is itself a RecordPath that does not match, then a null value will be set for the field. For instances where this is not the desired behavior, RecordPath predicates can be used to filter the fields that match so that no fields will be selected. See RecordPath Predicates for more information.

    Below, we lay out some examples in order to provide clarity about the Processor’s behavior. For all the examples below, consider the example to operate on the following set of 2 (JSON) records:

    [
  {
    "id": 17,
    "name": "John",
    "child": {
      "id": "1"
    },
    "siblingIds": [
      4,
      8
    ],
    "siblings": [
      {
        "name": "Jeremy",
        "id": 4
      },
      {
        "name": "Julia",
        "id": 8
      }
    ]
  },
  {
    "id": 98,
    "name": "Jane",
    "child": {
      "id": 2
    },
    "gender": "F",
    "siblingIds": [],
    "siblings": []
  }
]

    For brevity, we will omit the corresponding schema and configuration of the RecordReader and RecordWriter. Otherwise, consider the following set of Properties are configured for the Processor and their associated outputs.

    Example 1 - Replace with Literal

    Here, we will replace the name of each Record with the name ‘Jeremy’ and set the gender to ‘M’:

    Property Name Property Value
    Replacement Value Strategy Literal Value
    /name Jeremy
    /gender M

    This will yield the following output:

    [
  {
    "id": 17,
    "name": "Jeremy",
    "child": {
      "id": "1"
    },
    "gender": "M",
    "siblingIds": [
      4,
      8
    ],
    "siblings": [
      {
        "name": "Jeremy",
        "id": 4
      },
      {
        "name": "Julia",
        "id": 8
      }
    ]
  },
  {
    "id": 98,
    "name": "Jeremy",
    "child": {
      "id": 2
    },
    "gender": "M",
    "siblingIds": [],
    "siblings": []
  }
]

    Note that even though the first record did not have a “gender” field in the input, one will be added after the “child” field, as that’s where the field is located in the schema.

    Example 2 - Replace with RecordPath

    This example will replace the value in one field of the Record with the value from another field. For this example, consider the following set of Properties:

    Property Name Property Value
    Replacement Value Strategy Record Path Value
    /name /siblings[0]/name

    This will yield the following output:

    [
  {
    "id": 17,
    "name": "Jeremy",
    "child": {
      "id": "1"
    },
    "siblingIds": [
      4,
      8
    ],
    "siblings": [
      {
        "name": "Jeremy",
        "id": 4
      },
      {
        "name": "Julia",
        "id": 8
      }
    ]
  },
  {
    "id": 98,
    "name": null,
    "child": {
      "id": 2
    },
    "gender": "F",
    "siblingIds": [],
    "siblings": []
  }
]

    Example 3 - Replace with Relative RecordPath

    In the above example, we replaced the value of field based on another RecordPath. That RecordPath was an “absolute RecordPath,” meaning that it starts with a “slash” character (/) and therefore it specifies the path from the “root” or “outermost” element. However, sometimes we want to reference a field in such a way that we defined the RecordPath relative to the field being updated. This example does just that. For each of the siblings given in the “siblings"array, we will replace the sibling’s name with their id’s. To do so, we will configure the processor with the following properties:

    Property Name Property Value
    Replacement Value Strategy Record Path Value
    /siblings[*]/name ../id

    Note that the RecordPath that was given for the value starts with .., which is a reference to the parent. We do this because the field that we are going to update is the “name” field of the sibling. To get to the associated “id” field, we need to go to the “name” field’s parent and then to its “id” child field. The above example results in the following output:

    [
  {
    "id": 17,
    "name": "Jeremy",
    "child": {
      "id": "1"
    },
    "siblingIds": [
      4,
      8
    ],
    "siblings": [
      {
        "name": "Jeremy",
        "id": 4
      },
      {
        "name": "Julia",
        "id": 8
      }
    ]
  },
  {
    "id": 98,
    "name": null,
    "child": {
      "id": 2
    },
    "gender": "F",
    "siblingIds": [],
    "siblings": []
  }
]

    Example 4 - Replace Multiple Values

    This example will replace the value of all fields that have the name “id”, regardless of where in the Record hierarchy the field is found. The value that it uses references the Expression Language, so for this example, let’s assume that the incoming FlowFile has an attribute named “replacement.id” that has a value of “91”:

    Property Name Property Value
    Replacement Value Strategy Literal Value
    //id ${replacement.id}

    This will yield the following output:

    [
  {
    "id": 91,
    "name": "John",
    "child": {
      "id": "91"
    },
    "siblingIds": [
      4,
      8
    ],
    "siblings": [
      {
        "name": "Jeremy",
        "id": 91
      },
      {
        "name": "Julia",
        "id": 91
      }
    ]
  },
  {
    "id": 91,
    "name": "Jane",
    "child": {
      "id": 91
    },
    "gender": "F",
    "siblingIds": [],
    "siblings": []
  }
]

    It is also worth noting that in this example, some of the “id” fields were of type STRING, while others were of type INT. This is okay because the RecordReaders and RecordWriters should handle these simple type coercions for us.

    Example 5 - Use Expression Language to Modify Value

    This example will capitalize the value of all ’name’ fields, regardless of where in the Record hierarchy the field is found. This is done by referencing the ‘field.value’ variable in the Expression Language. We can also access the field.name variable and the field.type variable.

    Property Name Property Value
    Replacement Value Strategy Literal Value
    //name ${field.value:toUpper()}

    This will yield the following output:

    [
  {
    "id": 17,
    "name": "JOHN",
    "child": {
      "id": "1"
    },
    "siblingIds": [
      4,
      8
    ],
    "siblings": [
      {
        "name": "JEREMY",
        "id": 4
      },
      {
        "name": "JULIA",
        "id": 8
      }
    ]
  },
  {
    "id": 98,
    "name": "JANE",
    "child": {
      "id": 2
    },
    "gender": "F",
    "siblingIds": [],
    "siblings": []
  }
]
Properties
Dynamic Properties
Relationships
Name Description
success FlowFiles that are successfully transformed will be routed to this relationship
failure If a FlowFile cannot be transformed from the configured input format to the configured output format, the unchanged FlowFile will be routed to this relationship
Writes Attributes
Name Description
record.index This attribute provides the current row index and is only available inside the literal value expression.
record.error.message This attribute provides on failure the error message encountered by the Reader or Writer.
Use Cases
  • Combine multiple fields into a single field.
    Description
    Combine multiple fields into a single field.
    Keywords
    combine, concatenate, recordpath
    Configuration
    "Replacement Value Strategy" = "Record Path Value"

A single additional property is added to the Processor. The name of the property is a RecordPath identifying the field to place the result in.
The value of the property uses the CONCAT Record Path function to concatenate multiple values together, potentially using other string literal values.
For example, to combine the `title`, `firstName` and `lastName` fields into a single field named `fullName`, we add a property with the name `/fullName` and a value of `CONCAT(/title, ' ', /firstName, ' ', /lastName)`
  • Change the value of a record field to an explicit value.
    Description
    Change the value of a record field to an explicit value.
    Keywords
    change, update, replace, transform
    Configuration
        "Replacement Value Strategy" = "Literal Value"

    A single additional property is added to the Processor. The name of the property is a RecordPath identifying the field to place the result in.
    The value of the property is the explicit value to set the field to. For example, we can set any field with a name of `txId`, regardless of its level in the data's hierarchy,     to `1111-1111` by adding a property with a name of `//txId` and a value of `1111-1111`
  • Copy the value of one record field to another record field.
    Description
    Copy the value of one record field to another record field.
    Keywords
    change, update, copy, recordpath, hierarchy, transform
    Configuration
        "Replacement Value Strategy" = "Record Path Value"

    A single additional property is added to the Processor. The name of the property is a RecordPath identifying the field to update.
    The value of the property is a RecordPath identifying the field to copy the value from.
    For example, we can copy the value of `/identifiers/all/imei` to the `identifier` field at the root level, by adding a property named     `/identifier` with a value of `/identifiers/all/imei`.
  • Enrich data by injecting the value of an attribute into each Record.
    Description
    Enrich data by injecting the value of an attribute into each Record.
    Keywords
    enrich, attribute, change, update, replace, insert, transform
    Configuration
    "Replacement Value Strategy" = "Literal Value"

A single additional property is added to the Processor. The name of the property is a RecordPath identifying the field to place the result in.
The value of the property is an Expression Language expression that references the attribute of interest. We can, for example, insert a new field name `filename` into each record by adding a property named `/filename` with a value of `${filename}`
  • Change the format of a record field's value.
    Description
    Change the format of a record field's value.
    Notes
    Use the RenameRecordField Processor in order to change a field's name.
    Keywords
    change, update, replace, insert, transform, format, date/time, timezone, expression language
    Configuration
    "Replacement Value Strategy" = "Literal Value"

A single additional property is added to the Processor. The name of the property is a RecordPath identifying the field to update.
The value is an Expression Language expression that references the `field.value` variable. For example, to change the date/time format of a field named `txDate` from `year-month-day` format to `month/day/year` format, we add a property named `/txDate` with a value of `${field.value:toDate('yyyy-MM-dd'):format('MM/dd/yyyy')}`. We could also change the timezone of a timestamp field (and insert the timezone for clarity) by using a value of `${field.value:toDate('yyyy-MM-dd HH:mm:ss', 'UTC-0400'):format('yyyy-MM-dd HH:mm:ss Z', 'UTC')}`.
See Also