Field Expansion

Expansion

There are two kinds of expansion in pipes. The first happens at processing-time involving the context and the second happens at run-time where we extract values from data fields.

Context Expansion

Any pipe has associated parameters, like how often to sample, what network interface to probe, etc. It is a good idea to give these values names, and define them in the context section of a pipe; as described in contexts these parameters can then be overriden.

Double-braces are used to enclose context variables ({{ name }}).

The pipe context predefines some default variables:

  • name the name of the target
  • target the target id
  • pipe the name of the pipe

Context variables can be used to simplify regular expressions:

name: temp
context:
  INT: '(\d+)'
input:
  exec:
    command: echo "today date is 2019/07/22"
actions:
- extract:
    remove: true
    pattern: '{{INT}}/{{INT}}/{{INT}}'
    output-fields:
    - year
    - month
    - day
- convert:
    auto: true
output:
    write: console

Context variable definitions may themselves include context variables:

context:
    PORT: 3030
    ADDR: '0.0.0.0:{{PORT}}'

So {{ADDR}} will expand to 0.0.0.0:3030.

It is useful to give names to arbitrary values, but the real power of context variables is that they can be overriden, per-target, per-pipe, per-tag, etc.

name: ping
context:
    interval: 5s
    address: google.com
input:
    exec:
        command: ping {{address}}
        interval: '{{interval}}'
output:
    write: console

This pipe can be pointed at different sites and made to poll at different rates by controlling the pipe's context.

NOTE changing a pipe's context requires it to be restarted.

'nested' variables can be rendered.

name: temp
context:
    object:
        person:
            name: frodo
            age: 50
input:
    exec:
        command: echo 'hello {{object.person.name}} your age is {{object.person.age}}'
output:
    write: console

(Although object.person.age is a number, a variable expansion always reasults in a text value)

Likewise, can access the elements of an array:

name: temp
context:
    array:
    - one
    - two
    - 3
input:
    exec:
        command: echo 'we have {{array.0}}, {{array.1}} and {{array.2}} {{array}}'
        json: true
output:
    write: console

{{array}} by default renders as '"one","two",3' - i.e strings are quoted.

The variable can be followed by an explicit delimiter, so {{array ' '}} will render as '"one" "two" 3'.

A wildcard expression allows a particular field to be extracted from an array of objects:

name: templ
context:
    hosts:
    - port: 10
      name: foo
    - port: 20
      name: boo
    - port: 30
      name: bar
input:
    text: host port {{hosts.*.port ' and host port '}}
output:
    write: console
# host port 10 and host port 20 and host port 30

Arrays may be merged:

name: temp
context:
  birds: [swallow,thrush]
  animals: [cat,dog]
input:
  exec:
    command: echo {{merge(birds,animals)}}
output:
    write: console
#  {"_raw":"swallow,thrush,cat,dog"}

Template Expansion

An alternative context expansion method that does function like a full templating language is also provided. These are defined in a special top-level section called templates.

name: templ
context:
    value:
    - 10
    - 20
    - 30
templates:
- name: one
  definition: '{{ value | join(sep=" // ") }}'
input:
    text: 'hello $T{one}'
output:
    write: console
# hello 10 // 20 // 30

The full power of Tera templates is available.

name: snifftest
context:
  interface: eth0
  inverted: false
  OID: '.1.3.6.1.2.1.2.2.1'
templates:
   - name: snmpd
     definition: |
        {% if inverted -%}
            {%- set INIF = 16 -%}
            {%- set OUTIF = 10 -%}
        {%- else -%}
            {%- set INIF = 10 -%}
            {%- set OUTIF = 16 -%}
        {%- endif -%}
        {{OID}}.{{INIF}}.interface {{OID}}.{{OUTIF}}.interface
input:
    text: 'snmpdelta $T{snmpd}'
output:
    write: console
# snmpdelta .1.3.6.1.2.1.2.2.1.10.interface .1.3.6.1.2.1.2.2.1.16.interface

Event Field Expansion (${variable})

At a number of places in the DSL it is possible to expand strings containing ${field} references.

For example, given {"date":"2018-06-19","time":"05:31:07"} we can construct the full ISO time like so:

- add:
    output-fields:
    - '@timestamp': '${date}T${time}Z'

(As with context expansions, the result is always a string)

Actions where field expansion is allowed:

  • add - output-fields, template and template-file
  • time - when, output-fields
  • exec - command
  • output - see outputs below
  • inputs

add is special since the field names can also be expanded.

Outputs where field expansion happens:

  • exec - command (but only if stdin: false and/or input-field defined)
  • http-post - url (but only if batch: 0)
  • redis - key
  • http-server - body

Inputs do not normally receive events (except when run as actions) so event field expansions are not present. But scheduled inputs have access to schedule variables

  • exec - command
  • http-poll - query values, address and body (if method is POST)