Context

Consider the following simple Pipe, which pings a site and extracts the ping times:

name: ping
input:
  exec:
    command: 'ping -i 4 google.com'
actions:
  - extract:
      input-field: _raw
      remove: true
      pattern: '=(\S+) ms$'
      output-fields:
        - msec
  - convert:
      - msec: num
  - add:
      output-fields:
        - site: google.com
output:
  exec:
    command: cat

The following steps can be used to create a new site and add this Pipe:

$ hotrod pipes add --file ping.yml
$ hotrod targets add Office
$ hotrod targets update Office --add-pipe ping

This Pipe has two arbitrary values: the site itself and the time between pings.

So, add a context section:

name: ping
context:
  addr: google.com
  interval: 4
input:
  exec:
    command: 'ping -i {{interval}} {{addr}}'
actions:
  - extract:
      input-field: _raw
      remove: true
      pattern: '=(\S+) ms$'
      output-fields:
        - msec
  - convert:
      - msec: num
  - add:
      output-fields:
        - site: '{{addr}}'
output:
  exec:
    command: cat

Any context values can be expanded in {{}} (double-braces).

Running this, after hotrod pipes update --file ping.yml to update:

{"msec":48.3,"site":"google.com"}
{"msec":49,"site":"google.com"}
{"msec":48.7,"site":"google.com"}
....

This is a useful way to avoid repeating yourself, but the real power of contexts is that they can be overriden:

$ hotrod context global addr=yahoo.com
# .... run as before
{"msec":374,"site":"yahoo.com"}
{"msec":317,"site":"yahoo.com"}
{"msec":362,"site":"yahoo.com"}
{"msec":306,"site":"yahoo.com"}
{"msec":351,"site":"yahoo.com"}
....

You can now add another site, and deploy the ping Pipe to it:

$ hotrod targets add Durban
site Durban has id 8c71be6f-b756-428c-8d35-cbe7be0dcfbe
$ hotrod targets update Durban --add-pipe ping

Again, it will ping "yahoo.com" since the global context overrides the original Pipe context - which we call the local context:

There is something special about Durban and we want it to ping Johannesburg.

hotrod targets context Durban addr=panoptix.co.za

A per-target context overrides global context, since it is more specialized.

We can create yet another target, Cape Town, then attach the tag "seaside" to it, as well as Durban.

$ hotrod targets add 'Cape Town' --tags seaside
site Cape Town has id 537c7521-9151-4384-b8c3-0103c417958c
$ hotrod targets update 'Cape Town' --add-pipe ping
$ hotrod targets update Durban --add-tag seaside
$ hotrod targets list
 name      | id                                   | tags      | pipes  | last seen
-----------+--------------------------------------+-----------+--------+-----------
 Office    | a5bb18fc-6d5a-43fc-a5fa-7835f14d6713 |           | ping   |
 Joburg    | 9568fdba-f4f6-4d7e-a6ba-dcfcef64cf95 |           | uptime |
 Durban    | f98ea805-4210-4ba2-a898-80148e3aaf5a | seaside   | ping   |
 Cape Town | 537c7521-9151-4384-b8c3-0103c417958c | seaside   | ping   |

Can attach context to a tag:

hotrod context tag seaside addr=jhb.co.za

That way, Cape Town gets "jhb.co.za" from the tag context - Durban remains "panoptix.co.za" since target context always wins.

$ hotrod targets show 'Cape Town'
id: 537c7521-9151-4384-b8c3-0103c417958c
tenant: default
tags: seaside
pipes: ping

$ hotrod targets show Durban
id: 8c71be6f-b756-428c-8d35-cbe7be0dcfbe
tenant: default
tags: seaside
pipe: ping
context:
---
addr: panoptix.co.za

Tag contexts are a useful way for a whole group of targets to acquire different settings from the global default.

Finally, all instances of a pipe running on different targets share a context, which is settable with hotrod pipes context. So we can point all our pings at a new address by setting addr in the pipe context, except where the target context overrides.

To summarize: a higher-priority context overrides a lower-priority context, with target contexts the highest priority and local context the lowest priority

  • target
  • pipe
  • tag
  • global
  • local

There are three context variables always available to a pipe:

  • pipe the name of the pipe itself
  • name the target name
  • target the target identifier
  • tenant the tenant (group owning the resource)

So it is common for a pipe to enrich its data with the name of the target it is running on:

add:
  output-fields:
  - name: '{{name}}'

Please note that overriding a context variable causes the redeployment of the effected pipes, since context expansion happens server-side.

Multiple hosts in a single Pipe

If we wanted a Target to ping multiple hosts, then one solution would be to make copies of ping.yml as ping1.yml, ping2.yml. This would be irritating because all these files need to be changed together. The foreach feature handles this transparently:

name: pinga
context:
  addresses:
  - google.com
  - yahoo.com
foreach:
  addr: addresses
input:
  exec:
    command: ping -i 2 {{addr}}
...

Again, the default hosts go in the context as addresses, but as an array.

foreach here says that each addr should come from this context array.

The result is that two pipe services will be created on the target, one with ping -i 2 google.com and another with ping -i 2 yahoo.com. The pipe services will be called ping-00 and ping-01.

Effectively this creates two copies of the Pipe definition for the Target.

The array may be nested, using the same dot notation for context expansions:

context:
  my_context:
    addresses:
    - google.com
    - yahoo.com
foreach:
  addr: my_context.addresses

We now want to ping the same hosts, but through either one of the available interfaces on the machine:

name: pinga
context:
  addresses:
    - google.com
    - yahoo.com
  interfaces:
    - eth0
    - eth1
foreach:
  addr: addresses
  interface: interfaces
input:
  exec:
    command: ping -i 2 {{addr}} -I {{interface}}

This generates four pipe services for the possible combinations:

  • ping -i 2 google.com -I eth0
  • ping -i 2 google.com -I eth1
  • ping -i 2 yahoo.com -I eth0
  • ping -i 2 yahoo.com -I eth1

Another form of foreach allows you to go over nested arrays in the context:

name: test1
context:
  hosts:
    - spoof: 192.168.66.38
      targets:
        - 1.1.1.1
        - 2.2.2.2
      port: 22
    - spoof: 192.168.66.39
      targets:
        - 3.3.3.3
        - 4.4.4.4
      port: 22      
foreach:
  host: hosts[targets]
input:
  exec:
    command: echo {{ host.spoof }} {{ host.targets }}:{{ host.port }}
    raw: true
output:
  write: console
# 192.168.66.38 1.1.1.1:22
# 192.168.66.38 2.2.2.2:22
# 192.168.66.39 3.3.3.3:22
# 192.168.66.39 4.4.4.4:22

Aliases

An alias is a way to refer to an input, output, or set of actions.

For example, consider test.alias:

name: enrich
actions:
- add:
    output-fields:
      - name: '{{name}}'
      - tenant: '{{tenant}}'
- time:
    output-field: '@timestamp'

It looks very much like a Pipe definition, except that it does not need to have all the three components of a Pipe (input, actions, and output).

This is because it is substituted into a Pipe definition, so:

name: uptime-enrich
input:
  exec:
    command: uptime
    interval: 2s
actions:
  - extract:
      input-field: _raw
      pattern: 'load average: (\S+), (\S+), (\S+)'
        output-fields: [m1, m5, m15]
  - alias: enrich
output:
    write: console

So all the site-specific enrichments can be wrapped up in a alias definition, and reused in any pipe.

Once the alias file has been updated with hotrod aliases update test.alias, uptime-enrich.yml can be loaded in the usual way.

Aliases can also provide 'canned' definitions of inputs and outputs, which is useful for testing.