exec

Execute arbitrary commands

Some things to take note of:

  • Shell expansion ($VAR) will take place using the Pipe environment
  • Will usually wrap the output as JSON in the form {"_raw":...}
  • Any linefeeds will be removed from the command, so this works fine:
    exec:
      command: |
        docker ps
          -all
          --no-trunc
          --quiet
          --size
    
Field NameDescriptionTypeDefault
commandA shell command to be executedstring-
rawDo not wrap incoming data in JSON format (i.e. `{"_raw":"some data"})boolfalse
countNumber of times to run the commandduration-
no-strip-linefeedsDo not strip newline characters (\n) automatically on multi-line commandsboolfalse
ignore-line-breaksDo not treat separate output lines as distinct eventsboolfalse
resultOffers a way to collect the full output of a command: stdout, stdin, and exit status--
batchFor when a number of events need to be marked as belonging to a distinct group--
envAccepts a YAML file containing variables that would be added to environment of the commandstring-
intervalHow often to run the commandduration-
cronHow often to run the commandcron-
immediateRun as soon as invoked, instead of waiting for the specified cron intervalboolfalse
random-offsetSets a random offset to the schedule, then sticks to itduration-
windowFor resources that need a time window to be specified--
retryFor operations that could potentially fail--

command

A shell command to be executed

Type: string

Example

action:

exec:
  command: echo 'one two'

output:

{"_raw":"one two"}

raw

Do not wrap incoming data in JSON format (i.e. `{"_raw":"some data"})

This is useful when incoming data is already in JSON format.

Type: bool

Example: Enabled

action:

exec:
  command: echo one
  raw: true

output:

one

Example: Disabled (default)

action:

exec:
  command: echo one

output:

{"_raw":"one"}

Example: Enabled, with JSON data

action:

exec:
  command: echo '{"one":1}'
  raw: true

output:

{"one":1}

Example: Disabled, with JSON data (note the escapes)

action:

exec:
  command: echo '{"one":1}'

output:

{"_raw":"{\"one\":1}"}

count

Number of times to run the command

Type: duration

Example

action:

exec:
  command: echo 'one two'
  interval: 1s
  count: 3

output:

{"_raw":"one two"}
{"_raw":"one two"}
{"_raw":"one two"}

no-strip-linefeeds

Do not strip newline characters (\n) automatically on multi-line commands

Type: bool

Example: Enabled

action:

exec:
  command: |
    echo one
    echo two
  no-strip-linefeeds: true

output:

{"_raw":"one"}
{"_raw":"two"}

Example: Enabled, with a more realistic example (this will fail as one line)

action:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true

output:

{"_raw":"1"}
{"_raw":"2"}
{"_raw":"3"}

Example: Disabled

action:

exec:
  command: |
    echo one
    echo two

output:

{"_raw":"one echo two"}

ignore-line-breaks

Do not treat separate output lines as distinct events

Type: bool

Example: Enabled

action:

exec:
  command: |
    echo one
    echo two
  no-strip-linefeeds: true
  ignore-line-breaks: true
  interval: 1s
  count: 1

output:

{"_raw":"one\ntwo"}

Example: Disabled

action:

exec:
  command: |
    echo one
    echo two
  no-strip-linefeeds: true

output:

{"_raw":"one"}
{"_raw":"two"}

result

Offers a way to collect the full output of a command: stdout, stdin, and exit status

Field NameDescriptionTypeDefault
stdout-fieldField where stdout of command will be storedstring_raw
stderr-fieldField where stderr of command will be storedstring-
status-fieldField where exit staus of command will be storedstring-

stdout-field

Field where stdout of command will be stored

Type: string

Example

action:

exec:
  command: echo foo
  result:
    stdout-field: stdout
  interval: 1m

output:

{"stdout":"foo\n"}

stderr-field

Field where stderr of command will be stored

Type: string

Example

action:

exec:
  command: "echo \"this will return with an error\" 1>&2 && false"
  result:
    stderr-field: stderr
  interval: 1m

output:

{"_raw":"","stderr":"this will return with an error"}

status-field

Field where exit staus of command will be stored

Type: string

Example

action:

exec:
  command: echo foo
  result:
    status-field: status
  interval: 1m

output:

{"_raw":"foo\n","status":0}

batch

For when a number of events need to be marked as belonging to a distinct group

Field NameDescriptionTypeDefault
uuid-fieldField where generated uuid, the unique marker for the group, will be storedstring-
invocation-time-fieldField where invocation time will be storedstring-
completion-time-fieldField where completion (end of execution) time will be storedstring-
begin-marker-fieldField used to mark first event in the groupstring-
end-marker-fieldField used to mark last event in the groupstring-

uuid-field

Field where generated uuid, the unique marker for the group, will be stored

Type: string

Example

action:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    uuid-field: marker
  interval: 1m

output:

{"_raw":"foo","line-count":3,"line-num":1,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}

Example: For cases where event count is known, a simple counter is used, instead of uuid (useful for testing)

action:

exec:
  command: echo foo
  no-strip-linefeeds: true
  count: 3
  batch:
    uuid-field: marker
  interval: 1m

output:

{"_raw":"1","line-count":3,"line-num":1,"marker":"1"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"1"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"1"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"2"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"2"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"2"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"3"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"3"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"3"}

invocation-time-field

Field where invocation time will be stored

Type: string

Example

action:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    invocation-time-field: begin
  interval: 1m

output:

{"_raw":"1","line-count":3,"line-num":1,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"2","line-count":3,"line-num":2,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"3","line-count":3,"line-num":3,"begin":"2020-01-17T09:55:09.135Z"}

completion-time-field

Field where completion (end of execution) time will be stored

Type: string

Example

action:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    invocation-time-field: begin
    completion-time-field: end
  interval: 1m

output:

{"_raw":"1","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":1}
{"_raw":"2","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":2}
{"_raw":"3","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":3}

begin-marker-field

Field used to mark first event in the group

Type: string

Example

action:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    begin-marker-field: begin
  interval: 1m

output:

{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","line-count":3,"line-num":3}

end-marker-field

Field used to mark last event in the group

Type: string

Example

action:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    begin-marker-field: begin
    end-marker-field: end
  interval: 1m

output:

{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","end":true,"line-count":3,"line-num":3}

env

Accepts a YAML file containing variables that would be added to environment of the command

Type: string

Example

file: some-file.yml

one: 1
two: 2

action:

exec:
  command: echo $one $two
  env: some-file.yml

output:

{"_raw":"1 2"}

interval

How often to run the command

Type: duration

Example

action:

exec:
  command: echo 'once a day'
  interval: 1d

cron

How often to run the command

Type: cron

Example: Once a day

action:

exec:
  command: echo 'once a day'
  cron: '0 0 0 * * *'

Example: Once a day, using a convenient shortcut

action:

exec:
  command: echo 'once a day'
  cron: '@daily'

immediate

Run as soon as invoked, instead of waiting for the specified cron interval

Type: bool

Example: Run immediately on invocation, and thereafter at 10h every morning

action:

exec:
  command: echo 'hello'
  immediate: true
  cron: '0 0 10 * * *'

random-offset

Sets a random offset to the schedule, then sticks to it

This can help avoid the thundering herd problem, where you do not, for example, want to overload some service at 00:00:00

Type: duration

Example: Would fire up to a minute after every hour

action:

exec:
  command: echo 'hello'
  random-offset: 1m
  cron: '0 0 * * * *'

window

For resources that need a time window to be specified

Field NameDescriptionTypeDefault
sizeWindow sizeduration-
offsetWindow offsetduration-
start-timeAllows the windowing to start at a specified timetime-
highwatermark-fileSpecify file where timestamp would be stored in order to resume, for when Pipe has been restartedstring-

size

Window size

Type: duration

Example

action:

exec:
  command: echo 'one two'
  window:
    size: 1m

offset

Window offset

Type: duration

Example

action:

exec:
  command: echo 'one two'
  window:
    size: 1m
    offset: 10s

start-time

Allows the windowing to start at a specified time

It should in the following format: 2019-07-10 18:45:00.000 +0200

Type: time

Example

action:

exec:
  command: echo 'one two'
  window:
    size: 1m
    start-time: 10s

highwatermark-file

Specify file where timestamp would be stored in order to resume, for when Pipe has been restarted

Type: string

Example

action:

exec:
  command: echo 'one two'
  window:
    size: 1m
    highwatermark-file:: /tmp/mark.txt

retry

For operations that could potentially fail

Field NameDescriptionTypeDefault
countHow many attempts to make before declaring failureinteger-
pauseHow long to pause before re-tryingduration-
foreverkeep trying until success is declaredboolfalse

count

How many attempts to make before declaring failure

Type: integer

Example

action:

exec:
  command: echo 'one two'
  retry:
    count: 1

output:

{"_raw":"one two"}

pause

How long to pause before re-trying

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Type: duration

Example

action:

exec:
  command: echo 'one two'
  retry:
    count: 6
    pause: 10s

output:

{"_raw":"one two"}

forever

keep trying until success is declared

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Type: bool

Example

action:

exec:
  command: echo 'one two'
  retry:
    forever: true

output:

{"_raw":"one two"}