amqp

Read from AMQP queues

Field NameDescriptionTypeDefault
queueAn AMQP Queue--
uriAMQP address to listen tostringamqp://localhost//
routing-keyUsed in conjunction with topic exchange, to route datastring-
exchangeAn AMQP Exchange--
persistentLet messages be cached and survive server restartboolfalse
durableLet Queue or Exchange survive server restartboolfalse
rawAssume incoming data is in JSON format (do not quote)boolfalse
ignore-line-breaksDo not treat separate lines as distinct eventsboolfalse
batchFor when a number of events need to be marked as belonging to a distinct group--

queue

An AMQP Queue

Field NameDescriptionTypeDefault
nameQueue name--
passiveAssume Queue has already been declared, else errorboolfalse

name

Queue name

Example

action:

amqp:
  queue:
    name: queue-name

passive

Assume Queue has already been declared, else error

Type: bool

Example

action:

amqp:
  queue:
    name: some-queue
    passive: true

uri

AMQP address to listen to

Type: string

Example

action:

amqp:
  queue:
    name: queue-name
  uri: amqp://localhost//

routing-key

Used in conjunction with topic exchange, to route data

Type: string

exchange

An AMQP Exchange

Field NameDescriptionTypeDefault
nameExchange name--
typeExchange type--
passiveAssume Exchange has already been declared, else errorboolfalse

name

Exchange name

Example

action:

amqp:
  exchange:
    name: some-exchange
    type: fanout

type

Exchange type

There are 2 supported types:

  • fanout: For when same messages are to be consumed by multiple comsumers
  • topic: Topic-based pub-sub
  • direct: For when routing keys are to be used

Example

action:

amqp:
  exchange:
    name: some-exchange
    type: topic
  routing-key: some.key

passive

Assume Exchange has already been declared, else error

Type: bool

Example

action:

amqp:
  exchange:
    name: some-exchange
    passive: true

persistent

Let messages be cached and survive server restart

Type: bool

Example

action:

amqp:
  queue:
    name: some-queue
  persistent: true

durable

Let Queue or Exchange survive server restart

Type: bool

Example

action:

amqp:
  queue:
    name: some-queue
  durable: true

raw

Assume incoming data is in JSON format (do not quote)

Type: bool

Example

action:

amqp:
  queue:
    name: queue-name
  raw: true

ignore-line-breaks

Do not treat separate lines as distinct events

Type: bool

Example

action:

amqp:
  queue:
    name: some-queue
  ignore-line-breaks: true

batch

For when a number of events need to be marked as belonging to a distinct group

Field NameDescriptionTypeDefault
uuid-fieldField where generated uuid, the unique marker for the group, will be storedstring-
invocation-time-fieldField where invocation time will be storedstring-
completion-time-fieldField where completion (end of execution) time will be storedstring-
begin-marker-fieldField used to mark first event in the groupstring-
end-marker-fieldField used to mark last event in the groupstring-

uuid-field

Field where generated uuid, the unique marker for the group, will be stored

Type: string

Example

action:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    uuid-field: marker
  interval: 1m

output:

{"_raw":"foo","line-count":3,"line-num":1,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}

Example: For cases where event count is known, a simple counter is used, instead of uuid (useful for testing)

action:

exec:
  command: echo foo
  no-strip-linefeeds: true
  count: 3
  batch:
    uuid-field: marker
  interval: 1m

output:

{"_raw":"1","line-count":3,"line-num":1,"marker":"1"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"1"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"1"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"2"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"2"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"2"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"3"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"3"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"3"}

invocation-time-field

Field where invocation time will be stored

Type: string

Example

action:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    invocation-time-field: begin
  interval: 1m

output:

{"_raw":"1","line-count":3,"line-num":1,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"2","line-count":3,"line-num":2,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"3","line-count":3,"line-num":3,"begin":"2020-01-17T09:55:09.135Z"}

completion-time-field

Field where completion (end of execution) time will be stored

Type: string

Example

action:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    invocation-time-field: begin
    completion-time-field: end
  interval: 1m

output:

{"_raw":"1","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":1}
{"_raw":"2","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":2}
{"_raw":"3","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":3}

begin-marker-field

Field used to mark first event in the group

Type: string

Example

action:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    begin-marker-field: begin
  interval: 1m

output:

{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","line-count":3,"line-num":3}

end-marker-field

Field used to mark last event in the group

Type: string

Example

action:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    begin-marker-field: begin
    end-marker-field: end
  interval: 1m

output:

{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","end":true,"line-count":3,"line-num":3}