Scripting in Pipes

script, like add is mostly used to add new fields to events (enrichment). The difference is that script can calculate fields and can conditionally add them.

These expressions are in Lua format, e.g conditions like a > 2 and b < 1 and expressions like 2*a - 1. All event fields are available in these expressions. If the data is not 'flat' (e.g {"a":{"b":1}} then the value of 1 here can be accessed with a.b. Array values are accessed starting at one.

NOTE For this to work field names must be valid Lua variables - start with a letter, and contain only letters, digits and underscores. This restriction also applies to extract so we recommend this for all field names.

let versus set

With let you give a list of variable-expression pairs, but with set you provide constants. Context expansions are always available.

# input: {"a":10}
- script:
    let:
    - c: 2*a - 1
    set:
    - site: '{{name}}'
# output: {"a":10,"c":19,"site":"NAME"}

script set is pretty much what add output-fields does, except with add you can add structured data like arrays and objects.

Both script and add will not override existing fields by default. Force this with override: true

There may be a condition field - the condition must be true before any fields are written.

Extra Functions

  • round(x) returns the nearest integer to a floating point number, like round(tmillis/1000). Useful for converting bytes to kB, milliseconds since epoch to seconds since epoch, etc.
  • count() counter since pipe start
  • random(n) a random integer between 1 and n
  • pick_random(...) returns one of multiple arguments randomly
  • sum(acc,val,cond) and avg(acc,val,cond) running sum and average of value val.
  • sec_s() will return seconds since epoch, sec_ms() milliseconds since epoch.
  • cidr(addr, spec) will match an IPv4 network address against a CIDR specification like '10.0.0.0/24'.
  • ip2asn uses the Team Cymru services to match IP addresses to domain owner.
  • cond(condition, value1, value2) is a useful function that will return value1 if condition is true, otherwise returns value2. E.g. status: cond(istat > 0,"ok","error").
  • hashes:
    • md5(txt)
    • sha1(txt)
    • sha256(txt)
    • sha512(txt)
  • uuid() returns a Unique Identifier each time
  • encrypt(txt,key) encrypt text using AES-128-CBC with a key and encode as Base64
  • decrypt(txt,key) decrypt result of encrypt using same key
  • emit(v) write out data directly

NB Please note that any field with one of these names will overwrite the function!

The second argument of sum is the 'accumulator' where we keep the count - can be any name that isn't a field. (We have this because there may be more than one field being summed.)

(Note that filter is the only other place where Lua expressions occur.)

- script:
    let:
    - counter: count()
    - sumi: sum("s",counter)
- filter:
    condition: counter % 5 == 0
# output
    {"counter":5,"sumi":15}
    {"counter":10,"sumi":55}
    {"counter":15,"sumi":120}
    {"counter":20,"sumi":210}
    ....

Useful Standard Lua Functions

Processing Text

  • string.upper(txt) convert text to upper-case
  • string.lower(txt) convert text to lower-case
  • string.sub(txt,istart,iend) 'substring' of text
  • string.len(txt) length of text
  • string.find(txt,patt) find if a pattern matches
  • table.concat(txt,sep) join an array as text
name: upper
input:
  text: '{"msg":"hello dolly","arr":[1,2,3],"num":3.1423}'
actions:
- script:
    overwrite: true
    let:
    - msg: string.upper(msg)
    - first: string.sub(msg,1,4)
    - n: string.len(msg)
    - arr: table.concat(arr," ")
    - num: string.format("%.02f",num)
output:
  write: console
# {"msg":"HELLO DOLLY","arr":"1.0 2.0 3.0","num":"3.14","first":"HELL","n":11}

Processing Numbers

  • math.ceil(x) nearest greater integer (e.g. 1.4 -> 2)
  • math.floor(x) nearest smaller integer (e.g. 1.4 -> 1)
  • math.min(...) smallest of multiple values
  • math.max(...) largest of multiple values
  • math.log(x,b) log of x with base b
  • math.exp(x) exponential
  • math.sin, math.cos ... trigonometric functions (in radians)

See the Lua manual for the full list of available functions.

Extending script using Lua

If there is a file init.lua in the pipe directory, it will be loaded into the Lua state. Any global functions or variables become available.

-- init.lua
    function every(n)
       return count() % n == 0
    end

Then the counter example can be made prettier like so:

- script:
    let:
    - counter: count()
    - sumi: sum("s",counter)
- filter:
    condition: every(5)

init.lua will be loaded automatically, but you have to copy it together with the pipe. A complete pipe could look like this:

name: sumi
files:
- init.lua
input:
    exec:
        command: echo foo
        interval: 100ms
actions:
- remove: [_raw]
- script:
    let:
    - counter: count()
    - sumi: sum("s",counter)
- filter:
    condition: every(5)
output:
    write: console

A more interesting example involves unpacking the result of a Prometheus query. The task is to take a structured JSON document and create events for each retrieved data point - expanding:

{
    "status":"success",
    "data":{
        "resultType":"matrix",
        "result":[
            {
                "metric":{
                    "__name__":"probe_success",
                    "instance":"178.62.49.144:20000",
                    "job":"APP-F4ZASD-JD-STE-RU-3033"},
                    "values":[[1571997720,"0"],[1571997780,"0"],...]
                }
            },
            ...
        ]
    }
}

init.lua contains a function which is passed the whole document and uses the emit function to directly write out events:

function unpack_prom(result)
    for _,res in ipairs(result) do
        local t = res.metric
        for _,pair in ipairs(res.values) do
            t.time = pair[1]
            t.value = pair[2]
            emit(t)
        end        
    end
    return 'ok'    
end

And this little pipe exercises it:

- script:
    let:
    - a: unpack_prom(data.result)
- remove: [status,data]
# output
{"instance":"178.62.49.144:20000","__name__":"probe_success","time":1571997720,"value":"0","job":"APP-F4ZASD-JD-STE-RU-3033"}
....

Here we are interested in the side effect of the function!