HTTP Inputs and Outputs

Hotrod has several inputs and outputs that make it convenient to integrate with other servers over HTTP.

http-server input

This provides a little HTTP server, which can be used to define a webhook. There is both an input and an output variant; both of them run in the background listening for HTTP GET or POST requests.

As input, by default every request results in a JSON event like so:

  • method: the HTTP method (GET,POST,etc)
  • url: the full URL
  • query: the query parameters (e.g. "?p=foo&q=boo")
  • headers: the HTTP headers
  • body: the received body

The only required field is address. Running curl http://localhost:3030 results in:

input:
    http-server:
        address: 0.0.0.0:3030
# {"body":"","method":"GET","query":{},"url":"/","headers":{"accept":"*/*","host":"localhost:3030","user-agent":"curl/7.58.0"}}        

If only the body is needed, only-body: true. This will be quoted with _raw unless json-body: true.

This server does support secure HTTPS connections. cert is path to TLS certificate and key is path to TLS key.

It's possible to send a fixed response to any request with custom-response.

http-poll input

http-poll does a HTTP(S) GET or a POST request. The only required field is the address. Like with exec the response is quoted with _raw and can be passed through directly with raw: true.

You may provide a body with body: text and send it as POST with method: post.

Polling the server above like so:

input:
    http-poll:
        address: http://localhost:3030/bar
        interval: 2s
        query:
        - foo: hello
        headers:
        - content-type: text/plain

And its response will be:

{"body":"","method":"GET","query":{"foo":"hello"},"url":"/bar","headers":{"accept":"*/*","accept-encoding":"gzip","content-type":"text/plain","host":"localhost:3030","user-agent":"reqwest/0.9.24"}}

This is a scheduled input, so schedule vars are avaiable in address and query.

http-server output

http-server is very similar to the input form, except that it is fed events by the pipe. When an outside agent queries it, it will get the last event that was generated by the pipe. So there is no history: if it is polled at a slower rate than the pipe, then data will be lost.

For instance, this pipe executes the 'uptime' command repeatedly and extracts the last number (the 5-minute load average).

name: server
input:
    exec:
        command: uptime
        interval: 5s
actions:
- extract:
    remove: true
    pattern: '(\S+)$'
    output-fields: [min5]
output:
    http-server:
        address: 0.0.0.0:3030
        content-type: application/json
        path: /uptime

We can now use curl to 'scrape' this little server:

$ curl -v http://localhost:3030/uptime
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3030 (#0)
> GET /uptime HTTP/1.1
> Host: localhost:3030
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json
< transfer-encoding: chunked
< date: Wed, 15 May 2019 06:48:33 GMT
<
* Connection #0 to host localhost left intact
{"min5":"0.49"}

path is optional - without it, it will serve the same content with any URL. Likewise, content-type can be optionally specified.

You can GET the output as often as you like, but if you scrape faster than 5s in this case, you will get the last value created by the pipe.

By default, the body returned is the whole event generated by the pipe. body allows you to specify exactly what is returned. Here the body is the value of the _raw field.

http-server:
    address: 0.0.0.0:3030
    body: '${_raw}'

body can contain any field expansions.

As with the input, can set TLS certificate and key with

    cert: /path/to/my.cert
    key: /path/to/my.key

http-post output

http-post is the output equivalent to http-poll; the pipe's events are sent HTTP(s) request to a server.

batch is the maximum number of lines in each POST request; if we wait for more than timeout, then the request is posted with less than this number. Note that timeout is a time identifier (100ms, 1s, 1m). The default is 100ms.

This allows maximum throughput and ensuring minimum waiting on data.

output:
    http-post:
        batch: 50
        timeout: 300ms
        url: http://whatever:port/parms

Without the batch, it does a single unbuffered POST request. Some useful features then become available. By default, the body of the POST is the whole record, but can be provided with body-field which is a field name to be used instead (just like input-field with exec.)

The url can contain field expansions. These will be properly URL-encoded if needed.

name: poster
input:
    text: '{"msg":"hello dolly","path":"/foo", "query":"new dawn", "lines": "/one/two"}'
output:
    http-post:
        body-field: msg
        url: 'http://localhost:3030${path}?q=${query}&p=${lines}'

You may add HTTP headers:

input:
    http-post:
        url: http://localhost:3030
        headers:
        - content-type: text/plain

Without headers, the content-type defaults to 'application/json'.