Reducing Firewall Log Volumes

The following section assumes that you already have Hotrod (possibly deployed on some Bboxes) up and running, for more information related to getting up and running see our Getting Started Guide.

Suppose there is a line that looks like the following (a FortiGate firewall sample):

date=2017-11-15 time=11:44:16 logid="0000000013" type="traffic"
subtype="forward" level="notice" vd="vdom1" eventtime=1510775056
srcip=10.1.100.155 srcname="pc1" srcport=40772 srcintf="port12"
srcintfrole="undefined" dstip=35.197.51.42 dstname="fortiguard.com"
dstport=443 dstintf="port11" dstintfrole="undefined"
poluuid="707a0d88-c972-51e7-bbc7-4d421660557b" sessionid=8058
proto=6 action="close" policyid=1 policytype="policy"
policymode="learn" service="HTTPS" dstcountry="United States"
srccountry="Reserved" trandisp="snat" transip=172.16.200.2
transport=40772 appid=40568 app="HTTPS.BROWSER" appcat="Web.Client"
apprisk="medium" duration=2 sentbyte=1850 rcvdbyte=39898 sentpkt=25
rcvdpkt=37 utmaction="allow" countapp=1 devtype="Linux PC"
osname="Linux" mastersrcmac="a2:e9:00:ec:40:01"
srcmac="a2:e9:00:ec:40:01" srcserver=0 utmref=0-220586

It can happen that you may not be interested in all the fields, and the following Pipe definition will show how to select only those of interest:

- expand:
    input-field: _raw
    remove: true
    delim: ' '
    key-value:
      key-value-delim: '='
- filter:
    schema:
      - logid
      - eventtime
      - srcip
      - srcport
      - dstip
      - dstport
      - sentbyte
      - rcvdbyte

Coming out the other side of those actions would be the following:

{"logid":"0000000013","eventtime":"1510775056","srcip":"10.1.100.155","srcport":"40772","dstip":"35.197.51.42","dstport":"443","sentbyte":"1850","rcvdbyte":"39898"}

That is 20% of what the original text was (original is 799 bytes, and output is 164 bytes).

Further savings can be had by converting to CSV format.

- collapse:
    csv: true
    output-field: output

Output of that is the following:

{"output":"0000000013,1510775056,10.1.100.155,40772,35.197.51.42,443,1850,39898"}

That is a further reduction of 50% (from 164 bytes to 81 bytes).

This of course ignores the header field, but not very important because the same header can be shared with the receiving side (where another Pipe can be used for further processing).

Following shows how to include the headers:

- collapse:
    csv:
      header-field: header
    output-field: output

That results in the following:

{"output":"0000000013,1510775056,10.1.100.155,40772,35.197.51.42,443,1850,39898","header":"logid,eventtime,srcip,srcport,dstip,dstport,sentbyte,rcvdbyte"}