Monitoring firewall with ELK – Palo alto networks

Either you are a Cisco person or you are not! I am one of those who were and may be still am, however, I also really love technology and admire innovative technology. I was introduced to Palo alto networks firewalls back in 2012 while visiting a customer. The customer claimed that Cisco ASA are history as Palo alto network firewalls the so called next generation firewalls are here to stay and to replace the monopol of Cisco. However, at that time Checkpoint was also quite popular as they still are! Long story short, I tried using Palo alto network firewalls and became a fan! If there is a negative with their products then it must be that trial versions are not available easily, apart from that I cannot find a single thing to point on. Surely you can be picky about some small stuff, which I am going to be in this blog post, however, if you are using or are planning to use Palo alto network firewalls and are not planning to use Panorama (which small customers usually do not) then this blog post is for you!

We are going to be visualizing Palo alto networks firewall data traffic using ELK stack. There are several ways to do this, however in this post we are going to be focusing on using syslog format to be ingested to Elastic search using Logstash. I have also provided the Palo alto networks firewall Logstash filter but with some minor alterations. So lets get started at looking at how thing can be presented.

Palo alto firewall traffic overview in ELK dashboards

Palo alto networks firewall traffic overview in ELK dashboards
Palo alto networks firewall traffic overview in ELK dashboards

The image shows a dashboard that has been created in Kibana while utilizing data from Elastic search. The Palo alto networks firwall dashboard shows the traffic overview from all firewalls in the environment. Normally you will have at least 2 Palo alto networks firewalls in an active-passive cluster. In this scenario we have 4 firewalls in total. Two functioning as main corporate firewalls attached directly into the core network and then we have 2 more firewalls functioning as VPN providing firewalls. So the idea is to be able to view the traffic being generated by a single user throughout the value chain, i.e. all the traffic being produced by a single user on the corporate network either through directly connected devices or through devices connecting through VPN. UserID sharing/forwarding is essential in this setup so that when a user is first identified, the same user ID rules are applied throughout the session.

Palo alto networks single firewall traffic overview in ELK
Palo alto networks single firewall traffic overview in ELK

The image above is showing the same information, but for a single firewall. As the dashboard focuses on a single Palo alto networks firewall cluster i.e. on the active node the relevant information can be shown. Normally you will be interested in what kind of traffic is traversing the wire, which users and computers are generating this traffic, which hosts is this traffic being directed to, is the traffic being allowed or blocked etc. In the dashboard for Palo alto networks firewalls we also have a section showing traffic overview based on countries. As I was not able to include all the details in a single image it has to be added separately, however, this too is information which is included in the same dashboard.

Palo alto networks firewall traffic overview in ELK from countries
Palo alto networks firewall traffic overview in ELK from countries

Firewall traffic trend analysis

If you are using ELK or have used it, you might be aware of the handy function called timelion which makes it possible to compare data over time. Technically speaking it is a fuction in Kibana, which makes it possible to present data from Elastic search. The image below is showing dropped traffic compared for today vs yesterday, it is showing the traffic between 14:00-22:00 hours.

Palo alto networks firewall traffic trend analysis with ELK
Palo alto networks firewall traffic trend analysis with ELK

Timelion expression to create Palo alto networks firewall trend analysis for today vs yesterday is as following

.es(q=action:drop).lines(fill=2).color(red).label("Dropped traffic today"),
.es(q=action:drop,offset=-1d).lines(fill=2).color(purple).label("Blocked traffic yesterday")

If you want to utilize the true power of Kibana and Elastic search, you might want to use the built in function for trend.

Palo alto networks firewall traffic trend analysis with ELK over time
Palo alto networks firewall traffic trend analysis with ELK over time

Timelion expression to create Palo alto network firewall trend analysis over time is as following

.es(q=action:drop).lines(fill=2).color(red).label("Dropped traffic trend").trend()

As you can see, the trend shows that there are lesser number of packets being dropped in July as compared to June. This is correct and related to summer holidays. As there are less people at work, there is much lesser traffic and hence far less drop/deny events being logged by the firewall.

You might be thinking, well all of this can be done directly in the firewall GUI as well. Well that is not correct as you cannot have a dashboard consisting of traffic from all firewalls, not even the firewalls in the same cluster. Besides that, your active and passive firewalls have their own logs which are only available locally, hence you have no possibility to show the data by aggregating it natively with the tools shipped with the firewalls. Furthermore, you do not have the trend analysis feature nor do you have the comparison features. The most important fact is, when viewing traffic in/on the firewall management GUI you are utilizing firewall resources.

By analyzing Palo alto networks firewall traffic in ELK you are freeing up the management plane resources for the firewall. This capacity can be used for other important functions rather than being utilized for log and traffic analysis of your environment. Once the data has been shipped to Logstash, the firewall no longer need to spend a single CPU cycle to provide you traffic insight, this can be done by ELK stack.

Management plane in Palo alto networks firewalls are used for configuration and management related task, for example for Wildfire, download and processing of signature firewalls, configuration generation and commit actions, for sending syslog traffic etc.

Components required for Palo alto networks firewall dashboard

Needless to state you must have a working ELK environment which can accept syslog messages being sent from your Palo alto networks firewalls. If you have this in place your logs presented in Kibana should be similar to the image below

Palo alto networks firewall traffic fields in ELK
Palo alto networks firewall traffic fields in ELK

As you might have noticed we already have fields being identified, some of these are shown in the image above. The fields shown are

Time, dst_ip, dst_location, protocol, dst_port, event, src_location, type and user

It is important that the fields being identified have the correct datatype i.e. either if the field is being treated as String, Integer, IP and float etc. This effects the amount of storage that will be consumed in your ELK environment. In busy environment I have seen several million documents being created daily which normally results in a couple of gigabytes of data. This should basically align with you retention policy for your SELM solution. Assuming all this is sorted out lets move on to the most important building blocks of this solution.

Palo alto networks firewall syslog format

As you will be depending on data format in order to process it correctly it is important to understand the structure of the Palo alto networks firewalls syslog format. I will not spend time on protesting on syslog format and will rather put the focus on how Palo alto networks firewalls do this. You can either choose the default format or you can choose to adjust the format according to your needs. I prefer to use the former i.e. use the defaults. This is mainly because default format will be least taxing with regards to resource usage furthermore, if later down the road if the customer chooses to go for panorama or some other solution, you will not have to reconfigure the syslog format. This is by the way done per firewall cluster along with the log forwarding profile.

Palo alto networks do a wonderful job of documenting the syslog format for each PAN OS version. Pay attention to the previous sentence, the syslog contents might change from version to version. This normally happens with major updates and not minor updates. Major updates are when for example moving from PAN OS version 8.9 to 9.0. During the same major version for example 9.0 to 9.1 and then from 9.1 to 9.2 the syslog fields remains the same. This is my perception based on experience working with Palo alto network firewalls and ELK. The image below shows the syslog description of a PAN OS version.

Palo alto firewall syslog field description
Palo alto firewall syslog field description

However the sad part is that this description is not 100% accurate as the actual log being sent from Palo alto networks firewalls differs a bit from the description. So be sure to check the raw syslog message after upgrading or before upgrading a production firewall where logging data is important to be archived. For PAN OS version 9.1 you can find the syslog field description as the following link:

https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/traffic-log-fields.html

Logstash configuration for Palo alto networks firewalls

I normally create different configuration files for each type of equipment being configured to ship log data to Elastic search. The base configuration for Logstash server is put in a separate configuration file and for all relevant other devices/solution configuration files are created. You might choose to do it another manner, but this is a structure which works quite well at least for me and has done so in a number of years now. The structure looks a bit like this

Palo alto firewall logstash config structure
Palo alto firewall logstash config structure

The configuration for Palo alto networks firewall being used in my environment is 16-PA.conf. This file consists of several sections. Lets explore each of these.

Palo alto firewall logstash filter section traffic
Palo alto firewall logstash filter section traffic

In the filter section the traffic is searching for PA220 in the syslog message. When this is found the logs being sent are surely from a Palo alto 220 firewall. If you are using a good naming convention for you firewalls, now is the time to enter the unique prefix. Initially spaces are being removed and then commas are being replaced by spaces. I have chosen the dissect filter as its performance is much higher than a grok filter. Another easy thing is just to place a question mark “?” in order to drop fields that you are not interested in.

By analyzing contents of Logstash filter for Palo alto networks firewalls and Palo alto networks description of syslog fields, you can see the difference between the description provided by Palo alto Networks vs the actual field/structure of syslog messages.

In order to utilize the dissect filter you have to install the plugin for Logstash. This is a trivial task which is done by executing a simple command on your Logstash host

 bin/logstash-plugin install logstash-filter-dissect

If you want to gather and process Threat logs besides Traffic logs, this must be configured in your Palo alto Networks firewall log forwarding profile. I have provided the configuration for both Traffic and Threat logs at the bottom of this page. Make sure to include the relevant sections in your Logstash configuration files. If you currently only have Traffic log data, it could still be beneficial to add the Threat log filters. This way your Logstash environment and configuration will be prepared to accept the traffic whenever you choose to turn this on or take it to use.

To monitor URLs you must enable Threat logs forwarding. In the newer versions of PAN OS you are provided very granular control of Syslog messages that should be forwarded. Hence default Syslog forwarding does not start forwarding Traffic and Threat data, only whatever you choose is sent. Furthermore you have to option to configure System and Configuration messages.

The more options you have the better control you might get! Thanks to Palo alto Networks for providing such granular control for log forwarding

The next section of my Logstash configuration file handles the system messages

Palo alto firewalls logstash filter section System
Palo alto firewalls logstash filter section System

The next section is the Configuration section, that handles configuration change message for Palo alto networks firewalls.

Palo alto firewalls logstash filter section Config
Palo alto firewalls logstash filter section Config

And the last section of Logstash configuration file points to Elastic Search instance and maybe to a file location for troubleshooting purposes, depending on what you need.

Palo alto firewalls logstash filter output section
Palo alto firewalls logstash filter output section

You can have several Elastic search servers in this section. Logstash will then be able to send data to both of them and not sending all the data to same Elastic search host. This provides better performance. The default port of 9200 can also be omitted, however, personally I like to keep the information visible. You should also include other filtering criteria before entering data into you Elastic search cluster for example you would not like to insert data if it had _grokparseerror or any other errors in the tag fields. The sections below include the complete Logstash filter configuration file for Palo alto networks firewalls running version PAN OS version 9.1 and above.

filter {

if [type] == "syslog" and [message] =~ "PA220" {

#Case 1 Traffic logs
if ([message] =~ "TRAFFIC" )
	{
                mutate
                        {
                                gsub => [
                                        # remove all spaces
                                        "message", " ", "",
                                        # replace comma with space
                                        "message", ",", " "
                                        ]
                                add_tag => ["TRAFFIC"]
                        }
	# now start matching and adding fields to Traffic log message
	dissect {
		mapping => { "message" => '%{?timestamp} %{?version} %{?RcvTime} %{?SrNum} %{type} %{event} %{?ThreatContentType} %{?GeneratedTime} %{src_ip} %{dst_ip} %{nat_src_ip} %{nat_dst_ip} %{fw_rule_name} %{src_usr} %{dst_usr} %{application} %{?VirSys} %{src_zone} %{dst_zone} %{ingress_int} %{egress_int} %{log_profile} %{?FutureUse} %{sess_id} %{?RptCnt} %{src_port} %{dst_port} %{nat_src_port} %{nat_dst_port} %{flags} %{protocol} %{action} %{bytes} %{sent_bytes} %{recv_bytes} %{pkts} %{?StartTime} %{duration} %{category} %{?FutureUse} %{?SequenceNum} %{flags} %{?SrcLocation} %{dst_country} %{?FutureUse} %{sent_pkts} %{recv_pkts} %{sess_end_reason} %{?DevGrpHirchyLvl1} %{?DevGrpHirchyLvl2} %{?DevGrpHirchyLvl3} %{?DevGrpHirchyLvl4} %{?VirSysName} %{device_name} %{?ActionSrc} %{?SrcVMUUID} %{?DstVMUUID} %{?TunnelIDIMSI} %{?MonitorTagIMEI} %{?ParentSessID} %{?ParentStartTime} %{?TunnelType} %{?SCTPAssociationID} %{?SCTPChunks} %{?SCTPChunksSent} %{?SCTPChunsRcvd} %{fw_rule_uuid} %{?HTTP2Connection} %{?LinkChangeCnt} %{?PolicyID} %{?LinkSwitches} %{?SD-WANCluster} %{?SD-WANDevType} %{?SD-WANSite} %{?DynUsrGrpName}' }
			}
	#delete unwanted fields
	mutate {
			remove_field => [ "message" ]
			}
	}

#Case 2 System logs
if ([message] =~ "SYSTEM" )
	{
        mutate
            {
                gsub => [
                #Remove leading space
                "message", "^\s*", ""
                ]
				add_tag => ["SYSTEM"]
            }

		dissect 
			{
				mapping => { "message" => '%{?timestamp},%{?version},%{?RcvTime},%{?SrNum},%{Type},%{ThreatContentType},%{?FutureUse},%{?GeneratedTime},%{?VirSys},%{EventID},%{Object},%{?FutureUse},%{?FutureUse},%{Module},%{Severity},%{Description},%{?SequenceNum},%{?ActionFlags},%{?DevGrpHirchyLvl1},%{?DevGrpHirchyLvl2},%{?DevGrpHirchyLvl3},%{?DevGrpHirchyLvl4},%{?VirSysName},%{?DevName}' }
			}
		#delete unwanted fields
		mutate 
			{
				remove_field => [ "message" ]
			}
	#end if message contains SYSTEM
	}

#Case 3 Config logs
if ([message] =~ "CONFIG" )
	{
        mutate
                {
                    gsub => [
                    " message", "^\s*", ""
                    ]
                    add_tag => ["CONFIG"]
                }
		dissect 
			{
				mapping => 
				{
					"message" => '%{?timestamp},%{?version},%{?RcvTime},%{SrNum},%{Type},%{Subtype},%{?FutureUse},%{?GeneratedTime},%{Host},%{?VirSys},%{Command},%{AdminUser},%{GUI},%{Result},%{ConfigPathObject},%{?SequenceNum},%{ActionFlags},%{?DevGrpHirchyLvl1},%{?DevGrpHirchyLvl2},%{?DevGrpHirchyLvl3},%{?DevGrpHirchyLvl4},%{?VirSysName},%{DevName}'
				}
			}

	}

	#delete unwanted fields
	mutate 
		{
			remove_field => [ "message" ]
		}
#end if message contains CONFIG
	mutate {
				add_tag => ["Firewall", "Paloalto"]
			}

#Outer if
}
}



output {
	#Here you should also add other tags, for example device_type, device_name or whatever fields you are using 
	# if [type] == "syslog" {
elasticsearch {
    hosts => [ "10.10.10.10:9200" ]
    }
	#Uncomment for testing and checking values
	#stdout { codec => json }
	#file {
		#path => [ "/var/log/logstash/syslogRAW.log" ]
	#     }
}

If you want to include Threat logs then include the code below in you configuration file

if [type] == "rsyslog" and [message] =~ "THREAT" and [message] !~ "logstash"
{
        dissect {
                mapping =>
                        {
                                "message" => '%{?junk},%{?junk},%{?junk},%{type},%{sub_type},%{?junk},%{?junk},%{src_ip},%{dst_ip},%{src_ip_nat},%{dst_ip_nat},%{rule},%{user},%{?dst_user},%{application},%{?virtual_system},%{?ad_group},%{dst_zone},%{ingress_int},%{egress_int},%{loggingprofile},%{?junk},%{?session_id},%{?repeat_count},%{src_port},%{dst_port},%{src_port_nat},%{dst_port_nat},%{?flags},%{protocol},%{action},%{url_file},%{threat_id},%{category},%{severity},%{traffic_direction},%{?sequence_number},%{?actionflag},%{src_location},%{dst_location},%{?junk},%{?content_type},%{?pcap_id},%{?junk},%{?junk},%{?junk},%{?junkusragent},%{?junkfiletype},%{?junkxforwardedfor},%{?junkreferere},%{?junksender},%{?junksubject},%{?junkrecepient},%{?junkreportid},%{?junkdevgrphilvl1},%{?junkdevgrphilvl2},%{?junkdevgrphilvl3},%{?junkdevgrphilvl4},%{?junkvirsysname},%{device},%{?junkfutureuse},%{?junksrcVMuuid},%{?junkhttpmethod},%{?junktunnelidimsi},%{?junkmonitortagimei},%{?junkparentsessionid},%{?junkparentstarttime},%{?junktunneltype},%{?junkthreatcategory},%{?junkthreatcategory},%{?junkcontentver},%{?junkfutureuse},%{?junksctpassosiation},%{?junk},%{?junk},%{?junk},%{?junk},%{?junk}'
                        }
                }

        mutate {
                 convert => [ "src_port", "integer" ]
                 convert => [ "dst_port", "integer" ]
                 convert => [ "src_port_nat", "integer" ]
                 convert => [ "dst_port_nat", "integer" ]
               }



}

When using the filters about you get the logs formated as in the image below

Palo alto firewalls syslog data format details
Palo alto firewalls syslog data format details

Palo alto networks – Logstash filter input

The input filter is quite simple at least for the lab environment. But working with customers, I normally set up several different ports for logstash to listen on. In large environments you will most probably have several different logstash servers handling ingestion and performing formatting. For example, you might have one or two for the network segment for firewalls, and you might have a couple for switches, and yet others for load balancers etc.

input {
  udp {
    host => "10.10.10.10"
    port => 10514
    type => "syslog"
  }
}

Additional resources

To read more about Threat logs Syslog message structure for Palo alto networks firewalls you can visit https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/threat-log-fields.html

To read more about dissect filter for Logstash and Elastic’s general recommendations visit their blog post at https://www.elastic.co/blog/logstash-dude-wheres-my-chainsaw-i-need-to-dissect-my-logs

5 Thoughts on “Monitoring firewall with ELK – Palo alto networks

    1. Hi Mike,
      It consists of host (logstash ip), protocol(udp), port (10514) and of type syslog. I can upgrade the blog post to include this as well.

      Best regards,
      Rao

Leave a Reply

Your email address will not be published. Required fields are marked *