Elasticsearch Cluster for – Monitoring Palo alto firewalls

Recently I received a comment from someone asking me to share my solution for Index Management for ELK. This comment was posted on the blog post regarding how to monitor firewall traffic using Elastic stack

Which lead to writing a blog post Index Management for ELK – monitoring Palo alto firewalls. After writing this blog post, I was contacted by a customer who needed either to scale up or scale out their ELK cluster as the amount of data being saved is more than initially anticipated and the customer is not able to retain data for more than half a year or so. This tells me only one thing, design, design, plan, redesign and then implement! Well actually the customer in question initially only wanted to retain the data for at most 3 months, but the value provided by ELK made them realize that the data should be retained for a longer period so that the full power of ELK can be unlashed and so that features like trend analyses and big data like analyses can be performed.

I think, lets make this post a bit more interesting describing what the ELK stack cluster is and how it functions. Folks at ELK are quite innovate and active i.e. they frequently develop new versions and iterations of ELK. Furthermore, they keep on adding features and perform a lot of changes. This also applies to the cluster functionality. The old configurations do keep on functioning in existing clusters, however, new terms so to speak setting are introduced in new versions. If you are one of those Star wars or Star track fans, you will love the analogy they use. Its just about checking out the settings files for Elasticsearch.

ELK Cluster

So even though I am using the term ELK Cluster, I mean Elasticsearch Cluster. Neither Logstash nor Kibana are clustered, at least not in any of the installations I have performed or seen. Besides these other products are soo light weight and easy to configure that you might not even ever need the service in clustered instance. So now that this has been clarified, lets look at the possible solution.

In term of numbers, there are no limits on number of nodes that can be a part of a Cluster. At least that is the case when it comes to official description from Elastic.co, on the other hand, Amazon support clusters of up to 200 nodes if you are hosting your Elasticsearch cluster in Amazon. Elasticsearch support states that they have customers running clusters with up to 300 nodes. Quite impressive if you ask me! But again, as with almost all Microsoft products, the use case and your requirements are the driving force and the reason that sort of dictates the number of nodes required in the cluster. A cluster is normally used to store all data, rightly so is data stored in indices and shards etc. but in the end of the day the final entity where data ends up into is a Cluster. You might not agree with the analogy that I am using, but my point is, if you are gathering logs from Nginx and lets say you want to gather log data from your in house developed applications, both those datatypes might not be fit to store data in the same Elasticsearch cluster. You could save the data in the same Cluster if you wanted to, but you might be better of creating a totally different Cluster for each purpose. It all boils down to how you are planning to ingest and at consume that data that is being stored.

A bit of terminology before we can dig deeper into the cluster settings. As with almost all cluster type, you must have a single master node in the cluster. You can have several eligible master, i.e. these nodes can function as masters if the master node is not available. But at any given time, there is only one active master node. The reason is simple, there are some functions that needs to be performed by the master and other functions or jobs can be shared between the remaining nodes. Then you have the nodes that function as ingestion nodes. These nodes ingest data into Elasticsearch. You normally do not want to make the master node and ingestion node, even tough it will participate or rather say contribute by hosting and storing shards, but you do not want to strain it by configuring it as an ingestion node. In a small environment, it might be totally legitimate to make even the master node as an ingestion node.

However your cluster design becomes, do make sure that your Logstash host utilize the full power of Elasticsearch cluster. You can configure Logstash to send data to multiple nodes. Logstash load balances the requests and the nodes calculate the hash value of the data or documents so to say, before ingesting them into Elasticsearch, hence making sure that the data is only entered once. No wonder so many fall in love with Elasticsearch!

Adding new cluster nodes

Unless someone requests me to post a description of how to set up the cluster, I am just going to assume that you are able to set up a Elasticsearch cluster without my elaboration on a process that is heavily documented and even quite welly documented by elastic.co. What I do want to share though is a technique that I use quite often use when I am engaged to extend and existing cluster to either scale up or scale out. I prefer the scale out method, in this manner there is no downtime or risk of breaking something while adding more storage to your cluster. As sad it makes me to write it, but most of the Elasticsearch cluster I encounter are Linux based, where ELK components are configured on a Ubuntu or similar platform, renown for its minimum maintenance and maximum stability/availability.

If the Cluster nodes are VMs, which the normally are, the job becomes on of the easiest jobs, while the only thing I need to do is to copy, or right term would be to clone the VM and then perform some changes before bringing it online. After the VM has been cloned on your favorite virtualizing platform Hyper V, just joking, on Vmware, you are ready to start removing temporary files.

You would at a minimum configure a new name for you VM i.e. within the operating system, set a new static IP or dynamic if you are using these in your environment, and then delete all Elasticsearch locally stored data. This is normally stored under

/var/lib/elasticsearch/nodes/0/indicies as shown in the picture below

Elasticsearch documents folder
Elasticsearch documents folder

Here you can delete all the files, as when introducing the new node into the cluster it will automatically adjust the shards on each node. If you choose not to delete the files, the node will not be accepted in the cluster either as it will contain data already present on other node.

In addition to the above folder you must also delete all the files present in folder /var/lib/elasticsearch/nodes/0/_state and all files under the folder /var/lib/elasticsearch/nodes/0/nodelock

Once this has been done you can update the elasticsearch.yml file contents to reflect that changes that you have made to the cluster.

Contents of elasticsearch.yml

When adding a node, I perform following changes in the file:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: zeglory-dot-com
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-4
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.0.14
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
discovery.seed_hosts: ["192.168.1.11", "192.168.1.12", "192.168.1.13", "192.168.1.14", "192.168.1.15"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
cluster.initial_master_nodes: ["192.168.1.11", "192.168.1.12"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
node.data: true
node.ingest: true
node.master: false
discovery.zen.minimum_master_nodes: 2

As you can see in the settings file, We are configuring node-4 and have assigned it the IP address of 192.168.1.14. We also can see that this is a data and ingest node, while it is not a master node.

Status of Elasticsearch Cluster in Kibana

The status of Cluster can be viewed in Kibana, it shows the status while new nodes is added and while shards are being moved from one node to another one.

Kibana addition of Elasticsearch node 4
Kibana addition of Elasticsearch node 4

In the picture above you see the introduction of node 4 into the environment, You can see that Shards have started to move from the three existing nodes to node-4. I personally make sure that the cluster have normalized before I start adding any more nodes. Theoretically the introduction of node 5 could have been simultaneously, however, given the fact that the custoer is in production, the safest way is to add one node at a time.

Kibana addition of Elasticsearch node 5
Kibana addition of Elasticsearch node 5

After node 4 has been added and the environment is converged, we just repeat the entire process of cloning and configuring. If you are more of a CLI type of a person you can use the Cluster log file. It is found in the same place as other Elasticsearch logfiles under /var/logs/elasticsearch/ and the file consists of the Cluster name my file would have been zeglory-dot-com. The log files shows when new nodes are being added or are being removed and when the cluster roles are being transferred internally.

Elasticsearch Cluster log file
Elasticsearch Cluster log file

Hope as always that this blog post helps someone out there in need. I know for sure that I spent quite a long time learning ELK, and still I feel like there are miles left just to come to the beginner level of the fascinating and complex toolset.

Leave a Reply

Your email address will not be published. Required fields are marked *