SSL monitoring using SCOM with Powershell and dot net (.net) – Sealed MP

Many years ago I wrote a blog post about how to bulk monitor SSL certificates using Operations manager 2012, Operations Manager 2012 R2. The same theme or topic is still valid as a lot of people end up using calendar or Excel sheets to keep track of expiring certificates. Other use relatively expensive software to do this just, while it can be done almost free of cost as long as you have SCOM/Operations Manager or are willing to set u a Windows server with Powershell. The idea came as a result of a request from a customer, who wanted to bulk monitor website certificates using SSL only. This is mostly applicable to proxy servers and load balancers as well, depending on what design has been implemented. If you are using wildcard certificates, monitoring is quite simple, but if you use specific certificates with SSL offloading, this might become an administrative nightmare for you. Anyways, back then, I created a MP which ran on the RMS to monitor URLs. There are basically 3 main things that were monitored

  1. If a SSL/TLS certificate is valid i.e. its validity date is okay
  2. If a SSL/TLS certificate is about to expire within 30 days
  3. If a SSL/TLS certificate has expired

However, the approach had it flaws. First of all, RMS should not be performing this action, It should be possible for any server running the SCOM agent to perform this action. Second, the entire process must be as automated as possible. Only thing an admin should be responsible for doing is to provide a list of URLs that should be monitored. I received feed back regarding RMS not being able to connect to DMZ servers or internet or in general connect to servers over the WAN due to custom ports, so this was a solution but still causing problems at least for a number of users. As the MP was never shared, I ended up sharing it on email and ab course deployed it to customers. However the MP was far from perfect and was causing some issues, hence it was not shared with the community in a broad manner. In addition, it was not possible to upload the .mp file either. I see that people still are looking for this functionality as requests are being registered to my almost 8 years old blog post. So here comes the recipe along with all the necessary files. I have also created a sealed MP so that overrides do not intervene with the original MP code. During the last few days I realized that you forget a lot of things during 8 years, as I almost has forgotten how to author MPs. I ended up using several days on the process, but all is well that ends well right? I did manage to create the MP, with a custom discovery and better set of rules. So here I am sharing the code or Powershell script that performs most of the work along with CSV file structure that the script utilizes. Although, the solution works best with Operations Manager, it can still be used without SCOM or Operations Manager.

In Operations Manager/SCOM (has been tested on 2012, 2012R2, but will probably work on 07 and 2017-2018 versions as well), after the MP has been imported and initial discovery process has completed you get alerts as shown below. Do not worry about the dates/time as my SCOM environment had to be run in 2013 due to activation challenges. The entire solution has been updated/created during June 2020. It has been retested as well. The solution in SCOM looks like

Shows how alerts are presented in SCOM
SCOM console SSL monitoring view

When you import the MP you also get a state view, showing Windows servers functioning as watcher agents(just borrowing the term from SCOM). The idea is that, you specify which SCOM agents should perform remote monitoring of SSL certificates and you should provide a CSV file containing URLs. Each of these proxy monitors or agents will now start generating alerts as shown in the figure above. A state view containing all agents functioning as SSL certificate remote monitoring hosts is made available. In my test environment I am using my DB server as a remote SSL certificate monitoring host. It is here where the most code runs, which is forwarded to SCOM management servers that generate alerts and eventually notify with subscription channels.

State view showing all SCOM agents functioning as remote SSL monitoring agents
SCOM SSL remote monitoring hosts state view

How to setup the solution in SCOM

To start using the solution in SCOM you have to download the MP which is provided on this site. Although there are some additional prerequisites in order to make the solution work. In order to make the discovery process to work you have to create a registry key. If you want to set up several SCOM agents as remote SSL monitoring hosts you will have to create this registry key on each host. This might be a requirement if you do not want to use same SCOM agent to monitor SSL/TLS certificates in production and test environment, or in internal LAN, DMZ, WAN and on internet. You have to create a key under HKLM\SOFTWARE\SSLMonitoring

Certificate monitoring SCOM registry key
Certificate monitoring SCOM registry key

You can also create a registry file in Notepad by adding this content and saving it as .reg extension. The file is also provided in the blog.

Certificate monitoring SCOM registry file
Certificate monitoring SCOM registry file
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\SSLMonitoring]

If you make sure that this registry value is populated prior to importing the MP the agent will show up on the state view, indicating an instance of class “zeglory.com SSL Monitoring host”. The discovery runs every 24 hours by default, but can be overridden to run less frequently or more often as you might require.

SSL monitoring SCOM Discovery rule
SSL monitoring SCOM Discovery rule

After a successful initial discovery, the rules within the management pack will start to create the file structure needed to monitor URLs. The MP will create a directory on the root of C: drive, C:\serverlist

Certificate monitoring SCOM folder structure
Certificate monitoring SCOM folder structure

The folder will further be populated with two files. This is done automatically by the MP so you do not need to perform any further actions. The GenerateEvents.ps1 file is utilized to run the script every 12 hours. However, the MonitoredServers.csv file must be populated manually.

Certificate monitoring SCOM folder files
Certificate monitoring SCOM folder files

The contents of CSV file must be set manually. You have to enter servername,port, as headings for CSV file and values for URLs containing SSL/TLS certificate to be monitored. In my environment I am monitoring zeglory.com running on port 443, and SCOM2012dc.contoso.no on port 443.

Certificate monitoring SCOM CSV content
Certificate monitoring SCOM CSV content

The management pack generates following rules, which can be overridden according to needs.

SSL monitoring SCOM Monitoring rules
SSL monitoring SCOM Monitoring rules

If the CSV file is missing or has accidentally been deleted an alert is generated, this is checked each time the Powershell script is run, which is by default each 12 hours.

Certificate monitoring SCOM CSV file missing
Certificate monitoring SCOM CSV file missing

The alert description describes what file is missing. If the contents of the CSV file is malformed the another alert is generated

Certificate monitoring SCOM CSV content error
Certificate monitoring SCOM CSV content error

The alert also enlists the line on which the error has been detected so that you can easily edit the CSV file along with the error. In the example alert we can see that monitored port has not been defined, hence the error states line 2 as well as an empty value for port. Other rules include reporting of when the Powershell script was successfully run.

SSL monitoring SCOM Monitoring rule supression
SSL monitoring SCOM Monitoring rule supression

When a certificate is not monitored or a server is unreachable an alert is generated

Certificate monitoring SCOM failed
Certificate monitoring SCOM failed

When a SSL certificate is valid the following information alert is generated. This alert can be disabled as it will generate a large number of informational alerts one for each monitored website/URL.

Certificate monitoring SCOM valid certificate
Certificate monitoring SCOM valid certificate
private certificate monitoring SCOM
private certificate monitoring SCOM

Sadly I have no expired certificates in the lab environment, as it is running in an isolated environment without internet access I am unable to post example of the scenarios certificate about to expire and certificate expired. However, these check for SSL certificate validity periode which is less than 30 days or value which is in minus, that means that the certificate expired for a number of days ago. Rules can be found under the authoring pane of Operations Manager console

If you want to manually trigger a check of SSL/TLS certificate status and see that alerts are being generated you can run the script on the SCOM agent hosting the Zeglory.com SSL remote monitoring host role. This can be done by running GenerateEvents.ps1. Be aware of the execution policy, that it might block the script execution. Execution policy will though not be a problem for SCOM rules, as that part has been taken care off. Running the script will create events in the event log as show below

SSL monitoring SCOM Event log
SSL monitoring SCOM Event log

The following event IDs have been utilized in the solution, which are monitored by SCOM/operations manager

EventIDs used:
EventID 65001 = CSV file not found – Warning (2)
EventID 65002 = CSV file is empty – Warning (2)
EventID 65003 = CSV file contains errors (1)
EventID 65010 = Could not connect to host or find host
EventID 65020 = Certificate Expired
EventID 65021 = Certificate about to expire
EventID 65022 = Certificate OK
EventID 65030 = Last runtime for this script

If you have any comments or questions feel free to contact me. If you have any suggestions or feedback let me know, and I will try to accommodate the request or proposal, however, this will be subject to amount of ongoing activities in my daily life. Below you will find the powershell script and other files utilized in this blog post, including the management pack.

#Define constants
$Filepath = "C:\serverlist\MonitoredServers.csv"
$EventSource = "Script"

function checkCsvFile ##Checks for existence of CSV file
{
Param()

if (!(Test-Path -path $Filepath))
 {
 $msg = "File not found: " + $Filepath
  $bcheckCsvFile = $false
 createEventlogObj 2 $msg 65001     #CSV File not found!
  }
 else
 {
 $bcheckCsvFile = $true
 }
  return $bcheckCsvFile
}# function readFromCsv ends here
function createEventlogObj     ##Creates and instance of eventlog and writes to logfiles    
{
param ($inputEventType,$OutputMessage,$EventID) #Can be 1 for error and 2 for warning and 3 for information!
#EventSource
#Should take a parameter which assigns Information(3),warning(2) or error(1)
$EventLog=new-object System.Diagnostics.EventLog("Application")
$EventLog.Source= "Script"
$Information=[System.Diagnostics.EventLogEntryType]::Information
$Warning=[System.Diagnostics.EventLogEntryType]::Warning
$Error=[System.Diagnostics.EventLogEntryType]::Error

if ($inputEventType -eq 1)
    {
    #$OutputMessage
    $Severity = $Error
    $EventLog.WriteEntry($OutputMessage,$Severity,$EventID)
    }
   
    elseif($inputEventType -eq 2)
    {
    $Severity = $Warning
    $EventLog.WriteEntry($OutputMessage,$Severity,$EventID)
    }
    elseif($inputEventType -eq 3)
    {
    $Severity = $Information
    $EventLog.WriteEntry($OutputMessage,$Severity,$EventID)
    }
   
    else
    {
    #Condition is not possible to occur! Future use
    $Eventlog = $Null
    $Severity = $Null
    }

}# createEventlogObj function ends here
function readCsvFile{           #####Reads in data from CSV file
param()
$csv = Import-csv -path $Filepath -header ("servername","port") -Delimiter ","
$CurrentLine = 0
$csvCount = $csv.count

foreach ($line in $csv) {
$CurrentLine = $CurrentLine + 1
       if($($line.servername -ne "servername")-AND ($line.servername -ne "")-AND ($line.port -ne "")){#Do not process headers in csv file
            getCertificateDetails $line.servername $line.port #Call function and pass values
            } #If condition ends here
        Else
            {
            # Write to event log with event
                if($CurrentLine -NE 1){
                    $msg = "Formatting error on line: "+ $CurrentLine
                    $ExtendedMsg = $msg + " Data in input file is: " + $line
  if($CurrentLine -LT $csvCount)
                     {createEventlogObj 1 $ExtendedMsg 65003} #Call function and log error to application log
                  }# If ends -Do not Log error for 1.st line
            }

} #Foreach condition ends here
}#readCsvFile function ends here

function getCertificateDetails([String]$ConnectedServer="localhost",[String]$ConnectedPort=443)
{
#Now check servername and perform connection test
    if (($ConnectedServer -NE "") -AND ($ConnectedServer -NE $Null))
        {
           $HostConnection = new-object system.net.sockets.tcpclient($ConnectedServer,$ConnectedPort)
         
            if (($HostConnection -EQ $null) -OR($HostConnection -EQ "")){
                 $msg = "Failed to connect to host: " + $ConnectedServer +" on port: "+$ConnectedPort
                    #Logg this to eventlog
                    #Variables to pass 2, Write-host text, EventID
                    createEventlogObj 2 $msg 65010 #Call function and log error to application log
                }
                Else
                {
                $stream = new-object system.net.security.sslstream($HostConnection.getstream())
                #send hostname on cert to try SSL negotiation 
                #Write-host "The value returned by stream is:"$stream
     
                        if (($stream -NE $null) -AND ($stream -NE "")){
                            $stream.authenticateasclient($ConnectedServer) 
                            $certificate = $stream.get_remotecertificate()          
                       
                        #Now check the certificate for values.........
              
                        $validto = $certificate.getexpirationdatestring()
                        #write-host "Value in validto is:"$validto
                        $subject = $certificate.get_subject()    
                        $issuer = $certificate.get_issuer()    
                       
     
                        
                $DaysToExpire = New-Timespan $(Get-Date) $($validto)
               
                    if($DaysToExpire -le 0){
                    # Certificate is Expired - Error -Log details to event viwer and make scom generate alert!
                    $msg = "Certificate for " + $ConnectedServer + " has expired. It was valid to: "+ $validto + " Certificate was issued by:" + $issuer + " subject is: " + $subject
                        createEventlogObj 1 $msg 65020 #Call function and log error to application log
                    }
                    elseif(($DaysToExpire -le 30) -AND ($DaysToExpire -GE 1))
                    {
                    #Certificate about to expire - warning
                    $msg = "Certificate for " + $ConnectedServer + " is about to expire in: " + $DaysToExpire + " days and should be renewed. It is valid to: "+ $validto + " Certificate is issued by: " + $issuer + " effected certificate subject is: " + $subject
                    createEventlogObj 2 $msg 65021 #Call function and log error to application log
                    }
                    else
                    {
                    #Certificate valid - Information
                    $msg = "Certificate for: " + $ConnectedServer + " is valid for " + $DaysToExpire + " days "
                    createEventlogObj 3 $msg 65022 #Call function and log to application log
                    #Write information events with days left for expiray? Or do nothing!
                    }
                  
                   } #If $stream value is zero test
                }#If $HostConnection ends here
        }
Else
    {# $ConnectedServer value is empty or Null- Log to eventlog
    #write-host "Do nothing! No hostname!"
    }
}#Function getCertificateDetails end here


$procceed = checkCsvFile #Call function to check if CSV file exists... Must also check if empty, wrong format etc
if ($procceed -eq $true)
{
#Open CSV file and repeat the function for all servers..........
readCsvFile #Call function to open CSV file and then report status of all certs
}
else
{
#Write-host "CSV file not found! Check eventlog for details!"
}
$msg = "Done processing csv file at: " + $(Get-Date)
createEventlogObj 3 $msg 65030 #Call function and log error to application log

#****************************Script ends***********************************

Sample monitored servers.csv file

Registry file: Rename SSLMonitoring to .reg

MP file: zeglory.com.ssl.monitoring.mp downloadable from OneDrive as it is not possible to upload the .mp file extension to this site

https://1drv.ms/u/s!AlZ3jzcY8BZMgp5Sy4pvNBncgYVV_g?e=RUR6gN

Leave a Reply

Your email address will not be published. Required fields are marked *