Many years ago I wrote a blog post about how to bulk monitor SSL certificates using Operations manager 2012, Operations Manager 2012 R2. The same theme or topic is still valid as a lot of people end up using calendar or Excel sheets to keep track of expiring certificates. Other use relatively expensive software to do this just, while it can be done almost free of cost as long as you have SCOM/Operations Manager or are willing to set u a Windows server with Powershell. The idea came as a result of a request from a customer, who wanted to bulk monitor website certificates using SSL only. This is mostly applicable to proxy servers and load balancers as well, depending on what design has been implemented. If you are using wildcard certificates, monitoring is quite simple, but if you use specific certificates with SSL offloading, this might become an administrative nightmare for you. Anyways, back then, I created a MP which ran on the RMS to monitor URLs. There are basically 3 main things that were monitored
- If a SSL/TLS certificate is valid i.e. its validity date is okay
- If a SSL/TLS certificate is about to expire within 30 days
- If a SSL/TLS certificate has expired
However, the approach had it flaws. First of all, RMS should not be performing this action, It should be possible for any server running the SCOM agent to perform this action. Second, the entire process must be as automated as possible. Only thing an admin should be responsible for doing is to provide a list of URLs that should be monitored. I received feed back regarding RMS not being able to connect to DMZ servers or internet or in general connect to servers over the WAN due to custom ports, so this was a solution but still causing problems at least for a number of users. As the MP was never shared, I ended up sharing it on email and ab course deployed it to customers. However the MP was far from perfect and was causing some issues, hence it was not shared with the community in a broad manner. In addition, it was not possible to upload the .mp file either. I see that people still are looking for this functionality as requests are being registered to my almost 8 years old blog post. So here comes the recipe along with all the necessary files. I have also created a sealed MP so that overrides do not intervene with the original MP code. During the last few days I realized that you forget a lot of things during 8 years, as I almost has forgotten how to author MPs. I ended up using several days on the process, but all is well that ends well right? I did manage to create the MP, with a custom discovery and better set of rules. So here I am sharing the code or Powershell script that performs most of the work along with CSV file structure that the script utilizes. Although, the solution works best with Operations Manager, it can still be used without SCOM or Operations Manager.
In Operations Manager/SCOM (has been tested on 2012, 2012R2, but will probably work on 07 and 2017-2018 versions as well), after the MP has been imported and initial discovery process has completed you get alerts as shown below. Do not worry about the dates/time as my SCOM environment had to be run in 2013 due to activation challenges. The entire solution has been updated/created during June 2020. It has been retested as well. The solution in SCOM looks like
When you import the MP you also get a state view, showing Windows servers functioning as watcher agents(just borrowing the term from SCOM). The idea is that, you specify which SCOM agents should perform remote monitoring of SSL certificates and you should provide a CSV file containing URLs. Each of these proxy monitors or agents will now start generating alerts as shown in the figure above. A state view containing all agents functioning as SSL certificate remote monitoring hosts is made available. In my test environment I am using my DB server as a remote SSL certificate monitoring host. It is here where the most code runs, which is forwarded to SCOM management servers that generate alerts and eventually notify with subscription channels.
How to setup the solution in SCOM
To start using the solution in SCOM you have to download the MP which is provided on this site. Although there are some additional prerequisites in order to make the solution work. In order to make the discovery process to work you have to create a registry key. If you want to set up several SCOM agents as remote SSL monitoring hosts you will have to create this registry key on each host. This might be a requirement if you do not want to use same SCOM agent to monitor SSL/TLS certificates in production and test environment, or in internal LAN, DMZ, WAN and on internet. You have to create a key under HKLM\SOFTWARE\SSLMonitoring
You can also create a registry file in Notepad by adding this content and saving it as .reg extension. The file is also provided in the blog.
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\SSLMonitoring]If you make sure that this registry value is populated prior to importing the MP the agent will show up on the state view, indicating an instance of class “zeglory.com SSL Monitoring host”. The discovery runs every 24 hours by default, but can be overridden to run less frequently or more often as you might require.
After a successful initial discovery, the rules within the management pack will start to create the file structure needed to monitor URLs. The MP will create a directory on the root of C: drive, C:\serverlist
The folder will further be populated with two files. This is done automatically by the MP so you do not need to perform any further actions. The GenerateEvents.ps1 file is utilized to run the script every 12 hours. However, the MonitoredServers.csv file must be populated manually.
The contents of CSV file must be set manually. You have to enter servername,port, as headings for CSV file and values for URLs containing SSL/TLS certificate to be monitored. In my environment I am monitoring zeglory.com running on port 443, and SCOM2012dc.contoso.no on port 443.
The management pack generates following rules, which can be overridden according to needs.
If the CSV file is missing or has accidentally been deleted an alert is generated, this is checked each time the Powershell script is run, which is by default each 12 hours.
The alert description describes what file is missing. If the contents of the CSV file is malformed the another alert is generated
The alert also enlists the line on which the error has been detected so that you can easily edit the CSV file along with the error. In the example alert we can see that monitored port has not been defined, hence the error states line 2 as well as an empty value for port. Other rules include reporting of when the Powershell script was successfully run.
When a certificate is not monitored or a server is unreachable an alert is generated
When a SSL certificate is valid the following information alert is generated. This alert can be disabled as it will generate a large number of informational alerts one for each monitored website/URL.
Sadly I have no expired certificates in the lab environment, as it is running in an isolated environment without internet access I am unable to post example of the scenarios certificate about to expire and certificate expired. However, these check for SSL certificate validity periode which is less than 30 days or value which is in minus, that means that the certificate expired for a number of days ago. Rules can be found under the authoring pane of Operations Manager console
If you want to manually trigger a check of SSL/TLS certificate status and see that alerts are being generated you can run the script on the SCOM agent hosting the Zeglory.com SSL remote monitoring host role. This can be done by running GenerateEvents.ps1. Be aware of the execution policy, that it might block the script execution. Execution policy will though not be a problem for SCOM rules, as that part has been taken care off. Running the script will create events in the event log as show below
The following event IDs have been utilized in the solution, which are monitored by SCOM/operations manager
EventIDs used:
EventID 65001 = CSV file not found – Warning (2)
EventID 65002 = CSV file is empty – Warning (2)
EventID 65003 = CSV file contains errors (1)
EventID 65010 = Could not connect to host or find host
EventID 65020 = Certificate Expired
EventID 65021 = Certificate about to expire
EventID 65022 = Certificate OK
EventID 65030 = Last runtime for this script
If you have any comments or questions feel free to contact me. If you have any suggestions or feedback let me know, and I will try to accommodate the request or proposal, however, this will be subject to amount of ongoing activities in my daily life. Below you will find the powershell script and other files utilized in this blog post, including the management pack.
#Define constants
$Filepath = "C:\serverlist\MonitoredServers.csv"
$EventSource = "Script"
function checkCsvFile ##Checks for existence of CSV file
{
Param()
if (!(Test-Path -path $Filepath))
{
$msg = "File not found: " + $Filepath
$bcheckCsvFile = $false
createEventlogObj 2 $msg 65001 #CSV File not found!
}
else
{
$bcheckCsvFile = $true
}
return $bcheckCsvFile
}# function readFromCsv ends here
function createEventlogObj ##Creates and instance of eventlog and writes to logfiles
{
param ($inputEventType,$OutputMessage,$EventID) #Can be 1 for error and 2 for warning and 3 for information!
#EventSource
#Should take a parameter which assigns Information(3),warning(2) or error(1)
$EventLog=new-object System.Diagnostics.EventLog("Application")
$EventLog.Source= "Script"
$Information=[System.Diagnostics.EventLogEntryType]::Information
$Warning=[System.Diagnostics.EventLogEntryType]::Warning
$Error=[System.Diagnostics.EventLogEntryType]::Error
if ($inputEventType -eq 1)
{
#$OutputMessage
$Severity = $Error
$EventLog.WriteEntry($OutputMessage,$Severity,$EventID)
}
elseif($inputEventType -eq 2)
{
$Severity = $Warning
$EventLog.WriteEntry($OutputMessage,$Severity,$EventID)
}
elseif($inputEventType -eq 3)
{
$Severity = $Information
$EventLog.WriteEntry($OutputMessage,$Severity,$EventID)
}
else
{
#Condition is not possible to occur! Future use
$Eventlog = $Null
$Severity = $Null
}
}# createEventlogObj function ends here
function readCsvFile{ #####Reads in data from CSV file
param()
$csv = Import-csv -path $Filepath -header ("servername","port") -Delimiter ","
$CurrentLine = 0
$csvCount = $csv.count
foreach ($line in $csv) {
$CurrentLine = $CurrentLine + 1
if($($line.servername -ne "servername")-AND ($line.servername -ne "")-AND ($line.port -ne "")){#Do not process headers in csv file
getCertificateDetails $line.servername $line.port #Call function and pass values
} #If condition ends here
Else
{
# Write to event log with event
if($CurrentLine -NE 1){
$msg = "Formatting error on line: "+ $CurrentLine
$ExtendedMsg = $msg + " Data in input file is: " + $line
if($CurrentLine -LT $csvCount)
{createEventlogObj 1 $ExtendedMsg 65003} #Call function and log error to application log
}# If ends -Do not Log error for 1.st line
}
} #Foreach condition ends here
}#readCsvFile function ends here
function getCertificateDetails([String]$ConnectedServer="localhost",[String]$ConnectedPort=443)
{
#Now check servername and perform connection test
if (($ConnectedServer -NE "") -AND ($ConnectedServer -NE $Null))
{
$HostConnection = new-object system.net.sockets.tcpclient($ConnectedServer,$ConnectedPort)
if (($HostConnection -EQ $null) -OR($HostConnection -EQ "")){
$msg = "Failed to connect to host: " + $ConnectedServer +" on port: "+$ConnectedPort
#Logg this to eventlog
#Variables to pass 2, Write-host text, EventID
createEventlogObj 2 $msg 65010 #Call function and log error to application log
}
Else
{
$stream = new-object system.net.security.sslstream($HostConnection.getstream())
#send hostname on cert to try SSL negotiation
#Write-host "The value returned by stream is:"$stream
if (($stream -NE $null) -AND ($stream -NE "")){
$stream.authenticateasclient($ConnectedServer)
$certificate = $stream.get_remotecertificate()
#Now check the certificate for values.........
$validto = $certificate.getexpirationdatestring()
#write-host "Value in validto is:"$validto
$subject = $certificate.get_subject()
$issuer = $certificate.get_issuer()
$DaysToExpire = New-Timespan $(Get-Date) $($validto)
if($DaysToExpire -le 0){
# Certificate is Expired - Error -Log details to event viwer and make scom generate alert!
$msg = "Certificate for " + $ConnectedServer + " has expired. It was valid to: "+ $validto + " Certificate was issued by:" + $issuer + " subject is: " + $subject
createEventlogObj 1 $msg 65020 #Call function and log error to application log
}
elseif(($DaysToExpire -le 30) -AND ($DaysToExpire -GE 1))
{
#Certificate about to expire - warning
$msg = "Certificate for " + $ConnectedServer + " is about to expire in: " + $DaysToExpire + " days and should be renewed. It is valid to: "+ $validto + " Certificate is issued by: " + $issuer + " effected certificate subject is: " + $subject
createEventlogObj 2 $msg 65021 #Call function and log error to application log
}
else
{
#Certificate valid - Information
$msg = "Certificate for: " + $ConnectedServer + " is valid for " + $DaysToExpire + " days "
createEventlogObj 3 $msg 65022 #Call function and log to application log
#Write information events with days left for expiray? Or do nothing!
}
} #If $stream value is zero test
}#If $HostConnection ends here
}
Else
{# $ConnectedServer value is empty or Null- Log to eventlog
#write-host "Do nothing! No hostname!"
}
}#Function getCertificateDetails end here
$procceed = checkCsvFile #Call function to check if CSV file exists... Must also check if empty, wrong format etc
if ($procceed -eq $true)
{
#Open CSV file and repeat the function for all servers..........
readCsvFile #Call function to open CSV file and then report status of all certs
}
else
{
#Write-host "CSV file not found! Check eventlog for details!"
}
$msg = "Done processing csv file at: " + $(Get-Date)
createEventlogObj 3 $msg 65030 #Call function and log error to application log
#****************************Script ends***********************************Sample monitored servers.csv file
Registry file: Rename SSLMonitoring to .reg
MP file: zeglory.com.ssl.monitoring.mp downloadable from OneDrive as it is not possible to upload the .mp file extension to this site