Monday, 28 July 2014

Using PowerShell 3 to get Clustered ONTAP Command History and Send to Syslog Server

Note: See the following updates from the 17th August:
Command History to Syslog for CDOT Version 2: Part 2/2 - The Script

Getting PowerShell 3 (If you’ve not already got it)

Firstly, check you’ve got PowerShell 3 using the command:
$PSVersionTable.PSVersion

In the example below we only have PowerShell 2.

PS C:\Users\administrator.LAB> $PSVersionTable.PSVersion

Major  Minor  Build  Revision
-----  -----  -----  --------
2      0      -1     -1

PowerShell 3 is built into Windows 2012. For Windows 7 SP1+ and Windows Server 2008R2 SP1+, you can download PowerShell 3 from:


Testing the Script

You can get a free syslog from here:


Example Outputs

I did try to get the outputs from 7-Mode and C-Mode looking roughly the same.

Image: Example 7-Mode Output to Syslog
Image: Example of C-Mode Output to Syslog Using the Script
The Script

As always, formatted for blogger (definitely displays all the characters correctly, including greater and less than symbols, if using Google Chrome). Note: Just over a day was spent on this endeavor - when I get time I intend to revisit and improve the script.

###############################################################################
## Command History to Syslog for CDOT (v1.1 - July 23rd 2014)                ##
## ==========================================================                ##
##                                                                           ##
## Description:                                                              ##
## In CDOT 8.2.1 it is not possible to send the contents of the              ##
## command-history.log to a syslog server natively. This script gets around  ##
## this by simply reading and comparing reads of the command-history.log     ##
## files as obtained over the SPI and sending the differences. It can also   ##
## handle when the command-history.log recycles itself.                      ##
##                                                                           ##
## Additional notes:                                                         ##
## + It's not real time, but, if you set a scheduled task to run this        ##
## regularly, it's not too far off.                                          ##
## + This uses all native PowerShell 3 commands. It doesn't use any          ##
## DataONTAP PowerShell toolkit cmdlets.                                     ##
## + The user connecting to the SPI needs only HTTP application, but admin   ##
## role (doesn't work with HTTP and readonly role.)                          ##
## ::> sec login cre -user syslogger -app http -auth password -role admin    ##
###############################################################################

###############################################################################
## Setup Variables                                                           ##
###############################################################################

$currentWorkingPath  = (pwd).path
$whoAmI              = $env:username
$credentialsFile     = $currentWorkingPath + "\command-history.toSyslogCreds_" + $whoAmI + ".txt"
$clusterNameFQDNorIP = "NACLU5"
$nodeNames           = @("NACLU5N1","NACLU5N2") # Separate by , for more nodes (case sensitive)
$URLtoSPI            = "https://" + $clusterNameFQDNorIP + "/spi/"
$syslogServer = "10.10.10.17"

###############################################################################
## Store File Names for Locally Stored Command History Logs                  ##
###############################################################################

# We store 3 command-history.logs (per node) - why?
# We check the latest against the previous and send differences to syslog, but ...
# if the file size of the latest is smaller than the previous we know the logs have rotated on CDOT, so ...
# we compare the last saved command-history.log with the previous one, send all the differences,
# and we then send all the latest file's messages
$savedLogFileName = @()

foreach ($node in $nodeNames){
$savedLogFileName   += $currentWorkingPath + "\command-history." + $node + ".latest.log"
$savedLogFileName   += $currentWorkingPath + "\command-history." + $node + ".previous.log"
$savedLogFileName   += $currentWorkingPath + "\command-history." + $node + ".lastCycled.log"}

###############################################################################
## A Fix for Invoke-Web Request not liking some SSL Certificates             ##
###############################################################################

# This add-type is to get around SSL certificate errors with Invoke-WebRequest
# From: http://stackoverflow.com/questions/11696944/powershell-v3-invoke-webrequest-https-error
add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
            ServicePoint srvPoint, X509Certificate certificate,
            WebRequest request, int certificateProblem) {
            return true;
        }
    }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

###############################################################################
## FUNCTION: Send-SyslogMessage                                              ##
## This was written by:                                                      ##
##                      Kieran Jacobsen                                      ##
## Original link:                                                            ##
## http://aperturescience.su/blog/2014/7/1/sending-syslog-messages-from-powershell.html
## This function is totally perfect - I've not made any changes to it (apart ##
## from compacting it a little.) The function needs to be in the script and  ##
## before the main part of the program.                                      ##
###############################################################################

Add-Type -TypeDefinition @"
public enum Syslog_Facility
{
kern,
user,
mail,
daemon,
auth,
syslog,
lpr,
news,
uucp,
clock,
authpriv,
ftp,
ntp,
logaudit,
logalert,
cron,
local0,
local1,
local2,
local3,
local4,
local5,
local6,
local7,
}
"@

Add-Type -TypeDefinition @"
public enum Syslog_Severity
{
Emergency,
Alert,
Critical,
Error,
Warning,
Notice,
Informational,
Debug
}
"@

function Send-SyslogMessage {
<#
.SYNOPSIS: Sends a SYSLOG message to a server running the SYSLOG daemon
.DESCRIPTION: Sends a message to a SYSLOG server as defined in RFC 5424. A SYSLOG message contains not only raw message text,
but also a severity level and application/system within the host that has generated the message.

.PARAMETER Server: Destination SYSLOG server that message is to be sent to
.PARAMETER Message: Our message
.PARAMETER Severity: Severity level as defined in SYSLOG specification, must be of ENUM type Syslog_Severity
.PARAMETER Facility: Facility of message as defined in SYSLOG specification, must be of ENUM type Syslog_Facility
.PARAMETER Hostname: Hostname of machine the mssage is about, if not specified, local hostname will be used
.PARAMETER Timestamp: Timestamp, myst be of format, "yyyy:MM:dd:-HH:mm:ss zzz", if not specified, current date & time will be used
.PARAMETER UDPPort: SYSLOG UDP port to send message to

.INPUTS: Nothing can be piped directly into this function
.OUTPUTS: Nothing is output

.EXAMPLE:
Send-SyslogMessage mySyslogserver "The server is down!" Emergency Mail
Sends a syslog message to mysyslogserver, saying "server is down", severity emergency and facility is mail

.NOTES
NAME: Send-SyslogMessage
AUTHOR: Kieran Jacobsen
LASTEDIT: 2014 07 01

.LINK: https://github.com/kjacobsen/PowershellSyslog
.LINK: http://aperturescience.su
#>
[CMDLetBinding()]
Param
(
[Parameter(mandatory=$true)] [String] $Server,
[Parameter(mandatory=$true)] [String] $Message,
[Parameter(mandatory=$true)] [Syslog_Severity] $Severity,
[Parameter(mandatory=$true)] [Syslog_Facility] $Facility,
[String] $Hostname,
[String] $Timestamp,
[int] $UDPPort = 514
)

# Create a UDP Client Object
$UDPCLient = New-Object System.Net.Sockets.UdpClient
$UDPCLient.Connect($Server, $UDPPort)

# Evaluate the facility and severity based on the enum types
$Facility_Number = $Facility.value__
$Severity_Number = $Severity.value__
Write-Verbose "Syslog Facility, $Facility_Number, Severity is $Severity_Number"

# Calculate the priority
$Priority = ($Facility_Number * 8) + $Severity_Number
Write-Verbose "Priority is $Priority"

# If no hostname parameter specified, then set it
if (($Hostname -eq "") -or ($Hostname -eq $null)){$Hostname = Hostname}

# If the hostname hasn't been specified, then we will use the current date and time
if (($Timestamp -eq "") -or ($Timestamp -eq $null)){$Timestamp = Get-Date -Format "yyyy:MM:dd:-HH:mm:ss zzz"}

# Assemble the full syslog formatted message
$FullSyslogMessage = "<{0}>{1} {2} {3}" -f $Priority, $Timestamp, $Hostname, $Message

# create an ASCII Encoding object
$Encoding = [System.Text.Encoding]::ASCII

# Convert into byte array representation
$ByteSyslogMessage = $Encoding.GetBytes($FullSyslogMessage)

# If the message is too long, shorten it
if ($ByteSyslogMessage.Length -gt 1024){$ByteSyslogMessage = $ByteSyslogMessage.SubString(0, 1024)}

# Send the Message
$UDPCLient.Send($ByteSyslogMessage, $ByteSyslogMessage.Length)
}

# Note: I've made no changes to the above function except compacting it a bit.

###############################################################################
## FUNCTION: Convert-MessageHistoryDate2SyslogFormat                         ##
## e.g. "Tue Jul 22 2014 09:13:40 +01:00" to "yyyy:MM:dd:-HH:mm:ss zzz"      ##
###############################################################################

function Convert-MessageHistoryDate2SyslogFormat {

Param( [Parameter(mandatory=$true)] [String] $cdotDateFormat )
$syslogDateFormat  = $cdotDateFormat.substring(11,4) # yyyy
$syslogDateFormat += ":"
$month             = $cdotDateFormat.substring(4,3)
$months            = @("Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec")
$i=0

do {
$iToString = [string]($i+1)
if ($i -lt 9){$iToString = "0" + $iToString}
if ($month -eq $months[$i]){$syslogDateFormat += $iToString} # MM
$i++
} while ($i -lt 12)

$syslogDateFormat += ":"
$syslogDateFormat += $cdotDateFormat.substring(8,2) # dd
$syslogDateFormat += ":-"
$syslogDateFormat += $cdotDateFormat.substring(16,2) # HH
$syslogDateFormat += ":"
$syslogDateFormat += $cdotDateFormat.substring(19,2) # mm
$syslogDateFormat += ":"
$syslogDateFormat += $cdotDateFormat.substring(22,2) # ss
$syslogDateFormat += " +"
$syslogDateFormat += $cdotDateFormat.substring(26,5) # zzz
return $syslogDateFormat}

###############################################################################
## FUNCTION: FormAndSendToSyslog                                             ##
###############################################################################

function FormAndSendToSyslog {

Param(
[Parameter(mandatory=$true)] [String] $syslogger,
[Parameter(mandatory=$true)] [String] $inputLine,
[Parameter(mandatory=$true)] [String] $nodeFrom
)

# We need to process the date into syslog format
$cdotDateTime = $inputLine.substring(27,31)
$syslogDateTime = Convert-MessageHistoryDate2SyslogFormat $cdotDateTime
# We remove the  "00000046.00000b3c 00001b27 " and date from the command-history lines
$outputLine = $inputLine.remove(0,59)
"$outputLine Debug local7 $nodeFrom $syslogDateTime" # Write Output To Screen
$Sent2Syslog = Send-SyslogMessage $syslogger $outputLine Debug local7 $nodeFrom $syslogDateTime}

###############################################################################
## 1) Check for credentials or prompt                                        ##
###############################################################################

# Check for credentials File
$isThereACredentialsFile = Test-Path $credentialsFile

# If no credentials file, prompt and write one
if(!$isThereACredentialsFile){
$getCredential          = Get-Credential -Credential $cotA1
$username               = $getCredential.username
$securePassword         = $getCredential.password
$credential             = New-Object System.Management.Automation.PsCredential($username, $securePassword)
$securePasswordString   = $securePassword | ConvertFrom-SecureString
$credentialsFileContent = @($username,$securePasswordString)
$credentialsFileContent | Set-Content $credentialsFile}

# If there is a credentials file, read it
if($isThereACredentialsFile){
$fileContent          = Get-Content $credentialsFile
$username             = $fileContent[0]
$securePasswordString = $fileContent[1]
$securePassword       = $securePasswordString | ConvertTo-SecureString
$credential           = New-Object System.Management.Automation.PsCredential($username, $securePassword)}

###############################################################################
## 2) Cycle Through all the Nodes and Send Syslog Updates                    ##
###############################################################################

# Count the node we're on
$count = 0
foreach ($node in $nodeNames){

###############################################################################
# 2a) Check for stored command-history files                                  #
###############################################################################

"Checking node $node"
$countX3 = $count * 3
$isThereALatestSavedCommandHistoryFile   = Test-Path $savedLogFileName[$countX3 + 0]
$isThereAPreviousSavedCommandHistoryFile = Test-Path $savedLogFileName[$countX3 + 1]

# If there is a Latest Saved Command History File, we rename it
if ($isThereALatestSavedCommandHistoryFile){
if ($isThereAPreviousSavedCommandHistoryFile){Remove-Item $savedLogFileName[$countX3 + 1]}
Rename-Item $savedLogFileName[$countX3 + 0] $savedLogFileName[$countX3 + 1]}

# Get a new LatestSavedCommandHistoryFile
$URLtoMlog = $URLtoSPI + $node + "/etc/log/mlog/"
$LatestCommandHistoryFileURL = $URLtoMlog + "command-history.log"
invoke-webrequest -uri $LatestCommandHistoryFileURL -credential $credential -OutFile $savedLogFileName[$countX3 + 0]

###############################################################################
# 2b) Compare logs                                                            #
###############################################################################

# If there was no a Previous Saved Command History File, the for loop effectively ends here.
# If there was a Previous Saved Command History File, we continue!
if ($isThereAPreviousSavedCommandHistoryFile){

$sizeOfLatestFile   = (Get-Item $savedLogFileName[$countX3 + 0]).length
$sizeOfPreviousFile = (Get-Item $savedLogFileName[$countX3 + 1]).length

# If the files are the same size (no update), the for loop effectively ends here.
# If the files are different sizes, we continue!
if ($sizeOfLatestFile -ne $sizeOfPreviousFile){

# If the latest file is greater than the previous, get the differences and transmit them
if ($sizeOfLatestFile -gt $sizeOfPreviousFile){
$differences = Compare-Object $(Get-Content $savedLogFileName[$countX3 + 0]) $(Get-Content $savedLogFileName[$countX3 + 1])
foreach ($difference in $differences){
$message = $difference.InputObject
FormAndSendToSyslog $syslogServer $message $node}}

# If the latest file is smaller than the previous, we need to do read from an archived log file
if ($sizeOfLatestFile -lt $sizeOfPreviousFile){

# We get the list of command-history links from mlog and find the last archived log
$linksInMlogPage           = invoke-webrequest -uri $URLtoMlog -credential $credential
$linksInMlog               = $linksInMlogPage.links | foreach {$_.href}
$commandHistoryLogLinks    = $linksInMlog | where-object {$_ -match 'command-history'}
$countCommandHistoryLogs   = $commandHistoryLogLinks.count
$penultimateCmdHistFile    = $commandHistoryLogLinks[$countCommandHistoryLogs-2]

# Then we save that log file
$penultimateCmdHistFileURL = $URLtoMlog + $penultimateCmdHistFile
invoke-webrequest -uri $penultimateCmdHistFileURL -credential $credential -OutFile $savedLogFileName[$countX3 + 2]

# Check for differences between that and the previous one
$differences = Compare-Object $(Get-Content $savedLogFileName[$countX3 + 1]) $(Get-Content $savedLogFileName[$countX3 + 2])
foreach ($difference in $differences){
$message = $difference.InputObject
FormAndSendToSyslog $syslogServer $message $node}

# Finally send what's in the new log
$newContent = Get-Content $savedLogFileName[$countX3 + 0]
foreach ($line in $newContent){FormAndSendToSyslog $syslogServer $line $node}

} # END if ($sizeOfLatestFile -lt $sizeOfPreviousFile)
# Note: The circumstance where command-history.log.9999999999 goes back to command-history.log.000000000 has not been handled (would take a long time if log cycles once a day.)
} # END if ($sizeOfLatestFile -ne $sizeOfPreviousFile)
} # END if ($isThereAPreviousSavedCommandHistoryFile)
$count++
} # END foreach ($node

Minimum Permission User for Joining Clustered ONTAP Clusters to OnCommand Performance Manager 1.0

Following on from the previous blog post where we created a minimum permission user account for joining clusters to OCUM 6.1, this unofficial blog post outlines how to do the same for OPM 1.0. As before, the role was built starting with a read-only account, then looking at the command-history.log and seeing what commands were error-ing regards lack of access, then adding in just the required permissions.

Note: The role will need to be reviewed for later releases of OnCommand Performance Manager and Clustered Data ONTAP (tested with 8.2.1.)

Creating the OCPM Role

The commands below create a role called ocpm:

CLUSTERNAME::>
security login role create -role ocpm -cmddirname DEFAULT -access readonly
security login role create -role ocpm -cmddirname "cluster application-record" -access all
security login role create -role ocpm -cmddirname "volume modify" -access all
security login role create -role ocpm -cmddirname "storage disk show" -access all

Note: The role is constructed resolving ONTAP errors as seen in the command-history.log. Insufficient privileges errors were seen for “cluster-application-record-create”, “volume-modify-iter”, and “storage-shelf-list-info”. Adding “volume modify all” also effects "volume create" and "volume show". Adding “storage disk show all” also affects "storage disk modify".

Creating the OCPM User

The command below creates a user called ocpm:

CLUSTERNAME::>
security login create -username ocpm -application ontapi -role ocpm -authmethod password

Additional Information

In CDOT 8.2.1, OPM needs to be able to enable QoS counters on volumes. A volume that does not have a QoS policy will get added to the hidden _Performance_Monitor_volumes QoS Policy Group - which the user described above has the ability to do.

Example:

CLUSTERNAME::> vol show -fields qos-policy-group -type RW
vserver       volume qos-policy-group
------------- ------ ----------------
VSERVER1      testvol
                     _Performance_Monitor_volumes

Minimum Permission User for Joining Clustered ONTAP Clusters to OnCommand Unified Manager 6.1

If you’re looking to create an account with minimum permissions for joining Clustered ONTAP clusters (tested with 8.2.1) to OnCommand Unified Manager 6.1, this unofficial blog post might help. The role was built starting with a read-only account, then looking at the command-history.log and seeing what commands were error-ing regards lack of access, then adding in just the required permissions.

Note: The role will need to be reviewed for later releases of OnCommand Unified Manager and Clustered Data ONTAP.

Requirements for the User Account

The user account needs to support the following features:

- Allow the monitoring abilities of OCUM to work
- Allow SnapRestore to function for restores within the same read-write volume
- Allow NDMP restore from other volumes (e.g. SnapVault/DR read-only volumes.)

Pre-Requisites

The role requires that ‘vserver services ndmp’ is turned on for the cluster in order for NDMP restores to work.

Commands:

CLUSTERNAME::>
vserver services ndmp show -vserver CLUSTERNAME -fields enable
vserver services ndmp on -vserver CLUSTERNAME

Note: Creating an OCUM user that can also turn NDMP on, results in a pretty unrestricted account, like the below. This is due to certain hardcoding of permissions to roles in CDOT 8.2.X.

UserName   Application Method   Role Name
---------- ----------- -------- ---------
ocum       console     password admin   
ocum       ontapi      password admin   
ocum       ssh         password backup  

Creating the OCUM Role

The commands below create a role called ocum:

CLUSTERNAME::>
security login role create -role ocum -cmddirname DEFAULT -access readonly
security login role create -role ocum -cmddirname "volume file show-disk-usage" -access all
security login role create -role ocum -cmddirname "volume snapshot restore-file" -access all
security login role create -role ocum -cmddirname "storage aggregate check_spare_low" -access all

Note 1: The first and fourth lines are required for OCUM monitoring. Line 4 was because seeing an alert for “aggr-check-spare-low” with “Insufficient privileges” in the command-history.log.
Note 2: The second and third lines are required for SnapRestore to function.
Note 3: I was considering adding the 5th line below because seeing some errors from “storage-shelf-list-info” in the command-history.log with “Enclosure services not ready at this time” - I put this down to testing on a SIM though. Adding “storage disk show all” also affects "storage disk modify".
::> security login role create -role ocum -cmddirname "storage disk show" -access all

Creating the OCUM User

The commands below create a user called ocum:

CLUSTERNAME::>
security login create -username ocum -application ontapi -role ocum -authmethod password
security login create -username ocum -application ssh -role backup -authmethod password

Note: The second line is required because only users with application ssh and the role admin or backup can run the command “vserver services ndmp generate-password” which is required for NDMP restores to function (also the backup role comes with “vserver services ndmpall access.)

Saturday, 19 July 2014

How to Use PowerShell to Health Check Event Logs in Clustered ONTAP

This post presents a function that can be used to health check event logs in Clustered ONTAP.

<#######################
HC-EventLogShow Function
========================
The function takes 2 or 3 arguments:
1) HC-EventLogShow SEVERITY LAST?HOURS
2) HC-EventLogShow SEVERITY LAST?HOURS NODE

It returns as an array the output of the Clustershell command "event log show", for the chosen SEVERITY (and optionally NODE), and going back for the chosen LAST?HOURS. The main feature of the script is that it suppresses messages where they repeat and records the number of repetitions, so you just see all the unique events.

The function returns the information as an array with columns for: TIME, NODE, COUNT (Occurrences), EVENT. Like previous HC (Health Check) scripts, the idea is that this array can then be manipulated to presented the information in whatsoever way is desired (e.g. an Excel spreadsheet.)

Note 1: Requires a pre-existing connection to a CDOT cluster
Note 2: Severities are (in order of most to least severe):
EMERGENCY, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFORMATIONAL, DEBUG
Note 3: For more information on "Get-NcEmsMessage", see section at end of this script "Investigating Get-NcEmsMessage"

HC-EventLogShowBO (Basic Output) Function
=========================================
This function exists only to display the output in a basic format.
1) HC-EventLogShowBO SEVERITY LAST?HOURS
2) HC-EventLogShowBO SEVERITY LAST?HOURS NODE
############################################>

function HC-EventLogShow {

if (!$args[0]){return null} # Need a SEVERITY argument
if (!$args[1]){return null} # Need an HOURS argument
if ($args[2]){$node = $args[2]} # NODE argument is optional

# We use a template to make the script more efficient byonly pulling the information we need
$attributes = Get-NcEmsMessage -Template
$attributes.Event = ""
$attributes.Node = ""
$attributes.Time = ""

# If we have a NODE argument, use it, otherwise don't!
if (!$args[2]){$messages = Get-NcEmsMessage -Severity $args[0] -StartTime (Get-Date).AddHours(-$args[1]) -Attributes $attributes}
if ($args[2]){$messages = Get-NcEmsMessage -Severity $args[0] -StartTime (Get-Date).AddHours(-$args[1]) -Attributes $attributes -Node $node}
# Note: We don't check for case sensitivity of the node name - this must be entered correctly.
      
$count = $messages.count
# If you add all the event repetitions recorded, it should equal this number.

# We create an array big enough to contain all the messages but it shouldn't need to be that big due to duplicates!
$outputArray = New-Object 'object[,]' 5,$count
# We start off with an array with no rows this increases every time we find a unique one.
$arrayYsize = 0

# In the array, our X fields are:
# TIME (0), NODE (1), EVENT (2), and COUNT (Occurrences) (3)
      
# Cycle through all the messages
foreach ($message in $messages){

$i=0
$match = $null # Sets Match to false per new message

# A do loop comparing node, then event, and recording a match if both do.
:Check4Match do {

if ($message.Node -eq $outputArray[1,$i]){
if ($message.Event -eq $outputArray[2,$i]){
$match = "true"
$outputArray[3,$i]++ # Accumulate the counter
break Check4Match # Have found a match - stop checking!
}
}

$i++                      
} while ($i -lt ($arrayYsize-1))

# If no match, create a new line in the array.
if (!$match){
$outputArray[0,$arrayYsize]=$message.TimeDT
$outputArray[1,$arrayYsize]=$message.Node
$outputArray[2,$arrayYsize]=$message.Event
$outputArray[3,$arrayYsize]=1
$arrayYsize++
}

} # END of foreach ($message in $messages)

, $outputArray
# returns the array

}

############################
      
function HC-EventLogShowBO {

if(!$args[0]){return} # Need a SEVERITY argument
if(!$args[1]){return} # Need an HOURS argument

if (!$args[2]){$output = HC-EventLogShow $args[0] $args[1]}
if ($args[2]){$output = HC-EventLogShow $args[0] $args[1] $args[2]}

"OUTPUT"
"======"
"TIME # NODE # COUNT # EVENT"

$i=0
$data = "true"
while($data) {
      
$TimeDT = $output[0,$i]
$node = $output[1,$i]
$count = $output[3,$i]
$event = $output[2,$i]
"$TimeDT # $node # $count # $event"
$i++
if ($output[0,$i] -eq $null){$data = $null}
      
}

}

<# Investigating Get-NcEmsMessage
$EMS_Event.EmsSeverity                   
$EMS_Event.Event                         
$EMS_Event.EventXmlLen                   
$EMS_Event.EventXmlLen                    
$EMS_Event.EventXmlLenSpecified          
$EMS_Event.KernelGen                     
$EMS_Event.KernelGenSpecified            
$EMS_Event.KernelSeqNum                  
$EMS_Event.KernelSeqNumSpecified         
$EMS_Event.MessageName                    
$EMS_Event.NcController                  
$EMS_Event.Node                          
$EMS_Event.NumSuppressedSinceLast        
$EMS_Event.NumSuppressedSinceLastSpecified
$EMS_Event.SeqNum                        
$EMS_Event.Severity                      
$EMS_Event.Source                        
$EMS_Event.Time                          
$EMS_Event.TimeDT                        
#>