A full /var/log can cause various problems on an affected ESX host – unable to snapshot a guest, unable to enable HA agent …
One potential culprit for filling up /var/log, can be the cma.log file generated by the HP Insight Management agents.
Example:
[root@esx01 /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 4.9G 1.9G 2.8G 41% /
/dev/sda2 2.0G 2.0G 0 100% /var/log
/dev/cciss/c0d0p1 198M 68M 121M 36% /boot
Investigation uncovers that the cma.log is taking up nearly all the space:
[root@esx01 /]# cd /var/log/hp-snmp-agents/
[root@esx01 hp-snmp-agents]# ls -s
total 1690984
1690984 cma.log 0 cmaX.log
- or as seen via Veeam Fast SCP (below) -
Fix:
From the folder /var/log/hp-snmp-agents, use the rm command to delete cma.log
[root@esx01 hp-snmp-agents]# rm cma.log
rm: remove regular file `cma.log'? y
(or delete via Veeam Fast SCP)
Then change to the /etc/init.d directory, and run the command ‘service hp-snmp-agents restart’ to restart the HP Management Agents (the cma.log file is invisible, and remains in /var/log until this service is restarted.) This command will not affect live guests in any way.
[root@esx01 hp-snmp-agents]# cd /etc/init.d
[root@esx01 init.d]# service hp-snmp-agents restart
Shutting down NIC Agent Daemon (cmanicd): [ OK ]
Shutting down Storage Event Logger (cmaeventd): [ OK ]
Shutting down FCA agent (cmafcad): [FAILED]
Shutting down SAS agent (cmasasd): [ OK ]
Shutting down IDA agent (cmaidad): [ OK ]
Shutting down IDE agent (cmaided): [ OK ]
Shutting down SCSI agent (cmascsid): [ OK ]
Shutting down Health agent (cmahealthd): [ OK ]
Shutting down Standard Equipment agent (cmastdeqd): [ OK ]
Shutting down Host agent (cmahostd): [ OK ]
Shutting down Threshold agent (cmathreshd): [ OK ]
Shutting down RIB agent (cmasm2d): [ OK ]
Shutting down Rack Infrastructure Info Srv (cpqriisd): [FAILED]
Shutting down Rack agent (cmarackd): [FAILED]
Shutting down Performance agent (cmaperfd): [ OK ]
Shutting down SNMP Peer (cmapeerd): [ OK ]
Starting Health agent (cmahealthd): [ OK ]
Starting Standard Equipment agent (cmastdeqd): [ OK ]
Starting Host agent (cmahostd): [ OK ]
Starting Threshold agent (cmathreshd): [ OK ]
Starting hp-ilo:
Already started hpilo module. [ OK ]
Starting RIB agent (cmasm2d): [ OK ]
Starting hp-ilo:
Already started hpilo module. [ OK ]
Starting Rack Infrastructure Info Srv (cpqriisd): [ OK ]
Starting Rack agent (cmarackd): [ OK ]
Starting Performance agent (cmaperfd): [ OK ]
Starting SNMP Peer (cmapeerd): [ OK ]
Starting Storage Event Logger (cmaeventd): [ OK ]
Starting FCA agent (cmafcad): [ OK ]
Starting SAS agent (cmasasd): [ OK ]
Starting IDA agent (cmaidad): [ OK ]
Starting IDE agent (cmaided): [ OK ]
Starting SCSI agent (cmascsid): [ OK ]
Starting NIC Agent Daemon (cmanicd): [ OK ]
Finally, re-run the df –h to check space on /var/log has been freed up
[root@esx01 init.d]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 4.9G 1.9G 2.8G 41% /
/dev/sda2 2.0G 318M 1.6G 17% /var/log
/dev/cciss/c0d0p1 198M 68M 121M 36% /boot
One potential culprit for filling up /var/log, can be the cma.log file generated by the HP Insight Management agents.
Example:
[root@esx01 /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 4.9G 1.9G 2.8G 41% /
/dev/sda2 2.0G 2.0G 0 100% /var/log
/dev/cciss/c0d0p1 198M 68M 121M 36% /boot
Investigation uncovers that the cma.log is taking up nearly all the space:
[root@esx01 /]# cd /var/log/hp-snmp-agents/
[root@esx01 hp-snmp-agents]# ls -s
total 1690984
1690984 cma.log 0 cmaX.log
- or as seen via Veeam Fast SCP (below) -
Fix:
From the folder /var/log/hp-snmp-agents, use the rm command to delete cma.log
[root@esx01 hp-snmp-agents]# rm cma.log
rm: remove regular file `cma.log'? y
(or delete via Veeam Fast SCP)
Then change to the /etc/init.d directory, and run the command ‘service hp-snmp-agents restart’ to restart the HP Management Agents (the cma.log file is invisible, and remains in /var/log until this service is restarted.) This command will not affect live guests in any way.
[root@esx01 hp-snmp-agents]# cd /etc/init.d
[root@esx01 init.d]# service hp-snmp-agents restart
Shutting down NIC Agent Daemon (cmanicd): [ OK ]
Shutting down Storage Event Logger (cmaeventd): [ OK ]
Shutting down FCA agent (cmafcad): [FAILED]
Shutting down SAS agent (cmasasd): [ OK ]
Shutting down IDA agent (cmaidad): [ OK ]
Shutting down IDE agent (cmaided): [ OK ]
Shutting down SCSI agent (cmascsid): [ OK ]
Shutting down Health agent (cmahealthd): [ OK ]
Shutting down Standard Equipment agent (cmastdeqd): [ OK ]
Shutting down Host agent (cmahostd): [ OK ]
Shutting down Threshold agent (cmathreshd): [ OK ]
Shutting down RIB agent (cmasm2d): [ OK ]
Shutting down Rack Infrastructure Info Srv (cpqriisd): [FAILED]
Shutting down Rack agent (cmarackd): [FAILED]
Shutting down Performance agent (cmaperfd): [ OK ]
Shutting down SNMP Peer (cmapeerd): [ OK ]
Starting Health agent (cmahealthd): [ OK ]
Starting Standard Equipment agent (cmastdeqd): [ OK ]
Starting Host agent (cmahostd): [ OK ]
Starting Threshold agent (cmathreshd): [ OK ]
Starting hp-ilo:
Already started hpilo module. [ OK ]
Starting RIB agent (cmasm2d): [ OK ]
Starting hp-ilo:
Already started hpilo module. [ OK ]
Starting Rack Infrastructure Info Srv (cpqriisd): [ OK ]
Starting Rack agent (cmarackd): [ OK ]
Starting Performance agent (cmaperfd): [ OK ]
Starting SNMP Peer (cmapeerd): [ OK ]
Starting Storage Event Logger (cmaeventd): [ OK ]
Starting FCA agent (cmafcad): [ OK ]
Starting SAS agent (cmasasd): [ OK ]
Starting IDA agent (cmaidad): [ OK ]
Starting IDE agent (cmaided): [ OK ]
Starting SCSI agent (cmascsid): [ OK ]
Starting NIC Agent Daemon (cmanicd): [ OK ]
Finally, re-run the df –h to check space on /var/log has been freed up
[root@esx01 init.d]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 4.9G 1.9G 2.8G 41% /
/dev/sda2 2.0G 318M 1.6G 17% /var/log
/dev/cciss/c0d0p1 198M 68M 121M 36% /boot
Instead of removing the cma.log and restarting the HP management daemons afterwards, you can also do:
ReplyDeleteecho >/var/log/hp-snmp-agents/cma.log
This will replace the current open file with an empty one, freeing the space without restarting the daemons.