Note: The version of OnCommand Balance here is 4.1
A VMware vCenter Server 4.0 had been added to Balance (Discovery > vCenter Servers > + Add vCenter Server). Sometime later, a new vCenter Server 5.1 is built (inside the vCenter Server 4.0), and then the old vCenter Server 4.0 is decommissioned and the new vCenter Server 5.1 acquires all the hosts and guests. Now, quite understandably, Balance is a bit confused. So we delete the old vCenter 4.0, and try and add in the new vCenter 5.1, but we get this error:
“A vCenter Server is already being monitored with the same URI.”
Well the vCenter 5.1 was already discovered due to being a part of the old vCenter 4.0.
We refresh the - Discovery > vCenter Servers - page, and notice another error:
“An error occurred attempting to retrieve the data”
Now, thinking a reboot might sort it out, we reboot the Balance appliance, but the web interface does not come up!
Connecting to the Balance Advanced Management Tool (AMT) over SSH (e.g. PuTTY to IP_OF_BALANCE_APPLIANCE port 22) or via the console, we are presented with:
Current Status: Not running
WARNING: the product is running in MAINTENANCE MODE
WARNING: The product is NOT RUNNING.
WARNING: The GUI is NOT AVAILABLE.
WARNING: New data will NOT BE COLLECTED.
WARNING: Contact Customer Support for assistance.
Note i: Balance AMT default login username = netapp and password = netapp
Note ii: (For completeness) The Web UI default login username = admin and password = password
Note iii: Please note the warning to ‘Contact Customer Support for assistance.’
Subsequent reboots, same thing.
In case you’re wondering - yes, maintenance mode is not selected when the appliance boots!
Image: Balance boot without maintenance mode
Further troubleshooting via the main menu option 5) System Configuration:
1 ) Update Balance
2 ) Hypervisor Tools
3 ) IP Management Options
4 ) Balance Backup Options
5 ) System Configuration
x ) Exit
Please enter your choice: 5
And option 1 for Display Balance Status:
1 ) Display Balance Status
2 ) Reboot Operating System
3 ) Shutdown Operating System
4 ) Change Management Tool Password
5 ) Change 'selfmonitor' Password
b ) Back
x ) Exit
Please enter your choice: 1
Reveals another error message:
ERROR: Balance Monitor is in Maintenance Mode.
use “/etc/init.d/bpm clearmaint” to clear
Now, at this stage you would be forgiven for thinking that just clearing Maintenance Mode will fix the problem in this scenario, it doesn’t quite!
Apologies ladies and gentlemen - it would not be appropriate for me to publish the fix that worked in this case on a public blog, since it involves going into a hidden menu from the Balance AMT Main Menu. Nevertheless, I can outline what the fix involved. Best answer though is to contact customer support :-)
The fix involves:
1) Via the Balance AMT:
Disable Data Collection
Clear Maintenance Mode
2) After the reboot, the Web UI should now come up and you can delete the new vCenter which had been incorrectly discovered:
- Click Discovery > Servers
- Click ‘Unmonitored Servers’
- Click ‘Edit vCenter settings’
- Click Delete
3) Then - via the Balance AMT - re-enable Data Collection and reboot the Balance VA.
4) Finally, we need to rediscover the vCenter:
- Click Discovery > vCenter Servers
- Click ‘+ Add vCenter Server’
And wait for it to do its discovery!
Note i: The re-discovery might take a long time.
Note ii: If not much configuration has been done on the Balance VA, a re-deploy is a quicker fix.