Friday, 10 December 2010

Nexenta Community 3.0.3 / 3.0.4 Web UI Stops Working!

Scenario:

Running five Nexenta (Community Version 3.03 and 3.04) storage boxes (installed on reclaimed HP DL380 G4's,) and on all five the Web UI has stopped working.

Below follows a fix to get the Web UI working again, additional commands, and two examples


Fix:

Part 1/2

Use putty or similar to SSH to the Nexenta box and use the root login

From the UNIX shell (#) run these commands:

1 root@nexentabox:/volumes# svcadm enable -rs apache2
2 root@nexentabox:/volumes# svcadm restart nmv
3 root@nexentabox:/volumes# svcadm restart nms
4 root@nexentabox:/volumes#

Note 1: Command from line 1 only needs to be run one time
Note 2: These commands are perfectly safe to run during the working day

If in the NMC shell ($), to get to the UNIX shell (#) run these commands:

nmc@nexentabox:/$ option expert_mode=1 -s
nmc@nexentabox:/$ !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command(s). Warning: using low-level Unix commands is not recommended! Execute? Yes
root@nexentabox:/volumes#


Part 2/2



A promising fix (courtesy of my colleague Alfredo) to stop the Web UI failing in the future (or at least reduce the rate of it happening) is to change the 'Seconds between Retrieves' time on the Status → General → General Status and Details pane to 100 or more (default is 5)


Note 3: The HP DL380 G4s used are not on the HCL for OpenSolaris - http://www.sun.com/bigadmin/hcl/data/os/ - apart from problems with the Web UI, Nexenta runs better than the old Openfiler installs it has replaced, with a much greater feature set


Some other commands used in Nexenta

UNIX shell (#)

Reboot # shutdown -y -i6 -g0
Reboot (older command that still works)  # sync; sync; init 6
Shutdown # shutdown -y -i5 -g0
Shutdown (older command that still works) # sync; sync; init 5

NMC shell ($)

$ setup appliance upgrade nms (to upgrade Web UI)
$ setup appliance upgrade (to upgrade base OS s/ware in Nexenta Community Edition)
$ setup appliance init (re-run through the network setup)


Example 1 where SSH enters into the NMC shell

login as: root
Using keyboard-interactive authentication.
Password:
Last login: Thu Dec 9 07:00:13 2010 from 172.23.123.234
*****************************************************************
* Management Console *
* Version 3.0.3-4 *
* *
* press TAB-TAB to list and complete available options *
* *
* type help for help *
* exit to exit local NMC, remote NMC, or group mode *
* q[uit] or Ctrl-C exit NMC dialogs *
* q[uit] or Ctrl-C exit NMC text viewer *
* *
* option -h help on NMC options *
* -h help on any command *
* ? brief summary *
* help keyword [-q] locate NMC commands *
* help -k [-q] same as above *
* setup usage combined 'setup' man pages *
* show usage combined 'show' man pages *
* *
* type help and press TAB-TAB *
* *
* Management GUI: https://10.11.12.13:2000/ *
* *
*****************************************************************
nmc@flake:/$ option expert_mode=1 -s

nmc@flake:/$ !bash
You are about to enter the Unix ("raw") shell and execute low-level Unix command (s). Warning: using low-level Unix commands is not recommended! Execute? Yes

root@flake:/volumes# svcadm enable -rs apache2
root@flake:/volumes# svcadm restart nmv
root@flake:/volumes# svcadm restart nms
root@flake:/volumes#


Example 2: where SSH connection appears to be unresponsive after logging in - wait a few minutes and the SYSTEM NOTICE “Failed to initialize NMC” pops up and the prompt enters the UNIX shell

login as: root
Using keyboard-interactive authentication.
Password:
Last login: Fri Dec 10 08:30:42 2010 from 172.23.117.160

* * *
SYSTEM NOTICE

Failed to initialize NMC:
no introspection data available for method 'get_props' in object '/Root/App liance', and object is not cast to any interface

Suggested possible recovery actions:
- Reboot into a known working system checkpoint
- Run 'svcadm clear nms'; then try to re-login
Suggested troubleshooting actions:
- Run 'svcs -vx' and collect output for further analysis
- Run 'dmesg' and look for error messages
- View "/var/log/nms.log" for error messages
- View "/var/svc/log/application-nms:default.log" for error messages

Entering UNIX shell. Type 'exit' to go back to NMC login...
root@ripple:~# svcadm enable -rs apache2
root@ripple:~# svcadm restart nmv
root@ripple:~# svcadm restart nms
root@ripple:~#

5 comments:

  1. or
    export_mode=!
    /etc/init.d/apache2 restart

    updating to:
    appers to have fixed it:

    NMS Version 3.0.4 (r8917)
    NMC Version 3.0.4 (r8917)
    NMV Version 3.0.4 (r8960)
    OS Version 3.0.4

    ReplyDelete
  2. The use of "sync;sync" prior to reboot is no longer necessary (and hasn't been for the past 10+ years). The init command will do the right thing.

    ReplyDelete
  3. I have 2 e1000g devices with both set in the /kernel/drv/e1000g.conf file for a max MTU of 3 or 1500-16298

    1 NIC plays nice and it allows the larger MTU. The other NIC, no matter what i do will not go above 1500 MTU:

    Any ideas, Thank you!

    root@keeper:/volumes# dladm show-linkprop -p mtu e1000g0
    LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
    e1000g0 mtu rw 9000 1500 1500-16298
    root@keeper:/volumes# dladm show-linkprop -p mtu e1000g1
    LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
    e1000g1 mtu rw 1500 1500 1500
    root@keeper:/volumes#

    ReplyDelete
  4. Hi matthew@vocys.com

    Alas, I do not have an answer to your question. If you have been through the Nexenta forum and found no answer, there are many alternative free iSCSI products out there that might work - Openfiler, OSNexus, OPen-E, ....

    Cheers

    ReplyDelete
  5. With 3 e1000g interfaces I am still experiencing this problem on 2 out of the 3 interfaces. My add-on PCI-E Intel card displays the correct MTU range of 1500-16298.

    The 2 built-in Intel interfaces show only MTU of 1500.

    my /kernel/drv/e1000g.conf and /var/lib/dpkg/alien/sunwintgige/reloc/kernel/drv/e1000g.conf (If I do not set this one too, the above conf file gets overwritten) are set to: MaxFrameSize=3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3;

    And root@keeper:/kernel/drv# dladm show-link LINK CLASS MTU STATE BRIDGE OVER e1000g2 phys 1500 up -- -- e1000g0 phys 9000 up -- -- e1000g1 phys 1500 up -- --

    How do I get the e1000g driver to allow max MTU on all 3 e1000g interfaces?

    Thank you, Matthew

    ReplyDelete