Posting here for ease of reference:
Source → Jitun replying to post @ forums.itrc.hp.com
Gigabit Ethernet support
Each storage node comes equipped with at least two copper Gigabit Ethernet ports (802.3ab). To take advantage of full-duplex 1GbE capabilities, the cabling infrastructure must be Cat5e or Cat6 to the storage nodes. Server connections and switch interconnects can also be done via fiber cabling instead of Cat5e or Cat6 cabling. For 10Gigabit implementations, Cat-6a or Cat-7 cabling is usually required for use with distances over 55 meters.
Fully subscribed non-blocking backplanes
In order to achieve maximum performance on the HP P4000 SAN, it is important to select a switch that has a fully subscribed backplane, which means that the backplane must be capable of supporting all ports in full-duplex mode. For instance, if the switch has 24 1 Gb ports, it will require a 48 Gb backplane to support full-duplex Gigabit communications.
Adequate per-port buffer cache
For optimal switch performance, HP recommends that the switch have at least 512 KB of buffer cache per port. For example, if the switch has 48 1 Gb ports, the recommendation is to have at least 24 MB of buffer cache dedicated to those ports. If the switch aggregates cache among a group of ports (for example, 1 MB of cache per 8 ports), space your storage modules and servers appropriately to avoid cache oversubscription.
Enable Flow Control on network switches and adapters
Flow Control ensures a receiver can make the sender pace its speed and is important in avoiding data loss. IP storage networks are unique in the amount of sustained bandwidth that is required to maintain adequate performance levels under heavy workloads. Gigabit Ethernet flow control (802.3x) technology should be enabled on the switch to eliminate receive and/or transmit buffer cache pressure. The storage nodes should also be set to have flow control enabled. Note: Some switch manufacturers do not recommend configuring flow control when using jumbo frames, or vice versa. For optimal performance, HP recommends implementing flow control over jumbo frames. Flow control is required when using DSM/MPIO.
Ensure spanning tree algorithm for detecting loops is turned off
Loop detection introduces a delay in making a port become usable for data transfer and may lead to application timeouts. If supported by the switch infrastructure, HP recommends implementing Rapid Spanning Tree for faster Spanning Tree convergence. If configurable on the switch, consider disabling spanning tree on the storage node and server switch ports so that they do not participate in the Spanning Tree convergence protocol timing. Enable Rapid Spanning Tree protocol convergence or PortFast (Cisco) on the storage node and server switch ports.
HP recommends implementing a separate subnet or VLAN for the IP storage network. If you are implementing VLAN technology within the switch infrastructure, you will typically need to enable VLAN tagging (802.1q) and/or VLAN trunking (802.1q) or Cisco Inter-Switch Link (ISL)).
Basic IP routing
The storage nodes can access external services such as DNS, SMTP, SNMP, Syslog, and so on. In order to allow this traffic outside the IP storage network, an IP route is required from the IP storage network to the LAN environment. Also, if the storage nodes are to be managed from a remote network, an IP route must exist to the storage nodes from the management station. Finally, if remote copy functionality is going to be used, the remote copy traffic must be routable between the primary/remote sites.
Disable unicast storm control on iSCSI ports
Most switches have unicast storm control disabled by default. If your switch has this enabled, you should disable this on the ports connected to iSCSI hosts and targets to avoid packet loss.
Segregate SAN and LAN traffic
iSCSI SAN interfaces should be separated from other corporate network traffic (LAN). Servers should use dedicated NICs for SAN traffic. Deploying iSCSI disks on a separate network helps to minimize network congestion and latency. Additionally, iSCSI volumes are more secure when SAN and LAN traffic can be separated using port based VLANs or physically separate networks (different physical switches.)
Configure additional Paths for High Availability; use either Microsoft MPIO or MCS (multiple connections per session) with additional NICs in the server to create additional connections to the iSCSI storage array through redundant Ethernet switch fabrics.
Unbind File and Print Sharing from the iSCSI NIC
on the NICs which connect only to the iSCSI SAN, unbind File and Print Sharing.
Use Gigabit Ethernet connections for high speed access to storage
Congested or lower speed networks can cause latency issues that disrupt access to iSCSI storage and applications running on iSCSI devices. In many cases, a properly designed IP-SAN can deliver better performance than internal disk drives. iSCSI is suitable for WAN and lower speed implementations including replication where latency and bandwidth are not a concern.
Use Server class NICs
It is recommended to use NICs which are designed for enterprise networking and storage applications.
Use Jumbo Frames if supported in your network infrastructure
Jumbo Frames can be used to allow more data to be transferred with each Ethernet transaction and reduce the number of frames. This larger frame size reduces the overhead on both your servers and iSCSI targets. For end to end support, each device in the network needs to support Jumbo frames including the NIC and Ethernet switches.
High network latency can be the primary cause of slow I/O performance, or worse, iSCSI drive disconnects
It is important to keep network latency on your HP P4500 Multi-Site SAN subnet below two milliseconds. Many factors can contribute to increasing network latency, but two are most common:
i: Distance between storage cluster modules
ii: Router hops between storage cluster modules
Configuring the HP P4500 Multi-Site SAN on a single IP subnet with Layer 2 switching will help to lower the network latency between storage cluster modules.