Egenera Blog

As Ethernet speeds reached Fibre Channel’s 1Gb/s and IP became a defacto global protocol for small and large enterprises alike, it became clear that wrapping SCSI commands in TCP/IP packets may be the next standard for block transfer over networks. Even more exciting was that iSCSI would run on everyone’s favorite data center network, Ethernet. So with that promise, iSCSI became a full fledged IETF standard in 2002.

iSCSI has surely gained steady momentum and market share, but not at a pace in line with earlier expectations. Certainly, it has not taken foothold in the large enterprise SAN market which has continued to deploy Fibre Channel. Of course, it did not help that Fibre channel speeds also got upgraded to 2 and then 4 Gb/s while Ethernet remained at 1 Gb/s.

iSCSI also saw several technical challenges running over Ethernet. Block transfer protocols are highly susceptible to latency and loss. Ethernet networks being a best effort datagram service do not have delivery guarantees. Ethernet switches with rather shallow buffers have a nasty habit of dropping packet when there is congestion. Worse yet, the iSCSI storage array becomes the congestion point in a switched Ethernet network since all clients (or iSCSI initiators) want to simultaneously access it.

Another issue is that TCP/IP’s congestion management is really optimized for long haul networks (with large round trip delays). If you want high performance iSCSI on a LAN, initiator and target TCP stacks must be modified and tuned for operation on lower bandwidth-delay product networks. Finally, high performance iSCSI really wants to use Jumbo Ethernet packets with larger I/O sizes (like 64KB) to reduce the per-packet processing overhead on end-nodes. Well, running Jumbo packets on Ethernet switches with only a couple of MB of packet memory can result in, yep you guessed it, more drops if the network is not purposely designed.

Enter 10 Gb/s Ethernet and a new generation of Ethernet NICs with Transport Offload Engines (TOE) and iSCSI protocol offload in HW. This gave iSCSI the long awaited boost it needed to hit the main street. A multitude of start-up companies now offer compelling iSCSI 10G NICs with TCP and protocol acceleration capabilities. Also carrier class features like Weighted Random Early Discard (WRED) and traffic shaping have made their way into LAN switches that allow high performance TCP on LANs.

The recent Data Center Ethernet (DCE) congestion management initiative proposed by IEEE8021au/az has also opened doors for FC vendors to bring their beloved Fibre Channel packets and run it over 10 Gb/s Ethernet networks. Cisco has been leading the effort for Fibre Channel over Ethernet standard (FCoE) and already has products in the market. All these new DCE features also help iSCSI performance over Ethernet just as well but I do not hear anyone talking about that!

Ethernet has proven time and time again that it is the Borg that assimilates all other competing LAN transports. So, what does the FC community do? If you cannot beat it, embrace it. FC switch vendors like Brocade can offer new switches that have FC and FCoE Ethernet ports. FC HBA vendors like Qlogic and Emulex now get to play in the perceived to be enormous 10 Gb/s Ethernet NIC market. Furthermore, they will claim that their “trusted” drivers and Fibre channel “secret sauce” on THEIR Ethernet NICs will be better than traditional Ethernet NIC vendors’ FCoE offering since they have no FC experience. Brilliant!

Just when you might think that things are finally start looking brighter for high performance iSCSI as the storage protocol of choice over 10 Gb/s Ethernet LAN, it is again faced with stiff competition. The good old Fibre Channel is back in new clothes playing in iSCSI’s backyard. Perhaps the most compelling positive that FCoE has going it for it is that the standard provisioning and management tools can manage FCoE infrastructure like they do FC SANs. iSCSI has going for it a year or so of lead to prove itself on 10 GigE before NIC vendors offer FCoE.

With 2 Ethernet based SAN protocols, I would be willing wager a bet that enterprise SAN is slowly but surely moving to Ethernet. Who wants to bet?

Tags: 10 Gigabit Ethernet, Converged I/O, Data Center Ethernet, Egenera FCoE, iSCSi