FRAMINGHAM, Mass. — Cisco Systems Inc.’s FabricPath data center Ethernet technology is designed to combine traditional, Spanning Tree-based Ethernet with a next-generation architecture that uses a link-state protocol to allow for multiple active paths. Deploying multiple active paths in a data center network is required to flatten the infrastructure to reduce latency and better support traffic flow between server racks.
The following attempts to boil down the key aspects of FabricPath on a Cisco Nexus 7000 core data center switch, as described in a Cisco whitepaper. Several features of FabricPath are vital to its operation, including:
• replacing Spanning Tree with a link-state protocol;
• interacting with Spanning Tree domains;
• use of virtual PortChannel (vPC) to activate parallel paths and remove Spanning Tree blocks;
• defining processes as virtual device contexts (VDC);
• conversational MAC address learning and use of switch IDs (SID), and;
• VLAN designations.
Use of a link-state protocol
Using or replacing Spanning Tree with a link-state protocol is intended to overcome the limitations of Spanning Tree in data center and cloud environments – chiefly, the inability to use multiple active parallel paths in an Ethernet network. Spanning Tree allows only one path to be active between any two nodes and blocks the rest, which is unsuitable for low latency, Ethernet-based fabrics in data centers and cloud environments.
Virtually every vendor addressing the data center fabric switching market proposes augmenting or replacing Spanning Tree with a link-state protocol, and standards like TRILL and Shortest Path Bridging are being defined to do just that. Brocade Communications Systems uses TRILL in the data plane of its VCS architecture but the control plane is based on Fabric Shortest Path First, an ANSI standard used by all Fibre Channel SAN fabrics as the link-state routing protocol.
Cisco pitches FabricPath as a “superset” of TRILL. FabricPath also uses a Shortest Path First (SPF) routing protocol to determine reachability and path selection in the FabricPath domain. Cisco says its SPF implementation is an industry standard IS-IS routing protocol with FabricPath-specific extensions, like exchanging switch ID (SID) reachability instead of IP prefixes. FabricPath also employs equal-cost multipath (ECMP) forwarding to make use of all available bandwidth.
Arista Networks also uses ECMP for Layer 3 fabrics, and Multichassis Link Aggregation for Layer 2 deployments.
Spanning Tree interaction
Even though it replaces Spanning Tree with a link-state protocol, FabricPath can also support and interact with Spanning Tree domains. FabricPath edge switches transmit and process Spanning Tree Bridge Protocol Data Units (BPDU) and participate in building the Spanning Tree forwarding topology in each connected Spanning Tree domain.
But BPDUs and Spanning Tree topology change notifications are not transmitted on FabricPath core ports, and no BPDUs are forwarded or tunneled through the FabricPath domain by default. FabricPath isolates each Spanning Tree domain, and topology changes in one Spanning Tree domain are not propagated to other domains connected to the same FabricPath fabric.
The entire FabricPath domain appears as a single Spanning Tree bridge to any connected Spanning Tree domain because all FabricPath switches share a common FabricPath ID. Each FabricPath edge switch is configured as the root for all Spanning Tree devices connecting to the fabric.
Virtual PortChannel
The foundation of FabricPath’s multiple active link capability is Cisco’s virtual PortChannel (vPC) technique. The technology lets a single Ethernet device connect simultaneously to two discrete Nexus 7000 switches while treating these parallel connections as a single logical PortChannel interface.
The result is active-active forwarding paths and the removal of Spanning Tree blocked links, relegating Spanning Tree to a fail-safe role to prevent creation of a network loop.
But vPC has its own limitations: although vPC provides active-active forwarding, only two active parallel paths are possible, according to Cisco [Nasdaq: CSCO]. No provision is made for the addition of a third or fourth aggregation-layer switch to increase density or bandwidth. And vPC offers no means by which VLANs can be extended, which Cisco says is also a limitation of traditional Spanning Tree designs.
Virtual Device Contexts
Every FabricPath control-plane protocol and functional block runs in its own protected memory space as individual processes, for stability and fault isolation purposes. Some processes run on the supervisor engine while others run on the individual I/O modules.
Many supervisor engine processes run in protected memory as independent virtual device contexts (VDC). Example VDCs include the SPF protocol; the Dynamic Resource Allocation Protocol for SIDs and forwarding tag values; the IGMP protocol for building multicast forwarding databases; and routing information bases for unicast and multicast Layer 2 forwarding.
I/O modules house MAC and SID tables, unicast and multicast forwarding tables, and MAC address table management. Interfaces to FabricPath can be “Classic Ethernet,” Cisco says. But interfaces within FabricPath – core ports – always encapsulate Ethernet frames in a 16-byte FabricPath header and forward based on SID table lookups, not MAC address learning. FabricPath core switches forward frames exclusively on the outer destination address of the FabricPath header.
Conversational MAC learning
Edge switches in Fabric Path do learn MAC addresses but in a way that Cisco says conserves MAC address table space. FabricPath’s conversational learning technique defines two types of MAC addresses: local and remote. Local addresses are for a device directly connected to the FabricPath edge switch. Remote addresses are for those devices connected to a different FabricPath switch.
The FabricPath edge switch learns the source MAC address as a local MAC address entry for Ethernet frames received from a directly connected access or trunk port. This behavior is consistent with a Spanning Tree switch, Cisco says.
For unicast frames received with FabricPath encapsulation, the switch learns the source MAC address of the frame as a remote MAC address entry only if the destination MAC address matches an already learned local MAC address entry — the switch learns remote MAC addresses only if the remote device is having a bidirectional conversation with a locally connected device. Unknown unicast frames being flooded in the FabricPath network do not necessarily trigger learning on edge switches, nor do broadcast frames. But broadcast frames such as ARP messages are used to update any existing source MAC address entries already in the table.
Multicast frames also trigger learning on edge switches, since several LAN protocols rely on source MAC address learning from multicast frames for proper forwarding.
VLAN designations and operation
VLANs can also be configured in Classic Ethernet and FabricPath mode, and conversational learning occurs automatically in VLANs configured for FabricPath mode. When a frame is carried across a FabricPath core port, the frame includes an IEEE 802.1Q tag with the original VLAN ID of the frame. So FabricPath core ports behave like 802.1Q trunks, and the VLAN ID is still used in the FabricPath core to control forwarding and flooding behavior, similar to Classic Ethernet networks, according to Cisco.
Edge ports that bridge FabricPath and Ethernet domains are always FabricPath interfaces belonging to FabricPath VLANs. Cisco FabricPath switches will have the same VLAN IDs as the Classic Ethernet switch connected to it, but with the VLAN mode configured as FabricPath — VLANs configured for Classic Ethernet mode cannot be carried over a FabricPath core port.
As easy as Spanning Tree?
Although IS-IS forms the basis of FabricPath, users need not be IS-IS experts, Cisco notes. Users can enable FabricPath interfaces and begin forwarding FabricPath encapsulated frames in much the same way they can activate Spanning Tree and interconnect switches, Cisco ensures.