From Proprietary to Open-Standards Fabric

When Pluribus Netvisor was introduced to AMS-IX, it was a revolutionary solution offering an unconventional approach for managing VXLAN fabrics, by distributing the function of the controller to every node of the fabric. This meant that the whole fabric could be monitored and managed from any node, without the need for any external controller/orchestrator. For example, one could simply check from which switch and port a specific IP or MAC address was learned by entering a command like “port-show ip 2001:db8::1” or “port-show mac 02:11:22:33:44:55”.

Similarly, service provisioning operations were simplified too. Even though VXLAN tunnels were static, the provisioning complexity was completely under the hood. For example, to extend tunnels for a specific VXLAN to a new leaf from all the leaves already participating in the VXLAN, a simple command like “vtep-vxlan-add name leaf-24 vxlan 100” could be used.

However, as it is often the case, this convenience came at a cost which is discussed below.

Considering that every Pluribus Netvisor node is a hard-linked part of the fabric, every node must run the same NOS version. This requirement complicates upgrades significantly. Most importantly, this requires upgrading and rebooting the whole fabric during upgrade maintenances as doing it node by node would result in two or more isolated fabric instances, not compatible with each other.

Because many things can go wrong when the whole network is rebooted into a new major version, a lot of efforts must be put in validating the upgrade procedure and the switches’ readiness for the upgrade. For instance, together with the Pluribus Networks engineers, it took several months for us to prepare our switches for a major version upgrade that took place in 2021.

Namely, we had to RMA several switches due to degraded SSDs metrics. We also had to upgrade the SSD firmware from the ONIE environment on all the switches and physically power-cycle them for the changes to take effect. Similarly, some switches needed physical power-cycling due to the signs of an unstable I2C bus. In addition, we had to re-install the NOS on several switches because some drives were partitioned differently during the initial setup. All that was needed to eliminate potential risks of losing nodes during the major upgrade.

Adding new management nodes was getting increasingly difficult over the years. To join a new node to the fabric means that a node must replay all the transactions that the fabric has seen in its lifetime. For example, if you had to add a new switch, it had to create and delete fabric-wide objects such as VXLANs and respective tunnels that were in use, even temporarily, since the creation of the fabric. Repeating around 2k transactions (as of 2022) was extremely unreliable, and newly added nodes were crashing before they could successfully replay all fabric transactions, making further expansion impractical.

It was already mentioned that the distributed fabric functionality allowed NOC engineers to run commands in the fabric-wide context, and it was extremely useful for read/get operations. Now consider the impact if an operator forgot to switch into a local context before disabling a port or VLAN on a specific switch. Yes, it happened at least once: the same port was disabled on all the nodes at the same time, and this did cause an internal incident. Also, the fact that the CLI was in fabric-wide context by default did not help at all.

Because the fabric configuration must be consistent among all the nodes, fabric-wide write operations to the configuration database are blocked when there are offline nodes. For example, if the node C is temporarily offline, Netvisor will not allow adding new VXLAN tunnels between the nodes A and B even if there are no intentions to provision any new tunnels on the node C. Therefore, any node added to the fabric was considered an important member of the fabric irrespectively of the importance of the node’s role in general.

MLAGs offer a way to elegantly multihome end devices to multiple leaves. Traditionally, vendors have had their own proprietary implementations, but the idea is that from the end devices’ perspective, they have a normal LACP LAG of two or more ports that are connected to different switches.

In most cases, switches offering multihoming services to edge devices must synchronise states with each other, so a dedicated cluster link/LAG is normally used for that. That is not an exception for Pluribus Netvisor. Not only do such links involve extra ports and cables, but they also become critical points of the whole fabric. If the cluster link is down for some reason, the switches enter a “split-brain” situation when they assume they are alone, so they start forwarding traffic that they might not be supposed to, introducing loops and storms.

On the other hand, the EVPN standard was designed with multihoming in mind, so MLAGs (or Ethernet Segments (ES) in EVPN parlance) are supported natively and without the need for dedicated cluster links as the ES states are synchronised with the EVPN signalling naturally. That was one of the major reasons to define EVPN as a hard requirement for the next generation of the fabric.

There is little to note here, except that because the fabric was managed by a proprietary distributed controller, it could not be extended with other NOSes. It is fair to mention that starting from Netvisor version 6.1.0, EVPN support was added, but the version 6 was not supported by our hardware.