4. Allowed Traffic Types and Configurations
The Technical Specifications state the following:
There are only three ethertypes allowed:
0x0800 - IPv4
0x0806 - ARP
0x86dd - IPv6
This implies IEEE 802.3 compliance, not 802.2, so no LLC encapsulation!
Only one MAC address allowed on a port, i.e. all frames sent towards the AMS-IX should have exactly one unique MAC address.
The only non-unicast traffic allowed is:
Broadcast ARP.
Multicast ICMPv6 Neighbour Discovery (ND) packets. (NOTE: this does not include Router Advertisement (ND-RA) packets!)
AMS-IX member equipment should only reply to ARP queries for IP addresses of their directly connected AMS-IX interface. In other words, proxy ARP is not allowed.
Traffic for link-local protocols is not allowed, except for ARP and IPv6 ND (see above).
IP packets addressed to AMS-IX peering LAN's directed broadcast address shall not be automatically forwarded to AMS-IX ports.
The speed and duplex setting of 10baseT and 100baseTX ports must be statically configured, i.e. auto-negotiation should be disabled.
4.1. Physical L2 Topology
The AMS-IX rules dictate that only one MAC address is allowed behind a port. This means that you have to be extremely careful when connecting a device that can act as a L2 device. In general, we do not recommend using L2 devices between a member's router and the AMS-IX switch, except when used as a media converter.
The reason for allowing only one MAC address that we want no additional L2 network behind the AMS-IX ports. Extended L2 networks are not under the control of the AMS-IX, but instabilities in a L2 network behind the AMS-IX switches can and typically do have a negative impact on the whole exchange. Forwarding loops and spanning tree topology changes are good examples of this. By enforcing the one-MAC-address-per-port rule, we effectively prevent forwarding loops and STP traffic from intermediate L2 devices.
In short, an intermediate L2 device may only bridge frames from the member's router to the AMS-IX port (so we see only one MAC address) and should otherwise be completely invisible. No connected device should bridge frames from other devices onto the AMS-IX, or talk STP on its AMS-IX interface.
4.1.1. Connecting a L3 Device
The most preferred way of connecting to the AMS-IX is directly through a L3 device (router), see the diagram below.
This is your best chance of not leaking MAC addresses or STP traffic and it greatly increases the stability of the network.
4.1.2. Connecting Through a L2 Device
We neither recommend nor encourage connecting your router through a L2 device, but if you do so, keep the following in mind:
You must make absolutely sure that only traffic to/from your L3 router's interface goes to/from the AMS-IX port.
You must disable spanning tree on your link to AMS-IX.
 | On all intermediate L2 devices, consider using explicitly defined port-based VLANs for production ports. It forces you to understand your topology and reduces the chances of a nasty surprise further down the road. In particular, we strongly recommend using a dedicated VLAN for the path from your router to the AMS-IX. |
4.1.3. Connecting a L2/L3 Hybrid
The L2/L3 hybrid switch/router requires careful configuration in order to prevent unwanted traffic from leaking onto the exchange. As with intermediate L2 devices, you need to keep the following in mind:
You must make absolutely sure that your AMS-IX port is configured as a “router only” port.
You must disable Spanning Tree on your link to AMS-IX.
| On a L2/L3 hybrid device, it is a good idea to put the AMS-IX connected interface (untagged) in a separate (non-default) port-based VLAN without spanning tree and with no other ports in it. This is the best way to ensure that no traffic from other ports will be bridged onto the AMS-IX port. |
4.2. Commonly Seen Illegal Traffic and Setup
Any traffic other than the types mentioned in the previous section is deemed to be illegal traffic. In this section we will list some of the more common types of violations we see at the AMS-IX and give some arguments as to why it is considered unwanted.
4.2.1. Multiple MAC addresses
Since the AMS-IX operates on the principle of one router per port, there should be one MAC address visible behind each port. Some members connect through intermediate switches, or use a L2/L3 hybrid device. If these devices are not configured properly, they can cause forwarding loops, STP instabilites, and lots of unwanted traffic on the exchange. There is no excuse for these devices to leak traffic, and there is no necessity to talk STP on the link to the AMS-IX. Hence, by enforcing the one-MAC-address rule, we also enforce these issues. Beware that this rule is enforced automatically, so if you leak traffic from another MAC address, your legitimate traffic may be blocked (depending on which MAC address the switch sees first) or your port may be shut down for a few minutes.
4.2.2. Spanning Tree (STP)
This point is closely related to the previous point. The device(s) connected to the AMS-IX port are not allowed to be visible as L2 bridges. This means that they should not speak STP (spanning tree) or any other (proprietary) L2 specific protocol.
4.2.3. Routing protocols: EIGRP, OSPF, RIP, IS-IS
The only routing protocol allowed on the AMS-IX is BGP. There is no valid reason for interior routing protocols to appear on the shared medium. These protocols only cause unnecessary multicast and broadcast traffic.
4.2.4. (Cisco) Keepalive
By default Cisco routers and switches periodically test their (Fast) Ethernet links by sending out Loopback frames (ethertype 0x9000) addressed to themselves. Call it a “L2 self-ping” if you will. In a switched environment it can be used to test the functionality of the switch and/or keep the router's MAC address in the switch's address table.
In the AMS-IX environment, this is not useful since we use MAC timeouts that are larger than the typical BGP and/or ARP timeouts. In fact, the keepalives a may actually cause port security violations if they are being sent by an intermediate switch.
4.2.5. Discovery protocols: CDP, EDP
Various vendors (e.g. Extreme, Cisco) tend to ship their boxes as gregarious devices: by default they announce their existence out of all their interfaces and try to find family members. CDP (Cisco) and EDP (Extreme) are examples of this, but there are others.
The only reason for running discovery protocols is to support certain types of autoconfiguration. Autoconfiguration on an Internet Exchange is a very bad idea. Hence, there is absolutely no reason to run discovery protocols on your AMS-IX interface. Discovery protocols typically cause unwanted broadcast or multicast traffic.
4.2.6. Non-unicast IPv4: IGMP, DHCP, TFTP
On the ISP peering LAN, the only non-unicast traffic that is allowed is the ARP query.
Sometimes we see equipment trying to get a configuration through broadcast TFTP, or configure themselves through DHCP. We will leave it to the reader to consider why this is a bad idea.
Other equipment has IGMP turned on by default (or by accident). The Peering LAN is for unicast IP traffic only, so there is no point in configuring multicast on the AMS-IX interface.
4.2.7. Proxy ARP
Since traffic over the AMS-IX is exchanged based on BGP routes, there is no reason to answer ARP queries for any other IP address(es) than those that are configured on your AMS-IX interface.
Unfortunately, some vendors (e.g. Cisco) ship their products with proxy ARP enabled by default.
Proxy ARP is not only sloppy, it can lead to unwanted traffic on your network. Consider that if you have it enabled at the AMS-IX, it's likely to be enabled at other peering points, allowing parties on both sides to use you as a transit.
Proxy ARP is not allowed.