5. Cisco Configuration Hints
Cisco's philosophy seems to be similar to that of some PC OS vendors: enable as many protocols and features as possible by default, so the device works out-of-the-box in most situations. Unfortunately, this means that a lot of unnecessary features are turned on that, while harmless in LAN or corporate environments, can cause undesired traffic on an Internet exchange.
Typical things that need to be disabled are: autoconfiguration protocols (DHCP, BOOTP, TFTP config download over the AMS-IX interface), CDP, DEC MOP, IP redirects, IP directed broadcasts, proxy ARP, IPv6 Router Advertisements, keepalive.
Intermediate switches or hybrid devices will also need to disable VTP, STP, etc.
5.1. Global Config
! Do not run a DHCP server/relay agent no service dhcp ! Older IOS versions require this instead of the above. no ip bootp server ! Do not download configs through TFTP no service config ! Do not run CDP no cdp run |
5.2. Interface Config
! Don't do redirects -- if they don't know ! how to route properly, tough luck! no ip redirects ! Don't run proxy ARP on your AMS-IX interface no ip proxy-arp ! Don't run CDP on your AMS-IX interface no cdp enable ! Directed broadcasts are evil. no ip directed-broadcast ! Disable the DEC drek if you haven't done so globally yet. no mop enable ! For (Fast)Ethernet: no auto-negotiation on your connection. ! no negotiation auto ! duplex half duplex full ! L2 keepalives are useless on the AMS-IX no keepalive |
5.3. Layer 2 Config
It is difficult to give a complete guide for Cisco products, because of the many different types of devices and (IOS) software versions. When in doubt, consult your documentation.
5.3.1. 29xx and 35xx Series
If you use a Cisco Layer 2 device (such as the 2900 and 3500 series), you have to turn off VTP (VLAN Trunking Protocol), DTP (Dynamic Trunking Protocol), LLDP, and UDLD.
In global config mode:
vtp mode transparent ! no spanning-tree vlan 1200 ! If you don't need LLDP, disable globally no lldp run ! If you don't need CDP, disable globally no cdp run ! vlan 1200 name AMS-IX ! interface IfIdent description Interface to AMS-IX switchport access vlan 1200 switchport mode access switchport nonegotiate no keepalive speed nonegotiate no udld enable ! If CDP has not been disabled globally: no cdp enable ! If LLDP has not been disabled globally: no lldp receive no lldp transmit ! If you do not want to shut off STP: spanning-tree bpdufilter enable end |
5.3.2. Catalyst 6500 Series
CatOS and IOS are different beasts, so for Catalyst switches, the following applies:
set vtp mode off set port name IfIdent My AMS-IX Port set cdp disable IfIdent set udld disable IfIdent set trunk IfIdent off dot1q set spantree bpdu-filter IfIdent enable set vlan 1200 name My_AMS-IX_Vlan set vlan 1200 IfIdent |
If, for some reason, you cannot afford to turn off VTP globally, the only way to turn it off on individual ports seems to be by using l2pt:
set port l2protocol-tunnel IfIdent vtp enable |
Depending on your CatOS platform, you may or may not be able to do this.
5.4. Cisco Aggregated Links
5.4.1. Catalyst 6500 Series
Configure the port-channel as on, not negotiate or desirable as the AMS-IX switches do not have LACP enabled nor do they speak PAgP.
Some modules do not support more than 1 Gbps of traffic under certain conditions across an aggregated link. Please see the Cisco documentation for more details.
Load-balancing over four ports may result in an unequal distribution due to bug CSCsg80948.
! Here is an example configuration: interface GigabitEthernet1/1 description AMS-IX Link 1 no ip address no ip redirects no ip proxy-arp no keepalive no cdp enable channel-group 1 mode on ! interface GigabitEthernet1/2 description AMS-IX Link 2 no ip address no ip redirects no ip proxy-arp no keepalive no cdp enable channel-group 1 mode on ! interface Port-channel1 description AMS-IX aggregated link ip address 195.69.14x.y 255.255.254.0 no ip redirects no ip proxy-arp no keepalive ! |
5.4.2. GSR Series
Do not set a static MAC address on the Port-channel interface. This causes CEF inconsistencies and other assorted failures.
Link aggregation and IPv6 do not seem to play well together. Cisco advises against trying this.
Some changes will result in a different MAC address getting chosen for the aggregated link (likely such as reloading a linecard, if it contains the first port in the bundle). This will keep your ports dysfunctional due to port security on the AMS-IX switches and you will have to contact the AMS-IX NOC in such cases to fix this.
Some restrictions apply to what features are supported on link bundles (e.g. sampled NetFlow only on ISE/Engine4+; no uRPF). Also not all line cards support link bundling, and if traffic towards AMS-IX comes in on such an interface you will experience suboptimal load-balancing. Please see the Cisco documentation for more details.
Support for link bundling on Engine 5 linecards will come in 12.0(33)S.
Cisco Engineering have a special train called "Phase 3" (lb-eft-ph3) that is purported to also provide functionality such as MAC address accounting for Port-Channel interfaces. This seems to have been integrated into 12.0(32)S, but IPv6 does not seem to be supported yet.
Below follows a list of Cisco Bug IDs (ddts) related to link aggregation that you need to consider when choosing an appropriate IOS image.
CSCee27396
present in 12.0(26)S1; fixed in 12.0(26)S3, 12.0(27)S2, 12.0(28)S1, 12.0(30)S
Symptoms: Over 90% CPU usage by CEF Scanner on all linecards and %TFIB-7-SCANSABORTED errors occur when configuring a link bundle. Also, the router sends traffic to MAC addresses taken from its ARP table seemingly at random, instead of to the appropriate next-hop's MAC address.
CSCef12828
present in post-CSCee27396; fixed in 12.0(26)S4, 12.0(27)S3, 12.0(28)S1, 12.0(30)S
Symptoms: When traffic passes through a router, the router blocks traffic for certain prefixes behind a port-channel link.
CSCdz33664
present in 12.0(25)S3, 12.0(26)S1, 12.0(27)S2, 12.0(28)S; fixed in 12.0(25)S4
Symptoms: An HSRP state change on any Engine2 interface causes a microcode bundle flap on all other Engine2 linecards, preventing load balancing to work due to vanilla microcode getting loaded.
CSCee81071
present in 12.0(26)S3, 12.0(27)S2, 12.0(29)S
Symptoms: Router sends Ethernet frames with a source MAC address of beef.f00d.beef and destination MAC address f00d.beef.f00d (which is the pattern scribbled in unallocated memory in GSR linecards), with what looks to be a legitimate payload of transit traffic. This is one of the symptoms of CSCee27396.
CSCeb38014
present in 12.0(26)S5; fixed in 12.0(26)S5, 12.0(27)S
Symptoms: The BGP Router process flushes the BGP tables for each peer when you change one neighbor's description. This pegs the GRP CPU at 99% for quite a while.
CSCeg31951
present in 12.0(31)S; fixed in 12.0(31)S2 (CSCei53226)
IOS (at least in the PRP code) places each individual public peer in its own update-group if remove-private-as is configured on a peer. Needless to say, this scales badly for a router connected to an Internet exchange. (Try "show ip bgp replication".)
A collection of hearsay follows for recent IOS images for the GSR/PRP regarding link aggregation. AMS-IX does not run any GSRs. Please take this information with appropriately-sized grains of salt.
12.0(24)S2 is not advisable (not many specifics known but they include CSCef89562 and CSCee33045)
12.0(24)S6 boots but load-balancing is completely off
12.0(25)S* until S3 have CSCdz33664
12.0(26)S* until S4 have CSCef89562, where Engine4+ linecards can have continuously flapping interfaces, but is also somewhat required for Quadra linecards
12.0(26)S3 has CSCee27396 integrated but not CSCef12828, which leads to traffic blackholing
12.0(27)S* until S3 have CSCef89562 as well
12.0.(27)S1 has a problem where it sends traffic to random destinations
12.0(27)S2 has CSCee27396 integrated but not CSCef12828
12.0(27)S4 reportedly works reasonably well on PRP2s
12.0(28)S1 has problems with Engine2 linecards (CSCef78098) and Engine4+ (CSCef89562)
12.0(28)S2 reportedly works better but still sometimes emits beef.f00d.beef frames on normal ports with only an IPv6 address configured
12.0(30)S has only been observed to exhibit CSCef12828-like symptoms in conjunction with broken hardware, and also to still sometimes emit frames from MAC beef.f00d.beef.
Routers occasionally still send out frames with beef.f00d.beef as MAC source address on interfaces with an IPv6 but no IPv4 address configured, even on regular links.
Due to the massive amount of feature requests there will be both a 12.0(32)S and a new 12.0(32)SY train.
You can check for incorrect next-hops by attaching to the linecard and executing show controllers rewrite and show adjacency internal and comparing the two rewrite strings for a certain peer's IPv4 address (suffix the commands with | begin 195.69.14a.b). The first six bytes of the returned long hex string should be the peer's MAC address, and equal for all three occurrences.
! An example configuration follows: ! interface Port-channel1 description AMS-IX Aggregated Link ip address 195.69.14x.y 255.255.254.0 no ip redirects no ip directed-broadcast no ip proxy-arp channel-group minimum active 1 no channel-group bandwidth control-propagation hold-queue 150 in ! interface GigabitEthernet1/2/1 no keepalive no negotiation auto channel-group 1 no cdp enable ! interface GigabitEthernet1/2/2 no keepalive no negotiation auto channel-group 1 no cdp enable ! |
Specifying a hold-queue value is optional, but setting it to the amount of ports in an aggregated link multiplied by 75 is advised.
show interfaces Port-channel 1 will display keepalives enabled even though they are not; also, the BIA (burnt-in address, shown as 0000.0000.0000) can be ignored.
Please contact the AMS-IX NOC if you disable autonegotiation on Gigabit Ethernet ports as we may have to explicitly configure our switch for this.
5.5. Cisco 10GE Specifics
IOS supports no bgp fast-external-fallover and event dampening. The no bgp fast external-fallover tells the device to not act immediately on link flaps but wait for the BGP hold timers to expire before resetting sessions.
Newer versions of Cisco IOS even support ip bgp fast-external-fallover deny in a per-interface context.
Note that in practice we have found that the previously advised carrier-delay does not work as expected on Cisco equipment. We suggest you disable fast-external-fallover instead.
5.6. IPv6 Config
Responses on a ICMPv6 multicast listener queries result in bursts of ICMPv6 multicast listener reports. To prevent this configure no ipv6 mld router in interface context. Some other per-interface commands we recommend on a Cisco device:
! disable ICMPv6 multicast listener reports no ipv6 mld router ! disable IPv6 multicast forwarding no ipv6 mfib forwarding ! v6 ND-RA is unnecessary and undesired ipv6 nd suppress-ra ! disable PIM on a specified interface no ipv6 pim |