Last weekend the team were involved in a migration of a pair of customer WAN edge switches which provided fail-over routing for their WAN and point to point circuits.
The original configurations had been reviewed and migrated to a new pair of Cisco CAT 9K switches which would get installed into separate computer rooms (for resilience). The job for the very early hours was to swap out the existing switches and replace them with the newly configured devices, perform some testing and ideally get back to bed!
Topology
It was a relatively simple setup, one switch in each of the two computer room, connected to each other over a fiber trunk. The first switch (N71) had the point-to-point link to the head office and the second switch (N72) had the WAN connection. Both switches had connections back to internal firewall/IPS. During the initial migration, only a single link was used to connect the switches between the computer rooms – but a second port was ready to be connected.
The hardware change was completed without issue and access to the new switches showed all the relevant interfaces had come up. A good start.
Routing
When checking the routing tables on N71 we noticed that the P2P link was not being used, no routes were being learnt from the remote neighbor over the P2P, and all traffic was going via the MPLS link. A quick ping test showed we could see the neighbor over the P2P link, so the link was OK.
OSPF Neighbors
So we did the usual checks (interface status, interface configs and ping tests) and confirmed they all looked OK. But we were clearly not learning any routes from the P2P neighbor – and in fact the local switch was not even seeing the remote switch as an OSPF neighbor.
Time to look at the OSPF configuration:
I am sure you eagle-eyed engineers will have spotted the issue – the configuration was defaulting to “passive-interface” on all ports, meaning no OSPF neighboring by default. So our point to point port (G1/0/1) would not create an OSPF relationship with the remote switch – (not) working as designed!
A Quick Fix
A quick addition to the router configuration:
no passive-interface GigabitEthernet1/0/1
and everything sprang to life!
Summary
Later that day (after a little sleep) we had a quick look at our switch config generator and strangely it did have the statement included. We can only put this issue down to a cut and paste issue when applying the configuration to the switches. A classic PEBCAK.
From a lessons learnt perspective, this reinforces our approach to change management. We always try and have two engineers working on an installation or change – as working on your own can lead to tunnel vision and digging deep holes. A second pair of eyes is vital – especially at 5am!