This probably should have been one of my first posts for SDA but here we are. I've recently come to realize that there is a lot of misconception about how ARP works within the SDA solution - the defacto answer appears to be that it is a part of BUM traffic (broadcast, unknown unicast, multicast), and thus, it will be flooded (implying that there is a dependency on some form of replication, either head-end or via an underlay multicast infrastructure).
This is not true and it's time to bust that myth! We will continue to use the same topology, only this time, Host1 and Host2 are part of the same subnet (220.127.116.11/24) and the same VN - Corp_VN.
First things first - let's gather some preliminary data to establish a baseline before we start testing. Both hosts have generated some minimal traffic, due to which their IP addresses and macs are present in the IPDT (IP device tracking) tables and the LISP database.
LISP should have picked these up dynamically as well, so we should see these addresses in the LISP database as well. The allocated instance-ID for IPv4 is 4100 and for Ethernet, it is 8195.
Perfect. Since these are registered in the LISP database, we should see them in the site table on the control-plane as well. Let's quickly check Border1 for this:
One last thing to verify before we start our testing - let's make sure the hosts do not have each others IP address resolved already.
Only the gateway IPs are resolved and the hosts have no knowledge of each other. Naturally, our test is very simple - we will ping from Host1 to Host2.
Pings work too. Let's try to understand what happened here.
Host1 generates an ARP request for Host2s IP address, since it knows it is in the same subnet. This hits Edge1.
A packet capture on Gi1/0/23 on Edge1 confirms that the ARP is a broadcast as it comes into Edge1:
Here's where things get interesting. The ARP will not be flooded immediately. Edge1 will first send out a map-request for the IP address that is present in the "Target IP address" of the ARP packet:
The control-plane nodes maintain an address-resolution table which has IP-mac mapping entries for all hosts in the fabric. This is similar to how BGP EVPN, in a VXLAN fabric, maintains an EVPN ARP suppression table.
You can view this table using the following command:
When this map-request (generated from an ARP request) comes to the control-plane (Border1, in this case), it looks at this table for a hit. If there is a hit, it sends a map-reply with the mac address of the host.
Wireshark doesn't have the proper dissectors to understand this packet fully but here's the map-reply anyway:
When Edge1 gets this, it builds another map-request - this time, for the mac address it just got from the control-plane. This leads to a second round of map-request/map-reply. When Border1 gets the map-request, it looks at the Ethernet site table (since the request is for a mac address) and responds back with the RLOC associated with this mac (which is Edge2).
Inline, this looks like so:
Border1 replies back with the RLOC:
Inline, this looks like so:
At this point, Edge1 should have its LISP map-cache built as well:
Remember to look at the Ethernet instance-ID and not the IPv4 one. What happens now? The ARP is converted into a unicast message on Edge1 and is encapsulated with a destination IP of Edge2:
Inline, this looks like this:
Notice how the ARP is no longer a broadcast - it has been converted into a unicast ARP on Edge1 and encapsulated with a VXLAN header with a VNID of the Ethernet service (for that VLAN). The destination IP is of Loopback0 of Edge2.
Once Edge2 gets this, it decapsulates the packet and unicasts it to Host2. This entire process now happens in reverse when Host2 sends an ARP reply in response to the ARP request.
So, one final question to close this post out - when do we really use some form of replication for ARP? Naturally, when there is no hit in the address-resolution table of the control-plane. This also implies that this is perfect for silent hosts and this is exactly where L2 flooding comes into the picture.
With L2 flooding (and an underlay multicast infrastructure), we can flood ARPs through the fabric and force a silent host to respond, thus bringing it alive in the fabric. L2 flooding should be another post on its own so we'll get to it some day!
Thanks for reading, I hope this was informative! Till the next one, hopefully in better times (yes, this quarantine/social-distancing sucks but it is necessary).