Updated: May 11
This is going to be a new mini series that we'll do in preparation for the first open networking certification that Cumulus Networks introduced (see here - https://education.cumulusnetworks.com/certification-exam-registration).
Cumulus Networks has a great page (both free and paid content) where you can spend some time and learn all things Cumulus and open networking related (see here - https://education.cumulusnetworks.com/).
So, where do we begin? Every time I learn a new product, I start at the start. Things we learned way back when. Because Cumulus Linux is a native Linux distribution (and it's interface may be unfamiliar to many), we'll start with some very simple aspects of working with the box - basic port bring up/down, port configurations, gathering information about a port and finally an introduction to Cumulus' NCLU!
As a reference, we'll be working on the following topology:
With Linux networking, your interface configurations would be found in /etc/network/interfaces (there are several helpful pages that you can google and find to understand the syntax in this file so we're not going to go over that in too much detail). You can quickly view this via the 'cat' option.
On both boxes, with a blank configuration, we see:
This file can directly be modified to change the configuration of various interface or to introduce new logical interfaces. To demonstrate this, let's go ahead and configure swp1 on each box to be a L3 interface with an IP address in the subnet 10.0.0.0/24.
Bring these interfaces up using the 'ifup' option of the ifupdown2 module on both the switches.
To confirm the status of the interface, you can use 'ip link show' to list all interfaces or specify a particular interface to look at using 'ip link show swp1'
Looking at only swp1:
An interesting thing to note here - even if you shut down one side of the link (say, do a 'sudo ifdown swp1' on SPINE1), the other side would still show that the link is up. This behavior is specific to Cumulus VX in virtualized environments only and was confirmed by Cumulus folks (this is not just a Cumulus issue - this behavior is typical of virtual routers/switches and is seen across vendors).
"When using VBox there is a little switch in the middle of the link that holds it up."
You can poke this middle switch to bring the links down if you're specifically testing link failures. But we're not going to get into that here.
Clearly, manipulating these network configuration files can be a little tiresome and more importantly, prone to human error. You need to be aware of the kind of syntax that is used within these files and all of the different intricacies that go into the configuration here.
This is where Cumulus' NCLU comes in. NCLU (Network Command Line Utility) is essentially a CLI that takes you away from manual manipulation of network files and provides a helpful CLI for the same instead.
All NCLU commands start with 'net'. You can tab or use a question mark to get all the available options.
To add configuration to an interface, you can use the 'net add interface' CLI. For example, instead of manipulating the /etc/network/interfaces file directly to add an IP address to swp1, I can do this instead:
NCLU has three steps to it:
1. Configure using 'net add [del]' commands.
2. Confirm what is going to be configured using 'net pending'.
3. Push this configuration using 'net commit'.
A complete example of configuring an IP address on swp1 follows:
Quick note: from a link logging perspective, this is done via rsyslog. The defaults can be found in '/etc/rsyslog.d'. 22-linkstate.conf contains information on where link state changes are logged. By default, this goes to /var/log/linkstate. However, with Cumulus VX, you'd soon realize that there is no 'linkstate' file in /var/log. This is because switchd is responsible for this and switchd doesn't do much in Cumulus VX (as it is largely just an interface for the ASIC). Due to this, you cannot track link state changes in Cumulus VX. Shoutout to Eric Pulvino (https://twitter.com/EricPulvino) for clarifying this.
We will continue with bridging on Cumulus VX in part II. Each post in this series will show both NCLU configurations and manual changes needed to the relevant network files.