Cumulus Basics Part I - navigating the OS
This first blog on Cumulus introduces the reader to the basics of the operating system and Cumulus' NCLU.
Introduction
This is going to be a new mini series that we will do in preparation for the first open networking certification that Cumulus Networks introduced,see here.
Cumulus Networks has a great page (both free and paid content) where you can spend some time and learn all things Cumulus and open networking related, see here.
So, where do we begin? Every time I learn a new product, I start at the start. Things we learned way back when. Because Cumulus Linux is a native Linux distribution (and it’s interface may be unfamiliar to many), we’ll start with some very simple aspects of working with the box - basic port bring up/down, port configurations, gathering information about a port and finally an introduction to Cumulus' NCLU!
Topology
As a reference, we’ll be working on the following topology:
Basic interface configuration
With Linux networking, your interface configurations would be found in /etc/network/interfaces (there are several helpful pages that you can google and find to understand the syntax in this file so we’re not going to go over that in too much detail). You can quickly view this via the ‘cat’ option.
On both boxes, with a blank configuration, we see:
cumulus@cumulus:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*.intf
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
This file can directly be modified to change the configuration of various interface or to introduce new logical interfaces. To demonstrate this, let’s go ahead and configure swp1 on each box to be a L3 interface with an IP address in the subnet 10.0.0.0/24.
SW1:
cumulus@cumulus:/$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*.intf
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
auto swp1
iface swp1
address 10.0.0.1/24
SW2:
cumulus@cumulus:/$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*.intf
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
auto swp1
iface swp1
address 10.0.0.2/24
Bring these interfaces up using the ‘ifup’ option of the ifupdown2 module on both the switches.
cumulus@cumulus:/$ sudo ifup swp1
To confirm the status of the interface, you can use ‘ip link show’ to list all interfaces or specify a particular interface to look at using ‘ip link show swp1’.
cumulus@cumulus:~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 0c:db:6b:cc:20:00 brd ff:ff:ff:ff:ff:ff
3: swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 0c:db:6b:cc:20:01 brd ff:ff:ff:ff:ff:ff
4: swp2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:db:6b:cc:20:02 brd ff:ff:ff:ff:ff:ff
5: swp3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:db:6b:cc:20:03 brd ff:ff:ff:ff:ff:ff
6: swp4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:db:6b:cc:20:04 brd ff:ff:ff:ff:ff:ff
7: swp5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:db:6b:cc:20:05 brd ff:ff:ff:ff:ff:ff
8: swp6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 0c:db:6b:cc:20:06 brd ff:ff:ff:ff:ff:ff
Looking at only swp1:
cumulus@cumulus:~$ ip link show swp1
3: swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 0c:db:6b:cc:20:01 brd ff:ff:ff:ff:ff:ff
An interesting thing to note here - even if you shut down one side of the link (say, do a ‘sudo ifdown swp1’ on SPINE1), the other side would still show that the link is up. This behavior is specific to Cumulus VX in virtualized environments only and was confirmed by Cumulus folks (this is not just a Cumulus issue - this behavior is typical of virtual routers/switches and is seen across vendors).
“When using VBox there is a little switch in the middle of the link that holds it up.”
You can poke this middle switch to bring the links down if you’re specifically testing link failures. But we’re not going to get into that here.
Clearly, manipulating these network configuration files can be a little tiresome and more importantly, prone to human error. You need to be aware of the kind of syntax that is used within these files and all of the different intricacies that go into the configuration here.
Cumulus NCLU introduction
This is where Cumulus' NCLU comes in. NCLU (Network Command Line Utility) is essentially a CLI that takes you away from manual manipulation of network files and provides a helpful CLI for the same instead.
All NCLU commands start with ‘net’. You can tab or use a question mark to get all the available options.
cumulus@SPINE1:~$ net
abort : abandon changes in the commit buffer
add : add/modify configuration
clear : clear counters, BGP neighbors, etc
commit : apply the commit buffer to the system
del : remove configuration
example : detailed examples of common workflows
help : context sensitive information; see section below
pending : show changes staged in the commit buffer
rollback : revert to a previous configuration state
show : show command output
To add configuration to an interface, you can use the ‘net add interface’ CLI. For example, instead of manipulating the /etc/network/interfaces file directly to add an IP address to swp1, I can do this instead:
cumulus@cumulus:~$ net add interface swp1 ip address 10.0.0.1/24
NCLU has three steps to it:
- Configure using ‘net add [del]’ commands.
- Confirm what is going to be configured using ‘net pending’.
- Push this configuration using ‘net commit’.
A complete example of configuring an IP address on swp1 follows:
// net add
cumulus@SPINE1:/$ net add interface swp1 ip address 10.0.0.1/24
// net pending
cumulus@SPINE1:~$ net pending
--- /etc/network/interfaces 2019-03-05 16:40:58.268594116 +0000
+++ /run/nclu/ifupdown2/interfaces.tmp 2019-03-05 17:23:54.585798862 +0000
@@ -6,14 +6,12 @@
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
auto swp1
iface swp1
- address 10.1.1.1/24
-
-
-
+ address 10.0.0.1/24
+ address 10.1.1.1/24
net add/del commands since the last "net commit"
================================================
User Timestamp Command
------- -------------------------- ---------------------------------------------
cumulus 2019-03-05 17:23:51.321383 net add interface swp1 ip address 10.0.0.1/24
// net commit
cumulus@SPINE1:~$ net commit
--- /etc/network/interfaces 2019-03-05 16:40:58.268594116 +0000
+++ /run/nclu/ifupdown2/interfaces.tmp 2019-03-05 17:24:00.025799294 +0000
@@ -6,14 +6,12 @@
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
auto swp1
iface swp1
- address 10.1.1.1/24
-
-
-
+ address 10.0.0.1/24
+ address 10.1.1.1/24
net add/del commands since the last "net commit"
================================================
User Timestamp Command
------- -------------------------- ---------------------------------------------
cumulus 2019-03-05 17:23:51.321383 net add interface swp1 ip address 10.0.0.1/24
Quick note: from a link logging perspective, this is done via rsyslog. The defaults can be found in ‘/etc/rsyslog.d’. 22-linkstate.conf contains information on where link state changes are logged. By default, this goes to /var/log/linkstate. However, with Cumulus VX, you’d soon realize that there is no ‘linkstate’ file in /var/log. This is because switchd is responsible for this and switchd doesn’t do much in Cumulus VX (as it is largely just an interface for the ASIC). Due to this, you cannot track link state changes in Cumulus VX. Shoutout to Eric Pulvino for clarifying this.
We will continue with bridging on Cumulus VX in part II. Each post in this series will show both NCLU configurations and manual changes needed to the relevant network files.