reveal.js

In the NIC of time

Enabling high-performance edge applications with OpenStack, OVS, and SmartNICs

40% of all mobile traffic data

At Ericsson, we use OpenStack for virtual infrastructure management, as part of our NFVi solution.
Our customers are typically telecom operators ... well, and a taxi company ... in Dubai ... and Panasonic Avionics for entertainment on board of flights, and many other such examples.

What is NFVi?
For many years at Ericsson, we made SW and HW very dependent on each other. Customers bought whole racks of custom HW and the Ericsson SW applications deployed and running on top of that HW.
NFVi is part of the NFV framework (a network architecture concept) and it means, more or less, decoupling the SW from HW for network nodes using virtualization.
It means you can run the telecom applications(the SW) on any HW (like Dell, HP, Quanta, SuperMicro, Fujitsu servers, whatnot), in VMs or containers.
Basically, running telecom applications on Intel HW, 'cause that's so much better ... meltdown ... spectre ... zombieload

Traces of this decoupling of the network functions from proprietary hardware have been there for many years now.
Around 2003, I worked in an ISP. We used Cisco routers to do BGP with customers and the upstream provider. I was in awe when GNU Zebra came out and I could run BGP in a Linux box.
Fast forward to today, part of SDN, we use opendaylight with Quagga soft router for BGP ( Quagga is what followed after zebra, it is actually an extinct sub-specie of the African zebra.)

88% increase in mobile data traffic Q4-2017 to Q4-2018

EPC to the rescue

According to Ericsson Mobility Report, the monthly mobile data traffic increased to 88% between Q4 2017 and Q4 2018, this is mainly increasing traffic per smartphone in China. According to the same report, mobile traffic is 50% video today and it will increase to 75% video in 2020, driven by, amongst others, AR/VR applications.

EPC is handling all this traffic.
EPC is the equivalent of formerly used GPRS, it is there to make mobile data traffic possible.
It means you traverse it when surfing the internet from your mobile, when watching youtube, Netflix, GoT, playing Pokemon?

I work at E/// with infrastructure, so EPC people ask us for ways to solve this increasing traffic problem. How to cope with this amounts of traffic in a performant way?
We think smartNIC can be the answer.

There are quite a few types of smartNICs, ranging from thousands to a few hundred dollars(some you can buy on eBay).
Some smartNICs use an FPGA (Field Programmable Gate Array) mounted on the PCIe network card. Using FPGA boards provides flexibility, they can be easily programmed and updated once installed. (P4-Programmable Protocol-independent Packet Processor- can be used for programming packet forwarding planes.)

Some smartNICs do not use an FPGA, and are ASIC based instead, they are less flexible but still capable of doing lots of things, at a significantly lower price.

SmartNICs might use ARM CPUs in a RISC architecure (the kind of HW you have on your *smart*phone) or x86 CPUs, different amounts of RAM, they can consume more or less power.

SmartNICs can have different form factors HHHL(half hight half lenght) if you need to fit it in a 2Us compute or FHFL full hight full lenght for wider than 2Us computes.

You can opt for 2 or 4 X 25, 4X 50, 2 X 100 Gbps ports, that is two ports (cages) on a NIC wifh HHHL form factor and 4 ports on NICs with FHFL form factor.
You have options around the number of PCI lanes to be used, x8 or x16.

Is it a bird? Is it a plane?

It's SmartNIC !

My manager advised I remove this joke because it's lame. OK, I agree, it's lame but I kept it anyway :).

How should your architecture look like when using smartNICs.

You could use one smartNIC for data per compute, we're talking 2x100 Gbps here, that's *a lot* of bandwidth. In this case, high availability happens at compute level, not at network card level.

What if you have a dual socket system. If you plug a smartNIC in a PCIe slot, applications might not like crossing that QPI link (from 2017 called UPI) between the CPUs. In this case you can look into using a bifurcated smartNIC, that's basically splitting the card in two physical pieces that you can insert in two PCIe slots, one per NUMA node.

If you use two smartNICs, do you want two separate ovs controllers in your compute? will OpenStack even support that?

What is this smartNIC, anyway?

Is it an embedded linux? Is it a linux mini server?

When should you use a smartNIC?

Do you use more than 4CPUs for OVS per compute host?

Here comes an example where you can see the CPUs allocated to ovs under nohz_full ( confirmed with /etc/default/ovs-dpdk PMDs). If ovs is running on that CPU, the kernel will stop sending timer ticks to that CPU, running ovs rather than servicing interrupts and context switching. (but from my experience this works to a certain degree).

Here's one interesting example: on a poor dimensioned compute (where you're running many VMs and did not take into account that you need to reserve resources for device emulation for each VM, *on the host OS side*), if you run out of mem , oom killer will take down your ovs and with it shutting down all your VMs? With ovs in the NIC this will not happen, but of course you still need to properly dimension your system!!

Ironic Neutron Cyborg

Now the question is where are we in Openstack when it comes to integrating the wide range of smartNICs popping up on the market?

Work is done in several Openstack projects, like ironic, nova, neutron and cyborg.

For instance when it comes to neutron, we need changes in the Neutron OVS driver and Neutron OVS agent in order to bind the Neutron port for the baremetal host with the smartNIC.
This is needed so that neutron ovs agent can configure the OVS running on the smartNIC.

We can have neutron ovs agent running locally on the smartNIC or remotely and manages the OVS bridges for all baremetal smartNICs. (well, good luck with that)

There are many interesting questions raised, like how do you know which smartNIC belongs to which server, in ironic we configure hostname and ssh keys...

Openstack Cybborg works with providing managementAPIs to work with acceleration resources, like FPGA.

root@cic-1:~# openstack port create

usage: openstack port create [-h] [-f {json,shell,table,value,yaml}]

[-c COLUMN] [--max-width ] [--fit-width]

[--print-empty] [--noindent] [--prefix PREFIX]

--network [--description ]

[--device ]

[--mac-address ]

[--device-owner ]

[--vnic-type ] [--host ]

[--dns-name dns-name]

[--fixed-ip subnet=,ip-address=]

[--binding-profile ]

[--enable | --disable] [--project ]

[--project-domain ]

[--security-group | --no-security-group]

[--qos-policy ]

[--enable-port-security | --disable-port-security]

[--allowed-address ip-address=[,mac-address=]]

[--tag | --no-tag]

--vnic-type

VNIC type for this port (direct | direct-physical | macvtap | normal | baremetal, default: normal)

root@FPA1066GX-DA2:~# ovs-appctl dpif/dump-flows br0

in_port(14),eth(src=68:05:ca:91:36:b5,dst=68:05:ca:91:36:b5),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.1.1,proto=6),

tcp(src=1234,dst=5678), packets:0, bytes:0, used:7.340s, actions:drop

Performance

Kernel Space vs User Space

If you care about latency and packet processing performance there are a few options.

We basically need to overcome the limitations in the Linux kernel which is not ideal for *lots* of packet-processing.

Why is the Linux kernel a problem when we talk about latency and performance ( with performance, I mean higher throughput at a lower CPU cost)?

The networking stack inside the Linux kernel limits how many packets per second it can process, it was conceived to be slow, a decision taken 20 25 years ago, that packets should be delivered into sockets. If you want to be fast you dont do this upfront.

Too many packets per second means CPUs get busy just receiving packets, then either the packets are dropped or we CPU starve the applications.

When a packet comes in, it is stored in the RX ring buffer. The device driver sends a software interrrupt to the kernel. The packet gets stored in sk_buff to be hold up to MTU.
When all packets are received, they get sent to higher in the kernel stack. Then they get copied to the receiving process.

Note: Initially, a hard interrupt is used followed by sw interrupts.

Kernel bypass with DPDK

To get better performance, one can choose to bypass the kernel, fully or partially.

There are several kernel bypass options like:
DPDK, (that would be the poster child of kernel bypass)
Snabbswitch,
PF_RING,
Netmap. (I am personally more familiar with DPDK.)

With kernel bypass, we move the NIC to the user-space.
If the NIC is managed in the user-space, it means we skip things like context switching, networking layer processing, interrupts that happen in the kernel, IRQ storms and do the packet-processing in the user-space.

This is relevant at 10Gbps already. EPC today saturates 200Gbps already.

NUMA awareness together with CPU isolation needs to be considered as well if we need high performance.

Moving to userspace means losing the abstraction level the kernel provides for e.g. hw resources, it means you need to load own driver.
Moving to userspace means the kernel space is skipped together with the good stuff too like networking functionality that needs to be reimplemented now.

DPDK is great except ...

you need to reinvent the wheel

$ echo 0000:18:00.4 > /sys/bus/pci/devices/0000\:18\:00.4/driver/unbind

$ echo 0000:18:00.3 > /sys/bus/pci/devices/0000\:18\:00.3/driver/unbind

$ echo 0000:18:00.4 > modprobe vfio

$ echo 0000:18:00.3 > modprobe vfio_pci

$ mkdir -p /dev/hugepages/

$ mount -t hugetlbfs hugetlbfs /dev/hugepages/

$ echo 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

$ modprobe uio

$ insmod x86_64-native-linuxapp-gcc/kmod/igb_uio.ko

$ dpdk-devbind.py -b igb_uio 18:00.3 18:00.4

XDP and eBPF

$ grep CONFIG_BPF_SYSCALL /boot/config-4.15.0-46-generic

CONFIG_BPF_SYSCALL=y

prompt: XDP sockets

type: bool

depends on: CONFIG_BPF_SYSCALL

defined in net/xdp/Kconfig

found in Linux kernels: 4.18–4.20, 5.0–5.1, 5.2-rc+HEAD

So XDP is a hook in the Linux kernel, not a kernel bypass but a bypass of the network stack in the linux kernel. It is a programmable layer in the kernel network stack.
XDP operates at the same level as DPDK, directly on the packet buffers, operating on the packet before it is moved to the socket buffer.

While DPDK steals the whole NIC from linux kernel, XDP does not steal the NIC from the kernel. In XDP we put in a filter per receive queue, make decisions on the packets with zero copy to userspace. This way all the fun features in kernel are available.

XDP can be used in two modes: native mode, generic (SKB) mode
Native mode XDP (process packet right after DMA, just before SKB allocation), a driver hook , before memory allocation , small no of instructions executed before we start processing packets
Limited because we need support for XDP in the driver, but higher performance mode.

Generic mode
IT happens after DMA and SKB allocation, it means a larger number of instructions is executed before doing something with that packet, so this means significantly lower performance than native mode. Works on any NIC device, driver independent.

You can look at XDP and BPF with a dataplane/controlplane approach.
The dataplane is in the kernel and the control plane is in the userspace, where userspace load eBPF program.

(spanning tree daemon uses BPF to filter only BPDUs are comming on that socket)

XDP_DROP

XDP_PASS

XDP_TX

XDP_ABORTED

XDP_REDIRECT

These are actions that can be executed on a packet (that's the stuff you need to have in the NIC driver to support XDP)

DROP is awesome, useful for DDOS, fb uses it heavily and cloudflare

PASS it lets the packet pass, after packet inspection, this does not do DPDK , light weight as you program it , lets the packet go back to stack

TX send it immedietly back out on the port that it was received, for load balancer cases for example

ABORTED basically drop but what is extra is that you will get some log about it , useful for debugging for sysadmin or developer

REDIRECT to another port, to other CPUs, you can modify headers on the packet (TX and REDIRECT similar to the DPDK ones)

Limitation on support for jumbo frames in XDP , jumbo frames are supported, like 3kB but not 9kframes. Might be a problem for storage? (reason why this is a problem "one frame cannot exceed a page" constraint)

eBPF is a superpower

This is BPF!

eBPF to modify the kernel behaviour

Shamefully stolen from Alexei

smartNICs with Storage

I talked mostly about the data traffic so far and using smartNICs with the data traffic.

smartNICs make sense in a lot of cases, storage, video, big data, security. In the storage case, they can be used to accelerate storage apps and security, like encrypting data on the disk.
Using smartNICs with NVMeOF you can disagrate compute from storage and you can get local storage performance across the network, that is very close performance between local storage and remote storage.
Basically you attach a block device to your VM and acessing it is as fast as if it was local.
NVMeOF maintains the high performance of a local nvme SSD.

Many applications moving to cloud today are probably not in high demand of high number of IOPS, because they were written for HDDs to begin with.
But smartNICs with storage will be relevant in the future.

Ericsson customers running on Openstack

Telstra
RTA Dubai
Panasonic
Swisscom
Telefonica
2020 Olympics with NTT Docomo

Datacenters today?
Stacked up "desktop" computers filled with ... air?

It is my desire to leave you with a thought.
I came to the realization that most of the things we learn are provisional and in consequence they are open to recantation and refutation. (I have been raised in an eastern europe communist country and been lied to a significant part of my life).
I enjoy this path of questioning everything, why do we do things a certain way.

In 2001 my first job was sysadmin working for a big Eastern European Internet Service Provider
Many customers asked to move their web servers and mail servers on our premises.
It was because their services would access directly the big pipe and we had a generator. 2001 Eastern Europe meant lots of power outages ... daily.

They would bring desktop tower PCs that we would place on and under lined up tables against the wall. Soon enough, we ran out of physical space on those tables and bought racks.
We asked customers to buy rackable servers in order to host with us, then we ran out of space again, we build an awesome datacenter and somewhere 2005 we ran out of space there too. We started using VMs with kvm.

Fast forward, today our datacenters are collections of those stacked boxes that used to be desktop computers. To me, this is insane, does it make any sense to you? Should we go down the timeline of computer history and disagragate it all and have racks of compute, memory, storage, networking?

Thank you:
Florian Haas
Brendan Gregg
Jesper Dangaard Brouer
Alexei Starovoitov
Hakim El Hattab