BNG Article #1 — VPP and DPDK HQoS¶

Broadband Network Gateways (BNGs) are the source of plenty of controversy in the telco industry. They are the magic smoke-and-mirror boxes that you have to pay a lot of money for, almost all of which are massively over-specified and overpriced for the needs and means of a small ISP. There is a huge dead-zone in the market for BNGs that can handle more than 500 subscribers but less than 20,000 subscribers simultaneously, and in general the industry is very focused on hyperscale-sized companies. Juniper, Cisco, Nokia, rtBrick and various vBNG solutions are all guilty of taking this approach — and many engineers are cognisant of the massive drop in software quality from big vendors over the last decade in spite of the large price-tags they're commanding.

In reality, all these devices are is a standard router with some hacks thrown in to handle some telco-specific problems. Ok that might be trivialising it a bit, but let's dig into what we can do without the big vendors' solutions.

VPP¶

VPP is an open-source project that was originally created by Cisco, it's a programmable software data plane that runs on various architectures and operating systems (although we are particularly interested in it's x86 Linux flavour in this scenario). I have used it a number of times in the past in all kinds of experiments, but it sees heavy use commercially in a number of hardware and software products.

It does have its flaws, and being a project for “mass-consumption” means that being versatile is of very high importance to the project.

VPP Can't do Shaping¶

Anyone that has used VPP in an attempt to do subscriber aggregation has probably noticed that HQoS was removed from the VPP project a long time ago and there is no sign of it making a return any time soon. The reason for this is that “it doesn't work properly”, what the project maintainers meant by this is that the HQoS functionality didn't fit into the intended architecture of VPP properly. The VPP HQoS feature was actually just an implementation of the DPDK HQoS Scheduler within VPP's DPDK Plugin, which ultimately goes against the “graph node” ethos of VPP because it meant that HQoS could not be used on non-DPDK bound interfaces (such as Tunnel interfaces, or even RDMA interfaces).

Whilst I can appreciate why this decision was made, this also sucks for me because I don't care about using non-DPDK interfaces in this project. I'm not the first one to feel this way, as you can see the Meta Terragraph project came to the same conclusion and actually ported the HQoS functionality back into a newer version of VPP — albeit unfortunately only for NXP's ARM-based platforms such as Layerscape.

DPDK Can do Shaping¶

My C Programming skill-set is limited — I will gladly sit down and write handy scripts in PHP, or code more serious things in Golang, but C is another step above that and C is the language that DPDK applications need to be written in.

However, there are a number of very helpful examples that explain how to implement a HQoS scheduler, and it is actually relatively easy to write a DPDK app that just does the job of HQoS and nothing else.

Connecting VPP with other DPDK Applications¶

We've established my C skills aren't quite where they need to be to bring HQoS functionality back into the latest version of the VPP DPDK Plugin, so we need to look at another option.

As it turns out, DPDK applications are able to communicate via Memif interfaces, and it is actually possible to connect a Memif interface in VPP to another DPDK application that is handling HQoS. DPDK only supports Ethernet-type Memif interfaces, but that is fine as we do not need DPDK to do any routing, and it can still inspect the headers to classify traffic into the correct HQoS queues as needed.

I'm tempted to say this is dodgy in some way, but this is literally the kind of situation that Memif interfaces are intended for — very high speed network communications between two programs on the same machine.

The only real downside to this compared to implementing it in the VPP DPDK Plugin is that it requires separate CPU cores to be dedicated to VPP and to the DPDK HQoS app separately, and this will need to be done on a single CPU in NUMA (multi-socket) systems to avoid poor performance / latency related issues. Meta's TGHQoS implementation actually provides the option to dedicate a core/thread to HQoS enqueueing/dequeueing anyway, so perhaps this solution isn't too far off the mark.

Other Possibilities¶

XDP (REDIRECT to AF_XDP)¶

XDP is a way of “fast-tracking” packets in the Linux kernel. It doesn't bypass the kernel, but it effectively allows you to bypass most of it and bring the packet into user-space very quickly and efficiently upon ingress. Because of the requirement for HQoS, it would be necessary to write an entire user-space HQoS application for XDP to REDIRECT into via an AF_XDP socket to handle HQoS scheduling when transmitting packets. This user-space application could actually be a DPDK application, it could even be the same application as the one above using Memif interfaces, but adapted to use AF_XDP interfaces instead.

XDP (REDIRECT to NIC)¶

XDP_REDIRECT to a NIC may be possible, but it would be necessary to implement an HQoS Scheduler in eBPF, which I couldn't even tell you if it's possible to do so or not. On Linux systems the other choice is to use tc-qdisc on transmit to do HQoS Scheduling, but in my opinion using tc-qdisc in a carrier scenario is too inefficient (see QDisc Locking) for HQoS policies that are expected to deliver multiple multi-gigabit connections under a single parent QDisc to end-users. If you are dealing with <1Gbps bandwidths per tail then the XDP + tc-qdisc solution may actually be a good fit for you, in which case you should look into using LibreQoS. The developers have done a good job of making the most out of qdisc.

Not Shaping?¶

In the world of high bandwidth connections (>250Mbps), is it actually necessary to shape subscribers? The telecommunications engineering community at large has now seen and understands the negative effects of buffer-bloat on applications. Traffic policing enables congestion control algorithms built into TCP and other protocols to manage their own bandwidth by simply dropping packets instead of queueing them. This is exactly what a full queue is going to do on a shaper anyway, and end-users have Active Queue Management technologies such as CAKE available to them in an increasing numbers of firewalls and routers.

Future Plans¶

Unfortunately, I do have some problems with VPP:

It is intensely difficult to develop for if you are inexperienced with C
Developers who are experienced with VPP are often too busy to assist inexperienced developers
It isn't the most robust software — running an instance with minimal configuration for a few months with no traffic passing through it can still be enough to make it crash within a few months of run-time

The above has me leaning more toward trying to work with XDP, which is much more simple in many ways and has a larger community with more educational material for new developers.

I'm not quite sure where my next article on this topic will head, let's see!