Capturing Packets Faster in the 100Gbps World

Featured article By Dan Joe Barry, VP Positioning and Chief Evangelist, Napatech

The number of network end points is growing into the billions as the Internet of Things (IoT) expands its presence, spurred on inexpensive sensors, the rapid adoption of mobile devices and innovative minds. This presents engineers and administrators with the challenge of managing and protecting the network in real time, even as speeds increase up to 100Gbps.

To meet this challenge effectively, they need the tool of packet capture (PCAP). A mechanism for intercepting data packets that are traversing a computer network, PCAP is a common capability deployed within an organization to monitor security events and network performance, identify data leaks, troubleshoot issues and even perform forensic analysis to determine the impact of network breaches.

Though PCAP is an essential tool, existing PCAP systems using commodity network interface cards (NICs) are struggling to keep up with the demands of performing precision capture and replay at 10/40/100 Gbps speeds.

With that limitation in mind, providers have created solutions that can facilitate packet capture at speeds topping 100 Gbps. The use of network acceleration technology, coupled with open source network monitoring and capture solutions, can enable organizations to keep up with the demands of precision packet capture and replay on high-speed networks.

PCAP: What to Consider

Precision PCAP systems are able to give engineers and administrators an accurate, real-time view of what is happening within a network infrastructure. Likewise, these systems also provide organizations with the ability to re-create network events with high fidelity for verification and validation of architectural changes, troubleshooting and analysis.

What should you look for when researching high-speed PCAP solutions? Consider the combining of open source tools with the speed and accuracy of programmable logic. Here are three key factors when comparing your options:

– Avoiding latency: Seek out solutions that deliver hardware-based, high-precision time stamping with nanosecond resolution for every frame captured and transmitted. Hardware-based time stamping avoids the unpredictable latency inherent in software-based solutions and enables a communication flow to be recorded precisely as it occurs. Precision time protocol (PTP) can also be supported for accurate synchronization across distributed network probes.

– Performing at high speeds: To uphold capture and analysis performance at high speeds, implement technology that can identify and direct traffic flows immediately upon ingress. In doing so, the load on user-space applications can be minimized and administrators are provided with the ability to dynamically identify and direct data flows into specific CPU cores based on the type of traffic being analyzed.

– Capture and replay at line-rate: At speeds varying from 1 to 100 Gbps, FPGA-based network acceleration cards (NACs) are ideal for performing high-speed packet capture and replay. Moreover, NACs allow for precise inter-frame gap (IFG) control, which is critical when replaying captured traffic for troubleshooting or simulation of traffic flows.

Traditional PCAP

The standard approach for packet capture and analysis has been to use software tools. In this case, software is installed on a designated monitoring host and configured to poll packets from a commodity network adapter placed in promiscuous mode and connected to the network via a Switched Port Analyzer (SPAN) interface. A typical architecture for low-speed PCAP using a commodity network interface card (NIC) and libpcap is illustrated in figure 1 below:

Figure 1: Conventional PCAP Architecture

As the figure demonstrates, every time the network adapter receives an Ethernet frame, it generates an interrupt request and copies the data from the memory buffer on the adapter into kernel space. Normally the kernel space driver would determine if the packet is intended for this host and either drop the packet or pass it up the protocol stack until it reaches the user-space application it is destined for. However, when configured for promiscuous mode, all packets are captured in a kernel buffer regardless of destination host. Once the kernel buffer is full, a context switch is performed to transfer the data to a user-space buffer managed by libpcap, a system-independent interface for user-level packet capture, so that the data can be accessed by user-level applications.

The user-space buffer stays out of sight of user-level applications and is necessary to prevent applications from accessing kernel-managed memory. Given this architecture, it is clear that some amount of time will lapse between when a frame is received by the adapter and actually delivered to the user-space application for processing.

PCAP accuracy is not greatly affected by this time lapse at low data rates. However, at higher rates, this latency is compounded and CPUs become saturated trying to keep pace with incoming data, leading to capture loss and timing issues. A case in point: a 1 Gbps network link can move about 1.5 million packets per second, or one packet every 670 nanoseconds. Conversely, at 10 and 100 Gbps speeds, systems are processing one packet every 67 or 6.7 nanoseconds respectively.

Using a standard architecture, just capturing traffic at this rate is enough of a challenge without the added complexity of precise timing, categorization, flow identification and filtering. Performing lossless, high-fidelity packet capture, replay and real-time analysis of data flows at these rates requires a different approach to PCAP, one that moves the bulk of the data processing out of the user-space and into the hardware while also eliminating the inefficiency of user-to-kernel space interactions.

Today’s PCAP

PCAP can work on high-speed networks if a hardware-accelerated approach is adopted. The targeted use of programmable logic coupled with open source tools allows data to be accurately captured and processed within a network acceleration card (NAC) before it is passed into user-space applications. Figure 2 illustrates what an accelerated PCAP architecture might look like.

Figure 2: Accelerated PCAP Architecture

NACs use Field Programmable Gate Arrays (FPGAs) to perform in-line event processing and line-rate packet analysis in hardware at 1/10/40/100 Gbps speeds. Due to their programmable nature, FPGAs play an important role in, and are an ideal fit for, many different markets. These semiconductor devices are based around a matrix of configurable logic blocks (CLBs) connected via programmable interconnects. FPGAs can be reprogrammed to desired application or functionality requirements after manufacturing. Through the use of FPGA-based NACs, network administrators can immediately improve an organization’s ability to monitor and react to events that occur within its network infrastructure.

This type of PCAP framework leverages line-rate packet analysis to push most of the frame processing into the hardware of the capture device, which can be deployed within a commodity server or workstation, preserving CPU cycles for higher-level analysis. This approach ensures that by the time data is passed to the user-space buffer for access by applications it has already been time stamped, categorized, and filtered appropriately.

When devices like this are paired with open source applications, powerful – yet cost- effective – solutions can be built for a variety of purposes. In general, high-performance NACs enable easy in-house development of scalable, high-performance network applications over PCAP. Even complex payload analysis and network-wide correlation algorithms can be easily scaled by the effective flow-based load-balancing mechanism built into the NAC. The more complex analysis that the application performs, the more critical it is that the PCAP stream from the capture device has no packet drops and that the frames are in the correct order. Tasks like protocol reconstruction, reassembly, event detection and QoS calculations are severely impacted by insufficient PCAP performance.

It’s important to look for solutions that support IEEE 1588, or Precision Time Protocol (PTP). In this way, precise time synchronization is maintained in a distributed deployment where multiple accelerated PCAP probes are deployed throughout a network infrastructure. This allows frames to be merged from multiple ports on multiple NACs into a single, time-ordered analysis stream.

Sustaining temporal fidelity within the capture ensures that organizations can perform retrospective analysis of network events by replaying data in exactly the same way as it was captured, complete with precise timing and inter-frame gap control.

Offering a real-time view into a network and the ability to perform a retrospective review of activity is critical to understanding and measuring performance, identifying bottlenecks, troubleshooting issues, and securing the environment. As such, packet capture and analysis continues to play a critical role in managing and securing large and small-scale networks.

New Infrastructure for the Data Superhighway

Modern network fabrics run at 10/40/100 Gbps speeds and the standard methods of performing PCAP are not up to the challenge. This leads to large amounts of dropped packet data and imprecise collections.

To make PCAP perform well at such high speeds, the processing of captured packets must be pushed to the point of ingress. To do this, administrators can leverage hardware acceleration to maintain precise, lossless capture. A combination of open source software deployed on commodity servers and programmable logic can round out a new framework to enable PCAP on high-speed networks.

About the author:

Daniel Joseph Barry is VP Positioning and Chief Evangelist at Napatech and has over 20 years experience in the IT and Telecom industry. Prior to joining Napatech in 2009, Dan Joe was Marketing Director at TPACK, a leading supplier of transport chip solutions to the Telecom sector.  From 2001 to 2005, he was Director of Sales and Business Development at optical component vendor NKT Integration (now Ignis Photonyx) following various positions in product development, business development and product management at Ericsson. Dan Joe joined Ericsson in 1995 from a position in the R&D department of Jutland Telecom (now TDC). He has an MBA and a BSc degree in Electronic Engineering from Trinity College Dublin.