http://www.cnhalo.net/2016/09/13/linux-gro/
- GRO(Generic receive offload): 在napi poll里把小包封装成大包再递交给协议栈
- LRO: GRO的硬件实现(通过网卡的RSC功能)
http://lwn.net/Articles/358910/
- Why not irqbalance?
- 我们知道通过设置/proc/irq//smp_affinity， 可以设置中断的亲和性。 在用户态也有irqbalance来根据系统各cpu的实时情况来设置亲和性，从而达到中断的负载均衡。
- 但是irqbalance虽然能够利用多核计算特性， 但是显而易见的cache利用率非常低效。低端网卡硬件不识别网络流，即它只能识别到这是一个数据包，而不能识别到数据包的元组信息。如果一个数据流的第一个数据包被分发到了CPU1，而第二个数据包分发到了CPU2，那么对于流的公共数据，比如nf_conntrack中记录的东西，CPU cache的利用率就会比较低，cache抖动会比较厉害。对于TCP流而言，可能还会因为TCP串行包并行处理的延迟不确定性导致数据包乱序。因此最直接的想法就是将属于一个流的所有数据包分发了一个CPU上。
- Why RFS(Receive Flow Steering)
- 在使用RPS接收数据包之后，会在指定的CPU进行软中断处理，之后就会在用户态进行处理；如果用户态处理的CPU不在软中断处理的CPU，则会造成CPU cache miss，造成很大的性能影响。RFS能够保证处理软中断和处理应用程序是同一个CPU，这样会保证local cache hit，提升处理效率。RFS需要和RPS一起配合使用。
- 也就是RPS虽然能够利用多核特性，但是如果如果应用程序所在的CPU和RPS选择的CPU不是同一个的话，也会降低cache的利用。
- 因此，RFS是RPS的一个扩展补丁包，在RPS的基础上，解决了以上问题。
- 主要是在应用程序调用系统调用的时候，在一个全局的hash表上，用流的hash值，映射到当前的cpu上。
- 在流的下一个数据包到来的时候，可以查这个全局的hash表。
- out of order
当调度器调度应用程序到另一个cpu上的时候，根据RFS算法，数据包也要发送到这个新的cpu上，这时候过去的包在另一个cpu的软中断中处理。
同一个流的数据包同时发送到两个不同cpu的队列中，就会导致ooo乱序包。
因此RFS引入了另一个per rx队列的rps_flow_table, 具体实现在下文描述，总之如果cpu变更，且原来cpu中还有该flow的数据，就不会把数据包发送到新的cpu队列上。
- Why RPS(Receive Packet Steering)?
RPS distributes the load of received packet processing across multiple CPUs.
考虑以下场景:https://lwn.net/Articles/370153/
一个只有一个接收队列的网卡，收到数据包后中断到一个cpu，这时候就会导致数据包被串行处理。不能利用多核。
因此google提交了RPS的patch，在收到数据包后提交协议栈的时候(netif_receive_skb_internal)，根据/sys/class/net//queues/rx-/rps_cpus的设置，
把这个接收队列收到的数据包发送， 通过hash值分发到设置的cpu集合中。 hash(skb->hash)值一般由网卡根据数据包头部的四元组直接计算, 因此能够让同一个流在同一个cpu上处理
- RSS
- 启用RSS这种功能之后，网卡会有多个接收和发送队列，这些队列对被分配不同的CPU进行处理。RSS为网卡数据传输使用多核提供了支持，RSS在硬件/驱动级别实现多队列并且使用一个hash函数对数据包进行多队列分离处理，这个hash根据源IP、目的IP、源端口和目的端口进行数据包分布选择，这样同一数据流的数据包会被放置到同一个队列进行处理并且能一定程度上保证数据处理的均衡性。
RPS是和RSS类似的一个技术，区别在于RSS是网的硬件实现而RPS是内核软件实现。RPS帮助单队列网卡将其产生的SoftIRQ分派到多个CPU内核进行处理。在这个方案中，为网卡单队列分配的CPU只处理所有硬件中断，由于硬件中断的快速高效，即使在同一个CPU进行处理，影响也是有限的，而耗时的软中断处理会被分派到不同CPU进行处理，可以有效的避免处理瓶颈。
这边 有一片blog：https://blog.csdn.net/dog250/article/details/46705657 谈了一下 自己的看法
https://github.com/torvalds/linux/blame/d8ec26d7f8287f5788a494f56e8814210f0e64be/Documentation/networking/scaling.txt#L160-L164 关于 rfs rps xps rss 的介绍以及使用建议。目前觉得最重要的一条就是下面这一条
2、 For a single queue device, a typical RPS configuration would be to set the rps_cpus to the CPUs in the same memory domain of the interrupting CPU.
If NUMA locality is not an issue, this could also be all CPUs in the system. At high interrupt rate, it might be wise to exclude theinterrupting CPU
from the map since that already performs much work.
For a multi-queue system, if RSS is configured so that a hardware receive queue is mapped to each CPU,
then RPS is probably redundant and unnecessary. If there are fewer hardware queues than CPUs,
then RPS might be beneficial if the rps_cpus for each queue are the ones that share the same memory domain as the
interrupting CPU for that queue.
RPS
Receive Packet Steering (RPS) is similar to RSS in that it is used to direct packets to specific CPUs for processing. However, RPS is implemented at the software level, and helps to prevent the hardware queue of a single network interface card from becoming a bottleneck in network traffic.
RPS has several advantages over hardware-based RSS:
-
RPS can be used with any network interface card.
-
It is easy to add software filters to RPS to deal with new protocols.
-
RPS does not increase the hardware interrupt rate of the network device. However, it does introduce inter-processor interrupts.
RPS is configured per network device and receive queue, in the /sys/class/net/*device*/queues/*rx-queue*/rps_cpus file, where device is the name of the network device (such as eth0) and rx-queue is the name of the appropriate receive queue (such as rx-0).
The default value of the rps_cpus file is zero. This disables RPS, so the CPU that handles the network interrupt also processes the packet.
To enable RPS, configure the appropriate rps_cpus file with the CPUs that should process packets from the specified network device and receive queue.
The rps_cpus files use comma-delimited CPU bitmaps. Therefore, to allow a CPU to handle interrupts for the receive queue on an interface, set the value of their positions in the bitmap to 1. For example, to handle interrupts with CPUs 0, 1, 2, and 3, set the value of rps_cpus to 00001111 (1+2+4+8), or f (the hexadecimal value for 15).
For network devices with single transmit queues, best performance can be achieved by configuring RPS to use CPUs in the same memory domain. On non-NUMA systems, this means that all available CPUs can be used. If the network interrupt rate is extremely high, excluding the CPU that handles network interrupts may also improve performance.
For network devices with multiple queues, there is typically no benefit to configuring both RPS and RSS, as RSS is configured to map a CPU to each receive queue by default. However, RPS may still be beneficial if there are fewer hardware queues than CPUs, and RPS is configured to use CPUs in the same memory domain.
RFS
Receive Flow Steering (RFS) extends RPS behavior to increase the CPU cache hit rate and thereby reduce network latency. Where RPS forwards packets based solely on queue length, RFS uses the RPS backend to calculate the most appropriate CPU, then forwards packets based on the location of the application consuming the packet. This increases CPU cache efficiency.
RFS is disabled by default. To enable RFS, you must edit two files:
/proc/sys/net/core/rps_sock_flow_entries
Set the value of this file to the maximum expected number of concurrently active connections. We recommend a value of 32768 for moderate server loads. All values entered are rounded up to the nearest power of 2 in practice.
/sys/class/net/*device*/queues/*rx-queue*/rps_flow_cnt
Replace device with the name of the network device you wish to configure (for example, eth0), and rx-queue with the receive queue you wish to configure (for example, rx-0).
Set the value of this file to the value of rps_sock_flow_entries divided by N, where N is the number of receive queues on a device. For example, if rps_flow_entries is set to 32768 and there are 16 configured receive queues, rps_flow_cnt should be set to 2048. For single-queue devices, the value of rps_flow_cnt is the same as the value of rps_sock_flow_entries.
Data received from a single sender is not sent to more than one CPU. If the amount of data received from a single sender is greater than a single CPU can handle, configure a larger frame size to reduce the number of interrupts and therefore the amount of processing work for the CPU. Alternatively, consider NIC offload options or faster CPUs.
Consider using numactl or taskset in conjunction with RFS to pin applications to specific cores, sockets, or NUMA nodes. This can help prevent packets from being processed out of order.