[tldk][dpdk][dev] TLDK--基于dpdk的用户态协议栈传输层组件简单调研

如题，以下是一份简单的快速调研。

TLDK: Transport Layer Development Kit

一什么是TLDK

transport layer development kit

处理tcp/udp的库

不提供socket api
构建在dpdk的上层。
专注在tcp, udp不做完整的协议栈。
没有arp, ip, eth, 路由表等。

VPP的一个节点

VPP协议栈的组成部分。

非VPP框架也能用

专门补充了一下功能，用来支持非VPP环境。netlink agents, packaging等。

二 TLDK的背景

tldk是VPP的一个graph node，VPP是fd.io项目的核心技术，由思科提交。fd.io是托管在linux基金会的项目。
tldk应该与vpp有相同的license。

什么是fd.io

FD.io (Fast data – Input/Output) is a collection of several projects and libraries to amplify the transformation that began with Data Plane Development Kit (DPDK) to support flexible, programmable and composable services on a generic hardware platform. FD.io offers the Software Defined Infrastructure developer community a landing site with multiple projects fostering innovations in software-based packet processing towards the creation of high-throughput, low-latency and resource-efficient IO services suitable to many architectures (x86, ARM, and PowerPC) and deployment environments (bare metal, VM, container).

什么是vpp

A key component is the Vector Packet Processing (VPP) library donated by Cisco. This code is already running in products on the market today. The VPP library is highly modular, allowing for new graph nodes to be easily “plugged in” without changes to the underlying code base. This gives developers the potential to easily build any number of packet processing solutions with varying forwarding graphs.

In development since 2002, VPP is production code currently running in shipping products. It runs in user space on multiple architectures including x86, ARM, and Power architectures on both x86 servers and embedded devices. The design of VPP is hardware, kernel, and deployment (bare metal, VM, container) agnostic. It runs completely in userspace.

资料

扫盲概述：https://fd.io/about/
什么是VPP: https://wiki.fd.io/view/VPP/What_is_VPP%3F
VPP的基本架构：https://fd.io/technology/
tldk主页：https://wiki.fd.io/view/TLDK
tldk代码页：https://github.com/FDio/tldk
IRC Channel: #fdio, #fdio-tldk

三编译

kernel

version: 3.10.0-327.36.3.el7.x86_64

dpdk

version：tag v18.02-rc4

tldk

version: tag v18.02

代码结构

   $(TLDK_ROOT)
   |
   +----app
   |      |
   |      +-- nginx - a clone of nginx integrated with TLDK
   |          (refer to app/nginx/README.TLDK for more information)
   |
   +----lib 
   |      |
   |      +--libtle_dring - dring library
   |      |
   |      +--libtle_l4p - implementation of the TCP/UDP packet processing
   |      |
   |      +--libtle_timer - implementation of the timer library
   |
   +----examples
   |      |
   |      +--l4fwd - sample app to demonstrate and test libtle_l4p TCP/UDP
   |            usage (refer to examples/l4fwd/README for more information)
   |
   +----test - unit-tests
   |      |
   |      +--dring - UT for libtle_dring (standalone app)
   |      | 
   |      +--gtest - UT for libtle_dring, libtle_l4p and libtle_timer
   |      |     (googletest)
   |      |
   |      +--timer - UT for libtle_timer (standalone app)

步骤

export RTE_SDK=/root/Src/thirdparty/dpdk.git
export RTE_TARGET=x86_64-native-linuxapp-gcc

[root@T9 tldk.git]# make all

四接口

接口文件

tle_ctx.h
tle_dpdk_wrapper.h
tle_dring.h
tle_event.h
tle_tcp.h
tle_timer.h
tle_udp.h
tle_version.h

接口文档

doc/l4_api_desc.txt

主要接口

模型

数据结构

ctx
 |
 |-----> stream
 |-----> stream
 |          |
 |          |-------------->  属性 endpoint(addr, port)  # 类型 socket
 |          |
 |          `-------------->  方法 open/send/recv/close
 |
 |-----> dev  (网卡的抽象)
 |
 `-----> dev

初始化API

tle_ctx_create()
tle_ctx_destroy()
tle_add_dev()
tle_del_dev()

recv/send机制

类型：

callback
event

API：

tle_tcp_stream_recv()
tle_tcp_stream_send()
tle_tcp_readv()
tle_tcp_writev()

tle_udp_stream_recv()
tle_udp_stream_send()

BackEnd API

tle_tcp_rx_bulk()
tle_tcp_tx_bulk()
tle_tcp_process()

tle_udp_rx_bulk()
tle_udp_tx_bulk()

FrontEnd API

tle_tcp_stream_open(ctx, send_callback/event, recv_callback/event)
tle_tcp_stream_close()
tle_tcp_stream_abort() /* 重置tcp状态，不释放资源。 */

tle_udp_stream_open()
tle_udp_stream_close()

server side API

tle_tcp_stream_listen()
tle_tcp_stream_synreqs()
tle_tcp_stream_accept()

client side API

tle_tcp_stream_connect()

五例子程序

l4fwd

+----------------------------+                +-------------------------------+
|                   TLDK Box |                | Linux Box                     |
|                            |                |                               |
|                     port 0 +----------------+ port 0                        |
|                192.168.1.1 |                | 192.168.1.2                   |
|          2001:4860:b002::1 |                | 2001:4860:b002::2             |
|          AA:BB:CC:DD:EE:F1 |                | AA:BB:CC:DD:EE:F2             |
+----------------------------+                +-------------------------------+

运行方法

加载dpdk uio驱动，并绑定好网卡。
写配置文件

[root@D129 app]# cat b.conf 
port=0,addr=192.168.8.0,masklen=24,mac=82:8c:82:22:33:a5
[root@D129 app]# cat f.conf 
lcore=1,op=echo,laddr=192.168.8.100,lport=8088,raddr=0.0.0.0,rport=0

运行

# server
[root@D129 app]# ./l4fwd -l 0,1 -w 0000:00:04.0 -- -P -T -L -R 256 -S 256 -s 256 -b b.conf  -f f.conf -a  port=0,lcore=1,ipv4=192.168.8.100

# client
sudo arp -s 192.168.8.100 00:00:00:01:00:02

参数解读：
在我的虚拟机里，必须要显示指定-w
-R/-S 影响window大小。
-P/-a 可以不设置。

运行结果演示
client端的echo效果

client设备出口抓包

代码分析

nginx

TODO

六性能

TODO

七 VPP场景

TODO

八实现分析

TODO

九其他

约1w行代码。

┬─[tong@T7:~/Src/thirdparty/tldk.git]─[03:36:35 PM]
╰─>$ find ./lib/ -name '*.c' -exec cat {} ; |wc -l
5549
┬─[tong@T7:~/Src/thirdparty/tldk.git]─[03:36:38 PM]
╰─>$ find ./lib/ -name '*.h' -exec cat {} ; |wc -l
5480