一个linux内核模块移植到低版本时发生的异常

在3.10的内核版本下，有一个运行稳定的内核模块，移植到suse11的时候，编译正常，运行则直接出现crash：

<4>[  503.347297] CPU 0
<4>[  503.347300] Modules linked in: caq_sendmsg(EN) mysendmsg(EN) witdriver(EN) datalink(EN) w83627dhg(EN) tipc(EX) ossmod(EN) pagecachelimit(EN) xfs ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables x_tables ipmi_devintf ipmi_si ipmi_msghandler edd cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf af_packet bonding fuse loop dm_mod vhost_net macvtap ipv6 ipv6_lib macvlan tun kvm_intel kvm pcspkr ses enclosure usbhid hid i40e(EX) sg igb i2c_i801 iTCO_wdt iTCO_vendor_support mei dca mptctl ptp pps_core mptbase rtc_cmos acpi_power_meter container button ext3 jbd mbcache ttm drm_kms_helper drm i2c_algo_bit sysimgblt sysfillrect i2c_core syscopyarea ehci_hcd usbcore usb_common sd_mod crc_t10dif processor thermal_sys hwmon scsi_dh_hp_sw scsi_dh_alua scsi_dh_rdac scsi_dh_emc scsi_dh mpt3sas(EX) configfs scsi_transport_sas raid_class scsi_mod
<4>[  503.347404] Supported: No, Unsupported modules are loaded
<4>[  503.347408]
<4>[  503.347413] Pid: 30269, comm: 00-IFileSender Tainted: G           ENX 3.0.101-0.47.90-default #1 ZTE Grantley/S1008
<4>[  503.347422] RIP: 0010:[<ffffffff813e9830>]  [<ffffffff813e9830>] get_page+0x0/0x30
<4>[  503.347434] RSP: 0018:ffff88334a88b4e0  EFLAGS: 00010246
<4>[  503.347438] RAX: ffffffff81a77be0 RBX: ffff88198c904b80 RCX: ffff881f76b3b3f0
<4>[  503.347443] RDX: 0000000000001000 RSI: 0000000000003ebc RDI: 0020000000000000
<4>[  503.347449] RBP: ffff881f76b3b300 R08: 0000000000000000 R09: 0020000000000000
<4>[  503.347453] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
<4>[  503.347458] R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000000
<4>[  503.347464] FS:  00007f3678a69700(0000) GS:ffff88207fc00000(0000) knlGS:0000000000000000
<4>[  503.347470] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[  503.347474] CR2: 00007f3659bcceac CR3: 00000034bfc1e000 CR4: 00000000001407f0
<4>[  503.347479] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[  503.347484] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>[  503.347490] Process 00-IFileSender (pid: 30269, threadinfo ffff88334a88a000, task ffff88334a888540)
<0>[  503.347495] Stack:
<4>[  503.347498]  ffffffff813eb098 000000004a88b5d8 ffff882000000000 0020000000000000
<4>[  503.347510]  ffff883300000000 000000007ffd9e00 0000000000000000 0000000000000004
<4>[  503.347519]  000000004a888e50 ffffea005c719dd8 000005b4000000d0 ffff88198c904c88
<0>[  503.347528] Call Trace:
<4>[  503.347541]  [<ffffffff813eb098>] do_tcp_sendpages+0x438/0x530
<4>[  503.347556]  [<ffffffffa06ed4cd>] caq_sendmsg_for_file+0x20d/0x480 [caq_sendmsg]
<4>[  503.347583]  [<ffffffffa06ee574>] sendmsg_for_file+0x154/0x220 [caq_sendmsg]
<4>[  503.347594]  [<ffffffffa06ee801>] my_sys_sendmsg+0x1c1/0x208 [caq_sendmsg]
<4>[  503.347608]  [<ffffffff8146f5f2>] system_call_fastpath+0x16/0x1b
<4>[  503.347621]  [<00007f36ad754e4d>] 0x7f36ad754e4c

我们修改了一个函数：

tcp_sendpage，这个函数会在lock sock的情况下，再调用do_tcp_sendpage,suse11的源代码如下：

int tcp_sendpage(struct sock *sk, struct page *page, int offset,
         size_t size, int flags)
{
    ssize_t res;

    if (!(sk->sk_route_caps & NETIF_F_SG) ||
        !(sk->sk_route_caps & NETIF_F_ALL_CSUM))
        return sock_no_sendpage(sk->sk_socket, page, offset, size,
                    flags);

    lock_sock(sk);
    res = do_tcp_sendpages(sk, &page, offset, size, flags);
    release_sock(sk);
    return res;
}

而对应的3.10版本，则是如下：

int tcp_sendpage(struct sock *sk, struct page *page, int offset,
         size_t size, int flags)
{
    ssize_t res;

    if (!(sk->sk_route_caps & NETIF_F_SG) ||
        !(sk->sk_route_caps & NETIF_F_CSUM_MASK))
        return sock_no_sendpage(sk->sk_socket, page, offset, size,
                    flags);

    lock_sock(sk);
    res = do_tcp_sendpages(sk, page, offset, size, flags);
    release_sock(sk);
    return res;
}

一开始从crash排查，发现传入的page是NULL指针，然后开始在入参之前检查，发现bug_on没有触发。

最后通过merge代码，发现suse11的do_tcp_sendpages 和cgslv5的 do_tcp_sendpages 的定义不一样，第二个参数，高版本的不需要取地址，低版本需要取page的地址。

为了这么低级的bug查了几个小时，哎。

总结：

 不同内核版本移植模块的时候，对于修改的代码，最好先对比下函数，防止犯低级错误。