Suricata 性能优化

最近在用Stamus做流量分析，suricata丢包比较离谱，优化了下性能就好很多，记一下做了什么。

根据官方文档做了以下事情

我的平台是EPYC 7302P，256G DRAM，Connect-X 4 25G，4 x nvme SSD raid-z。参考文档的AMD部分进行优化。

ethtool和PCI IRQ配置

文档里要开的-K rxhash和ntuple不存在的，怀疑是新版本内核改名了，分别开启了receive-hashing和ntuple-filters

IRQ默认的32个comp IRQ已经摊到所有CPU上了，所有虽然下面写了这部分的逻辑但没调用

硬盘看起来符合持续大量写入的场景，应该适合iopoll模式，所以开启了iopoll(需要配合kmod参数，见下方）

形成脚本：

#!/bin/bash

INTERFACE="${INTERFACE-xxx}"

# NOTE this is only for certain system

# cpu EPYC 7302P, 0,16 1,17 ...
# eth: mlx5, 1 async + 32 comp
# nvme: 4T x 4

irq_opt_eth() {
    # not necessary?
    ETH_PCI_DEVICE="$(realpath /sys/class/net/${INTERFACE}/device | awk -F '/' '{print $NF}')"

    compirqs=($(cat /proc/interrupts | grep -E "mlx5_comp[0-9]+@pci:${ETH_PCI_DEVICE}" | awk -F ':' '{print $1}'))
    asyncirq="$(cat /proc/interrupts | grep -E "mlx5_async0@pci:${ETH_PCI_DEVICE}" | awk -F ':' '{print $1}' | xargs)"

    echo "set core 0-31 for ${ETH_PCI_DEVICE} async irq ${asyncirq}"
    echo "0-31" > /proc/irq/${asyncirq}/smp_affinity_list

    for i in "${!compirqs[@]}"; do
        echo "set core ${i} for ${ETH_PCI_DEVICE} comp irq ${compirqs[$i]}"
        echo "$i" > /proc/irq/${compirqs[$i]}/smp_affinity_list
    done
}

irq_opt_nvme() {
    # 用不了一点
    core=0
    for dir in /sys/block/nvme*
    do
        # # change 4T disks only
        # if [ "$(cat "$dir/size")" -lt 7001573552 ]; then
        #     continue
        # fi
        pcidevice="$(realpath "$dir/device/device" | awk -F '/' '{print $NF}')"
        # TiPro have 1 + 16 irqs
        irqs=($(cat /proc/interrupts | grep -E "IR-PCI-MSIX-${pcidevice}" | awk -F ':' '{print $1}'))
        # echo "set core 0-31 for ${pcidevice} nvme disk"
        echo "0-31" > /proc/irq/${irqs[0]}/smp_affinity_list
        for irq in "${irqs[@]:1}"; do
            # echo "set core ${core} for ${pcidevice} nvme disk irq ${irq}"
            chmod 0600 /proc/irq/${irq}/smp_affinity_list
            echo "${core}" > /proc/irq/${irq}/smp_affinity_list
            core=$((core + 1))
            if [ $core -ge 32 ]; then
                core=0
            fi
        done
    done
}

poll_nvme() {
    for dir in /sys/block/nvme*
    do
        # change 4T disks only
        if [ "$(cat "$dir/size")" -lt 7001573552 ]; then
            continue
        fi

        echo 1 > "$dir/queue/io_poll"
    done
}

ethtool_opt() {
    # queues maximum
    ethtool -L "${INTERFACE}" combined 32
    # offload options
    ethtool -K "${INTERFACE}" \
        ntuple-filters on \
        rx-vlan-offload off \
        receive-hashing on \
        rx-vlan-filter off \
        rx-udp_tunnel-port-offload off
    # hash
    ethtool -X "${INTERFACE}" \
        hfunc toeplitz \
        hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A

    # set rx-flow-hash
    for proto in tcp4 udp4 tcp6 udp6; do
        ethtool -N "${INTERFACE}" rx-flow-hash $proto sdfn
    done
}

poll_nvme
ethtool_opt

以上脚本放到sbin，通过service启动：

[Unit]
After=network.target
Wants=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/suricata_optimize.sh
User=root

[Install]
WantedBy=multi-user.target

给docker（我是在docker里跑的）加上依赖

### /etc/systemd/system/docker.service.d/override.conf
[Unit]
After=network-online.target nss-lookup.target docker.socket firewalld.service containerd.service time-set.target zfs.target suricata_optimize.service

kmod参数和sysctl

nvme要开polling，所以

# /etc/modprobe.d/nvme.conf
options nvme poll_queues=32

记得mkinitcpio -P

然后设置sysctl（具体有没用不太清楚

# /etc/sysctl.d/99-suricata-net-opts.conf
net.core.rmem_max=268435456
net.core.netdev_max_backlog=4096

suricata配置

改suricata配置的时候发现了个坑，需要修改suricata容器的ulimit

diff --git a/docker/compose.yml b/docker/compose.yml
index 7916eda..3bcd918 100644
--- a/docker/compose.yml
+++ b/docker/compose.yml
@@ -119,7 +119,10 @@ services:
             - suricata-run:/var/run/suricata/
             - ${PWD}/containers-data/suricata/etc:/etc/suricata
             - suricata-logrotate:/etc/logrotate.d/
-
+        ulimits:
+            memlock:
+                soft: -1
+                hard: -1
     scirius:
         container_name: scirius
         image: ghcr.io/stamusnetworks/scirius:${SCIRIUS_VERSION:-selks}

配置

• 据文档说服务器网卡这个好：af-packet.cluster-type: cluster_flow -> cluster_qm
• af-packet.defrag: yes -> no
• af-packet.mmap-locked: (commented) -> yes
• af-packet.tpacket-v3: (commented) -> yes
• af-packet.ring-size: (commented) -> 163840 这个设大了又没有改memlock就会炸，记得改ulimit
• af-packet.block-size: (commented) -> 1048576
• max-pending-packets: (commented) -> 8192
• threading.autopin: yes

效果

改之前丢包率高达30%（离谱） 改完后低于1%（数量级在0.1%左右）