From a413cb9b642e7b4e0c865030254c113f4c118b94 Mon Sep 17 00:00:00 2001 From: Le Yao Date: Tue, 14 Dec 2021 04:32:58 +0000 Subject: [PATCH] Enable QAT and document for it. Enable QAT with LKCF mode in openwrt. Guide for IPSec accelerated by QAT. Change-Id: I8f2d5d0a3e2dce9fafde65bab8055201efa24418 Signed-off-by: Le Yao --- .../cnf-openwrt/doc/Accelerated_by_Intel_QAT.md | 494 +++++++++++++++++++++ platform/cnf-openwrt/src/Dockerfile_sdewan.tpl | 3 +- .../cnf-openwrt/src/Dockerfile_sdewan_noproxy.tpl | 3 +- platform/cnf-openwrt/src/strongswan.conf | 36 ++ 4 files changed, 534 insertions(+), 2 deletions(-) create mode 100644 platform/cnf-openwrt/doc/Accelerated_by_Intel_QAT.md create mode 100644 platform/cnf-openwrt/src/strongswan.conf diff --git a/platform/cnf-openwrt/doc/Accelerated_by_Intel_QAT.md b/platform/cnf-openwrt/doc/Accelerated_by_Intel_QAT.md new file mode 100644 index 0000000..2c86ecc --- /dev/null +++ b/platform/cnf-openwrt/doc/Accelerated_by_Intel_QAT.md @@ -0,0 +1,494 @@ +Your distribution may already have the Intel® QAT drivers, these instructions guide the user on how to download the kernel sources, compile kernel driver modules against those sources, and load them onto the host. + +* [Introduction](#introduction) +* [Requirements](#requirements) + * [Hardware](#hardware) + * [Software](#software) +* [Quick Start](#quick-start) +* [SDEWAN CNF setup](#sdewan-cnf-setup) + +## Introduction + +Intel® QuickAssist Technology (Intel® QAT) is developed by Intel® and runs on Intel® architecture providing security and compression acceleration capabilities to improve performance and efficiency. It offloads workloads from the CPU to hardware. Server, networking, big data, and storage applications use Intel® QAT to offload compute-intensive operations, such as: + +- Symmetric cryptography functions, including cipher operations and authentication operations. +- Public key functions, including RSA, Diffie-Hellman, and elliptic curve cryptography. +- Compression and decompression functions, including DEFLATE. + +Intel® QAT has improved the function of in many areas, such as Hadoop* acceleration, OpenSSL integration, SDN and NFV Solutions Boost and so on. On a more detailed level, we can get the benefits from the following aspects: + +- 4G LTE and 5G encryption algorithm offload for mobile gateways and infrastructure. +- VPN traffic acceleration, with up to 50 Gbps crypto throughput and support for IPsec and SSL acceleration. +- Compression/decompression, with up to 24 Gbps throughput. +- I/O virtualization using PCI-SIG Single-Root I/O Virtualization (SR-IOV). + +--- + +## Requirements + +### Hardware + +- Intel® QAT device on your machine with the following forms: + - Chipset: Intel® C6xx series chipset + - PCIE: Intel® QuickAssist Adapter 89xx + - SoC: Intel Atom® C3000 processor series (Denverton NS) / Rangeley +- Ensure SR-IOV and VT-d are enabled in BIOS + +### Software + +- Enable IOMMU and reboot + + Enable IOMMU support by setting the correct [kernel parameter](https://wiki.archlinux.org/index.php/Kernel_parameter) depending on the type of CPU in use: + + - For Intel® CPUs (VT-d) set `intel_iommu=on` + - For AMD CPUs (AMD-Vi) set `amd_iommu=on` + + You should also append the `iommu=pt` parameter. This will prevent Linux from touching devices which cannot be passed through. + + > Edit the file /etc/default/grub and change the `GRUB_CMDLINE_LINUX` + + ```shell + vim /etc/default/grub + # Change and save + ... + GRUB_CMDLINE_LINUX="iommu=pt, intel_iommu=on, apparmor=0" + ... + + sudo grub2-mkconfig -o /boot/grub2/grub.cfg + sudo reboot + + # We can check it + dmesg | grep -i iommu + [ 0.000000] DMAR: IOMMU enabled + ``` + +- Install needed software package + + ```shell + # For CentOS + yum -y groupinstall "Development Tools" + yum -y install pciutils + yum -y install libudev-devel + yum -y install kernel-devel-$(uname -r) + yum -y install gcc + yum -y install openssl-devel + + # For Ubuntu + apt-get update + apt-get install pciutils-dev + + # For Fedora + dnf groupinstall "Development Tools" + dnf install gcc-c++ + dnf install systemd-devel + dnf install kernel-devel-`uname -r` + dnf install elfutils-devel + dnf install openssl-devel + ``` + +--- + +## Quick Start + +Notice that the `linux kernel` we used should be version >=4.16. Because the [bug](https://patchwork.ozlabs.org/project/netdev/patch/2f86c9d7c39cfad21fdb353a183b12651fc5efe9.1583311902.git.lucien.xin@gmail.com/) before v4.15 and fixed [here]( https://elixir.bootlin.com/linux/v4.16-rc1/source/net/xfrm/xfrm_device.c). + +1. Get the Intel® QAT driver from 01.org + + ```shell + mkdir qat_driver + cd qat_driver + wget https://downloadmirror.intel.com/649693/QAT.L.4.15.0-00011.tar.gz + tar zxvf QAT.L.4.15.0-00011.tar.gz + ``` + +2. Configure the Intel® QAT driver and enable LKCF + + ```shell + # You can list the configuration by this command + ./configure -h + # We need enable the Intel® QAT sriov + ./configure --enable-icp-sriov=host --enable-qat-lkcf + ``` + +3. Build driver and install driver and samples + + ```shell + make -j + make install + make samples-install + ``` + +4. Check the driver module + + ```shell + lsmod | grep qa + qat_c62xvf 16384 0 + qat_c62x 20480 0 + intel_qat 212992 3 qat_c62x,usdm_drv,qat_c62xvf + uio 20480 1 intel_qat + ``` + +5. Configure the QAT device + + SDEWAN CNF will use QAT PFs/VFs bound in kernel driver, we should configure them with kernel section enabled in file like `/etc/c6xx_dev*.conf` for QAT PFs and `/etc/c6xxvf_dev*.conf` for QAT VFs. You also can replace them with the following examples. + + ```shell + # An example for PF configuration + ################################################################ + # This file is provided under a dual BSD/GPLv2 license. When using or + # redistributing this file, you may do so under either license. + # + # GPL LICENSE SUMMARY + # + # Copyright(c) 2007-2021 Intel Corporation. All rights reserved. + # + # This program is free software; you can redistribute it and/or modify + # it under the terms of version 2 of the GNU General Public License as + # published by the Free Software Foundation. + # + # This program is distributed in the hope that it will be useful, but + # WITHOUT ANY WARRANTY; without even the implied warranty of + # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + # General Public License for more details. + # + # You should have received a copy of the GNU General Public License + # along with this program; if not, write to the Free Software + # Foundation, Inc., 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA. + # The full GNU General Public License is included in this distribution + # in the file called LICENSE.GPL. + # + # Contact Information: + # Intel Corporation + # + # BSD LICENSE + # + # Copyright(c) 2007-2021 Intel Corporation. All rights reserved. + # All rights reserved. + # + # Redistribution and use in source and binary forms, with or without + # modification, are permitted provided that the following conditions + # are met: + # + # * Redistributions of source code must retain the above copyright + # notice, this list of conditions and the following disclaimer. + # * Redistributions in binary form must reproduce the above copyright + # notice, this list of conditions and the following disclaimer in + # the documentation and/or other materials provided with the + # distribution. + # * Neither the name of Intel Corporation nor the names of its + # contributors may be used to endorse or promote products derived + # from this software without specific prior written permission. + # + # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + # + # + # version: QAT.L.4.15.0-00011 + ################################################################ + [GENERAL] + ServicesEnabled = cy;dc + + # Set the service profile to determine available features + # ===================================================================== + # DEFAULT CRYPTO COMPRESSION CUSTOM1 + # Asymmetric Crypto * * * + # Symmetric Crypto * * * + # Hash * * * * + # Cipher * * * + # MGF KeyGen * * + # SSL/TLS KeyGen * * * + # HKDF * * + # Compression * * * + # Decompression (stateless) * * * + # Decompression (stateful) * * + # Service Chaining * + # Device Utilization * * * + # Rate Limiting * * * + # ===================================================================== + ServicesProfile = DEFAULT + + ConfigVersion = 2 + + #Default values for number of concurrent requests*/ + CyNumConcurrentSymRequests = 512 + CyNumConcurrentAsymRequests = 64 + + #Statistics, valid values: 1,0 + statsGeneral = 1 + statsDh = 1 + statsDrbg = 1 + statsDsa = 1 + statsEcc = 1 + statsKeyGen = 1 + statsDc = 1 + statsLn = 1 + statsPrime = 1 + statsRsa = 1 + statsSym = 1 + + + # Specify size of intermediate buffers for which to + # allocate on-chip buffers. Legal values are 32 and + # 64 (default is 64). Specify 32 to optimize for + # compressing buffers <=32KB in size. + DcIntermediateBufferSizeInKB = 64 + + # This flag is to enable device auto reset on heartbeat error + AutoResetOnError = 0 + + ############################################## + # Kernel Instances Section + ############################################## + [KERNEL] + NumberCyInstances = 1 + NumberDcInstances = 0 + + # Crypto - Kernel instance #0 + Cy0Name = "IPSec0" + Cy0IsPolled = 0 + Cy0CoreAffinity = 0 + + ############################################## + # User Process Instance Section + ############################################## + [SSL] + NumberCyInstances = 6 + NumberDcInstances = 2 + NumProcesses = 1 + LimitDevAccess = 0 + + # Crypto - User instance #0 + Cy0Name = "SSL0" + Cy0IsPolled = 1 + # List of core affinities + Cy0CoreAffinity = 0 + + # Crypto - User instance #1 + Cy1Name = "SSL1" + Cy1IsPolled = 1 + # List of core affinities + Cy1CoreAffinity = 1 + + # Crypto - User instance #2 + Cy2Name = "SSL2" + Cy2IsPolled = 1 + # List of core affinities + Cy2CoreAffinity = 2 + + # Crypto - User instance #3 + Cy3Name = "SSL3" + Cy3IsPolled = 1 + # List of core affinities + Cy3CoreAffinity = 3 + + # Crypto - User instance #4 + Cy4Name = "SSL4" + Cy4IsPolled = 1 + # List of core affinities + Cy4CoreAffinity = 4 + + # Crypto - User instance #5 + Cy5Name = "SSL5" + Cy5IsPolled = 1 + # List of core affinities + Cy5CoreAffinity = 5 + + + # Data Compression - User instance #0 + Dc0Name = "Dc0" + Dc0IsPolled = 1 + # List of core affinities + Dc0CoreAffinity = 0 + + # Data Compression - User instance #1 + Dc1Name = "Dc1" + Dc1IsPolled = 1 + # List of core affinities + Dc1CoreAffinity = 1 + ``` + + **Changes compared with default configuration** + + ```shell + # An example for VF configuration + ################################################################ + # This file is provided under a dual BSD/GPLv2 license. When using or + # redistributing this file, you may do so under either license. + # + # GPL LICENSE SUMMARY + # + # Copyright(c) 2007-2021 Intel Corporation. All rights reserved. + # + # This program is free software; you can redistribute it and/or modify + # it under the terms of version 2 of the GNU General Public License as + # published by the Free Software Foundation. + # + # This program is distributed in the hope that it will be useful, but + # WITHOUT ANY WARRANTY; without even the implied warranty of + # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + # General Public License for more details. + # + # You should have received a copy of the GNU General Public License + # along with this program; if not, write to the Free Software + # Foundation, Inc., 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA. + # The full GNU General Public License is included in this distribution + # in the file called LICENSE.GPL. + # + # Contact Information: + # Intel Corporation + # + # BSD LICENSE + # + # Copyright(c) 2007-2021 Intel Corporation. All rights reserved. + # All rights reserved. + # + # Redistribution and use in source and binary forms, with or without + # modification, are permitted provided that the following conditions + # are met: + # + # * Redistributions of source code must retain the above copyright + # notice, this list of conditions and the following disclaimer. + # * Redistributions in binary form must reproduce the above copyright + # notice, this list of conditions and the following disclaimer in + # the documentation and/or other materials provided with the + # distribution. + # * Neither the name of Intel Corporation nor the names of its + # contributors may be used to endorse or promote products derived + # from this software without specific prior written permission. + # + # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + # + # + # version: QAT.L.4.15.0-00011 + ################################################################ + [GENERAL] + ServicesEnabled = cy;dc + + ConfigVersion = 2 + + #Default values for number of concurrent requests*/ + CyNumConcurrentSymRequests = 512 + CyNumConcurrentAsymRequests = 64 + + #Statistics, valid values: 1,0 + statsGeneral = 1 + statsDh = 1 + statsDrbg = 1 + statsDsa = 1 + statsEcc = 1 + statsKeyGen = 1 + statsDc = 1 + statsLn = 1 + statsPrime = 1 + statsRsa = 1 + statsSym = 1 + + ############################################## + # Kernel Instances Section + ############################################## + [KERNEL] + NumberCyInstances = 1 + NumberDcInstances = 0 + + # Crypto - Kernel instance #0 + Cy0Name = "IPSec0" + Cy0IsPolled = 0 + Cy0CoreAffinity = 0 + ``` + +6. Start QAT service + + ```shell + # Start QAT and QAT VFs service + service qat_service restart + service qat_service_vfs restart + # Check the devices status + adf_ctl status + ``` + +Then you can utilize the Intel® QAT VFs from your privileged docker container. + +--- + +## SDEWAN CNF setup + +We recommend you to deploy the SDEWAN CNF using the `sdewan_cnf` helm chart part in this [guide](https://github.com/akraino-edge-stack/icn-sdwan/blob/master/platform/deployment/README.md). And suppose you have successfully deployed two CNFs and the network between them is accessible. + +Note: We have two pre-build images `sdewan-cnf:qat` and `sdewan-cnf:qat-test`, you can use them to deploy you strongswan ipsec tunnel accelerated by your host Intel® QAT devices, what you should do is only replace the image field in Helm chart value. `qat-test` tag is used for performance testing. Of course, you can configure your custemer images based on `openwrt` to enbale QAT accelerating workload as the following guide. + +```shell +# We use `kubectl exec` to login to the CNF container and execute the following command +# under root to enable QAT and load testing tools + +opkg update +opkg install vim +opkg install iperf3 +opkg install pciutils +opkg install tcpdump +opkg install strongswan-mod-af-alg + +# Remove the alogrithm plugin not in linux kernel crypto framework +cd /etc/strongswan.d/charon +rm aes.conf des.conf sha1.conf sha2.conf fips-prf.conf md5.conf +``` + +Then we can configure the ipsec configuration and secrets for the tunnel established. + +```yaml +# Add the following section to each container under `/etc/ipsec.conf`, note that they +# would have an opposite value in left* and right* +conn con + authby=secret + auto=add + type=tunnel + leftid=l + left= + rightid=r + right= + ike=aes256-sha1-modp2048 + keyexchange=ikev2 + ikelifetime=28800s + esp=aes256-sha1-modp1536 +``` + +and add the following simple key to `/etc/ipsec.secrets` + +```yaml +l r : PSK 'AAABBCCDD' +``` + +Then we can start `ipsec` and load the connection to test the performance + +```shell +ipsec restart +ipsec up con + +# On one CNF, we start a `iperf3` server to receive the packages +iperfs -s +# On another CNF, we use `iperf3` as a client to send packages to server +# TCP +iperf3 -c -b 10g +# UDP +iperf3 -c -l 64k -b 10g -u + +``` + diff --git a/platform/cnf-openwrt/src/Dockerfile_sdewan.tpl b/platform/cnf-openwrt/src/Dockerfile_sdewan.tpl index e6f2c5c..5d6b7e2 100644 --- a/platform/cnf-openwrt/src/Dockerfile_sdewan.tpl +++ b/platform/cnf-openwrt/src/Dockerfile_sdewan.tpl @@ -17,9 +17,10 @@ RUN mkdir /var/lock && \ uci commit uhttpd && \ opkg install shadow-useradd shadow-groupadd shadow-usermod && \ opkg install mwan3 jq bash conntrack && \ - opkg install strongswan-default luasocket && \ + opkg install strongswan-default luasocket strongswan-mod-af-alg && \ opkg install luci-app-mwan3; exit 0 +COPY strongswan.conf /etc/strongswan.conf COPY system /etc/config/system COPY ipsec /etc/config/ipsec COPY ipsec_exec /etc/init.d/ipsec diff --git a/platform/cnf-openwrt/src/Dockerfile_sdewan_noproxy.tpl b/platform/cnf-openwrt/src/Dockerfile_sdewan_noproxy.tpl index 85d4582..c490692 100644 --- a/platform/cnf-openwrt/src/Dockerfile_sdewan_noproxy.tpl +++ b/platform/cnf-openwrt/src/Dockerfile_sdewan_noproxy.tpl @@ -14,9 +14,10 @@ RUN mkdir /var/lock && \ uci commit uhttpd && \ opkg install shadow-useradd shadow-groupadd shadow-usermod && \ opkg install mwan3 jq bash conntrack && \ - opkg install strongswan-default luasocket && \ + opkg install strongswan-default luasocket strongswan-mod-af-alg && \ opkg install luci-app-mwan3; exit 0 +COPY strongswan.conf /etc/strongswan.conf COPY system /etc/config/system COPY ipsec /etc/config/ipsec COPY ipsec_exec /etc/init.d/ipsec diff --git a/platform/cnf-openwrt/src/strongswan.conf b/platform/cnf-openwrt/src/strongswan.conf new file mode 100644 index 0000000..9e9bcfd --- /dev/null +++ b/platform/cnf-openwrt/src/strongswan.conf @@ -0,0 +1,36 @@ +# strongswan.conf - strongSwan configuration file +# +# Refer to the strongswan.conf(5) manpage for details +# +# Configuration changes should be made in the included files + +charon { + load_modular = yes + plugins { + include strongswan.d/charon/af-alg.conf + include strongswan.d/charon/attr.conf + include strongswan.d/charon/connmark.conf + include strongswan.d/charon/constraints.conf + include strongswan.d/charon/dnskey.conf + include strongswan.d/charon/gmp.conf + include strongswan.d/charon/kernel-netlink.conf + include strongswan.d/charon/nonce.conf + include strongswan.d/charon/pem.conf + include strongswan.d/charon/pgp.conf + include strongswan.d/charon/pkcs1.conf + include strongswan.d/charon/pubkey.conf + include strongswan.d/charon/random.conf + include strongswan.d/charon/rc2.conf + include strongswan.d/charon/resolve.conf + include strongswan.d/charon/revocation.conf + include strongswan.d/charon/socket-default.conf + include strongswan.d/charon/sshkey.conf + include strongswan.d/charon/stroke.conf + include strongswan.d/charon/updown.conf + include strongswan.d/charon/x509.conf + include strongswan.d/charon/xauth-generic.conf + include strongswan.d/charon/xcbc.conf + } +} + +include strongswan.d/*.conf -- 2.16.6