容器技术
Buildx 构建带 WAF 的 Openresty 多架构镜像
本博客介绍利用 docker buildx 源码构建跨平台带 waf 的 Openresty 镜像，重点介绍 buildx 在采用本地 qemu 方案编译时遇到的各种问题。
资源环境
资源列表如下：
| 资源 | 版本 | 其他说明 |
|---|---|---|
base image |
bitnami/minideb:bullseye | debian11 <=> ubuntu20.04 |
openresty |
1.21.4.2 | 当前最新 |
ModSecurity |
v3.0.10 | 当前最新 |
Registry |
harbor v2.5.0 |
< v2.7 ,不支持 attestations参数, 见Feature
[1]
|
qemu |
< 0.7 |
版本略低 |
centos 7 |
5.18.4 (x86_64) |
- |
| GCC | 4.8.5 |
版本过低 |
Dockerfile 编写
- 基础镜像选择社区稳定的
bitnami/minideb:bullseye，与社区 openresty 编译保持一致 - 所有源码均选择当前最新，以后可能会过期，慎重使用
- 国内
git下载源码容易失败，建议改为提前下载，COPY进去 WAF相关内容统一放置在/usr/local/openresty/nginx/modsec可以覆盖- 默认选择第三方插件静态编译，如果需要灵活性，可以选择动态编译，以便仅更新
waf模块，但我觉得容器不适合该方案 - 编译消耗大量资源，不建议本地
quem编译，效率太差，推荐使用远程节点，时间上能节省不少 - 第二阶段为了保证运行镜像 size 尽可能小 （最终size=175M），只安装了最终二进制文件检测到的缺失库，如果不介意镜像过大可以考虑将编译依赖都装进去
Dockerfile 内容如下:
# syntax = docker/dockerfile:1.4.0
FROM bitnami/minideb:bullseye AS builder
# ARG VERSION
ENV VERSION="1.21.4.2"
ENV BUILD_HOME=/opt
ENV OPENRESTY_INSTALL_DIR=/usr/local/openresty
ENV MODSECURITY_LIB_INSTALL_DIR=/usr/local/modsecurity
ENV MODSECURITY_MODULE_SRC=/usr/local/ModSecurity-nginx
WORKDIR $BUILD_HOME
RUN cp /etc/apt/sources.list /etc/apt/sources.list.bak
# Need syntax = docker/dockerfile:1.4.0 (only support buildx)
RUN <<EOF cat > /etc/apt/sources.list
deb http://mirrors.aliyun.com/debian/ bullseye main non-free contrib
deb-src http://mirrors.aliyun.com/debian/ bullseye main non-free contrib
deb http://mirrors.aliyun.com/debian-security/ bullseye-security main
deb-src http://mirrors.aliyun.com/debian-security/ bullseye-security main
deb http://mirrors.aliyun.com/debian/ bullseye-updates main non-free contrib
deb-src http://mirrors.aliyun.com/debian/ bullseye-updates main non-free contrib
deb http://mirrors.aliyun.com/debian/ bullseye-backports main non-free contrib
deb-src http://mirrors.aliyun.com/debian/ bullseye-backports main non-free contrib
EOF
# First line for openresty, the other line is modsecurity need
RUN apt-get update \
&& apt-get install -y libpcre3-dev libssl-dev perl make build-essential curl \
git g++ apt-utils autoconf automake build-essential \
libcurl4-openssl-dev libgeoip-dev liblmdb-dev libpcre++-dev \
libtool libxml2-dev libyajl-dev pkgconf wget zlib1g-dev
# RUN git config --global http.proxy http://127.0.0.1:1081
# RUN git config --global https.proxy http://127.0.0.1:1081
# Build libModSecurity (v3.0.10)
RUN git clone --depth 1 https://github.com/SpiderLabs/ModSecurity \
&& cd ModSecurity/ \
&& git submodule init \
&& git submodule update \
&& sh build.sh \
&& mkdir $MODSECURITY_LIB_INSTALL_DIR \
&& ./configure --prefix=$MODSECURITY_LIB_INSTALL_DIR \
&& make \
&& make install
# Build ModSecurity-nginx connector
RUN git clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git $MODSECURITY_MODULE_SRC
# dynamic module
# ENV OPENRESTY_BUILD_ARGS="--add-dynamic-module=$MODSECURITY_MODULE_SRC --prefix=$OPENRESTY_INSTALL_DIR"
# Here use 3rdPartyModules
ENV OPENRESTY_BUILD_ARGS="--add-module=$MODSECURITY_MODULE_SRC --prefix=$OPENRESTY_INSTALL_DIR"
RUN wget https://openresty.org/download/openresty-$VERSION.tar.gz
RUN tar zxf openresty-$VERSION.tar.gz \
&& mkdir $OPENRESTY_INSTALL_DIR \
&& cd openresty-$VERSION/ \
&& ./configure $OPENRESTY_BUILD_ARGS \
&& gmake \
&& gmake install
# Simple configuration prepara
ENV WAF_CONFIG_DIR=$OPENRESTY_INSTALL_DIR/nginx/modsec
RUN mkdir $WAF_CONFIG_DIR \
&& cp -rp ModSecurity/modsecurity.conf-recommended $WAF_CONFIG_DIR/modsecurity.conf \
&& cp -rp ModSecurity/unicode.mapping $WAF_CONFIG_DIR/ \
&& cd $WAF_CONFIG_DIR/ \
&& git clone https://github.com/SpiderLabs/owasp-modsecurity-crs.git \
&& cp -rp owasp-modsecurity-crs/crs-setup.conf.example owasp-modsecurity-crs/crs-setup.conf \
&& mv owasp-modsecurity-crs/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf.example owasp-modsecurity-crs/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf \
&& mv owasp-modsecurity-crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf.example owasp-modsecurity-crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf \
&& sed -i 's/^SecRuleEngine DetectionOnly/# SecRuleEngine DetectionOnly\nSecRuleEngine On/g' modsecurity.conf \
&& echo "Include $WAF_CONFIG_DIR/owasp-modsecurity-crs/crs-setup.conf" >> modsecurity.conf \
&& echo "Include $WAF_CONFIG_DIR/owasp-modsecurity-crs/rules/*.conf" >> modsecurity.conf
# Enable waf
# sed -i '/worker_processes/a\load_module modules\/ngx_http_modsecurity_module.so;' nginx.conf # for dynamic-module, here is thrid module
RUN cd $OPENRESTY_INSTALL_DIR/nginx/conf/ \
&& sed -i "/^http/a\ modsecurity on;\n modsecurity_rules_file $WAF_CONFIG_DIR/modsecurity.conf;" nginx.conf
# ------------------------------------------
# This is running container
FROM bitnami/minideb:bullseye
MAINTAINER owner
ENV OPENRESTY_HOME=/usr/local/openresty
ENV MODSECURITY_LIB_HOME=/usr/local/modsecurity
ENV MODSECURITY_MODULE_SRC_HOME=/usr/local/ModSecurity-nginx
WORKDIR $OPENRESTY_HOME
COPY --from=builder /etc/apt/sources.list /etc/apt/sources.list
COPY --from=builder /usr/local/openresty $OPENRESTY_HOME
COPY --from=builder /usr/local/modsecurity $MODSECURITY_LIB_HOME
COPY --from=builder /usr/local/ModSecurity-nginx $MODSECURITY_MODULE_SRC_HOME
# Here is lost lib (<ldd binary>)
RUN apt-get update \
&& apt-get install --no-install-recommends -y procps vim \
&& libgeoip-dev libcurl4-openssl-dev libxml2-dev libyajl-dev \
&& rm -rf /var/lib/apt/lists/*
ENV PATH="$PATH:/usr/local/openresty/bin:/usr/local/openresty/luajit/bin:/usr/local/openresty/nginx/sbin"
CMD ["/usr/local/openresty/bin/openresty", "-g", "daemon off;"]
# Use SIGQUIT instead of default SIGTERM to cleanly drain requests
# See https://github.com/openresty/docker-openresty/blob/master/README.md#tips--pitfalls
STOPSIGNAL SIGQUIT
编译命令如下:
docker buildx build --provenance=false --platform linux/amd64,linux/arm64 --push -t hub-me.com/infra/openresty-waf:v1.21.4.2m -o type=registry .
默认 waf 已经启用，可以访问如下两个连接进行测试，理论上第二个请求应该被拒绝
403 Forbbiden
http://127.0.0.1 http://127.0.0.1/?param="><script>alert(1);</script>
问题
以下记录了编译过程中遇到的问题，主要是 quem 版本问题。
多行文本如何导入文件
支持 cat 和 EOF 组合的文件写入方式出现在 dockerfile:1.4.0, （仅限 buildx 使用 here-documents）, 例如:
# syntax = docker/dockerfile:1.4.0 FROM bitnami/minideb:bullseye AS builder RUN <<EOF cat > /tmp/test.txt lin1 lin2 lin3 EOF
git 克隆的目标仓库过大出错
#15 0.154 Cloning into 'ModSecurity'... #15 151.0 error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated. #15 151.0 error: 7768 bytes of body are still expected #15 151.0 fetch-pack: unexpected disconnect while reading sideband packet #15 151.0 fatal: early EOF #15 151.1 fatal: index-pack failed ... error: failed to solve: process "/bin/sh -c git clone https://github.com/SpiderLabs/ModSecurity.git
解决方案:
降低深度，增加参数 --depth 1
git clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git
quem 版本过低，aarch64 进程编译报错
主要错误为:
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
#33 662.8 Processing triggers for libc-bin (2.31-13+deb11u6) ... #33 662.9 qemu: uncaught target signal 11 (Segmentation fault) - core dumped #33 662.9 Segmentation fault #33 662.9 qemu: uncaught target signal 11 (Segmentation fault) - core dumped #33 662.9 Segmentation fault #33 662.9 dpkg: error processing package libc-bin (--configure): #33 662.9 installed libc-bin package post-installation script subprocess returned error exit status 139 #33 662.9 Processing triggers for ca-certificates (20210119) ... #33 663.1 Updating certificates in /etc/ssl/certs... #33 677.4 0 added, 0 removed; done. #33 677.4 Running hooks in /etc/ca-certificates/update.d... #33 677.5 done. #33 677.6 Errors were encountered while processing: #33 677.6 libc-bin #33 677.7 E: Sub-process /usr/bin/dpkg returned an error code (1) ------ Dockerfile-M:30 -------------------- 29 | # for openresty with debian11/ubuntu20.04 30 | >>> RUN apt-get install -y libpcre3-dev libssl-dev perl make build-essential curl \ 31 | >>> git g++ apt-utils autoconf automake build-essential \ 32 | >>> libcurl4-openssl-dev libgeoip-dev liblmdb-dev libpcre++-dev \ 33 | >>> libtool libxml2-dev libyajl-dev pkgconf wget zlib1g-dev 34 | -------------------- error: failed to solve: process "/bin/sh -c apt-get install -y libpcre3-dev libssl-dev perl make build-essential curl git g++ apt-utils autoconf automake build-essential libcurl4-openssl-dev libgeoip-dev liblmdb-dev libpcre++-dev libtool libxml2-dev libyajl-dev pkgconf wget zlib1g-dev" did not complete successfully: exit code: 100
解决方案:
错误原因为 qemu 版本低于 0.7 ，详情查看 #1170 。
1. 源码编译升级，首先安装依赖
sudo apt update sudo apt install -y ninja-build libpixman-1-dev
sudo yum install -y glib2-devel glib2 ninja-build pixman-devel
2. 下载 qemu 7.0 开始编译 （大概需要1-2小时）
wget https://download.qemu.org/qemu-7.0.0.tar.xz tar xvJf qemu-7.0.0.tar.xz cd qemu-7.0.0 ./configure make
3. 运行 binfmt 容器
docker run --privileged --rm tonistiigi/binfmt --install all
检查 flags 字段值是否包含字母 F
$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64 enabled interpreter /usr/bin/qemu-aarch64-static flags: OC offset 0 magic 7f454c460201010000000000000000000200b700 mask ffffffffffffff00fffffffffffffffffeffffff
4. 如果结果不是OCF， 需要执行以下容器:
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes -c yes
无论是否包含 F, 都建议你执行第 4 步， 经测试，虽然结果有F，但是依然报同样的错，所以建议执行。
qemu-7.0.0 要求 GCC > 7 版本 或者 Clang > 6 版本， 由于我自己是 4.8.5，因此需要升级。
升级方案不选择源码编译，选择 devtoolset 工具更加方便。
以下是版本对应情况：
| version | gcc |
|---|---|
| devtoolset-3 | 4.x.x版本 |
| devtoolset-4 | 5.x.x版本 |
| devtoolset-6 | 6.x.x版本 |
| devtoolset-7 | 7.x.x版本 |
| devtoolset-8 | 8.x.x版本 |
| devtoolset-9 | 9.x.x版本 |
| devtoolset-10 | 10.x.x版本 |
安装
yum install centos-release-scl yum install devtoolset-10
启用
scl enable devtoolset-10 bash (or) source /opt/rh/devtoolset-10/enable
参考
- Openresty官方 编译安装介绍
- Openresty官方 下载
- ModSecurity-nginx connctor 动态模块编译和第三方模块编译
- Docker Hub bitnami/minideb
- docker-openresty/bullseye/Dockerfile 参考
- libModSecurity Compilation recipes for v3.x
- ModSecurity 验证参考方案
- qemu 版本低问题
- gcc 升级方法参考
脚注:
如未另行说明，那么本页面中的内容已根据 知识共享署名 4.0 许可 获得了许可，并且代码示例已根据 Apache 2.0 许可 获得了许可。内容随着时间推移，可能会过期，会定时更新或移除一些旧的内容。
最后更新时间 (CST)：2023-09-01
阅读更多
访问本博文的相关主题 容器技术 以获取更多的精彩内容。
