The py3-zmq package does not exist in alpine linux 3.11, used in both master and 1.11.4 . The py3-pyzmq package exists however:
https://pkgs.alpinelinux.org/packages?name=py3-pyzmq&branch=v3.11
COPY custom/app.ini /data/gitea/conf/app.ini
[...]
-RUN apk --no-cache add asciidoctor freetype freetype-dev gcc g++ libpng python-dev py-pip python3-dev py3-pip py3-zmq
+RUN apk --no-cache add asciidoctor freetype freetype-dev gcc g++ libpng python-dev py-pip python3-dev py3-pip py3-pyzmq
# install any other package you need for your external renderers
RUN pip3 install --upgrade pip