Hello all,
By default, FreeBSD restricts potentially unsafe AIO operations (as
determined by the target fd type) and operations like aio_read(2) will
“falsely” return EOPNOTSUPP to avoid a potentially dangerous operation
that can result in blocking the aio threadpool hanging the system or the
process, per aio(4).
I’ve observed in production with an nginx/1.23.3 instance (compiled with
--with-file-aio) running on FreeBSD 13.1-RELEASE-p5, configured with
`aio on;` (and `use kqueue;` though I suspect that is not relevant),
the following syslog entry:
pid 1125 (nginx) is attempting to use unsafe AIO requests - not
logging anymore
My curiosity got the best of me and I decided to allow unsafe aio
requests to see what would happen (`sysctl vfs.aio.enable_unsafe=1`).
It’s been about 24 hours and I haven’t noticed any ill or adverse
effects, at least judging by my scrutiny of the logs, though I intend to
continue to closely monitor this server and see what happens.
My question is whether or not nginx does anything “advanced” with aio
under FreeBSD, beyond using aio for operations on “sockets, raw disk
devices, and regular files on local filesystems,” which is the “safe”
list, again per aio(4), while other types of fds are blocked unless
unsafe aio is enabled.
On this server, nginx is serving static files from various zfs datasets
and is functioning as a reverse proxy to http and fastcgi upstreams. I
do have a few 3rd party modules statically compiled into nginx, so I'm
naturally limiting my question to core/stock nginx behavior to the best
of its developers’ knowledge :)
I don't have all the system logs but in a sample of the logs preserved
going back to November 2022 the "unsafe AIO" is not repeated anywhere,
leading me to _suspect_ that this isn't "normal" nginx behavior and that
I probably should *not* be enabling unsafe AIO - but curiosity is a hell
of a drug!
Thanks,
Mahmoud Al-Qudsi
NeoSmart Technologies
Hello!
On Tue, Jan 17, 2023 at 12:04:31PM -0600, Mahmoud Al-Qudsi wrote:
> Hello all,
>
> By default, FreeBSD restricts potentially unsafe AIO operations (as
> determined by the target fd type) and operations like aio_read(2) will
> “falsely” return EOPNOTSUPP to avoid a potentially dangerous operation
> that can result in blocking the aio threadpool hanging the system or the
> process, per aio(4).
>
> I’ve observed in production with an nginx/1.23.3 instance (compiled with
> --with-file-aio) running on FreeBSD 13.1-RELEASE-p5, configured with
> `aio on;` (and `use kqueue;` though I suspect that is not relevant),
> the following syslog entry:
>
> pid 1125 (nginx) is attempting to use unsafe AIO requests - not
> logging anymore
>
> My curiosity got the best of me and I decided to allow unsafe aio
> requests to see what would happen (`sysctl vfs.aio.enable_unsafe=1`).
> It’s been about 24 hours and I haven’t noticed any ill or adverse
> effects, at least judging by my scrutiny of the logs, though I intend to
> continue to closely monitor this server and see what happens.
>
> My question is whether or not nginx does anything “advanced” with aio
> under FreeBSD, beyond using aio for operations on “sockets, raw disk
> devices, and regular files on local filesystems,” which is the “safe”
> list, again per aio(4), while other types of fds are blocked unless
> unsafe aio is enabled.
>
> On this server, nginx is serving static files from various zfs datasets
> and is functioning as a reverse proxy to http and fastcgi upstreams. I
> do have a few 3rd party modules statically compiled into nginx, so I'm
> naturally limiting my question to core/stock nginx behavior to the best
> of its developers’ knowledge :)
>
> I don't have all the system logs but in a sample of the logs preserved
> going back to November 2022 the "unsafe AIO" is not repeated anywhere,
> leading me to _suspect_ that this isn't "normal" nginx behavior and that
> I probably should *not* be enabling unsafe AIO - but curiosity is a hell
> of a drug!
The only aio operation nginx uses is aio_read(), and it does
nothing "advanced" - just reads normal files which are being
served by nginx.
Further, nginx explicitly checks files being served, and rejects
non-regular files. As such, the "unsafe AIO" checks shouldn't be
triggered unless you are trying to serve something from non-local
file systems (well, you indeed shouldn't).
In general, if an aio_read() error happens, you should be able to
see corresponding error in nginx error log at the "crit" level.
The error will look like "[crit] ... aio_read("/path/to/file")
failed (45: Operation not supported)". It should make it possible
to find out what actually causes the error.
--
Maxim Dounin
http://mdounin.ru/
On Tue, Jan 17, 2023 at 6:00 PM Maxim Dounin <mdounin at mdounin.ru> wrote:
>
> The only aio operation nginx uses is aio_read(), and it does
> nothing "advanced" - just reads normal files which are being
> served by nginx.
>
> Further, nginx explicitly checks files being served, and rejects
> non-regular files. As such, the "unsafe AIO" checks shouldn't be
> triggered unless you are trying to serve something from non-local
> file systems (well, you indeed shouldn't).
>
> In general, if an aio_read() error happens, you should be able to
> see corresponding error in nginx error log at the "crit" level.
> The error will look like "[crit] ... aio_read("/path/to/file")
> failed (45: Operation not supported)". It should make it possible
> to find out what actually causes the error.
Hello and thanks for the reply!
This is exactly what I was hoping to hear; thanks for clarifying.
Due to a most unfortunate series of events (truncating the nginx error
log before setting up newsyslog(8) to manage it) I do not have the error
log for the three-hour window when this incident took place.
I do have an archive of error.log from shortly before then going all the
way back to 2016 and the only instances of the `[crit] .. aio_read(..)
failed (45: Operation not supported)` were six such errors from 2018 in
close succession, each for a different path, while nginx was trying to
read statically compressed .br versions of static assets as enabled by
ngx_brotli [0] (which doesn't make any `aio_read()` calls of its own).
I think it's improbable I'll see the error again anytime soon, but I now
know where to look if I do.
[0]: Specifically my mtime-enabled fork of ngx_brotli, open-sourced at
https://github.com/mqudsi/ngx_brotli
Thanks,
Mahmoud