Hello
What does it mean when nginx returns an http status code 0?
We see that cause we monitor nginx response status code, which is used as
front of our apps. The apps themselves can not return 0, only 200 or 500.
So it seems issue here comes from nginx itself which can not process the
connection under high peak of load, but why 0, is that expected?
Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20240504/42c707fc/attachment.htm>
Hi Sébastien,
thanks for the report.
On Sat, May 04, 2024 at 07:31:43PM +0200, Sébastien Rebecchi wrote:
>
> What does it mean when nginx returns an http status code 0?
>
> We see that cause we monitor nginx response status code, which is used as
> front of our apps. The apps themselves can not return 0, only 200 or 500.
> So it seems issue here comes from nginx itself which can not process the
> connection under high peak of load, but why 0, is that expected?
Could you please provide more details about the case:
- configuration file
- test cases that reproduces the issue
- additional details
Thank you.
--
Sergey A. Osokin
Hello!
On Sat, May 04, 2024 at 07:31:43PM +0200, Sébastien Rebecchi wrote:
> Hello
>
> What does it mean when nginx returns an http status code 0?
>
> We see that cause we monitor nginx response status code, which is used as
> front of our apps. The apps themselves can not return 0, only 200 or 500.
> So it seems issue here comes from nginx itself which can not process the
> connection under high peak of load, but why 0, is that expected?
Status code 0 as seen in nginx http access logs means that nginx
wasn't able to generate any reasonable status code, even some
generic failure like 500 (Internal Server Error), yet the request
was closed.
This usually happens due to some fatal issues, like unexpected
conditions (which might indicate a bug somewhere), unexpected
errors, or memory allocation errors if it wasn't possible to
return 500.
In most cases there should be additional details in the error log
explaining the reasons. If there aren't any, or reasons aren't
clear, it might be a good idea to dig further.
--
Maxim Dounin
http://mdounin.ru/
Hello!
There is nothing regarding this issue in nginx logs.
Now I think the issue is not with nginx itself, but in front of nginx, with
Linux itself.
We monitor using curl, and it seems that curl can print status code 0 when
it can not establish a connection with the server.
I think the Linux kernel is not configured properly to handle our
connection peaks. Looking at net.core.somaxconn, we have the default of
128. More generally, I will deep dive into this and possibly other kernel
settings to see if optimizing them will solve the problem. For information,
on each of our servers we have an average of around 80K simultaneous TCP
connections globally and around 35K in state ESTABLISHED (according to
netstat/ss -nta), and more during high peaks.
I saw this doc section which is a good starting point
https://www.nginx.com/blog/tuning-nginx/#Tuning-Your-Linux-Configuration
If you have any advice on other settings to increase, this would be very
appreciated.
I will keep you in touch about my investigations, to confirm at least that
there is no bug on nginx side, which i am quite confident about now.
Thank you very much for your help!
Le sam. 4 mai 2024 à 20:47, Maxim Dounin <mdounin at mdounin.ru> a écrit :
> Hello!
>
> On Sat, May 04, 2024 at 07:31:43PM +0200, Sébastien Rebecchi wrote:
>
> > Hello
> >
> > What does it mean when nginx returns an http status code 0?
> >
> > We see that cause we monitor nginx response status code, which is used as
> > front of our apps. The apps themselves can not return 0, only 200 or 500.
> > So it seems issue here comes from nginx itself which can not process the
> > connection under high peak of load, but why 0, is that expected?
>
> Status code 0 as seen in nginx http access logs means that nginx
> wasn't able to generate any reasonable status code, even some
> generic failure like 500 (Internal Server Error), yet the request
> was closed.
>
> This usually happens due to some fatal issues, like unexpected
> conditions (which might indicate a bug somewhere), unexpected
> errors, or memory allocation errors if it wasn't possible to
> return 500.
>
> In most cases there should be additional details in the error log
> explaining the reasons. If there aren't any, or reasons aren't
> clear, it might be a good idea to dig further.
>
> --
> Maxim Dounin
> http://mdounin.ru/
> --
> nginx mailing list
> nginx at freenginx.org
> https://freenginx.org/mailman/listinfo/nginx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20240506/d964221f/attachment.htm>
Hello
Just to close that conversation, it seems this was an error of our devops
in charge of alerting, who was using curl in a bad way.
Best regards,
Sébastien
Le lun. 6 mai 2024 à 11:33, Sébastien Rebecchi <srebecchi at kameleoon.com> a
écrit :
> Hello!
>
> There is nothing regarding this issue in nginx logs.
>
> Now I think the issue is not with nginx itself, but in front of nginx,
> with Linux itself.
> We monitor using curl, and it seems that curl can print status code 0 when
> it can not establish a connection with the server.
> I think the Linux kernel is not configured properly to handle our
> connection peaks. Looking at net.core.somaxconn, we have the default of
> 128. More generally, I will deep dive into this and possibly other kernel
> settings to see if optimizing them will solve the problem. For information,
> on each of our servers we have an average of around 80K simultaneous TCP
> connections globally and around 35K in state ESTABLISHED (according to
> netstat/ss -nta), and more during high peaks.
> I saw this doc section which is a good starting point
> https://www.nginx.com/blog/tuning-nginx/#Tuning-Your-Linux-Configuration
> If you have any advice on other settings to increase, this would be very
> appreciated.
>
> I will keep you in touch about my investigations, to confirm at least that
> there is no bug on nginx side, which i am quite confident about now.
>
> Thank you very much for your help!
>
> Le sam. 4 mai 2024 à 20:47, Maxim Dounin <mdounin at mdounin.ru> a écrit :
>
>> Hello!
>>
>> On Sat, May 04, 2024 at 07:31:43PM +0200, Sébastien Rebecchi wrote:
>>
>> > Hello
>> >
>> > What does it mean when nginx returns an http status code 0?
>> >
>> > We see that cause we monitor nginx response status code, which is used
>> as
>> > front of our apps. The apps themselves can not return 0, only 200 or
>> 500.
>> > So it seems issue here comes from nginx itself which can not process the
>> > connection under high peak of load, but why 0, is that expected?
>>
>> Status code 0 as seen in nginx http access logs means that nginx
>> wasn't able to generate any reasonable status code, even some
>> generic failure like 500 (Internal Server Error), yet the request
>> was closed.
>>
>> This usually happens due to some fatal issues, like unexpected
>> conditions (which might indicate a bug somewhere), unexpected
>> errors, or memory allocation errors if it wasn't possible to
>> return 500.
>>
>> In most cases there should be additional details in the error log
>> explaining the reasons. If there aren't any, or reasons aren't
>> clear, it might be a good idea to dig further.
>>
>> --
>> Maxim Dounin
>> http://mdounin.ru/
>> --
>> nginx mailing list
>> nginx at freenginx.org
>> https://freenginx.org/mailman/listinfo/nginx
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20240517/3e9e00c0/attachment.htm>