Hello,
I've noticed a recent change in behavior that occurred between releases
1.22.1 and 1.23.4, and I'm unable to find this particular change documented.
If an upstream proxy sends a response with duplicate Content-Length
headers, both of the same value, nginx 1.22.1 allows the response to be
sent to the client, removing the duplicate. However, nginx 1.23.4 (and
1.24.0) responds to the client with a 502. I did not explicitly test with
any other versions.
I won't try to make any claims on the correctness of either behavior, but
it is a change that may affect some workloads.
Here is an example response that exhibits the change:
"""
HTTP/1.1 200 OK
Server: http_tcp
Content-Length: 12
Content-Length: 12
Connection: Closed
Hello World\n
"""
------
jstimpson:[~/dev/c/nginx-1.22.1]: curl -i localhost
HTTP/1.1 200 OK
Server: nginx/1.22.1
Date: Wed, 19 Apr 2023 12:17:05 GMT
Content-Length: 12
Connection: keep-alive
Hello World
-----
jstimpson:[~/dev/c/nginx-1.23.4]: curl -i localhost
HTTP/1.1 502 Bad Gateway
Server: nginx/1.23.4
Date: Wed, 19 Apr 2023 12:13:09 GMT
Content-Type: text/html
Content-Length: 497
Connection: keep-alive
ETag: "643fd39e-1f1"
...<snip>...
---------
These tests were done with this simple config, the rest of the conf is
defaults.
location / {
proxy_pass http://localhost:4040;
}
Is this change intentional? Did I overlook it in the Changelog?
Thanks,
Jesse
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230419/338ab473/attachment.htm>
This sounds like your backend and nginx are both generating the header. NGINX hasn't changed to the point it would create two headers, but if your backend is adding the header as well as nginx then there's your problem.
Sent from my Galaxy
-------- Original message --------
From: Jesse Stimpson via nginx <nginx at nginx.org>
Date: 4/19/23 08:57 (GMT-05:00)
To: nginx at nginx.org
Cc: Jesse Stimpson <jstimpson at relaypro.com>
Subject: Duplicate Content-Length header with same value, recent change in behavior intentional?
Hello,
I've noticed a recent change in behavior that occurred between releases 1.22.1 and 1.23.4, and I'm unable to find this particular change documented.
If an upstream proxy sends a response with duplicate Content-Length headers, both of the same value, nginx 1.22.1 allows the response to be sent to the client, removing the duplicate. However, nginx 1.23.4 (and 1.24.0) responds to the client with a 502. I did not explicitly test with any other versions.
I won't try to make any claims on the correctness of either behavior, but it is a change that may affect some workloads.
Here is an example response that exhibits the change:
"""
HTTP/1.1 200 OK
Server: http_tcp
Content-Length: 12
Content-Length: 12
Connection: Closed
Hello World\n
"""
------
jstimpson:[~/dev/c/nginx-1.22.1]: curl -i localhost
HTTP/1.1 200 OK
Server: nginx/1.22.1
Date: Wed, 19 Apr 2023 12:17:05 GMT
Content-Length: 12
Connection: keep-alive
Hello World
-----
jstimpson:[~/dev/c/nginx-1.23.4]: curl -i localhost
HTTP/1.1 502 Bad Gateway
Server: nginx/1.23.4
Date: Wed, 19 Apr 2023 12:13:09 GMT
Content-Type: text/html
Content-Length: 497
Connection: keep-alive
ETag: "643fd39e-1f1"
...<snip>...
---------
These tests were done with this simple config, the rest of the conf is defaults.
location / {
proxy_pass http://localhost:4040;
}
Is this change intentional? Did I overlook it in the Changelog?
Thanks,
Jesse
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230419/db0ac108/attachment-0001.htm>
Yes, apologies for being unclear. Indeed, the upstream proxy is the one
generating the duplicate header. My message is primarily concerning the
behavior of nginx upon recognizing such a duplicate from the upstream. In
order to complete my testing, I wrote an http server as my upstream that
purposefully sent duplicate headers. I realize this consider misbehavior of
an upstream.
On Wed, Apr 19, 2023 at 9:06 AM Thomas Ward <teward at thomas-ward.net> wrote:
> This sounds like your backend and nginx are both generating the header.
> NGINX hasn't changed to the point it would create two headers, but if your
> backend is adding the header as well as nginx then there's your problem.
>
>
>
> Sent from my Galaxy
>
>
>
> -------- Original message --------
> From: Jesse Stimpson via nginx <nginx at nginx.org>
> Date: 4/19/23 08:57 (GMT-05:00)
> To: nginx at nginx.org
> Cc: Jesse Stimpson <jstimpson at relaypro.com>
> Subject: Duplicate Content-Length header with same value, recent change in
> behavior intentional?
>
> Hello,
>
> I've noticed a recent change in behavior that occurred between releases
> 1.22.1 and 1.23.4, and I'm unable to find this particular change documented.
>
> If an upstream proxy sends a response with duplicate Content-Length
> headers, both of the same value, nginx 1.22.1 allows the response to be
> sent to the client, removing the duplicate. However, nginx 1.23.4 (and
> 1.24.0) responds to the client with a 502. I did not explicitly test with
> any other versions.
>
> I won't try to make any claims on the correctness of either behavior, but
> it is a change that may affect some workloads.
>
> Here is an example response that exhibits the change:
>
> """
> HTTP/1.1 200 OK
> Server: http_tcp
> Content-Length: 12
> Content-Length: 12
> Connection: Closed
>
> Hello World\n
> """
> ------
>
> jstimpson:[~/dev/c/nginx-1.22.1]: curl -i localhost
> HTTP/1.1 200 OK
> Server: nginx/1.22.1
> Date: Wed, 19 Apr 2023 12:17:05 GMT
> Content-Length: 12
> Connection: keep-alive
>
> Hello World
>
> -----
>
>
> jstimpson:[~/dev/c/nginx-1.23.4]: curl -i localhost
> HTTP/1.1 502 Bad Gateway
> Server: nginx/1.23.4
> Date: Wed, 19 Apr 2023 12:13:09 GMT
> Content-Type: text/html
> Content-Length: 497
> Connection: keep-alive
> ETag: "643fd39e-1f1"
>
> ...<snip>...
>
>
> ---------
>
> These tests were done with this simple config, the rest of the conf is
> defaults.
>
> location / {
> proxy_pass http://localhost:4040;
> }
>
>
> Is this change intentional? Did I overlook it in the Changelog?
>
> Thanks,
> Jesse
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230419/b4711be7/attachment.htm>
I don't think NGINX has any default duplicate header handing and hasn't really. My guess is that you've simply not seen this behavior in the past.
You can suppress the backend header:
proxy_hide_header Content-Length;
Note that the behavior hasnt changed and only *certain* headers remain default-hidden from the backend - I don't believe Content-Length was ever one of them.
Sent from my Galaxy
-------- Original message --------
From: Jesse Stimpson <jstimpson at relaypro.com>
Date: 4/19/23 09:09 (GMT-05:00)
To: Thomas Ward <teward at thomas-ward.net>
Cc: nginx at nginx.org
Subject: Re: Duplicate Content-Length header with same value, recent change in behavior intentional?
Yes, apologies for being unclear. Indeed, the upstream proxy is the one generating the duplicate header. My message is primarily concerning the behavior of nginx upon recognizing such a duplicate from the upstream. In order to complete my testing, I wrote an http server as my upstream that purposefully sent duplicate headers. I realize this consider misbehavior of an upstream.
On Wed, Apr 19, 2023 at 9:06 AM Thomas Ward <teward at thomas-ward.net<mailto:teward at thomas-ward.net>> wrote:
This sounds like your backend and nginx are both generating the header. NGINX hasn't changed to the point it would create two headers, but if your backend is adding the header as well as nginx then there's your problem.
Sent from my Galaxy
-------- Original message --------
From: Jesse Stimpson via nginx <nginx at nginx.org<mailto:nginx at nginx.org>>
Date: 4/19/23 08:57 (GMT-05:00)
To: nginx at nginx.org<mailto:nginx at nginx.org>
Cc: Jesse Stimpson <jstimpson at relaypro.com<mailto:jstimpson at relaypro.com>>
Subject: Duplicate Content-Length header with same value, recent change in behavior intentional?
Hello,
I've noticed a recent change in behavior that occurred between releases 1.22.1 and 1.23.4, and I'm unable to find this particular change documented.
If an upstream proxy sends a response with duplicate Content-Length headers, both of the same value, nginx 1.22.1 allows the response to be sent to the client, removing the duplicate. However, nginx 1.23.4 (and 1.24.0) responds to the client with a 502. I did not explicitly test with any other versions.
I won't try to make any claims on the correctness of either behavior, but it is a change that may affect some workloads.
Here is an example response that exhibits the change:
"""
HTTP/1.1 200 OK
Server: http_tcp
Content-Length: 12
Content-Length: 12
Connection: Closed
Hello World\n
"""
------
jstimpson:[~/dev/c/nginx-1.22.1]: curl -i localhost
HTTP/1.1 200 OK
Server: nginx/1.22.1
Date: Wed, 19 Apr 2023 12:17:05 GMT
Content-Length: 12
Connection: keep-alive
Hello World
-----
jstimpson:[~/dev/c/nginx-1.23.4]: curl -i localhost
HTTP/1.1 502 Bad Gateway
Server: nginx/1.23.4
Date: Wed, 19 Apr 2023 12:13:09 GMT
Content-Type: text/html
Content-Length: 497
Connection: keep-alive
ETag: "643fd39e-1f1"
...<snip>...
---------
These tests were done with this simple config, the rest of the conf is defaults.
location / {
proxy_pass http://localhost:4040;
}
Is this change intentional? Did I overlook it in the Changelog?
Thanks,
Jesse
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230419/1e20ebcb/attachment-0001.htm>
Hello!
On Wed, Apr 19, 2023 at 08:57:28AM -0400, Jesse Stimpson via nginx wrote:
> I've noticed a recent change in behavior that occurred between releases
> 1.22.1 and 1.23.4, and I'm unable to find this particular change documented.
>
> If an upstream proxy sends a response with duplicate Content-Length
> headers, both of the same value, nginx 1.22.1 allows the response to be
> sent to the client, removing the duplicate. However, nginx 1.23.4 (and
> 1.24.0) responds to the client with a 502. I did not explicitly test with
> any other versions.
>
> I won't try to make any claims on the correctness of either behavior, but
> it is a change that may affect some workloads.
[...]
> Is this change intentional? Did I overlook it in the Changelog?
Yes, it is. It is part of a large work to implement proper
handling of multiple header lines with the same name, as done in
nginx 1.23.0. The particular commit is:
http://hg.nginx.org/nginx/rev/2bf7792c262e
The particular modification isn't in the change log as the
upstream response in question is obviously invalid and should
never be accepted in the first place, and the change is more or
less a minor cleanup work.
--
Maxim Dounin
http://mdounin.ru/