Hello,
I have a nginx proxy in front of systemd-nspawn containers.
The IP's of the containers are dynamic.
When I start nginx *after* the containers it works.
When the IP of the container is changed while nginx is running i get a
"Bad Gateway" error.
The config looks like:
server {
server_name foobar.example.com
...
location / {
resolver 127.0.0.53 valid=10s;
...
proxy_pass http://container;
}
}
nginx is 1.1.18 so it should work as documented in
http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver
The workaround there
https://stackoverflow.com/questions/42720618/docker-nginx-stopped-emerg-11-host-not-found-in-upstream/52319161#52319161
doesn't work.
I have also try to config a upstream backend and the resolver in the
server part or in the http part.
The errors are: "upstream timed out" or "container could not be resolved
(3: Host not found)"
Whats wrong there?
Best Regards
Am 18.04.23 um 14:58 schrieb mailinglist at unix-solution.de:
> nginx is 1.1.18
> Whats wrong there?
released in March 2012
couldn't you use a "not so historic" version?
Andreas
Sorry, my mistake.
1.18.0 debian11 is installed.
Am 18.04.23 um 19:21 schrieb A. Schulze via nginx:
>
>
> Am 18.04.23 um 14:58 schrieb mailinglist at unix-solution.de:
>> nginx is 1.1.18
>> Whats wrong there?
> released in March 2012
>
> couldn't you use a "not so historic" version?
>
> Andreas
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> https://mailman.nginx.org/mailman/listinfo/nginx
Hello!
On Tue, Apr 18, 2023 at 02:58:08PM +0200, mailinglist at unix-solution.de wrote:
> Hello,
> I have a nginx proxy in front of systemd-nspawn containers.
> The IP's of the containers are dynamic.
>
> When I start nginx *after* the containers it works.
> When the IP of the container is changed while nginx is running i get a
> "Bad Gateway" error.
>
> The config looks like:
>
> server {
> server_name foobar.example.com
> ...
> location / {
> resolver 127.0.0.53 valid=10s;
> ...
> proxy_pass http://container;
> }
> }
>
> nginx is 1.1.18 so it should work as documented in
> http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver
> The workaround there
> https://stackoverflow.com/questions/42720618/docker-nginx-stopped-emerg-11-host-not-found-in-upstream/52319161#52319161
> doesn't work.
>
> I have also try to config a upstream backend and the resolver in the
> server part or in the http part.
> The errors are: "upstream timed out" or "container could not be resolved
> (3: Host not found)"
>
> Whats wrong there?
The "resolver" directive is only used when the "proxy_pass"
directive contain variables, and therefore nginx isn't able to
determine upstream addresses when parsing configuration. Quoting
http://nginx.org/r/proxy_pass:
: Parameter value can contain variables. In this case, if an
: address is specified as a domain name, the name is searched among
: the described server groups, and, if not found, is determined
: using a resolver.
The "workaround", however, should work fine. Something like this
should be enough:
resolver 127.0.0.53 valid=10s;
set $upstream "http://container";
proxy_pass $upstream;
Note though that there should be an actual DNS name which can be
resolved with the DNS server configured with the "resolver"
directive. The "container could not be resolved (3: Host not
found)" error suggests that your DNS server does not return
anything for the name - you may want to dig further into systemd
to find out why it doesn't return anything.
--
Maxim Dounin
http://mdounin.ru/
Hi,
On 18/04/2023 13:58, mailinglist at unix-solution.de wrote:
> Hello,
> I have a nginx proxy in front of systemd-nspawn containers.
> The IP's of the containers are dynamic.
>
> When I start nginx *after* the containers it works.
> When the IP of the container is changed while nginx is running i get a
> "Bad Gateway" erro
>
> The config looks like:
>
> server {
> server_name foobar.example.com
> ...
> location / {
> resolver 127.0.0.53 valid=10s;
> ...
> proxy_pass http://container;
You need to use a variable in the proxy_pass here instead to do dynamic
proxy pass. This force nginx to re-resolve the hostname. The variable's
value should be set the target hostname (looks like it's 'containers'
from your config).
something like
set $container_hostname 'containers';
proxy_pass http://$containers_hostname;
Note, you can't target an upstream server group with this technique - it
has to be a hostname that the dns resolver returns.
If you need more advanced re-resolving, such as the ability to use
upstream server groups and resolve servers within it, NGINX Plus has
this feature.
In fact the patches from NGINX Plus that do the dynamic re-resolving are
already on the devel mailing list - just not integrated.
https://mailman.nginx.org/pipermail/nginx-devel/2023-February/4MCLSVRK7EX6DNKHFZN6CA4SKZUSA3GA.html
So it can also be obtained by comping from source with that set of
patches applied.
This is another alternative upstream resolver - although not as good, as
it requires requests to initiate a re-resolve.
https://www.nginx.com/resources/wiki/modules/domain_resolve/
> }
> }
>
> nginx is 1.1.18 so it should work as documented in
> http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver
> The workaround there
> https://stackoverflow.com/questions/42720618/docker-nginx-stopped-emerg-11-host-not-found-in-upstream/52319161#52319161 doesn't work.
>
> I have also try to config a upstream backend and the resolver in the
> server part or in the http part.
> The errors are: "upstream timed out" or "container could not be resolved
> (3: Host not found)"
>
> Whats wrong there?
> Best Regards
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> https://mailman.nginx.org/mailman/listinfo/nginx
Hello Maxim,
that does not work for me.
root at tower:~# resolvectl query icinga2
icinga2: 192.168.145.217 -- link: ve-icinga2
-- Information acquired via protocol LLMNR/IPv4 in 105.7ms.
-- Data is authenticated: no
root at tower:~# getent hosts icinga2
192.168.145.217 icinga2
root at tower:~# cat /etc/resolv.conf
nameserver 127.0.0.53
options edns0
root at tower:~#
location / {
proxy_redirect off;
proxy_set_header Host $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Host $remote_addr;
proxy_set_header X-Real-IP $remote_addr;
# proxy_pass https://icinga2;
resolver 127.0.0.53 valid=10s;
set $upstream "https://icinga2";
proxy_pass $upstream;
# WebSocket support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
2023/04/18 22:16:28 [error] 779163#779163: *26641 icinga2 could not be
resolved (3: Host not found),
503 bad gateway
Am 18.04.23 um 21:29 schrieb Maxim Dounin:
> Hello!
>
> On Tue, Apr 18, 2023 at 02:58:08PM +0200, mailinglist at unix-solution.de wrote:
>
>> Hello,
>> I have a nginx proxy in front of systemd-nspawn containers.
>> The IP's of the containers are dynamic.
>>
>> When I start nginx *after* the containers it works.
>> When the IP of the container is changed while nginx is running i get a
>> "Bad Gateway" error.
>>
>> The config looks like:
>>
>> server {
>> server_name foobar.example.com
>> ...
>> location / {
>> resolver 127.0.0.53 valid=10s;
>> ...
>> proxy_pass http://container;
>> }
>> }
>>
>> nginx is 1.1.18 so it should work as documented in
>> http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver
>> The workaround there
>> https://stackoverflow.com/questions/42720618/docker-nginx-stopped-emerg-11-host-not-found-in-upstream/52319161#52319161
>> doesn't work.
>>
>> I have also try to config a upstream backend and the resolver in the
>> server part or in the http part.
>> The errors are: "upstream timed out" or "container could not be resolved
>> (3: Host not found)"
>>
>> Whats wrong there?
>
> The "resolver" directive is only used when the "proxy_pass"
> directive contain variables, and therefore nginx isn't able to
> determine upstream addresses when parsing configuration. Quoting
> http://nginx.org/r/proxy_pass:
>
> : Parameter value can contain variables. In this case, if an
> : address is specified as a domain name, the name is searched among
> : the described server groups, and, if not found, is determined
> : using a resolver.
>
> The "workaround", however, should work fine. Something like this
> should be enough:
>
> resolver 127.0.0.53 valid=10s;
> set $upstream "http://container";
> proxy_pass $upstream;
>
> Note though that there should be an actual DNS name which can be
> resolved with the DNS server configured with the "resolver"
> directive. The "container could not be resolved (3: Host not
> found)" error suggests that your DNS server does not return
> anything for the name - you may want to dig further into systemd
> to find out why it doesn't return anything.
>
Hello!
On Tue, Apr 18, 2023 at 10:25:09PM +0200, basti wrote:
> Hello Maxim,
>
> that does not work for me.
>
> root at tower:~# resolvectl query icinga2
> icinga2: 192.168.145.217 -- link: ve-icinga2
>
> -- Information acquired via protocol LLMNR/IPv4 in 105.7ms.
> -- Data is authenticated: no
>
> root at tower:~# getent hosts icinga2
> 192.168.145.217 icinga2
>
> root at tower:~# cat /etc/resolv.conf
> nameserver 127.0.0.53
> options edns0
> root at tower:~#
>
And what "nslookup icinga2 127.0.0.53" shows?
> location / {
>
> proxy_redirect off;
> proxy_set_header Host $remote_addr;
> proxy_set_header X-Forwarded-For $remote_addr;
> proxy_set_header X-Forwarded-Host $remote_addr;
> proxy_set_header X-Real-IP $remote_addr;
> # proxy_pass https://icinga2;
> resolver 127.0.0.53 valid=10s;
> set $upstream "https://icinga2";
> proxy_pass $upstream;
>
> # WebSocket support
> proxy_http_version 1.1;
> proxy_set_header Upgrade $http_upgrade;
> proxy_set_header Connection "upgrade";
> }
>
> 2023/04/18 22:16:28 [error] 779163#779163: *26641 icinga2 could not be
> resolved (3: Host not found),
As previously mentioned, the message suggests that the name is not
available via DNS. This is to be resolved on the systemd side.
Unfortunately, I'm not familiar with systemd-nspawn and have no
idea how to do it properly and if it's at all possible.
--
Maxim Dounin
http://mdounin.ru/
Hello Maxim,
I have found.
nginx query ipv4 and ipv6 as expected.
systemd-resolved return servfail when there is no ipv6. I think it
should return nodata.
When I set ipv6=off in nginx all is fine.
Best Regards,
On 19.04.23 02:51, Maxim Dounin wrote:
> Hello!
>
> On Tue, Apr 18, 2023 at 10:25:09PM +0200, basti wrote:
>
>> Hello Maxim,
>>
>> that does not work for me.
>>
>> root at tower:~# resolvectl query icinga2
>> icinga2: 192.168.145.217 -- link: ve-icinga2
>>
>> -- Information acquired via protocol LLMNR/IPv4 in 105.7ms.
>> -- Data is authenticated: no
>>
>> root at tower:~# getent hosts icinga2
>> 192.168.145.217 icinga2
>>
>> root at tower:~# cat /etc/resolv.conf
>> nameserver 127.0.0.53
>> options edns0
>> root at tower:~#
>>
>
> And what "nslookup icinga2 127.0.0.53" shows?
>
>> location / {
>>
>> proxy_redirect off;
>> proxy_set_header Host $remote_addr;
>> proxy_set_header X-Forwarded-For $remote_addr;
>> proxy_set_header X-Forwarded-Host $remote_addr;
>> proxy_set_header X-Real-IP $remote_addr;
>> # proxy_pass https://icinga2;
>> resolver 127.0.0.53 valid=10s;
>> set $upstream "https://icinga2";
>> proxy_pass $upstream;
>>
>> # WebSocket support
>> proxy_http_version 1.1;
>> proxy_set_header Upgrade $http_upgrade;
>> proxy_set_header Connection "upgrade";
>> }
>>
>> 2023/04/18 22:16:28 [error] 779163#779163: *26641 icinga2 could not be
>> resolved (3: Host not found),
>
> As previously mentioned, the message suggests that the name is not
> available via DNS. This is to be resolved on the systemd side.
>
> Unfortunately, I'm not familiar with systemd-nspawn and have no
> idea how to do it properly and if it's at all possible.
>