Hi list, I have a situation where I am looking for the best way to
solve it, I have a load balancer with nginx 1.22.1 and behind it I
have three backend servers:
/ -> App1
/
load balancer. ----/ --> App2
/
/---> App3
if shutdown app1, the load balancer keeps sending traffic to app1 ,
and the clients are in a lag waiting for app1 to respond, I think the
load balancer should send all those clients to app2 and app3, but it
doesn't.
it put me in research mode :) and the nginx version can't do that,
it's only in the plus version, correct me if I'm wrong, but "Voilà"
https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#hc_active
, It gave me hope when reading that this option could help me, that by
adding it to the nginx configuration it would not be able to continue
sending requests, but it is not like that.
logs:
2023/01/12 18:07:38 [error] 26895#26895: *834 no live upstreams while
connecting to upstream, client: 44.210.106.130, server: demo.app.com,
request: "GET /aaaaaaaaaaaaaaaaaaaaaaaaaqr HTTP/1.1", upstream:
"http://paginaweb/aaaaaaaaaaaaaaaaaaaaaaaaaqr", host: "demo.app.com",
referrer: "http://173.255.X.X:80/aaaaaaaaaaaaaaaaaaaaaaaaaqr"
2023/01/12 18:07:38 [error] 26895#26895: *832 no live upstreams while
connecting to upstream, client: 44.210.106.130, server: demo.app.com,
request: "GET /99vt HTTP/1.1", upstream: "http://paginaweb/99vt",
host: "demo.app.com", referrer: "http://173.255.X.X:80/99vt"
2023/01/12 18:07:38 [error] 26895#26895: *838 no live upstreams while
connecting to upstream, client: 44.210.106.130, server: demo.app.com,
request: "GET /99vu HTTP/1.1", upstream: "http://paginaweb/99vu",
host: "173.255.X.X:443"
2023/01/12 18:07:39 [error] 26895#26895: *841 no live upstreams while
connecting to upstream, client: 44.210.106.130, server: demo.app.com,
request: "GET /99vu HTTP/1.1", upstream: "http://paginaweb/99vu",
host: "demo.app.com", referrer: "http://173.255.X.X:80/99vu"
2023/01/12 20:24:16 [error] 26895#26895: *6206 upstream prematurely
closed connection while reading response header from upstream, client:
167.94.138.62, server: demo.app.com, request: "GET / HTTP/1.1",
upstream: "http://104.237.138.27:80/", host: "demo.app.com"
2023/01/12 20:24:16 [error] 26895#26895: *6206 no live upstreams while
connecting to upstream, client: 167.94.138.62, server: demo.app.com,
request: "GET / HTTP/1.1", upstream: "http://paginaweb/", host:
"demo.app.com"
2023/01/12 20:24:26 [error] 26895#26895: *6220 no live upstreams while
connecting to upstream, client: 167.248.133.46, server: demo.app.com,
request: "GET / HTTP/1.1", upstream: "http://paginaweb/", host:
"demo.app.com"
2023/01/12 21:14:27 [error] 26895#26895: *7559 no live upstreams while
connecting to upstream, client: 162.221.192.26, server: demo.app.com,
request: "GET / HTTP/1.1", upstream: "http://paginaweb/", host:
"173.255.X.X"
#### CONFIG LB ####
upstream paginaweb {
ip_hash;
server 96.X.X.X:80 weight=1 fail_timeout=30s max_fails=3;
server 104.X.X.X:80 weight=1 fail_timeout=30s max_fails=3;
server 190.X.X.X:80 weight=1 fail_timeout=30s max_fails=3;
keepalive 5;
}
server {
server_tokens off;
listen 80;
server_name demo.app.com;
pagespeed unplugged;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name demo.app.com;
access_log /var/www/vhost/demo.app.com/logs/access.log specialLog;
error_log /var/www/vhost/demo.app.com/logs/error.log;
add_header Cache-Control "max-age=86400, public";
ssl_certificate /etc/letsencrypt/live/demo.app.com/fullchain.pem;
# managed by Certbot
ssl_certificate_key
/etc/letsencrypt/live/demo.app.com/privkey.pem; # managed by Certbot
ssl_trusted_certificate /etc/letsencrypt/live/demo.app.com/chain.pem;
# ssl_certificate /etc/nginx/ssl/nginx-selfsigned.crt;
# ssl_certificate_key /etc/nginx/ssl/nginx-selfsigned.key;
ssl_protocols TLSv1.3 TLSv1.2;
ssl_ciphers
ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-A
ES256-SHA384;
ssl_prefer_server_ciphers on;
ssl_stapling on;
ssl_stapling_verify on;
add_header Strict-Transport-Security "max-age=31557600; includeSubDomains";
add_header X-Xss-Protection "1; mode=block" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
keepalive_requests 1000;
keepalive_timeout 75 75;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 30m;
gzip on;
gzip_types text/plain text/css text/xml text/javascript;
gzip_proxied any;
gzip_vary on;
client_max_body_size 5120m;
location / {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Real_IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-WAf-Proxy true;
real_ip_header X-Real-IP;
proxy_http_version 1.1;
proxy_pass http://paginaweb;
proxy_redirect off;
# health_check;
}
}
any idea?
--
rickygm
http://gnuforever.homelinux.com
Hello!
On Thu, Jan 12, 2023 at 09:30:12PM -0600, Rick Gutierrez wrote:
> Hi list, I have a situation where I am looking for the best way to
> solve it, I have a load balancer with nginx 1.22.1 and behind it I
> have three backend servers:
>
> / -> App1
> /
> load balancer. ----/ --> App2
> /
> /---> App3
>
> if shutdown app1, the load balancer keeps sending traffic to app1 ,
> and the clients are in a lag waiting for app1 to respond, I think the
> load balancer should send all those clients to app2 and app3, but it
> doesn't.
>
> it put me in research mode :) and the nginx version can't do that,
> it's only in the plus version, correct me if I'm wrong, but "Voilà"
> https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#hc_active
> , It gave me hope when reading that this option could help me, that by
> adding it to the nginx configuration it would not be able to continue
> sending requests, but it is not like that.
Certainly nginx can do that. By default, all user requests are
used by nginx to detect any upstream server failures, and re-route
requests to other available servers.
Active health checks, which are indeed only available in the
commercial version, are only different that they also use requests
generated periodically by nginx-plus itself. This might improve
service to some real clients in some specific cases, but not
generally required.
> logs:
>
> 2023/01/12 18:07:38 [error] 26895#26895: *834 no live upstreams while
> connecting to upstream, client: 44.210.106.130, server: demo.app.com,
> request: "GET /aaaaaaaaaaaaaaaaaaaaaaaaaqr HTTP/1.1", upstream:
> "http://paginaweb/aaaaaaaaaaaaaaaaaaaaaaaaaqr", host: "demo.app.com",
> referrer: "http://173.255.X.X:80/aaaaaaaaaaaaaaaaaaaaaaaaaqr"
> 2023/01/12 18:07:38 [error] 26895#26895: *832 no live upstreams while
> connecting to upstream, client: 44.210.106.130, server: demo.app.com,
> request: "GET /99vt HTTP/1.1", upstream: "http://paginaweb/99vt",
> host: "demo.app.com", referrer: "http://173.255.X.X:80/99vt"
The errors indicate that all your upstream servers were not
responding properly, and were either all tried for the particular
request, or were disabled based on "fail_timeout=30s max_fails=3"
in your configuration.
Usually looking into other errors in the logs makes it immediately
obvious what actually happened. Alternatively, you may want to
further dig into what happened with the requests by logging the
$upstream_addr and $upstream_status variables (see
https://nginx.org/r/$upstream_addr and
https://nginx.org/r/$upstream_status for details).
[...]
--
Maxim Dounin
http://mdounin.ru/
El vie, 13 ene 2023 a las 17:51, Maxim Dounin (<mdounin at mdounin.ru>) escribió:
>
>
>
> Certainly nginx can do that. By default, all user requests are
> used by nginx to detect any upstream server failures, and re-route
> requests to other available servers.
>
> Active health checks, which are indeed only available in the
> commercial version, are only different that they also use requests
> generated periodically by nginx-plus itself. This might improve
> service to some real clients in some specific cases, but not
> generally required.
ok
>
> > logs:
> >
> > 2023/01/12 18:07:38 [error] 26895#26895: *834 no live upstreams while
> > connecting to upstream, client: 44.210.106.130, server: demo.app.com,
> > request: "GET /aaaaaaaaaaaaaaaaaaaaaaaaaqr HTTP/1.1", upstream:
> > "http://paginaweb/aaaaaaaaaaaaaaaaaaaaaaaaaqr", host: "demo.app.com",
> > referrer: "http://173.255.X.X:80/aaaaaaaaaaaaaaaaaaaaaaaaaqr"
> > 2023/01/12 18:07:38 [error] 26895#26895: *832 no live upstreams while
> > connecting to upstream, client: 44.210.106.130, server: demo.app.com,
> > request: "GET /99vt HTTP/1.1", upstream: "http://paginaweb/99vt",
> > host: "demo.app.com", referrer: "http://173.255.X.X:80/99vt"
>
> The errors indicate that all your upstream servers were not
> responding properly, and were either all tried for the particular
> request, or were disabled based on "fail_timeout=30s max_fails=3"
> in your configuration.
>
> Usually looking into other errors in the logs makes it immediately
> obvious what actually happened. Alternatively, you may want to
> further dig into what happened with the requests by logging the
> $upstream_addr and $upstream_status variables (see
> https://nginx.org/r/$upstream_addr and
> https://nginx.org/r/$upstream_status for details).
>
Thanks. The problem here is that crowdsec was blocking my connection.
I also found an interesting module that does almost the same thing as
the commercial version.
https://github.com/yaoweibin/nginx_upstream_check_module , I'm already
evaluating it
--
rickygm
http://gnuforever.homelinux.com