ssl preread for postgres connection

E
  • 14 May '23
for some context, I recently I tried configuring nginx as a tcp proxy that
routes
connections based on sni to multiple upstream services

the server only exposes one tcp port, and receives all connections there,
for example
a connection to redis.example.com:1234 would be proxy_pass'ed to some port
in the
machine, a connection to www.example.com:1234 to another, etc.

i used nginx itself to terminate the tls for all services for convenience

the problem:
now here is the issue, 1: postgres does some weird custom ssl stuff, which
means I
cannot terminate the ssl from within nginx, and 2: doing a tcp pass through
without
the ssl termination, and attempting to use ssl_preread and
$ssl_preread_server_name
_does not_ work for postgres connections (the module fails to extract the
server name)

what I attempted:
what I first thought of was to expand on the ssl_preread module to support
postgres
connections, I went into the source code and found that the module inserts
a handler into
the `NGX_STREAM_PREREAD_PHASE`
I tried looking into the buffer in this phase and no useful data showed up,
I then tried to
insert a second handler into the `NGX_STREAM_CONTENT_PHASE` and first
noticed
it is never used or initialised to begin with, so I did that, but then it
looks like no buffer
is ever available in this phase

any input, pointers, or suggestions are really welcomed

thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230513/ec63e752/attachment.htm>
R
  • 14 May '23
Hi Eduard,

On Sat, May 13, 2023 at 10:43:59PM -0600, Eduard Vercaemer wrote:
> for some context, I recently I tried configuring nginx as a tcp proxy that
> routes
> connections based on sni to multiple upstream services
> 
> the server only exposes one tcp port, and receives all connections there,
> for example
> a connection to redis.example.com:1234 would be proxy_pass'ed to some port
> in the
> machine, a connection to www.example.com:1234 to another, etc.
> 
> i used nginx itself to terminate the tls for all services for convenience
> 
> the problem:
> now here is the issue, 1: postgres does some weird custom ssl stuff, which
> means I
> cannot terminate the ssl from within nginx

In this case there must be an SSL error logged in nginx error log.
Can you post it?

> 2: doing a tcp pass through
> without
> the ssl termination, and attempting to use ssl_preread and
> $ssl_preread_server_name
> _does not_ work for postgres connections (the module fails to extract the
> server name)
> 
> what I attempted:
> what I first thought of was to expand on the ssl_preread module to support
> postgres
> connections, I went into the source code and found that the module inserts
> a handler into
> the `NGX_STREAM_PREREAD_PHASE`
> I tried looking into the buffer in this phase and no useful data showed up,

Incoming data is written to c->buffer as long as the handler returns NGX_AGAIN.
You just have to wait long enough and have large enough buffer (see directive
preread_buffer_size).

> I then tried to
> insert a second handler into the `NGX_STREAM_CONTENT_PHASE` and first
> noticed
> it is never used or initialised to begin with, so I did that, but then it
> looks like no buffer
> is ever available in this phase
> 
> any input, pointers, or suggestions are really welcomed

If you want to register a content phase handler, assign it to cscf->handler.
A good example is ngx_stream_return() in src/stream/ngx_stream_return_module.c.

--
Roman Arutyunyan
M
  • 14 May '23
Hello!

On Sun, May 14, 2023 at 09:55:54AM +0400, Roman Arutyunyan wrote:

> Hi Eduard,
> 
> On Sat, May 13, 2023 at 10:43:59PM -0600, Eduard Vercaemer wrote:
> > for some context, I recently I tried configuring nginx as a tcp proxy that
> > routes
> > connections based on sni to multiple upstream services
> > 
> > the server only exposes one tcp port, and receives all connections there,
> > for example
> > a connection to redis.example.com:1234 would be proxy_pass'ed to some port
> > in the
> > machine, a connection to www.example.com:1234 to another, etc.
> > 
> > i used nginx itself to terminate the tls for all services for convenience
> > 
> > the problem:
> > now here is the issue, 1: postgres does some weird custom ssl stuff, which
> > means I
> > cannot terminate the ssl from within nginx
> 
> In this case there must be an SSL error logged in nginx error log.
> Can you post it?

Postgres uses their own protocol with STARTTLS-like interface to 
initiate SSL handshake, see here:

https://www.postgresql.org/docs/current/protocol-flow.html#id-1.10.6.7.12

That is, it's not going to work with neither SSL termination, nor 
SSL preread, and needs an implementation of the Postgres protocol.

[...]

-- 
Maxim Dounin
http://mdounin.ru/
J
  • 14 May '23
Hello,

> On Sun, 14 May 2023 17:33:10 +0300
> Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
> 
> On Sun, May 14, 2023 at 09:55:54AM +0400, Roman Arutyunyan wrote:
> 
> > Hi Eduard,
> > 
> > On Sat, May 13, 2023 at 10:43:59PM -0600, Eduard Vercaemer wrote:
> > > for some context, I recently I tried configuring nginx as a tcp
> > > proxy that routes
> > > connections based on sni to multiple upstream services
> > > 
> > > the server only exposes one tcp port, and receives all
> > > connections there, for example
> > > a connection to redis.example.com:1234 would be proxy_pass'ed to
> > > some port in the
> > > machine, a connection to www.example.com:1234 to another, etc.
> > > 
> > > i used nginx itself to terminate the tls for all services for
> > > convenience
> > > 
> > > the problem:
> > > now here is the issue, 1: postgres does some weird custom ssl
> > > stuff, which means I
> > > cannot terminate the ssl from within nginx
> > 
> > In this case there must be an SSL error logged in nginx error log.
> > Can you post it?
> 
> Postgres uses their own protocol with STARTTLS-like interface to 
> initiate SSL handshake, see here:
> 
> https://www.postgresql.org/docs/current/protocol-flow.html#id-1.10.6.7.12
> 
> That is, it's not going to work with neither SSL termination, nor 
> SSL preread, and needs an implementation of the Postgres protocol.
> 
> [...]
> 

Out of curiosity I looked into what 'others' had done for Postgres's
application level negotiation.

https://github.com/envoyproxy/envoy/issues/10942

OP, it might be possible for you to hack this into ssl_preread.c in
ngx_stream_ssl_preread_handler in a similar fashion to that workaround.

It seems you just need to listen / wait for the SSLRequest magic
message bytes, send the 'fake' response, then do the normal handshake
logic.

https://www.postgresql.org/docs/current/protocol-message-formats.html

The other issue is if you want TLS from NGINX -> Postgresql Upstream
you'd need another hack somewhere in ngx_stream_proxy_module.c 
(or a custom content handler as mentioned above).