Postgresql spawning a ridiculous number of postmaster processes

postgresql

For some reason postgres is spawning >700 postmaster processes for handling database requests and the postgres log file if full of 'unexpected EOF on client connection', 'incomplete startup packet' and 'sorry, too many clients already'. netstat tells me that all the open connections are local and I'm pretty sure are coming from postgres internally. This particular instance has been running just fine for the last 230 days or so and nothing has changed configuration wise. Any thoughts on where I should be looking to try and resolve this issue?

This is my first time diagnosing a problem like this so if there is any steps I can take to help narrow down the cause that would be helpful as well.

UPDATE: Turns out there was an instance of tomcat running on a remote machine that was trying to connect to postgres through a ssh tunnel causing all hell to break loose.

Best Answer

Postgres spawns a postmaster for every connection. This is the way it's supposed to work. If you're spawning a ton of postmaster processes you have something initiating a ton of connections, and if that's not what you expect you probably have a broken/badly-behaved application creating a mess (a common example: One that keeps re-initializing the DB connection, but never closes it. You'll leak a session for every connection initialization).

If this is recent, start with "What Changed?" If not, start looking at all your apps that talk to Postgres (roughly in order of usage volume) and see if you can find the bad apple.

Re: your log messages --

  • unexpected EOF on client connection
    Something caused an established Postgres connection to die without properly closing (the postmaster may hang around for a while cleaning up the resulting mess). Find out what caused the connection to die (did the process initiating it crash? Is the application poorly-written and exiting before cleaning up its DB handles?) and fix it

  • incomplete startup packet
    Something connected to Postgres and didn't initiate a proper Postgres handshake/startup. Usually the resulting postmaster goes away in a few seconds.
    Find what's doing this and fix it. (This could be your monitoring system, someone port-scanning your network, or a badly written application somewhere)

  • sorry, too many clients already
    I think this one is self-explanatory. Fix the above and it will probably go away, otherwise consider increasing max_connections in postgresql.conf.
Related Topic