Linux – could not write temporary statistics file “pg_stat_tmp/pgstat.tmp”: No space left on device

debianlinuxpostgresql

The posgtres sudden stopped on the server and the logs shows the following error:

2015-08-18 02:29:22 IST ERROR:  could not extend file "base/197893/198337": No space left on device
2015-08-18 02:29:22 IST HINT:  Check free disk space.
2015-08-18 02:29:22 IST STATEMENT:  UPDATE "data_shipmentpptlmapping" SET "pptl_id" = '2558', "time" = '2015-08-17 20:40:47.163837+00:00', "bag_seal_status" = 'close', "
shipment_id_id" = 728518 WHERE "data_shipmentpptlmapping"."id" = 557513 
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary statistics file "pg_stat_tmp/pgstat.tmp": No space left on device
2015-08-18 02:29:25 IST LOG:  could not write temporary 2015-08-18 04:29:04 IST LOG:  database system shutdown was interrupted; last known up at 2015-08-18 03:35:25 IST

And here is the postgres details:

postgres=# SHOW data_directory;
        data_directory        
------------------------------
 /var/lib/postgresql/9.1/main
(1 row)

postgres=# SHOW temp_tablespaces;
 temp_tablespaces 
------------------

(1 row)

postgres=# SELECT version()
postgres-# ;
                                                version                                                 
--------------------------------------------------------------------------------------------------------
 PostgreSQL 9.1.14 on x86_64-unknown-linux-gnu, compiled by gcc-4.7.real (Debian 4.7.2-5) 4.7.2, 64-bit
(1 row)

Updating the information..
The disk usage result of the server..

➜  ~  df -h 
Filesystem                                              Size  Used Avail Use% Mounted on
rootfs                                                  909G  465G  398G  54% /
udev                                                     10M   10M     0 100% /dev
tmpfs                                                   1.6G  616K  1.6G   1% /run
/dev/disk/by-uuid/9ca82aa1-92c7-458c-a76d-9ce5c074f5dd  909G  465G  398G  54% /
tmpfs                                                   5.0M     0  5.0M   0% /run/lock
tmpfs                                                   4.8G   24M  4.8G   1% /run/shm
tmpfs                                                   4.8G     0  4.8G   0% /tmp

Adding the information ls -l /var/lib/postgresql/9.1/main.

total 56
drwx------ 8 postgres postgres 4096 Sep 28  2014 base
drwx------ 2 postgres postgres 4096 Aug 18 04:29 global
drwx------ 2 postgres postgres 4096 Aug 13 23:58 pg_clog
drwx------ 4 postgres postgres 4096 Aug 12  2014 pg_multixact
drwx------ 2 postgres postgres 4096 Aug 18 04:29 pg_notify
drwx------ 2 postgres postgres 4096 Aug 12  2014 pg_serial
drwx------ 2 postgres postgres 4096 Aug 19 18:24 pg_stat_tmp
drwx------ 2 postgres postgres 4096 Aug 19 05:34 pg_subtrans
drwx------ 2 postgres postgres 4096 Aug 12  2014 pg_tblspc
drwx------ 2 postgres postgres 4096 Aug 12  2014 pg_twophase
-rw------- 1 postgres postgres    4 Aug 12  2014 PG_VERSION
drwx------ 3 postgres postgres 4096 Aug 19 17:34 pg_xlog
-rw------- 1 postgres postgres  133 Aug 18 04:29 postmaster.opts
-rw------- 1 postgres postgres   93 Aug 18 04:29 postmaster.pid
lrwxrwxrwx 1 root     root       36 Aug 12  2014 server.crt -> /etc/ssl/certs/ssl-cert-snakeoil.pem
lrwxrwxrwx 1 root     root       38 Aug 12  2014 server.key -> /etc/ssl/private/ssl-cert-snakeoil.key

Best Answer

You are out of storage where pg_stat_tmp/pgstat.tmp resides. Which is exactly what the error message tells you.