If a file that has been deleted is still open by a process, the space will not be reclaimed until the process closes the file (or is killed). If you can not identify the process that is holding a file open, then a reboot will help as that will close all running processes (and so close all open files).
Another consideration is filesystem corruption. As this is your root filesystem you may need to reboot and force a filesystem check on restart (shutdown -rF now
). Be sure that you are configured to perform a non-interactive scan+fix unless you have KVM access or similar (so you can interact during the boot process) otherwise your remote machine will hang waiting for local input if it finds errors during the check.
Edit: (as per question in comment)
If you know the process that is holding the file open, you can just restart that particular process (either via service stop/start/restart scripts or by killing and restarting more manually) rather than the whole instance.
Also some programs support being able to reset themselves without restarting which will usually include closing and restarting log files (solving your problem if it is indeed due to a deleted log file that is still open) in response to being sent a SIGHUP signal (via kill
). Resetting processes this way is sometimes preferable as it reduces (often to zero) the amount of time a server process is unable to accept new connections. This is often what happens when you run /etc/init.d/<service> reload
instead of /etc/init.d/<service> restart
(if fact I've seen restart
implemented this way so to do a proper full reset you have to do /etc/init.d/<service> stop; /etc/init.d/<service> start
).
None of the other solutions will work if the volume is used as a root (bootable) device.
The newly created disk is missing the boot partition, so it would need to have GRUB installed and some flags set up correctly before an instance can use it as a root volume.
My (as of today, working) solution for shrinking a root volume is:
Background: We have an instance A, whose root volume we want to shrink. Let's call this volume VA. We want to shrink VA from 30GB to let's say 10GB
- Create a new ec2 instance, B, with the same OS as the instance A. Also kernels must match so upgrade or downgrade as needed. As storage, pick a volume that's the same type as VA, but with a size of 10GB. (or whatever your target size is). So now we have an instance B which uses this new volume (let's call it VB) as a root volume.
- Once the new instance (B) is running. Stop it and detach it's root volume (VB).
NOTE: The following steps are mostly taken from @bill 's solution:
Stop the instance you want to resize (A).
Create a snapshot of the volume VA and then create a "General Purpose SSD" volume from that snapshot. This volume we'll call it VASNAP.
Spin a new instance with amazon Linux, we'll call this instance C. We will just use this instance to copy the contents of VASNAP to VB. We could probably also use instance A to do these steps, but I prefer to do it in an independent machine.
Attach the following volumes to instance C.
/dev/xvdf for VB.
/dev/xvdg for VASNAP.
Reboot instance C.
Log onto instance C via SSH.
Create these new directories:
mkdir /source /target
- Format VB's main partition with an ext4 filesystem:
mkfs.ext4 /dev/xvdf1
If you get no errors, proceed to Step 11. Otherwise, if you do not have /dev/xvdf1
, you need to create the partition by doing the following i-vii:
i) If /dev/xvdf1
does not exist for whatever reason, you need to create it. First enter:
sudo fdisk /dev/xvdf
.
ii) Wipe disk by entering:
wipefs
iii) Create a new partition by entering:
n
iv) Enter p
to create primary partition
v) Keep pressing enter to continue with default settings.
vi) When it asks for a command again, enter w
to write changes and quit.
vii) Verify you have the /dev/xvdf1
partition by doing:
lsblk
You should see something like:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 250G 0 disk
└─xvda1 202:1 0 250G 0 part
xvdf 202:80 0 80G 0 disk
└─xvdf1 202:81 0 80G 0 part
xvdg 202:96 0 250G 0 disk
└─xvdg1 202:97 0 250G 0 part
Now proceed to Step 11.
- Mount it to this directory:
mount -t ext4 /dev/xvdf1 /target
- This is very important, the file system needs an e2label for Linux to recognize it and boot it, use "e2label /dev/xvda1" on an active instance to see what it should be, in this case the label is: "/"
e2label /dev/xvdf1 /
- Mount VASNAP on /source:
mount -t ext4 /dev/xvdg1 /source
- Copy the contents:
rsync -vaxSHAX /source/ /target
Note: there is no "/" following "/target". Also, there may be a few errors about symlinks and attrs, but the resize was still successful
- Umount VB:
umount /target
Back in AWS Console: Dettach VB from instance C, and also dettach VA from A.
Attach the new sized volume (VB) to the instance as: "/dev/xvda"
Boot instance A, now it's root device is 10GB :)
Delete both instances B and C, and also all volumes but VB, which is now instance A's root volume.
Best Answer
You need to extend your partition first. "resize2fs" is a filesytem tool, not a partitioning tool. Use a tool like fdisk or parted on your base disk (xvda) to first extend your partition to the end of the disk.
To do this using parted: https://www.gnu.org/software/parted/manual/html_node/parted_31.html
To do this using fdisk: https://access.redhat.com/articles/1190213
After doing this, you may then resize your filesystem.
While it is technically possible to do this to an online root partition, I would not recommend it. This is best done while the native system is offline (using a live system to operate on it instead).
Always back up your data before doing anything like this.