Cannot remove an empty folder with no user using by rm -rf

I was having the same issue, I was trying and failing to delete

/usr/local/tomcat/data

with the error rm: cannot remove/usr/local/tomcat/data': Device or resource busy`

until I noticed that df -h said

/dev/vda3              20G  172M   20G   1% /usr/local/tomcat/data

that is, I had a partition mounted to that point. Mystery solved.

It sounds to me like you may have had an enormous number of files in the directory. I've seen very similar symptoms to what you describe with very large directories.

There may have been some sort of issue (possibly a configuration issue with logrotate) where you ended up with millions of files. Once you get into the millions, most filesystems start having issues. Just doing an ls can take an enormous amount of time, and may even die if you wait long enough. Commands like rm will similarly have issues due to the sheer number of files. Similarly, things like logrotate will appear to hang because doing simple operations on the directory will take a really long time to complete.

One way to verify this is to do an ls -l on the directory above (/var/log/apache2/ in this case), and view the size of the directory entry (not the contents). For example (from a server I have, where there are a few directories with many files):

$ ls -lh
rwxrwxr-x 2 ironport ironport  11M Apr 21 19:37 oma01syslog01/
drwxrwxr-x 2 ironport ironport  12M Apr 21 19:37 oma01syslog02/
drwxrwxr-x 2 ironport ironport 5.8M Mar 17 12:30 sat01syslog01/
drwxrwxr-x 2 ironport ironport 4.0K Jun 29  2011 swn01syslog01/

$ for DIR in * ; do echo -n "$DIR:  " ; find $DIR -type f -print | wc -l ; done
oma01syslog01:  204332
oma01syslog02:  195500
sat01syslog01:  70960
swn01syslog01:  0

Notice the size of the directory entry as compared to the number of files in it.

Your best bet is to try to view the files with something like find. It doesn't try to pull the whole list before displaying them, and can allow you to query and operate on a directory without the issues you see in ls and rm. Some useful examples are below.

Number of files:

find ./directory/name/here -type f -print | wc -l

View a sample of files in the directory:

find ./directory/name/here -type f -print | head -20 | xargs ls -lh

Remove all files matching a pattern from the directory:

find ./directory/name/here -type f -iname '*.log' -print | xargs rm -rf

Note: I just noticed your comment, where you showed an ls from one directory up:

drwxr-xr-x 2 root root 126406656 2011-10-17 18:51 vhosts

This confirms my suspicions. 126406656 bytes is 120MB. That's a pretty huge directory entry (from my example above, I had an 11MB directory entry with 200k files), and suggests that you had millions of files in that directory.