Chmod/chown/mkdir: Operation not permitted as root

amazon-linuxchownpermissionsroots3fs

I get Operation not permitted for root user on one server but not on the other "identical" server.

Running on Amazon Linux 1.

Server 1:

[root@preprod-1 ]# chown root:root /s3mnt/outliers/
chown: changing ownership of ‘/s3mnt/outliers/’: Operation not permitted

[root@preprod-1 ]# chmod -x /s3mnt/outliers/
chmod: changing permissions of ‘/s3mnt/outliers/’: Operation not permitted

[root@preprod-1 ]# cd /s3mnt/outliers

[root@preprod-1 outliers]# mkdir test2
mkdir: cannot create directory ‘test2’: Operation not permitted

[root@preprod-1 ]# ls -ld /s3mnt/outliers
drwxrwxrwx 1 brutus brutus 0 Aug  2  2017 /s3mnt/outliers

[root@preprod-1 ]# ls -ld /s3mnt/
drwxrwxrwx 1 brutus brutus 0 Jan  1  1970 /s3mnt/

[root@preprod-1 ]# sestatus
SELinux status:                 disabled

Server 2:

[root@prod-1-1 via]# ls -ld /s3mnt/outliers
d--------- 1 brutus brutus 0 Jan  1  1970 /s3mnt/outliers

[root@prod-1-1 via]# ls -ld /s3mnt/
drwxrwxrwx 1 brutus brutus 0 Jan  1  1970 /s3mnt/

[root@prod-1-1 via]# cd /s3mnt/outliers

[root@prod-1-1 outliers]# mkdir test2

[root@prod-1-1 outliers]# ls -ld test2/
drwxr-xr-x 1 brutus brutus 0 Nov 25 12:45 test2/

[root@prod-1-1 outliers]# chmod -x /s3mnt/outliers/test2/

[root@prod-1-1 outliers]# ls -ld test2/
drw-r--r-- 1 brutus brutus 0 Nov 25 12:45 test2/

[root@prod-1-1 outliers]# chown root:root test2

[root@prod-1-1 outliers]# ls -ld test2/
drw-r--r-- 1 brutus brutus 0 Nov 25 12:45 test2/

[root@prod-1-1 ]# sestatus
SELinux status:                 disabled

Mounted as rw per mount cmd on both servers. Both servers have identical /etc/fstab.

Setup on both servers is identical:

lrwxrwxrwx   1 root   root      14 May 21  2018 s3mnt -> /storage/s3mnt 
drwxr-xr-x   7 brutus brutus  4096 May 21  2018 storage 

# mount 
s3fs on /storage/s3mnt type fuse.s3fs (rw,relatime,user_id=0,group_id=0,allow_other) 

When changing permissions on the mountpoint I get the expected changing ownership of Input/output error. So it's not that. Wish it was 🙁

What else can I check to find out why I am running into the above issues on one server but not the other?

Best Answer

It turned out to be an issue with S3 bucket policy denying access to the S3 mount.