I recently worked with a file from a link, now it is indexed in my "Shared with me" folder and is taking up space in my drive folder. It lists itself as a shared drive 
but I can't delete it
and if I try to remove my self from the user permissions I can't
So it's stuck in my Google drive "Shared with me" section, taking up storage space on my drive, I don't have permission to remove myself, and I can't get into contact with the owner to have them delete me from the permitted users list.
What do I do?
Best Answer
The item isn't taking up storage space, as the storage quota comes off the organization that owns the Shared Drive. It is however displayed in Shared with Me, and that is your concern. So, it is more a problem of organization/clutter. Under usual circumstances, one would be able to remove the item from Shared with Me, but that option is disabled. I confirm that I'm able to reproduce the problem.
The best way forward is to leave feedback to the Google Drive developers about how you don't like this problem. You can do that by clicking on the ? icon on the top right of drive.google.com, then clicking "Send feedback". Alternatively, in the mobile app, tap the menu, then tap "Help and feedback", then "Send feedback".