Linux tools to find duplicate files

You could use Files/FilesMatch and a regular expression:

<Files ~ "\.txt$">
    Order allow,deny
    Deny from all
</Files>

This is how .htpasswd is protected.

or redirect any access of .txt to a 404:

RedirectMatch 404 \.txt$

There's no 100% accurate way really, but there's a way to give a good guess.

There is a python library chardet which is available here: https://pypi.python.org/pypi/chardet

e.g.

See what the current LANG variable is set to:

$ echo $LANG
en_IE.UTF-8

Create a filename that'll need to be encoded with UTF-8

$ touch mÉ.txt

Change our encoding and see what happens when we try and list it

$ ls m*
mÉ.txt
$ export LANG=C
$ ls m*
m??.txt

OK, so now we have a filename encoded in UTF-8 and our current locale is C (standard Unix codepage).

So start up python, import chardet and get it to read the filename. I'm use some shell globbing (i.e. expansion through the * wildcard character) to get my file. Change "ls m*" to whatever will match one of your example files.

>>> import chardet
>>> import os
>>> chardet.detect(os.popen("ls m*").read())
{'confidence': 0.505, 'encoding': 'utf-8'}

As you can see, it's only a guess. How good a guess is shown by the "confidence" variable.