Ftp – File encoding altered by FTP transfer

There's no 100% accurate way really, but there's a way to give a good guess.

There is a python library chardet which is available here: https://pypi.python.org/pypi/chardet

e.g.

See what the current LANG variable is set to:

$ echo $LANG
en_IE.UTF-8

Create a filename that'll need to be encoded with UTF-8

$ touch mÉ.txt

Change our encoding and see what happens when we try and list it

$ ls m*
mÉ.txt
$ export LANG=C
$ ls m*
m??.txt

OK, so now we have a filename encoded in UTF-8 and our current locale is C (standard Unix codepage).

So start up python, import chardet and get it to read the filename. I'm use some shell globbing (i.e. expansion through the * wildcard character) to get my file. Change "ls m*" to whatever will match one of your example files.

>>> import chardet
>>> import os
>>> chardet.detect(os.popen("ls m*").read())
{'confidence': 0.505, 'encoding': 'utf-8'}

As you can see, it's only a guess. How good a guess is shown by the "confidence" variable.

Your editor probably did insert a BOM "Byte Order Mark" in the beginning of the file.

This is bad behaviour from your editor as BOMs are of no use in UTF-8 encoding, it is only usefull in UTF-16 or UTF-32. However, this behaviour seems to be common in several editors.

Either, there is a way to specify that you do not want the BOM in your file (e.g. in TextWrangler on a Mac, using the "UTF-8 no BOM" encoding), or you should remove the 2 first bytes using whatever app you can...