Mercurial convert filename encoding

crucibleencodingmercurial

I have Mercurial repositories running on Apache with mod_wsgi. Repositories have all filenames encoded in windows-1251. This encoding is used for historically reasons: they were converted to mercurial from svn, windows-1251 is default windows encoding for russian locale.

Now programmers want to use Crucible tool for code review. It can't undersand filenames in any other encoding than utf-8. So I need to convert them from windows-1251 to utf-8. Does anyone know how to do this? Mercurial convert extension doesn't have options to convert encodings.

hgweb.config:

[web]
#encoding = UTF-8
encoding = windows-1251
#allow_archive = gz, zip, bz2
allow_archive = zip
allow_push = *
push_ssl = false

[extensions]

[collections]
/data/mercurial = /data/mercurial

Best Answer

You are right that the convert extension doesn't support this in a nice way currently. That is, you cannot ask it to recode from encoding X to encoding Y. However, you can ask it to rename the files one by one for you! First create a file called rename.py with

import sys
for path in sys.stdin:
    old = path[:-1] # strip newline
    new = old.decode("cp1251").encode("utf-8")
    print 'rename "%s" "%s"' % (old, new)

Then run

$ hg manifest --all | python rename.py > rename.txt

This creates your file map. You can now use

$ hg convert --filemap rename.txt cp1251-repo utf-8-repo

to convert the repository into a new repository. In the new repository, it will look like the files have always been saved using UTF-8 file names.

Note: The file names are now stored as UTF-8 in the repository. This means that checkouts will look fine on moderns Linux machines. Windows, however, does not use UTF-8 file names. The FixUtf-8 extension must be used to make Mercurial convert the UTF-8 file names into UTF-16 on the fly. This will create readable file names on Windows too.

Note: Everybody will have to re-clone the new repository! Changing any part of the history inevitably changes all the changesets hashes too. So to pull this off, you need to either

make everybody push to the server,
convert the repositories on the server,
have people re-clone

make everybody run the above commands on their local repositories
convert the repositories on the server

Either way works since the conversion is deterministic and so your users can run it themselves if they have Python available. If they only have a TortoiseHg installation, then it's probably easiest if you convert for them on your server.

I looked at making the convert extension support this more directly and have sent a patch to the Mercurial mailinglist for more direct support for this.

Related Solutions

Linux – How to Determine Filename Language Encoding

There's no 100% accurate way really, but there's a way to give a good guess.

There is a python library chardet which is available here: https://pypi.python.org/pypi/chardet

e.g.

See what the current LANG variable is set to:

$ echo $LANG
en_IE.UTF-8

Create a filename that'll need to be encoded with UTF-8

$ touch mÉ.txt

Change our encoding and see what happens when we try and list it

$ ls m*
mÉ.txt
$ export LANG=C
$ ls m*
m??.txt

OK, so now we have a filename encoded in UTF-8 and our current locale is C (standard Unix codepage).

So start up python, import chardet and get it to read the filename. I'm use some shell globbing (i.e. expansion through the * wildcard character) to get my file. Change "ls m*" to whatever will match one of your example files.

>>> import chardet
>>> import os
>>> chardet.detect(os.popen("ls m*").read())
{'confidence': 0.505, 'encoding': 'utf-8'}

As you can see, it's only a guess. How good a guess is shown by the "confidence" variable.

Nginx – Mercurial not receiving push

I spent ages trying to get hgwebdir working with nginx and it is a bit of a hassle, because hgwebdir won't send the request to the browser for authentication. In the end I solved it like this:

server {
        listen 80;
        listen 10240;
        server_name code.zofrex.com;
        access_log /home/zofrex/websites/code.zofrex.com/logs/access.log;
        error_log /home/zofrex/websites/code.zofrex.com/logs/errors.log;

        location / {
                limit_except GET {
                        proxy_pass http://localhost:81;
                }
                fastcgi_pass 127.0.0.1:10001;
                include fastcgi_params;
        }

        location ~ ^/(HGBot|ZeroBotAHD|zoebot|FZeroZBot|RDPrototype|RSPS3000|zerobot|rmi)  {
                proxy_pass http://localhost:81;
        }

        include defaults;
}

server {
        listen 81;

        access_log /home/zofrex/websites/code.zofrex.com/logs/access_secure.log;
        error_log /home/zofrex/websites/code.zofrex.com/logs/errors_secure.log;

        location / {
                auth_basic           "Restricted";
                auth_basic_user_file /home/zofrex/passwords;
                fastcgi_pass 127.0.0.1:10001;
                include fastcgi_params;
        }

        include defaults;
}

This is the only way you can do it, really, as limit_except can't be combined with fastcgi_pass - in fact no conditionals can, but they be combined with proxy_pass. No idea why.

So how does this work? If a GET request comes in, it's handed off to hgwebdir straight away. If a POST (i.e. a push) comes in, the request upstreams to the proxy on port 81 which always prompts for basic auth, thus ensuring it happens.

Why the location with the huge regex? Those are protected projects that I want to require auth for even read access.

Why the listening on port 10240? I can't remember why that's there.

Oh, and I'm using wsgi with spawn-fcgi, which is why it's fastcgi_pass. This exact config probably won't be right for you but hopefully the same trick will solve your pushing woes.

Best Answer

Related Solutions

Linux – How to Determine Filename Language Encoding

Nginx – Mercurial not receiving push

Related Topic