C++ – fstream::open() Unicode or Non-Ascii characters don’t work (with std::ios::out) on Windows

cfstreamunicodeutf-8windows

In a C++ project, I want to open a file (fstream::open()) (which seems to be a major problem). The Windows build of my program fails miserably.

File "ä" (UTF-8 0xC3 0xA4)
```
std::string s = ...;
//Convert s
std::fstream f;
f.open(s.c_str(), std::ios::binary | std::ios::in); //Works (f.is_open() == true)
f.close();
f.open(s.c_str(), std::ios::binary | std::ios::in | std::ios::out); //Doesn't work
```
The string s is UTF-8 encoded, but then converted from UTF-8 to Latin1 (0xE4). I'm using Qt, so QString::fromUtf8(s.c_str()).toLocal8Bit().constData().

Why can I open the file for reading, but not for writing?
File "и" (UTF-8 0xD0 0xB8)

Same code, doesn't work at all.

It seems, this character doesn't fit in the Windows-1252 charset. How can I open such an fstream (I'm not using MSVC, so no fstream::open(const wchar_t*, ios_base::openmode))?

Best Answer

In Microsoft implementations of STL, there's a non-standard extension (overload) to allow unicode support for UTF-16 encoded strings.

Just pass UTF-16 encoded std::wstring to fstream::open(). This is the only way to make it work with fstream.

You can read more on what I find to be the easiest way to support unicode on windows here: http://utf8everywhere.org/

Related Solutions

Windows – How to set up an editor to work with Git on Windows

Update September 2015 (6 years later)

The last release of git-for-Windows (2.5.3) now includes:

By configuring git config core.editor notepad, users can now use notepad.exe as their default editor.
Configuring git config format.commitMessageColumns 72 will be picked up by the notepad wrapper and line-wrap the commit message after the user edits it.

See commit 69b301b by Johannes Schindelin (dscho).

And Git 2.16 (Q1 2018) will show a message to tell the user that it is waiting for the user to finish editing when spawning an editor, in case the editor opens to a hidden window or somewhere obscure and the user gets lost.

See commit abfb04d (07 Dec 2017), and commit a64f213 (29 Nov 2017) by Lars Schneider (larsxschneider).
Helped-by: Junio C Hamano (gitster).
^{(Merged by Junio C Hamano -- gitster -- in commit 0c69a13, 19 Dec 2017)}

launch_editor(): indicate that Git waits for user input

When a graphical GIT_EDITOR is spawned by a Git command that opens and waits for user input (e.g. "git rebase -i"), then the editor window might be obscured by other windows.
The user might be left staring at the original Git terminal window without even realizing that s/he needs to interact with another window before Git can proceed. To this user Git appears hanging.

Print a message that Git is waiting for editor input in the original terminal and get rid of it when the editor returns, if the terminal supports erasing the last line

Original answer

I just tested it with git version 1.6.2.msysgit.0.186.gf7512 and Notepad++5.3.1

I prefer to not have to set an EDITOR variable, so I tried:

git config --global core.editor "\"c:\Program Files\Notepad++\notepad++.exe\""
# or
git config --global core.editor "\"c:\Program Files\Notepad++\notepad++.exe\" %*"

That always gives:

C:\prog\git>git config --global --edit
"c:\Program Files\Notepad++\notepad++.exe" %*: c:\Program Files\Notepad++\notepad++.exe: command not found
error: There was a problem with the editor '"c:\Program Files\Notepad++\notepad++.exe" %*'.

If I define a npp.bat including:

"c:\Program Files\Notepad++\notepad++.exe" %*

and I type:

C:\prog\git>git config --global core.editor C:\prog\git\npp.bat

It just works from the DOS session, but not from the git shell.
(not that with the core.editor configuration mechanism, a script with "start /WAIT..." in it would not work, but only open a new DOS window)

Bennett's answer mentions the possibility to avoid adding a script, but to reference directly the program itself between simple quotes. Note the direction of the slashes! Use / NOT \ to separate folders in the path name!

git config --global core.editor \
"'C:/Program Files/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin"

Or if you are in a 64 bit system:

git config --global core.editor \
"'C:/Program Files (x86)/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin"

But I prefer using a script (see below): that way I can play with different paths or different options without having to register again a git config.

The actual solution (with a script) was to realize that:
what you refer to in the config file is actually a shell (/bin/sh) script, not a DOS script.

So what does work is:

C:\prog\git>git config --global core.editor C:/prog/git/npp.bat

with C:/prog/git/npp.bat:

#!/bin/sh
"c:/Program Files/Notepad++/notepad++.exe" -multiInst "$*"

#!/bin/sh
"c:/Program Files/Notepad++/notepad++.exe" -multiInst -notabbar -nosession -noPlugin "$*"

With that setting, I can do 'git config --global --edit' from DOS or Git Shell, or I can do 'git rebase -i ...' from DOS or Git Shell.
Bot commands will trigger a new instance of notepad++ (hence the -multiInst' option), and wait for that instance to be closed before going on.

Note that I use only '/', not \'. And I installed msysgit using option 2. (Add the git\bin directory to the PATH environment variable, but without overriding some built-in windows tools)

The fact that the notepad++ wrapper is called .bat is not important.
It would be better to name it 'npp.sh' and to put it in the [git]\cmd directory though (or in any directory referenced by your PATH environment variable).

How to use unicode characters in Windows command line

Try:

chcp 65001

which will change the code page to UTF-8. Also, you need to use Lucida console fonts.

Best Answer

Related Solutions

Windows – How to set up an editor to work with Git on Windows

launch_editor(): indicate that Git waits for user input

How to use unicode characters in Windows command line

Related Topic

`launch_editor()`: indicate that Git waits for user input