Windows – PowerShell stripping double quotes from command line arguments

command linepowershellwindows

Recently I have been having some trouble using GnuWin32 from PowerShell whenever double quotes are involved.

Upon further investigation, it appears PowerShell is stripping double quotes from command line arguments, even when properly escaped.

PS C:\Documents and Settings\Nick> echo '"hello"'
"hello"
PS C:\Documents and Settings\Nick> echo.exe '"hello"'
hello
PS C:\Documents and Settings\Nick> echo.exe '\"hello\"'
"hello"

Notice that the double quotes are there when passed to PowerShell's echo cmdlet, but when passed as an argument to echo.exe, the double quotes are stripped unless escaped with a backslash (even though PowerShell's escape character is a backtick, not a backslash).

This seems like a bug to me. If I am passing the correct escaped strings to PowerShell, then PowerShell should take care of whatever escaping may be necessary for however it invokes the command.

What is going on here?

For now, the fix is to escape command line arguments in accordance with these rules (which seem to be used (indirectly) by the CreateProcess API call which PowerShell uses to invoke .exe files):

To pass a double quote, escape with a backslash: \" -> "
To pass a one or more backslashes followed by a double quote, escape each backslash with another backslash and escape the quote: \\\\\" -> \\"
If not followed by a double quote, no escaping is necessary for backslashes: \\ -> \\

Note that further escaping of double quotes may be necessary to escape the double quotes in the Windows API escaped string to PowerShell.

Here are some examples, with echo.exe from GnuWin32:

PS C:\Documents and Settings\Nick> echo.exe "\`""
"
PS C:\Documents and Settings\Nick> echo.exe "\\\\\`""
\\"
PS C:\Documents and Settings\Nick> echo.exe "\\"
\\

I imagine that this can quickly become hell if you need to pass a complicated command line parameter. Of course, none of this documented in the CreateProcess() or PowerShell documentation.

Also note that this is not necessary to pass arguments with double quotes to .NET functions or PowerShell cmdlets. For that, you need only escape your double quotes to PowerShell.

Edit: As Martin pointed out in his excellent answer, this is documented in the CommandLineToArgv() function (which the CRT uses to parse the command line arguments) documentation.

Best Answer

It is a known thing:

It's FAR TOO HARD to pass parameters to applications which require quoted strings. I asked this question in IRC with a "roomful" of PowerShell experts, and it took hour for someone to figure out a way (I originally started to post here that it is simply not possible). This completely breaks PowerShell's ability to serve as a general purpose shell, because we can't do simple things like executing sqlcmd. The number one job of a command shell should be running command-line applications... As an example, trying to use SqlCmd from SQL Server 2008, there is a -v parameter which takes a series of name:value parameters. If the value has spaces in it, you must quote it...

...there is no single way to write a command line to invoke this application correctly, so even after you master all 4 or 5 different ways of quoting and escaping things, you're still guessing as to which will work when ... or, you can just shell out to cmd, and be done with it.

Bash Space-Separated (e.g., `--option argument`)

cat >/tmp/demo-space-separated.sh <<'EOF'
#!/bin/bash

POSITIONAL=()
while [[ $# -gt 0 ]]; do
  key="$1"

  case $key in
    -e|--extension)
      EXTENSION="$2"
      shift # past argument
      shift # past value
      ;;
    -s|--searchpath)
      SEARCHPATH="$2"
      shift # past argument
      shift # past value
      ;;
    -l|--lib)
      LIBPATH="$2"
      shift # past argument
      shift # past value
      ;;
    --default)
      DEFAULT=YES
      shift # past argument
      ;;
    *)    # unknown option
      POSITIONAL+=("$1") # save it in an array for later
      shift # past argument
      ;;
  esac
done

set -- "${POSITIONAL[@]}" # restore positional parameters

echo "FILE EXTENSION  = ${EXTENSION}"
echo "SEARCH PATH     = ${SEARCHPATH}"
echo "LIBRARY PATH    = ${LIBPATH}"
echo "DEFAULT         = ${DEFAULT}"
echo "Number files in SEARCH PATH with EXTENSION:" $(ls -1 "${SEARCHPATH}"/*."${EXTENSION}" | wc -l)
if [[ -n $1 ]]; then
    echo "Last line of file specified as non-opt/last argument:"
    tail -1 "$1"
fi
EOF

chmod +x /tmp/demo-space-separated.sh

/tmp/demo-space-separated.sh -e conf -s /etc -l /usr/lib /etc/hosts

Output from copy-pasting the block above

FILE EXTENSION  = conf
SEARCH PATH     = /etc
LIBRARY PATH    = /usr/lib
DEFAULT         =
Number files in SEARCH PATH with EXTENSION: 14
Last line of file specified as non-opt/last argument:
#93.184.216.34    example.com

Usage

demo-space-separated.sh -e conf -s /etc -l /usr/lib /etc/hosts

Bash Equals-Separated (e.g., `--option=argument`)

cat >/tmp/demo-equals-separated.sh <<'EOF'
#!/bin/bash

for i in "$@"; do
  case $i in
    -e=*|--extension=*)
      EXTENSION="${i#*=}"
      shift # past argument=value
      ;;
    -s=*|--searchpath=*)
      SEARCHPATH="${i#*=}"
      shift # past argument=value
      ;;
    -l=*|--lib=*)
      LIBPATH="${i#*=}"
      shift # past argument=value
      ;;
    --default)
      DEFAULT=YES
      shift # past argument with no value
      ;;
    *)
      # unknown option
      ;;
  esac
done
echo "FILE EXTENSION  = ${EXTENSION}"
echo "SEARCH PATH     = ${SEARCHPATH}"
echo "LIBRARY PATH    = ${LIBPATH}"
echo "DEFAULT         = ${DEFAULT}"
echo "Number files in SEARCH PATH with EXTENSION:" $(ls -1 "${SEARCHPATH}"/*."${EXTENSION}" | wc -l)
if [[ -n $1 ]]; then
    echo "Last line of file specified as non-opt/last argument:"
    tail -1 $1
fi
EOF

chmod +x /tmp/demo-equals-separated.sh

/tmp/demo-equals-separated.sh -e=conf -s=/etc -l=/usr/lib /etc/hosts

Output from copy-pasting the block above

FILE EXTENSION  = conf
SEARCH PATH     = /etc
LIBRARY PATH    = /usr/lib
DEFAULT         =
Number files in SEARCH PATH with EXTENSION: 14
Last line of file specified as non-opt/last argument:
#93.184.216.34    example.com

Usage

demo-equals-separated.sh -e=conf -s=/etc -l=/usr/lib /etc/hosts

To better understand ${i#*=} search for "Substring Removal" in this guide. It is functionally equivalent to `sed 's/[^=]*=//' <<< "$i"` which calls a needless subprocess or `echo "$i" | sed 's/[^=]*=//'` which calls two needless subprocesses.

Using bash with getopt[s]

getopt(1) limitations (older, relatively-recent getopt versions):

can't handle arguments that are empty strings
can't handle arguments with embedded whitespace

More recent getopt versions don't have these limitations. For more information, see these docs.

POSIX getopts

Additionally, the POSIX shell and others offer getopts which doen't have these limitations. I've included a simplistic getopts example.

cat >/tmp/demo-getopts.sh <<'EOF'
#!/bin/sh

# A POSIX variable
OPTIND=1         # Reset in case getopts has been used previously in the shell.

# Initialize our own variables:
output_file=""
verbose=0

while getopts "h?vf:" opt; do
  case "$opt" in
    h|\?)
      show_help
      exit 0
      ;;
    v)  verbose=1
      ;;
    f)  output_file=$OPTARG
      ;;
  esac
done

shift $((OPTIND-1))

[ "${1:-}" = "--" ] && shift

echo "verbose=$verbose, output_file='$output_file', Leftovers: $@"
EOF

chmod +x /tmp/demo-getopts.sh

/tmp/demo-getopts.sh -vf /etc/hosts foo bar

Output from copy-pasting the block above

verbose=1, output_file='/etc/hosts', Leftovers: foo bar

Usage

demo-getopts.sh -vf /etc/hosts foo bar

The advantages of getopts are:

It's more portable, and will work in other shells like dash.
It can handle multiple single options like -vf filename in the typical Unix way, automatically.

The disadvantage of getopts is that it can only handle short options (-h, not --help) without additional code.

There is a getopts tutorial which explains what all of the syntax and variables mean. In bash, there is also help getopts, which might be informative.

Windows – n equivalent of ‘which’ on the Windows command line?

Windows Server 2003 and later (i.e. anything after Windows XP 32 bit) provide the where.exe program which does some of what which does, though it matches all types of files, not just executable commands. (It does not match built-in shell commands like cd.) It will even accept wildcards, so where nt* finds all files in your %PATH% and current directory whose names start with nt.

Try where /? for help.

Note that Windows PowerShell defines where as an alias for the Where-Object cmdlet, so if you want where.exe, you need to type the full name instead of omitting the .exe extension.

Best Answer

Related Solutions

Bash – How to parse command line arguments in Bash

Bash Space-Separated (e.g., --option argument)

Output from copy-pasting the block above

Usage

Bash Equals-Separated (e.g., --option=argument)

Output from copy-pasting the block above

Usage

Using bash with getopt[s]

POSIX getopts

Output from copy-pasting the block above

Usage

Windows – n equivalent of ‘which’ on the Windows command line?

Related Topic

Bash Space-Separated (e.g., `--option argument`)

Bash Equals-Separated (e.g., `--option=argument`)