So, I've written tools with both a GUI and a CLI. The hard part was figuring out which one to open - in our case, though, the CLI version had required parameters, so I just opened the GUI if there weren't any parameters. Then, if they did want a console, call a function that looks something like:
private const int ATTACH_PARENT_PROCESS = -1;
private const int ERROR_INVALID_HANDLE = 6;
[DllImport("kernel32.dll", SetLastError = true)]
static extern bool AttachConsole(int dwProcessId);
[DllImport("kernel32.dll")]
static extern bool AllocConsole();
[DllImport("kernel32.dll")]
static extern bool FreeConsole();
private static bool StartConsole()
{
if (!AttachConsole(ATTACH_PARENT_PROCESS)) // try connecting to an existing console
{
if (Marshal.GetLastWin32Error() == ERROR_INVALID_HANDLE) // we don't have a console yet
{
if (!AllocConsole()) // couldn't create a new console, either
return false;
}
else
return false; // some other error
}
return true;
}
Returns whether the console was created. Don't forget to FreeConsole() when you're done!
In our case, of course, if we don't create a console, we create a GUI. It'd be just as easy to create either a console or no UI, though.
EDIT: That totally didn't answer the question in the edit that wasn't there when I started writing that, of course. Other than that our hack was just checking whether it was called with command-line parameters or not.
Best Answer
Microsoft designed console and GUI apps to be mutually exclusive. This bit of short-sightedness means that there is no perfect solution. The most popular approach is to have two executables (eg. cscript / wscript, java / javaw, devenv.com / devenv.exe etc) however you've indicated that you consider this "cheating".
You've got two options - to make a "console executable" or a "gui executable", and then use code to try to provide the other behaviour.
cmd.exe
will assume that your program does no console I/O so won't wait for it to terminate before continuing, which in interactive mode (ie not a batch) means displaying the next ("C:\>
") prompt and reading from the keyboard. So even if you use AttachConsole your output will be mixed withcmd
's output, and the situation gets worse if you try to do input. This is basically a non-starter.Contrary to belief, there is nothing to stop a console executable from displaying a GUI, but there are two problems.
The first is that if you run it from the command line with no arguments (so you want the GUI),
cmd
will still wait for it to terminate before continuing, so that particular console will be unusable for the duration. This can be overcome by launching a second process of the same executable (do you consider this cheating?), passing the DETACHED_PROCESS flag to CreateProcess() and immediately exiting. The new process can then detect that it has no console and display the GUI.Here's C code to illustrate this approach:
I compiled it with cygwin's gcc - YMMV with MSVC.
The second problem is that when run from Explorer, your program will for a split second display a console window. There's no programmatic way around this because the console is created by Windows when the app is launched, before it starts executing. The only thing you can do is, in your installer, make the shortcut to your program with a "show command" of SW_HIDE (ie. 0). This will only affect the console unless you deliberately honour the wShowWindow field of STARTUPINFO in your program, so don't do that.
I've tested this by hacking cygwin's "mkshortcut.exe". How you accomplish it in your install program of choice is up to you.
The user can still of course run your program by finding the executable in Explorer and double-clicking it, bypassing the console-hiding shortcut and seeing the brief black flash of a console window. There's nothing you can do about it.