On Solaris, how can I tell what processes are doing disk i/o on a particular device

lsof -i

Doesn't come installed by default, but you can get Solaris version from http://www.sunfreeware.com/

I know it's the time spent by the CPU waiting for a IO operations to complete, but what kind of IO operations precisely? What I am also not sure, is why it so important? Can't the CPU just do something else while the IO operation completes, and then get back to processing data?

Yes, the operating system will schedule other processes to run while one is blocked on IO. However inside that process, unless it's using asynchronous IO, it will not progress until whatever IO operation is complete.

Also what are the right tools to diagnose what process(es) did exactly wait for IO.

Some tools you might find useful

• iostat, to monitor the service times of your disks
• iotop (if your kernel supports it), to monitor the breakdown of IO requests per process
• strace, to look at the actual operations issued by a process

And what are the ways to minimize IO wait time?

• ensure you have free physical memory so the OS can cache disk blocks in memory
• keep your filesystem disk usage below 80% to avoid excessive fragmentation
• tune your filesystem
• use a battery backed array controller
• choose good buffer sizes when performing io operations