Linux – please explain the fio results – is O_SYNC|O_DIRECT misbehaving on linux

linuxlinux-kernelperformance

I'm going mad over figuring out what the problem could be with one of our storage boxes.
With a simple fio script I'm testing random writes using bs=1M and direct=1. The SSD is a Samsung 840pro attached to an LSI HBA (3Gbit/s ports).

This is the result I'm getting under FreeBSD 9.1:

WRITE: io=13169MB, aggrb=224743KB/s, minb=224743KB/s, maxb=224743KB/s, mint=60002msec,   maxt=60002msec

This is regardless of sync being set to 0 or 1.

On linux, this is the result with sync=0:

WRITE: io=14828MB, aggrb=253060KB/s, minb=253060KB/s, maxb=253060KB/s, mint=60001msec, maxt=60001msec

and with sync=1:

WRITE: io=6360.0MB, aggrb=108542KB/s, minb=108542KB/s, maxb=108542KB/s, mint=60001msec, maxt=60001msec

My understanding is that since I'm operating on the raw block device, O_SYNC should not make any difference – there's no filesystem, any barrier, anything between the writes and the drive itself. Especially with O_DIRECT|O_SYNC set.

Any ideas?

For reference, here's the fio script I'm testing with:

[global]
bs=1M
ioengine=sync
iodepth=4
size=16g
direct=1
runtime=60
filename=/dev/sdh
sync=1

[rand-write]
rw=randwrite
stonewall

Best Answer

Linux Kernel developer Christoph Hellwig answered Zoltan by email about O_SYNC on Linux block devices:

For consumer disks using O_SYNC on Linux does make a huge difference, because it flushes the disk write cache after completion to make sure the O_SYNC gurantees that data has hit physical storage are met.

It seems like FreeBSD might be missing that call.

Related Topic