Best way to ‘harden’ embedded ext4 file server against unexpected loss of power

data-protectionext4

First, a little background: my company makes an audio streaming device that is a headless, rack-mounted Linux box with a Solid State e-SATA drive attached. The drive is formatted with ext4. The users can connect to the system using Samba/CIFS to upload new audio files or access existing ones. There is also custom software for streaming out audio over the network.

This is all fine. The only problem is that the users are audio people, not computer people, and see the system as a 'black box', not as a computer. Which means that at the end of the day, they aren't going to ssh in to the box and enter "/sbin/shutdown -h"; they are just going to cut power to the rack and leave, and expect things to still work properly the next day.

Since ext4 has journalling, journal checksumming, etc, this mostly works. The only time it doesn't work is when someone uploads a new file via Samba and then cuts power to the system before the uploaded data has been fully flushed to the disk. In that case, they come in the next day and find that their new file has been truncated or is missing entirely, and are unhappy.

My question is, what is the best way to avoid this problem? Is there a way to get smbd to call "sync" at the end of every upload? (Performance on uploads isn't so important, since they only happen occasionally). Or is there a way to tell ext4 to automatically flush within a few seconds of any change to a file? (Again, performance can be sacrificed for safety here) Should I set a particular write-ordering mode, activate barriers, etc?

Best Answer

Mounting the filesystem with sync specified in fstab would probably help. I suspect someone will have a recommendation better suited for your particular application.

I begun initial research on filesystems used with flash storage, as I want to custom-build a home theater PC as an appliance. You may find a different storage solution better suited for your device. Unfortunately, I have yet to find something I prefer so I do not have a detailed recommendation there.

Edit 1

According to the smb.conf(5) manpage, it supports immediate syncing within SAMBA:

   strict sync (S)
          Many Windows applications (including the Windows 98
          explorer  shell)  seem  to  confuse flushing buffer
          contents to disk with doing a sync to  disk.  Under
          UNIX,  a  sync  call  forces the process to be sus-
          pended until the kernel has ensured that  all  out-
          standing  data  in  kernel  disk  buffers  has been
          safely stored onto stable  storage.  This  is  very
          slow  and  should only be done rarely. Setting this
          parameter to no (the default)  means  that  smbd(8)
          ignores  the  Windows  applications  requests for a
          sync call. There is only a  possibility  of  losing
          data  if  the operating system itself that Samba is
          running on crashes, so there is  little  danger  in
          this  default setting. In addition, this fixes many
          performance problems that people have reported with
          the new Windows98 explorer shell file copies.

          Default: strict sync = no

   sync always (S)
          This  is  a boolean parameter that controls whether
          writes will always be  written  to  stable  storage
          before  the  write call returns. If this is no then
          the server will be guided by the  client's  request
          in  each write call (clients can set a bit indicat-
          ing that a particular write should be synchronous).
          If this is yes then every write will be followed by
          a fsync()  call to ensure the data  is  written  to
          disk.  Note  that the strict sync parameter must be
          set to yes in order for this parameter to have  any
          affect.

          Default: sync always = no
Related Topic