Busybox on VMware ESXi: how to change syslogd options

Self-answer. It seems to be impossible to imitate syslogd behavior. After a lot of experiments, I provide updated snapshot of my research/guesses on syslog-ng options:

options {
        #####################################################################
        ### the flow of decisions for hostnames, syslog-ng 1.6.8:

        use_dns(yes);           # yes = first resolve the IP in $HOST_FROM (the message sender)

        keep_hostname(no);      # no = ignore $HOST embedded in the message (rare); overwrite $HOST with $HOST_FROM
                                #   note: RFC3164 allows embedding short hostname or IP, not FQDN

        use_fqdn(yes);          # yes = expand everything to FQDN, including local name
                                # Note syslogd behaviour is incompatible: use FQDN, but strip local
                                #   domain + strip "-s" domains + strip domains for "-l" hosts

        chain_hostnames(no);    # no = keep $FULLHOST same as $HOST; 
                                #   do not expand $FULLHOST into either "src@$HOST" for localhost, 
                                #   or to "$HOST/$HOST_FROM" for remote client

        #long_hostnames(no);    # synonym of chain_hostnames

        ### with default template, the resulting $FULLHOST is written to log
        #####################################################################

        check_hostname(yes);    # invalid chars?

        sync(0);                # sync immediately
};

I've found out that messages from my remote systems probably do not have hostname embedded, and this causes keep_hostname to be of no use.

I suspect that you're using consumer-grade drives for your storage? If so, these have onboard error recovery systems which will stall the volume while error recovery is being attempted. When this occurs, all storage serving can be delayed for a considerable period of time (10+ seconds).

In enterprise-grade drives, this 'feature' is disabled or never included, on the assumption that error recovery will be handled at the RAID-array level (RAID's implicitly assumed for enterprise deployments). For example, Western Digital refers to this feature (or removal of a feature!) as TLER - Time-Limited Error Recovery. In practical terms, it means a drive with TLER enabled won't stall out for an extended period of time to perform sector recovery/remapping/whatever.

So if you're running consumer drives, there's a fair chance you've hit an error on one of your disks, and it's repeatedly stalling out while it attempts to recover.

Solutions for this may be a bit tricky - I don't know if any third-party disk error scanners will support VMFS, and wouldn't risk pulling the disks and scanning them with ANYTHING, unless completely sure it won't trash the volume.