Downloading large files fails, Squid + c-icap + SquidClamAV

clamavsquid

I am building a new proxy server with Squid & SquidClamAV.

I am facing problems with downloading large files. I have set a maxsize of 52428800 bytes in squidclamav.conf – which should prevent the scanning of files bigger than 50MB.

Downloading anything up to 2GB through the proxy works fine, but files bigger than that produce the following error: ICAP protocol error – The system returned: (32) Broken pipe

In c-icap server log i get this: Fri Dec 2 10:19:36 2011, general, Bug in the service. Please report to the servive author!!!!

Downloading a file at sizes below 2GB produces the download window instantly to the browser. As expected.

When starting to download anything above 2GB, it takes some time before producing the error. Following /var/tmp I see that it actually starts to download the file up to 50MB (the maxsize set in squidclamav.conf!) and then stops and produces the error.

I don't understand why this happens with files that are bigger than the maxsize in squidclamav.

The server is a CentOS 6 32bit machine. C-icap is compiled from source (v0.1.7), squid is from CentOS, clamd+squidclamav from repoforge.org. Below I've added related config files.

/etc/squid/squid.conf:

...
icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_header X-Authenticated-User
icap_service service_req reqmod_precache \
     bypass=1 icap://127.0.0.1:1344/squidclamav
adaptation_access service_req allow all
icap_service service_resp respmod_precache \ 
     bypass=1 icap://127.0.0.1:1344/squidclamav
adaptation_access service_resp allow all

/etc/squidclamav.conf:

maxsize 52428800
redirect http://localproxy/detecteio.php
clamd_local /var/run/clamav/clamd.sock
timeout 1
logredir 0
dnslookup 1

/etc/c-icap.conf:

PidFile /var/run/c-icap/c-icap.pid
CommandsSocket /var/run/c-icap/c-icap.ctl
Timeout 300
MaxKeepAliveRequests 100
KeepAliveTimeout 600
StartServers 3
MaxServers 10
MinSpareThreads     10
MaxSpareThreads     20
ThreadsPerChild     10
MaxRequestsPerChild  0
Port 1344
ServerAdmin root@localhost
ServerName localproxy.domain.local
TmpDir /var/tmp
MaxMemObject 131072
DebugLevel 1
ModulesDir /usr/local/lib/c_icap
ServicesDir /usr/local/lib/c_icap
TemplateDir /usr/local/share/c_icap/templates/
TemplateDefaultLanguage en
LoadMagicFile /usr/local/etc/c-icap.magic
RemoteProxyUsers off
RemoteProxyUserHeader X-Authenticated-User
RemoteProxyUserHeaderEncoded on
ServerLog /var/log/c-icap_server.log
AccessLog /var/log/c-icap_access.log
Service squidclamav squidclamav.so
Service echo srv_echo.so

/etc/clamd.conf:

LogFile /var/log/clamav/clamd.log
LogFileMaxSize 0
LogTime yes
LogSyslog yes
PidFile /var/run/clamav/clamd.pid
TemporaryDirectory /var/tmp
DatabaseDirectory /var/clamav
LocalSocket /var/run/clamav/clamd.sock
FixStaleSocket yes
TCPSocket 3310
TCPAddr 127.0.0.1
MaxConnectionQueueLength 30
MaxThreads 50
ReadTimeout 300
User clamav
AllowSupplementaryGroups yes
ScanPE yes
ScanELF yes
DetectBrokenExecutables yes
ScanOLE2 yes
ScanMail yes
ScanArchive yes
ArchiveBlockEncrypted no

Best Answer

Generally, 32bit systems will have trouble dealing with files larger than 2GB, however, having the download capped at 50MB would suggest ClamAV is at fault.

Couple of suggestions;

1) Ensure Squid has been compiled with '--with-large-files'

2) Test the download locally on the box using curl or wget (rules out browser/network issue)

3) Try increasing the 'maxsize' directive in '/etc/squidclamav.conf' to see if you can download larger files.

4) Disable file scanning and re-test.

Related Topic