Warnings/Errors when running clamav/clamscan, scanning 3TB hard-drive

Question

What I'm trying to do:

I'm trying to scan my File-Server for malware, and I'm using clamav/clamscan, where the man page say's it can scan files up to 4GB.

This man page states:

--max-filesize=#n

Extract and scan at most #n kilobytes from each archive. You may pass the value in megabytes in format xM or xm, where x is a number. This option protects your system against DoS attacks (default: 25 MB, max: <4 GB)

--max-scansize=#n

Extract and scan at most #n kilobytes from each scanned file. You may pass the value in megabytes in format xM or xm, where x is a number. This option protects your system against DoS attacks (default: 100 MB, max: <4 GB)

My system is:

Newish hardware ASRock motherboard,

CPU: AMD Athlon(tm) II X2 270 Processor(3400MHz)

Memory: 4GB

OS: Debian Wheezy all updates.

Questions:

What am I doing wrong here?

What do those errors and warnings below mean?

Is there a fix for this behavior?

My case:

I've been trying to scan two 3TB hard-drives with clamscan for over a week now but it always gives the same errors(except Bytecode number varies):

LibClamAV Warning: [Bytecode JIT]: recovered from error
LibClamAV Warning: [Bytecode JIT]: JITed code intercepted runtime error!
LibClamAV Warning: Bytcode 38 failed to run: Time limit reached
LibClamAV Warning: [Bytecode JIT]: Bytecode run timed out, timeout flag set
LibClamAV Warning: [Bytecode JIT]: recovered from error
LibClamAV Warning: [Bytecode JIT]: JITed code intercepted runtime error!
LibClamAV Warning: Bytcode 38 failed to run: Time limit reached
LibClamAV Warning: [Bytecode JIT]: recovered from error
LibClamAV Warning: [Bytecode JIT]: Bytecode run timed out, timeout flag set
LibClamAV Warning: [Bytecode JIT]: JITed code intercepted runtime error!
LibClamAV Warning: Bytcode 38 failed to run: Time limit reached

after approx. 40-50 hours of scanning:

(Note that in the next snippet is the actual clamscan command I'm trying to run)

PID  USER PRI NI VIRT  RES  SHR S CPU% MEM%   TIME+ Command
2012 root 20  0 1903M 246M 1244 R 101. 6.6 47h27:45 clamscan -r -i --remove --max-filesize=4000M --max-scansize=4000M /DATA1/

I've tried to delete the files suggested in one forum where they suspected corruption in some of those files that is bytecode.cvd, main.cvd, daily.cld and re-download them(with the update tool):

root ~ # ls -ahl /usr/local/share/clamav/                                                
total 145M                                                                                  
drwxr-sr-x  2 clamav clamav 4.0K Mar 26 04:29 .
drwxrwsr-x 10 root   staff  4.0K Mar 20 01:59 ..
-rw-r--r--  1 clamav clamav  65K Mar 26 04:29 bytecode.cvd
-rw-r--r--  1 clamav clamav  83M Mar 26 04:29 daily.cld
-rw-r--r--  1 clamav clamav  62M Mar 18 01:17 main.cvd
-rw-------  1 clamav clamav  156 Mar 26 04:29 mirrors.dat
root ~ # rm -f /usr/local/share/clamav/bytecode.cvd /usr/local/share/clamav/daily.cld /usr/local/share/clamav/main.cvd
root ~ # freshclam
ClamAV update process started at Thu Mar 26 04:42:21 2015
Downloading main.cvd [100%]
main.cvd updated (version: 55, sigs: 2424225, f-level: 60, builder: neo)
Downloading daily.cvd [100%]
daily.cvd updated (version: 20242, sigs: 1358870, f-level: 63, builder: neo)
Downloading bytecode.cvd [100%]
bytecode.cvd updated (version: 247, sigs: 41, f-level: 63, builder: dgoddard)
Database updated (3783136 signatures) from db.UK.clamav.net (IP: 129.67.1.218)

---

I've also tried to set --max-filesize and --max-scansize lower per the forum post I found here where it states that there is a limit to files/scans size at 2.17GB:

clamscan -r -i --remove --max-filesize=2100M --max-scansize=2100M /DATA1/

but it gave the same errors.

---

The program is the latest from the official site: clamav-0.98.6 configured and compiled from source with these options:

./configure --enable-bzip2

I've tried to re-install the program and also at first I had more options set in the compilation(--enable-experimental, --with-dbdir=/usr/local/share/clamav)

The last option I know of is to uninstall this version and try the packages from my distributions repositories. But I would like to get this one working if at all possible.

UPDATE: I've also tried to install clamav from the repositories but it gives the same problems/errors.

---

I've found this, but it's old and doesn't seem to know what the problem is. And here but still not a definite answer or fix.

---

The drives I've been trying to scan are these:

# df -h

/dev/sdb1                  2.7T  2.6T  115G  96% /DATA1
/dev/sdc1                  2.7T  2.6T  165G  95% /DATA2

Here is fdisk:

# fdisk -l

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes                                                           
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors                                 
Units = sectors of 1 * 512 = 512 bytes                                                                  
Sector size (logical/physical): 512 bytes / 4096 bytes                                                  
I/O size (minimum/optimal): 4096 bytes / 4096 bytes                                                     
Disk identifier: 0x00000000                                                                             

   Device Boot      Start         End      Blocks   Id  System                                          
/dev/sdb1               1  4294967295  2147483647+  ee  GPT                                             
Partition 1 does not start on physical sector boundary.                                                 


WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes                                                           
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors                                 
Units = sectors of 1 * 512 = 512 bytes                                                                  
Sector size (logical/physical): 512 bytes / 4096 bytes                                                  
I/O size (minimum/optimal): 4096 bytes / 4096 bytes                                                     
Disk identifier: 0x00000000                                                                             

   Device Boot      Start         End      Blocks   Id  System                                          
/dev/sdc1               1  4294967295  2147483647+  ee  GPT                                             
Partition 1 does not start on physical sector boundary.

Possible cause:

It could be something related to memory/CPU that the system has but I don't have that information

I found this which states that clamscan loads the file to scan into memory and if there isn't enough memory it will fail. This is likely what is happening as I'm setting the scanner to scan files up-to 4Gigs and that's how much memory the system has.

Excerpt:

How big is that file? How much RAM (physical and swap separate, please) is installed on the scanning machine? Currently, ClamAV has a hard file limit of around 2.17GB. Because we're mapping the file into memory, if you don't have enough memory available to map the whole file, the memory mapping code (as currently implemented) will fail and the file won't be scanned.

One of our long-term goals is to investigate being able to properly support large files.

Possible solution:

Hope the above is the problem(not enough memory), then I can simply extend the systems memory to 8GB, but it's unlikely it is so simple because I tried to run those scans on a system with 12GB ram.

EDIT #1

Here is a run on another system with Fedora 21 + 12 GB RAM:

clamscan -r -i --remove --max-filesize=1700M --max-scansize=1700M --exclude=/proc --exclude=/sys --exclude=/dev /

---

LibClamAV Warning: [Bytecode JIT]: recovered from error
LibClamAV Warning: [Bytecode JIT]: JITed code intercepted runtime error!
LibClamAV Warning: [Bytecode JIT]: Bytecode run timed out, timeout flag set
LibClamAV Warning: Bytcode 27 failed to run: Time limit reached
LibClamAV Error: cli_scanxz: premature end of compressed stream
LibClamAV Error: cli_scanxz: premature end of compressed stream

---

----------- SCAN SUMMARY -----------
Known viruses: 3779101
Engine version: 0.98.6
Scanned directories: 101382
Scanned files: 744103
Infected files: 0
Total errors: 18419
Data scanned: 285743.78 MB
Data read: 394739.73 MB (ratio 0.72:1)
Time: 32171.073 sec (536 m 11 s)

when I ran those same scans on it with sizes set to 2100M-4000M it gave the same errors as mentioned in my original question.

Solution

I've found this(thanks to @FloHimself): Brief Re-introduction to ClamAV Bytecode Signatures, it's an good overview/supplement of some of the usages of the program and some useful options:

Excerpt:

Bytecode signatures are a specialized type of ClamAV signature which is able to perform additional processing of the scanned file and allow for more robust detection. Unlike the standard ClamAV signature types, bytecode signatures have a number of unique distinctions which need to be respected for their effective usage.

Trust

Bytecode signatures, by default, are considered untrusted. In fact, only bytecode signatures published by Cisco, in the bytecode.cvd are considered “trusted”. This means that the ClamAV engine will, by default, never load, trigger or execute untrusted bytecodes. One can bypass this safety mechanism by specifying the bytecode unsigned option to the engine but it should be noted that it is up to the user’s discretion on using untrusted bytecode signatures.

For clamscan, the command line option is

--bytecode-unsigned.

For clamd, one would need to specify BytecodeUnsigned yes to clamd.conf.

Timeout

Bytecode signatures are designed to only run for a limited amount of time designated by an internal timeout value. If execution time exceeds the value, the bytecode signature’s execution is terminated and the user is notified. The bytecode signature timeout value can be set by the user.

For clamscan, the command line is

--bytecode-timeout=[time in ms].

For clamd, one would specify BytecodeTimeout [time in ms] to clamd.conf.

And this is useful:

Issue Reporting

If anyone encounters issue with bytecode signatures, whether within the clambc-compiler or within ClamAV, they can report them to https://bugzilla.clamav.net/. Be sure to include the bytecode signature, bytecode source(if possible), and any other pieces of useful information.

---

Answer

The key seems to be to set the --bytecode-timeout= high so the scanner has time to scan the whole file. The default value is 60000 milliseconds/60 seconds, and I have set it to 190000 which works and doesn't give the timeout errors. This value could probably be set lower but it works for me. Tested on two systems that had the errors before the setting.

UPDATE:

Tested on three systems and many scans, the errors are gone with this setting for --bytecode-timeout.

Here is the new command:

clamscan -r -i --remove --max-filesize=4000M --max-scansize=4000M --bytecode-timeout=190000 /DATA1

Note:

I also upgraded the servers memory to 8GB, I'm not sure if clamscan loads the file to memory when it's being scanned but one post said that much and if so that is another consideration.

Comments

[FloHimself]

Are there any additional hints if you run clamscan with the --debug option?

[somethingSomething]

@FloHimself I've tried to run the --debug option, but "> file", ">> file" and "-l file" do not give any debug information and only contain the scan results. All the debug info goes to the terminal running the command.

[somethingSomething]

@FloHimself Yes your right, I'll make some scans and probably post some scan-logs. Thanks for the attention.

[somethingSomething]

I'm scanning the system again now with stderr going to a log file, and it's still going after 58½ hours....... could it be that those errors are not that fatal?, I'll post an edit with the log and if the scan was successful... hope someone comes along and enlightens me what those errors mean.

[FloHimself]

These warnings aren't fatal, that's why they are warnings. The clamav bytecode engine makes use of additional signatures to detect polymorphic viruses and malware. You may want to start reading about it here.

[FloHimself]

Thank you! In fact I'm no expert and just read about the bytecode stuff because it caught my attention. Your question is so detailed that it deserves a well written answer... which I can't offer. Regarding the warnings: possibly the warnings are caused by a bad bytecode signature. So depending on the goals it might be even wiser to lower the bytecode time-out (which defaults to 60s) to decrease the systems load. But that really depends on why the warnings are issued...

[user1133275]

It would be nice if the error out included the bytecode and file combination that timed out.

[FGiorlando]

probably a good idea to not have the --remove option shown in example code as someone copy pasting without checking may get a surprise!