How to determine compression method of a ZIP/RAR file

Choose the algorithm that is the quickest, since you probably care about doing this in real time. Generally for smaller blocks of data, the algorithms compress about the same (give or take a few bytes) mostly because the algorithms need to transmit the dictionary or Huffman trees in addition to the payload.

I highly recommend Deflate (used by zlib and Zip) for a number of reasons. The algorithm is quite fast, well tested, BSD licensed, and is the only compression required to be supported by Zip (as per the infozip Appnote). Aside from the basics, when it determines that the compression is larger than the decompressed size, there's a STORE mode which only adds 5 bytes for every block of data (max block is 64k bytes). Aside from the STORE mode, Deflate supports two different types of Huffman tables (or dictionaries): dynamic and fixed. A dynamic table means the Huffman tree is transmitted as part of the compressed data and is the most flexible (for varying types of nonrandom data). The advantage of a fixed table is that the table is known by all decoders and thus doesn't need to be contained in the compressed stream. The decompression (or Inflate) code is relatively easy. I've written both Java and Javascript versions based directly off of zlib and they perform rather well.

The other compression algorithms mentioned have their merits. I prefer Deflate because of its runtime performance on both the compression step and particularly in decompression step.

A point of clarification: Zip is not a compression type, it is a container. For doing packet compression, I would bypass Zip and just use the deflate/inflate APIs provided by zlib.

The answers from freedompeace, Kiyarash and Sam Vloeberghs:

.rar    application/x-rar-compressed, application/octet-stream
.zip    application/zip, application/octet-stream, application/x-zip-compressed, multipart/x-zip

I would do a check on the file name too. Here is how you could check if the file is a RAR or ZIP file. I tested it by creating a quick command line application.

<?php

if (isRarOrZip($argv[1])) {
    echo 'It is probably a RAR or ZIP file.';
} else {
    echo 'It is probably not a RAR or ZIP file.';
}

function isRarOrZip($file) {
    // get the first 7 bytes
    $bytes = file_get_contents($file, FALSE, NULL, 0, 7);
    $ext = strtolower(substr($file, - 4));

    // RAR magic number: Rar!\x1A\x07\x00
    // http://en.wikipedia.org/wiki/RAR
    if ($ext == '.rar' and bin2hex($bytes) == '526172211a0700') {
        return TRUE;
    }

    // ZIP magic number: none, though PK\003\004, PK\005\006 (empty archive), 
    // or PK\007\008 (spanned archive) are common.
    // http://en.wikipedia.org/wiki/ZIP_(file_format)
    if ($ext == '.zip' and substr($bytes, 0, 2) == 'PK') {
        return TRUE;
    }

    return FALSE;
}

Notice that it still won't be 100% certain, but it is probably good enough.

$ rar.exe l somefile.zip
somefile.zip is not RAR archive

But even WinRAR detects non RAR files as SFX archives:

$ rar.exe l somefile.srr
SFX Volume somefile.srr