Powershell Download via HTTP using Proxy and Checking Remote File Size Fails on Second File

httppowershell

I have a powershell script I'm developing, which will download files from a HTTP server going through a proxy and including a UN/PW on the server.

I have files downloading fine, but some files are very large, and so I've been adding a function to check whether the file has changed (checking size). This works on the first file. But it times out on the second file at this line: 

$test = $wc2.OpenRead($source) | Out-Null)

The code for my script, excluding credentials, is listed below. I'd appreciate advice as to how to properly close a connection (as I suspect is not happening) or on how to get a remote file's size via HTTP using Powershell.

Function getWebClient {
    $webClient = new-object System.Net.WebClient 

    $pxy = new-object System.Net.WebProxy $proxy  
    $pxy.Credentials = New-Object System.Net.NetworkCredential ("tnad\$proxy_un", "$proxy_pw") 
    $webClient.proxy=$pxy  
    $webClient.Headers.Add("user-agent", "Windows Powershell WebClient Header") 

    $WebClient.Credentials = New-Object System.Net.Networkcredential($un, $pw)

    return $WebClient
}

foreach ($ds in $datasetsTest) {
    Write-Host "Checking: $ds"
    $source = "$server$ds"
    $dest = "$destFolder$ds"

    #Test whether the destination file already exists, if it exists, compare file sizes.
    $destExists = Test-Path $dest

    if($destExists -eq $false) {
        Try {
            $wc1 = getWebClient
            $wc1.DownloadFile($source, $dest)
            $wc1.Dispose()
            Write-Host "File downloaded"  -foregroundcolor "green"
        } Catch {
            Write-Host "Downloading $ds failed..." -foregroundcolor "red"
        }
    } else {
        $localFileSize = (Get-Item $dest).length
        $wc2 = getWebClient
        $test = $wc2.OpenRead($source) | Out-Null
        $wc2.Dispose()

        $remoteFileSize = $wc2.ResponseHeaders["Content-Length"]

        Write-Host "Local file size: $localFileSize, remote file size: $remoteFileSize"
        if(!($localFileSize -eq $remoteFileSize)) {
            #file exists, but the server version changed...
            Try {
                Remove-Item $dest
                $wc3 = getWebClient
                $wc3.DownloadFile($source, $dest) 
                $wc3.Dispose($true)
                Write-Host "File downloaded"  -foregroundcolor "green"
            } Catch {
                Write-Host "Downloading $ds failed..." -foregroundcolor "red"
            }
        } else {
            Write-Host "Skipping, file exists and hasn't changed"  -foregroundcolor "magenta"
        }
    }
}

Best Answer

I can't comment on this particular issue per se, but I'd like to offer some general recommendations. I've written some quite mission critical PS code over the last few years, some of which interacts with remote telemetry equipment over HTTPS. This code either has to succeed or fail gracefully, with very verbose logging. The code, although running in Windows VMs, is treated as an "embedded system / appliance".

I'd sincerely recommend adding error-handling code to your script. Granted, the Internet is full of bang-bang-bang examples of PowerShell code, with each successive step assuming that the previous one succeeded. This is sometimes made worse by over-use of PS pipelines, e.g.: enumerate-something | foreach-object{ do-something }, where some of the do-something's fail, meaning you're often left in an impossible roll-back situation.

Anyway, enough babbling. Recommendations:

  1. Enable strict mode. This'll prevent any unintentional type-casting / coercion. Pre-declare your variables, with correct type definitions.
  2. Use -erroraction silentlyContinue where possible, and trap errors with "if ($?)" checks... or wrap calls that generate exceptions with try{} catch{} constructs. Not all PS functions raise exceptions!
  3. Write plenty of log data
  4. Identify actions that are more likely to fail and code retry logic where possible.

I'll stop there!

Good luck.

Related Topic