C# – Download when idle/low network utilization

One more, using urlretrieve:

import urllib
urllib.urlretrieve("http://www.example.com/songs/mp3.mp3", "mp3.mp3")

(for Python 3+ use import urllib.request and urllib.request.urlretrieve)

Yet another one, with a "progressbar"

import urllib2

url = "http://download.thinkbroadband.com/10MB.zip"

file_name = url.split('/')[-1]
u = urllib2.urlopen(url)
f = open(file_name, 'wb')
meta = u.info()
file_size = int(meta.getheaders("Content-Length")[0])
print "Downloading: %s Bytes: %s" % (file_name, file_size)

file_size_dl = 0
block_sz = 8192
while True:
    buffer = u.read(block_sz)
    if not buffer:
        break

    file_size_dl += len(buffer)
    f.write(buffer)
    status = r"%10d  [%3.2f%%]" % (file_size_dl, file_size_dl * 100. / file_size)
    status = status + chr(8)*(len(status)+1)
    print status,

f.close()

The source referenced by the OP has some credibility ...but what about Microsoft - what is the stance on struct usage? I sought some extra learning from Microsoft, and here is what I found:

Consider defining a structure instead of a class if instances of the type are small and commonly short-lived or are commonly embedded in other objects.

Do not define a structure unless the type has all of the following characteristics:

1. It logically represents a single value, similar to primitive types (integer, double, and so on).
2. It has an instance size smaller than 16 bytes.
3. It is immutable.
4. It will not have to be boxed frequently.

Microsoft consistently violates those rules

Okay, #2 and #3 anyway. Our beloved dictionary has 2 internal structs:

[StructLayout(LayoutKind.Sequential)]  // default for structs
private struct Entry  //<Tkey, TValue>
{
    //  View code at *Reference Source
}

[Serializable, StructLayout(LayoutKind.Sequential)]
public struct Enumerator : 
    IEnumerator<KeyValuePair<TKey, TValue>>, IDisposable, 
    IDictionaryEnumerator, IEnumerator
{
    //  View code at *Reference Source
}

*Reference Source

The 'JonnyCantCode.com' source got 3 out of 4 - quite forgivable since #4 probably wouldn't be an issue. If you find yourself boxing a struct, rethink your architecture.

Let's look at why Microsoft would use these structs:

1. Each struct, Entry and Enumerator, represent single values.
2. Speed
3. Entry is never passed as a parameter outside of the Dictionary class. Further investigation shows that in order to satisfy implementation of IEnumerable, Dictionary uses the Enumerator struct which it copies every time an enumerator is requested ...makes sense.
4. Internal to the Dictionary class. Enumerator is public because Dictionary is enumerable and must have equal accessibility to the IEnumerator interface implementation - e.g. IEnumerator getter.

Update - In addition, realize that when a struct implements an interface - as Enumerator does - and is cast to that implemented type, the struct becomes a reference type and is moved to the heap. Internal to the Dictionary class, Enumerator is still a value type. However, as soon as a method calls GetEnumerator(), a reference-type IEnumerator is returned.

What we don't see here is any attempt or proof of requirement to keep structs immutable or maintaining an instance size of only 16 bytes or less:

1. Nothing in the structs above is declared readonly - not immutable
2. Size of these struct could be well over 16 bytes
3. Entry has an undetermined lifetime (from Add(), to Remove(), Clear(), or garbage collection);

And ... 4. Both structs store TKey and TValue, which we all know are quite capable of being reference types (added bonus info)

Hashed keys notwithstanding, dictionaries are fast in part because instancing a struct is quicker than a reference type. Here, I have a Dictionary<int, int> that stores 300,000 random integers with sequentially incremented keys.

Capacity: 312874
MemSize: 2660827 bytes
Completed Resize: 5ms
Total time to fill: 889ms

Capacity: number of elements available before the internal array must be resized.

MemSize: determined by serializing the dictionary into a MemoryStream and getting a byte length (accurate enough for our purposes).

Completed Resize: the time it takes to resize the internal array from 150862 elements to 312874 elements. When you figure that each element is sequentially copied via Array.CopyTo(), that ain't too shabby.

Total time to fill: admittedly skewed due to logging and an OnResize event I added to the source; however, still impressive to fill 300k integers while resizing 15 times during the operation. Just out of curiosity, what would the total time to fill be if I already knew the capacity? 13ms

So, now, what if Entry were a class? Would these times or metrics really differ that much?

Capacity: 312874
MemSize: 2660827 bytes
Completed Resize: 26ms
Total time to fill: 964ms

Obviously, the big difference is in resizing. Any difference if Dictionary is initialized with the Capacity? Not enough to be concerned with ... 12ms.

What happens is, because Entry is a struct, it does not require initialization like a reference type. This is both the beauty and the bane of the value type. In order to use Entry as a reference type, I had to insert the following code:

/*
 *  Added to satisfy initialization of entry elements --
 *  this is where the extra time is spent resizing the Entry array
 * **/
for (int i = 0 ; i < prime ; i++)
{
    destinationArray[i] = new Entry( );
}
/*  *********************************************** */  

The reason I had to initialize each array element of Entry as a reference type can be found at MSDN: Structure Design. In short:

Do not provide a default constructor for a structure.

If a structure defines a default constructor, when arrays of the structure are created, the common language runtime automatically executes the default constructor on each array element.

Some compilers, such as the C# compiler, do not allow structures to have default constructors.

It is actually quite simple and we will borrow from Asimov's Three Laws of Robotics:

1. The struct must be safe to use
2. The struct must perform its function efficiently, unless this would violate rule #1
3. The struct must remain intact during its use unless its destruction is required to satisfy rule #1

...what do we take away from this: in short, be responsible with the use of value types. They are quick and efficient, but have the ability to cause many unexpected behaviors if not properly maintained (i.e. unintentional copies).