Python Iterators – Why Python Only Copies Individual Elements When Iterating a List

Having understood why Unicode is necessary (recommended reading: link, link :thanks @DanielB for the excellent links) and that character encodings are what computers use to represent real world characters, it becomes clear that when Python is reading or writing bytes representing text from a stream (which can be a file, a pipe, a socket, ...), it needs to know which character encoding is being used so those bytes are meaningful as human readable text.

Python uses ascii as the default encoding. The ascii encoding is inadequate for almost all languages (including English: there's no £ in ascii!): so you need to specify an alternate encoding when writing to streams if you intended to use any characters not part of ascii.

When reading from streams, you need to know which encoding was used to write to the stream and use the same encoding to read from it, otherwise the decoded result will be wrong. Try writing a string with Cyrillic characters with the ISO-8859-5 codec and reading it back with the UTF-8 codec: you'll see they don't match, because different byte sequences mean different characters in the two encodings.

So to answer your specific question,

out_file = open('example.txt', 'w')

is actually opening the file for writing using the ascii codec implicitly. If you want to specify another codec, you need to either use the encoding parameter of the open function in Python 3.x:

out_file = open('example.txt', 'a', 'utf-8')

or if you are still using Python 2.x (the latest is Python 2.7.3 at the time of this writing), you need to use the functions from the codecs module:

out_file = codecs.open('example.txt', 'a', 'utf-8')

since open in Python 2.x doesn't allow you to specify an encoding (you can use it to read the byte stream into a byte string in memory though, and then decode that string).

I'll expand my comment a bit. The List[T] data structure, from scala.collection.immutable is optimized to work the way an immutable list in a more purely functional programming language works. It has very fast prepend times, and it is assumed that you will be working on the head for almost all of your access.

Immutable lists get to have very fast prepend times due to the fact that they model their linked lists as a series of "cons cells". The cell defines a single value, and a pointer to the next cell (classic singly-linked-list style):

Cell [Value| -> Nil]

When you prepend to a list, you're really just making a single new cell, with the rest of the existing list being pointed to:

Cell [NewValue| -> [Cell[Value| -> Nil]]

Because the list is immutable, you're safe to do this without any actual copying. There's no danger of the old list changing and causing all the values in your new list to become invalid. However, you lose the ability to have a mutable pointer to the end of your list as a compromise.

This lends itself very well to recursively working on lists. Let's say you defined your own version of filter:

def deleteIf[T](list : List[T])(f : T => Boolean): List[T] = list match {
  case Nil => Nil
  case (x::xs) => f(x) match {
    case true => deleteIf(xs)(f)
    case false => x :: deleteIf(xs)(f)
  }
}

That's a recursive function that works from the head of the list exclusively, and takes advantage of pattern matching via the :: extractor. This is something you see a lot of in languages like Haskell.

If you really want fast appends, Scala provides a lot of mutable and immutable data structures to choose from. On the mutable side, you might look into ListBuffer. Alternatively, Vector from scala.collection.immutable has a fast append time.