Python – Searching a Unicode file using Python

encodingpythonunicode

Setup

I'm writing a script to process and annotate build logs from Visual Studio. The build logs are HTML, and from what I can tell, Unicode (UTF-16?) as well. Here's a snippet from one of the files:

c:\anonyfolder\anonyfile.c(17169) : warning C4701: potentially uninitialized local variable 'object_adrs2' used
c:\anonyfolder\anonyfile.c(17409) : warning C4701: potentially uninitialized local variable 'pclcrd_ptr' used
c:\anonyfolder\anonyfile.c(17440) : warning C4701: potentially uninitialized local variable 'object_adrs2' used

The first 16 bytes of the file look like this:

feff 003c 0068 0074 006d 006c 003e 000d

The rest of the file is littered with null bytes as well.

I'd like to be able to perform string and regular expression searches/matches on these files. However, when I try the following code I get an error message.

buildLog = open(sys.argv[1]).readlines()

for line in buildLog:
    match = u'warning'
    if line.find(match) >= 0:
        print line

The error message:

Traceback (most recent call last):
File "proclogs.py", line 60, in
if line.find(match) >= 0:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)

Apparently it's choking on the 0xff byte in 0xfeff at the beginning of the file. If I skip the first line, I get no matches:

buildLog = open(sys.argv[1]).readlines()

for line in buildLog[1:]: # Skip the first line.
    match = u'warning'
    if line.find(match) >= 0:
        print line

Likewise, using the non-Unicode match = 'warning' produces no results.

Question

How can I portably search a Unicode file using strings and regular expressions in Python? Additionally, how can I do so such that I can reconstruct the original file? (The goal is to be able to write annotations on the warning lines without mangling the file.)

Best Answer

Try using the codecs package:

import codecs
buildLog = codecs.open(sys.argv[1], "r", "utf-16").readlines()

Also you may run into trouble with your print statement as it may try to convert the strings to your console encoding. If you're printing for your review you could use,

print repr(line)

Related Solutions

Python – How to check whether a file exists without exceptions

If the reason you're checking is so you can do something like if file_exists: open_it(), it's safer to use a try around the attempt to open it. Checking and then opening risks the file being deleted or moved or something between when you check and when you try to open it.

If you're not planning to open the file immediately, you can use os.path.isfile

Return True if path is an existing regular file. This follows symbolic links, so both islink() and isfile() can be true for the same path.

import os.path
os.path.isfile(fname)

if you need to be sure it's a file.

Starting with Python 3.4, the pathlib module offers an object-oriented approach (backported to pathlib2 in Python 2.7):

from pathlib import Path

my_file = Path("/path/to/file")
if my_file.is_file():
    # file exists

To check a directory, do:

if my_file.is_dir():
    # directory exists

To check whether a Path object exists independently of whether is it a file or directory, use exists():

if my_file.exists():
    # path exists

You can also use resolve(strict=True) in a try block:

try:
    my_abs_path = my_file.resolve(strict=True)
except FileNotFoundError:
    # doesn't exist
else:
    # exists

Python – How to execute a program or call a system command

Use the subprocess module in the standard library:

import subprocess
subprocess.run(["ls", "-l"])

The advantage of subprocess.run over os.system is that it is more flexible (you can get the stdout, stderr, the "real" status code, better error handling, etc...).

Even the documentation for os.system recommends using subprocess instead:

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.

On Python 3.4 and earlier, use subprocess.call instead of .run:

subprocess.call(["ls", "-l"])