Python – Convert psycopg2 DictRow query to Pandas dataframe

pandaspsycopg2python

I would like to convert a psycopg2 DictRow query to a pandas dataframe, but pandas keeps complaining:

curs = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
curs.execute("SELECT * FROM mytable")
data = curs.fetchall()

print type(data)
print pd.DataFrame(list(data))

However, I always get an error although I specifically passed a list???

<type 'list'>
TypeError: Expected list, got DictRow

The result is the same if I do pd.DataFrame(data) Could someone please help me make this work?

It would also be nice if the column names of the dataframe worked (i.e. extract DictRow and pass them to the dataframe).

Update:
Since I need to process the data, I would like to use the data from the psycopg2 query as is and not the pandas approach, e.g. read_sql_query.

Best Answer

Hmm, I eventually found this hacky solution:

print pd.DataFrame([i.copy() for i in data])

The copy() function of the DictRow class will return an actual dictionary. With the list comprehension I create a list of (identical) dictionaries, that Pandas will happily accept.

I am still puzzled why list(data) produced a TypeError. Maybe someone can still enlighten me.

Related Solutions

Python – Convert bytes to a string

You need to decode the bytes object to produce a string:

>>> b"abcde"
b'abcde'

# utf-8 is used here because it is a very common encoding, but you
# need to use the encoding your data is actually in.
>>> b"abcde".decode("utf-8") 
'abcde'

Python – Selecting multiple columns in a Pandas dataframe

The column names (which are strings) cannot be sliced in the manner you tried.

Here you have a couple of options. If you know from context which variables you want to slice out, you can just return a view of only those columns by passing a list into the __getitem__ syntax (the []'s).

df1 = df[['a', 'b']]

Alternatively, if it matters to index them numerically and not by their name (say your code should automatically do this without knowing the names of the first two columns) then you can do this instead:

df1 = df.iloc[:, 0:2] # Remember that Python does not slice inclusive of the ending index.

Additionally, you should familiarize yourself with the idea of a view into a Pandas object vs. a copy of that object. The first of the above methods will return a new copy in memory of the desired sub-object (the desired slices).

Sometimes, however, there are indexing conventions in Pandas that don't do this and instead give you a new variable that just refers to the same chunk of memory as the sub-object or slice in the original object. This will happen with the second way of indexing, so you can modify it with the .copy() method to get a regular copy. When this happens, changing what you think is the sliced object can sometimes alter the original object. Always good to be on the look out for this.

df1 = df.iloc[0, 0:2].copy() # To avoid the case where changing df1 also changes df

To use iloc, you need to know the column positions (or indices). As the column positions may change, instead of hard-coding indices, you can use iloc along with get_loc function of columns method of dataframe object to obtain column indices.

{df.columns.get_loc(c): c for idx, c in enumerate(df.columns)}

Now you can use this dictionary to access columns through names and using iloc.

Best Answer

Related Solutions

Python – Convert bytes to a string

Python – Selecting multiple columns in a Pandas dataframe

Related Topic