I am interested in knowing how to convert a pandas dataframe into a NumPy array.
dataframe:
import numpy as np
import pandas as pd
index = [1, 2, 3, 4, 5, 6, 7]
a = [np.nan, np.nan, np.nan, 0.1, 0.1, 0.1, 0.1]
b = [0.2, np.nan, 0.2, 0.2, 0.2, np.nan, np.nan]
c = [np.nan, 0.5, 0.5, np.nan, 0.5, 0.5, np.nan]
df = pd.DataFrame({'A': a, 'B': b, 'C': c}, index=index)
df = df.rename_axis('ID')
gives
label A B C
ID
1 NaN 0.2 NaN
2 NaN NaN 0.5
3 NaN 0.2 0.5
4 0.1 0.2 NaN
5 0.1 0.2 0.5
6 0.1 NaN 0.5
7 0.1 NaN NaN
I would like to convert this to a NumPy array, as so:
array([[ nan, 0.2, nan],
[ nan, nan, 0.5],
[ nan, 0.2, 0.5],
[ 0.1, 0.2, nan],
[ 0.1, 0.2, 0.5],
[ 0.1, nan, 0.5],
[ 0.1, nan, nan]])
How can I do this?
As a bonus, is it possible to preserve the dtypes, like this?
array([[ 1, nan, 0.2, nan],
[ 2, nan, nan, 0.5],
[ 3, nan, 0.2, 0.5],
[ 4, 0.1, 0.2, nan],
[ 5, 0.1, 0.2, 0.5],
[ 6, 0.1, nan, 0.5],
[ 7, 0.1, nan, nan]],
dtype=[('ID', '<i4'), ('A', '<f8'), ('B', '<f8'), ('B', '<f8')])
or similar?
Best Answer
df.to_numpy()
is better thandf.values
, here's why.*It's time to deprecate your usage of
values
andas_matrix()
.pandas
v0.24.0
introduced two new methods for obtaining NumPy arrays from pandas objects:to_numpy()
, which is defined onIndex
,Series
, andDataFrame
objects, andarray
, which is defined onIndex
andSeries
objects only.If you visit the v0.24 docs for
.values
, you will see a big red warning that says:See this section of the v0.24.0 release notes, and this answer for more information.
* -
to_numpy()
is my recommended method for any production code that needs to run reliably for many versions into the future. However if you're just making a scratchpad in jupyter or the terminal, using.values
to save a few milliseconds of typing is a permissable exception. You can always add the fit n finish later.Towards Better Consistency:
to_numpy()
In the spirit of better consistency throughout the API, a new method
to_numpy
has been introduced to extract the underlying NumPy array from DataFrames.As mentioned above, this method is also defined on
Index
andSeries
objects (see here).By default, a view is returned, so any modifications made will affect the original.
If you need a copy instead, use
to_numpy(copy=True)
.pandas >= 1.0 update for ExtensionTypes
If you're using pandas 1.x, chances are you'll be dealing with extension types a lot more. You'll have to be a little more careful that these extension types are correctly converted.
This is called out in the docs.
If you need the
dtypes
in the result...As shown in another answer,
DataFrame.to_records
is a good way to do this.This cannot be done with
to_numpy
, unfortunately. However, as an alternative, you can usenp.rec.fromrecords
:Performance wise, it's nearly the same (actually, using
rec.fromrecords
is a bit faster).Rationale for Adding a New Method
to_numpy()
(in addition toarray
) was added as a result of discussions under two GitHub issues GH19954 and GH23623.Specifically, the docs mention the rationale:
to_numpy
aims to improve the consistency of the API, which is a major step in the right direction..values
will not be deprecated in the current version, but I expect this may happen at some point in the future, so I would urge users to migrate towards the newer API, as soon as you can.Critique of Other Solutions
DataFrame.values
has inconsistent behaviour, as already noted.DataFrame.get_values()
is simply a wrapper aroundDataFrame.values
, so everything said above applies.DataFrame.as_matrix()
is deprecated now, do NOT use!