I want to apply scaling (using StandardScaler() from sklearn.preprocessing) to a pandas dataframe. The following code returns a numpy array, so I lose all the column names and indeces. This is not what I want.
features = df[["col1", "col2", "col3", "col4"]]
autoscaler = StandardScaler()
features = autoscaler.fit_transform(features)
A "solution" I found online is:
features = features.apply(lambda x: autoscaler.fit_transform(x))
It appears to work, but leads to a deprecationwarning:
/usr/lib/python3.5/site-packages/sklearn/preprocessing/data.py:583:
DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17
and will raise ValueError in 0.19. Reshape your data either using
X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1)
if it contains a single sample.
I therefore tried:
features = features.apply(lambda x: autoscaler.fit_transform(x.reshape(-1, 1)))
But this gives:
Traceback (most recent call last): File "./analyse.py", line 91, in
features = features.apply(lambda x: autoscaler.fit_transform(x.reshape(-1, 1))) File
"/usr/lib/python3.5/site-packages/pandas/core/frame.py", line 3972, in
apply
return self._apply_standard(f, axis, reduce=reduce) File "/usr/lib/python3.5/site-packages/pandas/core/frame.py", line 4081, in
_apply_standard
result = self._constructor(data=results, index=index) File "/usr/lib/python3.5/site-packages/pandas/core/frame.py", line 226, in
init
mgr = self._init_dict(data, index, columns, dtype=dtype) File "/usr/lib/python3.5/site-packages/pandas/core/frame.py", line 363, in
_init_dict
dtype=dtype) File "/usr/lib/python3.5/site-packages/pandas/core/frame.py", line 5163, in
_arrays_to_mgr
arrays = _homogenize(arrays, index, dtype) File "/usr/lib/python3.5/site-packages/pandas/core/frame.py", line 5477, in
_homogenize
raise_cast_failure=False) File "/usr/lib/python3.5/site-packages/pandas/core/series.py", line 2885,
in _sanitize_array
raise Exception('Data must be 1-dimensional') Exception: Data must be 1-dimensional
How do I apply scaling to the pandas dataframe, leaving the dataframe intact? Without copying the data if possible.
Best Answer
You could convert the DataFrame as a numpy array using
as_matrix()
. Example on a random dataset:Edit: Changing
as_matrix()
tovalues
, (it doesn't change the result) per the last sentence of theas_matrix()
docs above:Note, indices are 10-19:
Now
fit_transform
the DataFrame to get thescaled_features
array
:Assign the scaled data to a DataFrame (Note: use the
index
andcolumns
keyword arguments to keep your original indices and column names:Edit 2:
Came across the sklearn-pandas package. It's focused on making scikit-learn easier to use with pandas.
sklearn-pandas
is especially useful when you need to apply more than one type of transformation to column subsets of theDataFrame
, a more common scenario. It's documented, but this is how you'd achieve the transformation we just performed.