note.nkmk.me

Convert pandas.DataFrame, Series and numpy.ndarray to each other

Posted: 2019-11-06 / Tags: Python, pandas, NumPy

pandas.DataFrame, pandas.Series and NumPy array numpy.ndarray can be converted to each other.

  • Convert DataFrame, Series to ndarray: values
  • Convert ndarray to DataFrame, Series
  • Notes on memory sharing (view and copy)
  • pandas 0.24.0 or later: to_numpy()

Note that pandas.DataFrame and pandas.Series also have as_matrix() that returns numpy.ndarray, but it has been deprecated since version 0.23.0.

Sponsored Link

Convert DataFrame, Series to ndarray: values

Both pandas.DataFrame and pandas.Series have valiues attribute that returns NumPy array numpy.ndarray. After pandas 0.24.0, it is recommended to use the to_numpy() method introduced at the end of this post.

pandas.DataFrame:

import numpy as np
import pandas as pd

df = pd.DataFrame(data=[[1, 2, 3], [4, 5, 6]], columns=['a', 'b', 'c'])
print(df)
#    a  b  c
# 0  1  2  3
# 1  4  5  6

a_df = df.values
print(a_df)
# [[1 2 3]
#  [4 5 6]]

print(type(a_df))
# <class 'numpy.ndarray'>

print(a_df.dtype)
# int64

pandas.Series:

s = df['a']
print(s)
# 0    1
# 1    4
# Name: a, dtype: int64

a_s = s.values
print(a_s)
# [1 4]

print(type(a_s))
# <class 'numpy.ndarray'>

print(a_s.dtype)
# int64

The data type dtype of numpy.ndarray is the same as dtype of the original pandas.DataFrame and pandas.Series.

df_f = pd.DataFrame([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]])
print(df_f)
#      0    1    2
# 0  0.1  0.2  0.3
# 1  0.4  0.5  0.6

a_df_f = df_f.values
print(a_df_f)
# [[0.1 0.2 0.3]
#  [0.4 0.5 0.6]]

print(type(a_df_f))
# <class 'numpy.ndarray'>

print(a_df_f.dtype)
# float64

Pandas.DataFrame with a mixture of numbers and strings returns ndarray whose dtype is object.

df_multi = pd.read_csv('data/src/sample_pandas_normal.csv')
print(df_multi)
#       name  age state  point
# 0    Alice   24    NY     64
# 1      Bob   42    CA     92
# 2  Charlie   18    CA     70
# 3     Dave   68    TX     70
# 4    Ellen   24    CA     88
# 5    Frank   30    NY     57

a_df_multi = df_multi.values
print(a_df_multi)
# [['Alice' 24 'NY' 64]
#  ['Bob' 42 'CA' 92]
#  ['Charlie' 18 'CA' 70]
#  ['Dave' 68 'TX' 70]
#  ['Ellen' 24 'CA' 88]
#  ['Frank' 30 'NY' 57]]

print(type(a_df_multi))
# <class 'numpy.ndarray'>

print(a_df_multi.dtype)
# object

If you extract DataFrame columns and get values, you can convert only certain columns to ndarray.

If you select only numeric columns, the type of ndarray will be that type instead of object.

a_df_int = df_multi[['age', 'point']].values
print(a_df_int)
# [[24 64]
#  [42 92]
#  [18 70]
#  [68 70]
#  [24 88]
#  [30 57]]

print(type(a_df_int))
# <class 'numpy.ndarray'>

print(a_df_int.dtype)
# int64

If you want to transpose, use T.

print(a_df_int.T)
# [[24 42 18 68 24 30]
#  [64 92 70 70 88 57]]

You can also extract only certain types of columns. For example, if you want to extract only int64 columns, you can write:

a_df_int = df_multi.select_dtypes(include=int).values
print(a_df_int)
# [[24 64]
#  [42 92]
#  [18 70]
#  [68 70]
#  [24 88]
#  [30 57]]

print(type(a_df_int))
# <class 'numpy.ndarray'>

print(a_df_int.dtype)
# int64

Note that the original pandas.DataFrame and the convertednumpy.ndarray may share memory, and changing one may change the other. It will be described later.

Convert ndarray to DataFrame, Series

The NumPy array numpy.ndarray can be specified as the first argument data of the pandas.DataFrame and pandas.Series constructors. As described later, numpy.ndarray and generated pandas.DataFrame, pandas.Series share memory.

pandas.Series()

If no other arguments are specified in the constructor, it will be a Series of the original ndarray type.

import numpy as np
import pandas as pd

a = np.arange(4)
print(a)
# [0 1 2 3]

s = pd.Series(a)
print(s)
# 0    0
# 1    1
# 2    2
# 3    3
# dtype: int64

You can specify index, name, dtype, etc.

index = ['A', 'B', 'C', 'D']
name = 'sample'
s = pd.Series(data=a, index=index, name=name, dtype='float')
print(s)
# A    0.0
# B    1.0
# C    2.0
# D    3.0
# Name: sample, dtype: float64

Specifying a multidimensional ndarray as data in the Series constructor results in an error.

a = np.arange(12).reshape((4, 3))
print(a)
# [[ 0  1  2]
#  [ 3  4  5]
#  [ 6  7  8]
#  [ 9 10 11]]

# s = pd.Series(a)
# print(s)
# Exception: Data must be 1-dimensional

It is possible to select rows or columns of two-dimentional ndarray and convert them to Series.

s = pd.Series(a[2])
print(s)
# 0    6
# 1    7
# 2    8
# dtype: int64

s = pd.Series(a.T[2])
print(s)
# 0     2
# 1     5
# 2     8
# 3    11
# dtype: int64

pandas.DataFrame()

As with Series, if you don't specify any other arguments in the constructor, you get a DataFrame of the original ndarray type.

a = np.arange(12).reshape((4, 3))
print(a)
# [[ 0  1  2]
#  [ 3  4  5]
#  [ 6  7  8]
#  [ 9 10 11]]

df = pd.DataFrame(a)
print(df)
#    0   1   2
# 0  0   1   2
# 1  3   4   5
# 2  6   7   8
# 3  9  10  11

You can specify index, columns, dtype, etc.

index = ['A', 'B', 'C', 'D']
columns = ['a', 'b', 'c']
df = pd.DataFrame(data=a, index=index, columns=columns, dtype='float')
print(df)
#      a     b     c
# A  0.0   1.0   2.0
# B  3.0   4.0   5.0
# C  6.0   7.0   8.0
# D  9.0  10.0  11.0
Sponsored Link

Notes on memory sharing (view and copy)

pandas.DataFrame orpandas.Series and numpy.ndarray converted to each other byvalues attribute or constructor may share memory with each other. If memory is shared, changing one changes the other. Be careful if you want to use each separately.

The following sample code and results are for pandas0.25.1.

values attribute

First, consider the following case.

df = pd.DataFrame(data=[[1, 2, 3], [4, 5, 6]], columns=['a', 'b', 'c'])
print(df)
#    a  b  c
# 0  1  2  3
# 1  4  5  6

a_values = df.values
print(a_values)
# [[1 2 3]
#  [4 5 6]]

values returns a view, changing one changes the other. This can be checked with np.shares_memory().

print(np.shares_memory(a_values, df))
# True

a_values[0, 0] = 100
print(a_values)
# [[100   2   3]
#  [  4   5   6]]

print(df)
#      a  b  c
# 0  100  2  3
# 1    4  5  6

It does not always return a view; for example, if each column has a different data type dtaype, a copy is returned.

df_if = pd.DataFrame(data=[[1, 0.1], [2, 0.2]], columns=['int', 'float'])
print(df_if)
#    int  float
# 0    1    0.1
# 1    2    0.2

print(df_if.dtypes)
# int        int64
# float    float64
# dtype: object

a_values_if = df_if.values
print(a_values_if)
# [[1.  0.1]
#  [2.  0.2]]

print(np.shares_memory(a_values_if, df_if))
# False

a_values_if[0, 0] = 100
print(a_values_if)
# [[100.    0.1]
#  [  2.    0.2]]

print(df_if)
#    int  float
# 0    1    0.1
# 1    2    0.2

In addition, when selecting a range, whether the view is returned or the copy is returned depends on the specification method.

print(df[['a', 'c']].values)
# [[100   3]
#  [  4   6]]

print(np.shares_memory(df[['a', 'c']].values, df))
# False

print(df.iloc[:, ::2].values)
# [[100   3]
#  [  4   6]]

print(np.shares_memory(df.iloc[:, ::2].values, df))
# True

You can explicitly create a copy with copy() of values.

a_values_copy = df.values.copy()
print(a_values_copy)
# [[100   2   3]
#  [  4   5   6]]

print(np.shares_memory(a_values_copy, df))
# False

a_values_copy[0, 0] = 10
print(a_values_copy)
# [[10  2  3]
#  [ 4  5  6]]

print(df)
#      a  b  c
# 0  100  2  3
# 1    4  5  6

pandas.DataFrame(), pandas.Series()

Passing numpy.ndarray to the constructor pandas.DataFrame() or pandas.Series(), two objects will share the memory. Changing one value changes the other.

a = np.array([[1, 2, 3], [4, 5, 6]])
print(a)
# [[1 2 3]
#  [4 5 6]]

df_a = pd.DataFrame(a, columns=['a', 'b', 'c'])
print(df_a)
#    a  b  c
# 0  1  2  3
# 1  4  5  6

print(np.shares_memory(a, df_a))
# True

a[0, 0] = 100
print(a)
# [[100   2   3]
#  [  4   5   6]]

print(df_a)
#      a  b  c
# 0  100  2  3
# 1    4  5  6

df_a.iat[1, 0] = 10
print(df_a)
#      a  b  c
# 0  100  2  3
# 1   10  5  6

print(a)
# [[100   2   3]
#  [ 10   5   6]]

If you don't want to share memory, use copy() of numpy.ndarray.

df_a_copy = pd.DataFrame(a.copy(), columns=['a', 'b', 'c'])
print(df_a_copy)
#      a  b  c
# 0  100  2  3
# 1   10  5  6

a[0, 0] = 1
print(a)
# [[ 1  2  3]
#  [10  5  6]]

print(df_a_copy)
#      a  b  c
# 0  100  2  3
# 1   10  5  6

pandas 0.24.0 or later: to_numpy()

The to_numpy() method has been added to pandas.DataFrame and pandas.Series in pandas 0.24.0. This method returns numpy.ndarray, similar to the values attribute above.

The official documentation recommends using the to_numpy() method instead of the values attribute, but as of version 0.25.1, using the values attribute does not issue a warning.

df = pd.DataFrame(data=[[1, 2, 3], [4, 5, 6]], columns=['a', 'b', 'c'])
print(df)
#    a  b  c
# 0  1  2  3
# 1  4  5  6

a = df.to_numpy()
print(a)
# [[1 2 3]
#  [4 5 6]]

print(type(a))
# <class 'numpy.ndarray'>

By default, like the values attribute, it may return a view (numpy.ndarray that shares memory).

print(np.shares_memory(df, a))
# True

a[0, 0] = 100
print(a)
# [[100   2   3]
#  [  4   5   6]]

print(df)
#      a  b  c
# 0  100  2  3
# 1    4  5  6

Setting copy to True returns a copy. The default is copy=False.

a_copy = df.to_numpy(copy=True)
print(a_copy)
# [[100   2   3]
#  [  4   5   6]]

print(np.shares_memory(df, a_copy))
# False

a_copy[0, 0] = 10
print(a_copy)
# [[10  2  3]
#  [ 4  5  6]]

print(df)
#      a  b  c
# 0  100  2  3
# 1    4  5  6

The default (copy=False) may return a copy instead of a view. copy=True always returns a copy, but copy=False does not always return a view.

If the values attribute returns a copy, to_numpy(copy=False) also returns a copy.

a_cols = df[['a', 'c']].to_numpy()
print(a_cols)
# [[100   3]
#  [  4   6]]

print(np.shares_memory(df, a_cols))
# False

Data type can be specified with the parameter dtype. A copy is also returned if the data type has changed.

a_f = df.to_numpy(dtype=float)
print(a_f)
# [[100.   2.   3.]
#  [  4.   5.   6.]]

print(np.shares_memory(df, a_f))
# False
Sponsored Link
Share

Related Categories

Related Posts