pandas数据处理常用函数demo之创建/行列操作/查看/文件操作

pandas是Python下强大的数据分析工具，这篇文章代码主要来自于
10 Minutes to pandas，我将示例代码进行了重跑和修改，基本可以满足所有操作，但是使用更高级的功能可以达到事半功倍的效果：原文如下：
http://pandas.pydata.org/pandas-docs/stable/10min.html
初次使用pandas，很多人最头痛的就是Merge, join等表的操作了，下面这个官方手册用图形的形式形象的展示出来了表操作的方式：
http://pandas.pydata.org/pandas-docs/stable/merging.html

创建dataframe

DataFrame和Series作为padans两个主要的数据结构，是数据处理的载体和基础。

def create():

    #create Series
    s = pd.Series([1,3,5,np.nan,6,8])
    print s

    #create dataframe
    dates = pd.date_range('20130101', periods=6)
    df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list('ABCD'))
    print df

#Creating a DataFrame by passing a dict of objects that can be converted to series-like.
    df2 = pd.DataFrame({ 'A' : 1.,
                        'B' : pd.Timestamp('20130102'),
                        'C' : pd.Series(1,index=list(range(4)),dtype='float32'),
                        'D' : np.array([3] * 4,dtype='int32'),
                        'E' : pd.Categorical(["test","train","test","train"]),
                        'F' : 'foo' })
    print df2
    #Having specific dtypes
    print df2.dtypes

查看dataframe属性

我们生成数据或者从文件加在数据后，首先要看数据是否符合我们的需求，比如行和列数目，每列的基本统计信息等，这些信息可以让我们认识数据的特点或者检查数据的正确性：

def see():

    dates = pd.date_range('20130101', periods=6)
    df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list('ABCD'))
    print df

    #See the top & bottom rows of the frame'''
    print df.head(2)
    print df.tail(1)

    #Display the index, columns, and the underlying numpy data,num of line and col
    print df.index
    print df.columns
    print df.values
    print df.shape[0]
    print df.shape[1]

    #Describe shows a quick statistic summary of your data
    print df.describe()

    #Transposing your data
    print df.T

    #Sorting by an axis,0 is y,1 is x,ascending True is zhengxv,false is daoxv
    print df.sort_index(axis=0, ascending=False)

    #Sorting by values
    print df.sort(column='B')

    #see valuenums
    print df[0].value_counts()
    print df[u'hah'].value_counts()

    #see type and change
    df.dtypes
    df[['two', 'three']] = df[['two', 'three']].astype(float)

选取数据

了解了数据基本信息后，我们可能要对数据进行一些裁剪。很多情况下，我们并不需要数据的全部信息，因此我们要学会选取出我们感兴趣的数据和行列，接下来的例子就是对数据的裁剪：

def selection():

    dates = pd.date_range('20130101', periods=6)
    df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list('ABCD'))
    print df

    #Selecting a single column, which yields a Series, equivalent to df.A
    print df['A']
    print df.A

    #Selecting via [], which slices the rows.
    print df[0:3]
    print df['20130102':'20130104']

    #Selection by Label

    #For getting a cross section using a label
    print df.loc[dates[0]]

    #Selecting on a multi-axis by label
    print df.loc[:,['A','B']]

    #Showing label slicing, both endpoints are included
    print df.loc['20130102':'20130104',['A','B']]

    #For getting a scalar value
    print df.loc[dates[0],'A']
    print df.at[dates[0],'A']


    #Selection by Position

    #Select via the position of the passed integers
    print df.iloc[3]

    #By integer slices, acting similar to numpy/python
    print df.iloc[3:5,0:2]

    #By lists of integer position locations, similar to the numpy/python style
    print df.iloc[[1,2,4],[0,2]]

    #For slicing rows explicitly
    print df.iloc[1:3,:]

    #For getting a value explicitly
    print df.iloc[1,1]
    print df.iat[1,1]


    #Boolean Indexing

    #Using a single column's values to select data.
    print df[df.A > 0]

    #Using the isin() method for filtering:
    df2 = df.copy()
    df2['E'] = ['one', 'one','two','three','four','three']
    print df2[df2['E'].isin(['two','four'])]

    #A where operation for getting.
    print df[df > 0]
    df2[df2 > 0] = -df2

    #Setting
    #Setting a new column automatically aligns the data by the indexes
    s1 = pd.Series([1,2,3,4,5,6], index=pd.date_range('20130102', periods=6))
    df['F'] = s1
    print df

    #Setting values by label/index
    df.at[dates[0],'A'] = 0
    df.iat[0,1] = 0
    print df

    #Setting by assigning with a numpy array
    df.loc[:,'D'] = np.array([5] * len(df))
    print df

文件操作

很多时候，我们的数据并不是自己生成的，而是从文件中读取的，数据文件则具有各种各样的来源，下面就展示如何加载和保存数据。pandas提供了多种API，可以加载txt/csv/libsvm等各个格式的数据，完全可以满足数据分析的需求

def file():
    ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
    df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index,
                      columns=['A', 'B', 'C', 'D'])
    pd.read_csv('foo.csv')
    df.to_csv('foo.csv')