Data aggregation and group operations in pandas

After loading,merging and preparing a dataset,you may need to compute group statistics or possibly pivot tables for reporting or visualization purpose.pandas provides a flexible groupby interface,enabling you to slice,dice and summarize datasets in a natural way.

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

Groupby mechanics

split-apply-combine:
In the first stage of the process,data contained in a pandas object,whether a Series,DataFrame,or otherwise,is split into groups based on one or more keys that you provide.The splitting is performed on a particular axis off an object.For example,a DataFrame can be grouped on its rows(axis=0) or its columns(axis=1).
Once the first step is done, a function is applied to each group,producing a new value.
Finally,the results of all those function applications are combined into a result object.

help(pd.Series.groupby)

Help on function groupby in module pandas.core.generic:

groupby(self, by=None, axis=0, level=None, as_index=True, sort=True, group_keys=True, squeeze=False, **kwargs)
    Group series using mapper (dict or key function, apply given function
    to group, return result as series) or by a series of columns.
    
    Parameters
    ----------
    by : mapping function / list of functions, dict, Series, or tuple /
        list of column names.
        Called on each element of the object index to determine the groups.
        If a dict or Series is passed, the Series or dict VALUES will be
        used to determine the groups
    axis : int, default 0
    level : int, level name, or sequence of such, default None
        If the axis is a MultiIndex (hierarchical), group by a particular
        level or levels
    as_index : boolean, default True
        For aggregated output, return object with group labels as the
        index. Only relevant for DataFrame input. as_index=False is
        effectively "SQL-style" grouped output
    sort : boolean, default True
        Sort group keys. Get better performance by turning this off.
        Note this does not influence the order of observations within each
        group.  groupby preserves the order of rows within each group.
    group_keys : boolean, default True
        When calling apply, add group keys to index to identify pieces
    squeeze : boolean, default False
        reduce the dimensionality of the return type if possible,
        otherwise return a consistent type
    
    Examples
    --------
    DataFrame results
    
    >>> data.groupby(func, axis=0).mean()
    >>> data.groupby(['col1', 'col2'])['col3'].mean()
    
    DataFrame with hierarchical index
    
    >>> data.groupby(['col1', 'col2']).mean()
    
    Returns
    -------
    GroupBy object

df=pd.DataFrame({'key1':['a','a','b','b','a'],
                'key2':['one','two','one','two','one'],
                'data1':np.random.randn(5),
                'data2':np.random.randn(5)});df

• The first method:
  Suppose you wanted to compute the mean of the data column using the labels from key1.There are a number of ways to do this.One is to access data1 and call groupby with the column(a Series) at key1:

grouped=df['data1'].groupby(df['key1'])

help(grouped)

Help on SeriesGroupBy in module pandas.core.groupby object:

class SeriesGroupBy(GroupBy)
 |  Class for grouping and aggregating relational data. See aggregate,
 |  transform, and apply functions on this object.
 |  
 |  It's easiest to use obj.groupby(...) to use GroupBy, but you can also do:
 |  
 |  ::
 |  
 |      grouped = groupby(obj, ...)
 |  
 |  Parameters
 |  ----------
 |  obj : pandas object
 |  axis : int, default 0
 |  level : int, default None
 |      Level of MultiIndex
 |  groupings : list of Grouping objects
 |      Most users should ignore this
 |  exclusions : array-like, optional
 |      List of columns to exclude
 |  name : string
 |      Most users should ignore this
 |  
 |  Notes
 |  -----
 |  After grouping, see aggregate, apply, and transform functions. Here are
 |  some other brief notes about usage. When grouping by multiple groups, the
 |  result index will be a MultiIndex (hierarchical) by default.
 |  
 |  Iteration produces (key, group) tuples, i.e. chunking the data by group. So
 |  you can write code like:
 |  
 |  ::
 |  
 |      grouped = obj.groupby(keys, axis=axis)
 |      for key, group in grouped:
 |          # do something with the data
 |  
 |  Function calls on GroupBy, if not specially implemented, "dispatch" to the
 |  grouped data. So if you group a DataFrame and wish to invoke the std()
 |  method on each group, you can simply do:
 |  
 |  ::
 |  
 |      df.groupby(mapper).std()
 |  
 |  rather than
 |  
 |  ::
 |  
 |      df.groupby(mapper).aggregate(np.std)
 |  
 |  You can pass arguments to these "wrapped" functions, too.
 |  
 |  See the online documentation for full exposition on these topics and much
 |  more
 |  
 |  Returns
 |  -------
 |  **Attributes**
 |  groups : dict
 |      {group name -> group labels}
 |  len(grouped) : int
 |      Number of groups
 |  
 |  Method resolution order:
 |      SeriesGroupBy
 |      GroupBy
 |      _GroupBy
 |      pandas.core.base.PandasObject
 |      pandas.core.base.StringMixin
 |      pandas.core.base.SelectionMixin
 |      builtins.object
 |  
 |  Methods defined here:
 |  
 |  agg = aggregate(self, func_or_funcs, *args, **kwargs)
 |  
 |  aggregate(self, func_or_funcs, *args, **kwargs)
 |      Apply aggregation function or functions to groups, yielding most likely
 |      Series but in some cases DataFrame depending on the output of the
 |      aggregation function
 |      
 |      Parameters
 |      ----------
 |      func_or_funcs : function or list / dict of functions
 |          List/dict of functions will produce DataFrame with column names
 |          determined by the function names themselves (list) or the keys in
 |          the dict
 |      
 |      Notes
 |      -----
 |      agg is an alias for aggregate. Use it.
 |      
 |      Examples
 |      --------
 |      >>> series
 |      bar    1.0
 |      baz    2.0
 |      qot    3.0
 |      qux    4.0
 |      
 |      >>> mapper = lambda x: x[0] # first letter
 |      >>> grouped = series.groupby(mapper)
 |      
 |      >>> grouped.aggregate(np.sum)
 |      b    3.0
 |      q    7.0
 |      
 |      >>> grouped.aggregate([np.sum, np.mean, np.std])
 |         mean  std  sum
 |      b  1.5   0.5  3
 |      q  3.5   0.5  7
 |      
 |      >>> grouped.agg({'result' : lambda x: x.mean() / x.std(),
 |      ...              'total' : np.sum})
 |         result  total
 |      b  2.121   3
 |      q  4.95    7
 |      
 |      See also
 |      --------
 |      apply, transform
 |      
 |      Returns
 |      -------
 |      Series or DataFrame
 |  
 |  count(self)
 |      Compute count of group, excluding missing values
 |  
 |  filter(self, func, dropna=True, *args, **kwargs)
 |      Return a copy of a Series excluding elements from groups that
 |      do not satisfy the boolean criterion specified by func.
 |      
 |      Parameters
 |      ----------
 |      func : function
 |          To apply to each group. Should return True or False.
 |      dropna : Drop groups that do not pass the filter. True by default;
 |          if False, groups that evaluate False are filled with NaNs.
 |      
 |      Examples
 |      --------
 |      >>> grouped.filter(lambda x: x.mean() > 0)
 |      
 |      Returns
 |      -------
 |      filtered : Series
 |  
 |  nlargest(self, n=5, keep='first')
 |      Return the largest `n` elements.
 |      
 |      Parameters
 |      ----------
 |      n : int
 |          Return this many descending sorted values
 |      keep : {'first', 'last', False}, default 'first'
 |          Where there are duplicate values:
 |          - ``first`` : take the first occurrence.
 |          - ``last`` : take the last occurrence.
 |      take_last : deprecated
 |      
 |      Returns
 |      -------
 |      top_n : Series
 |          The n largest values in the Series, in sorted order
 |      
 |      Notes
 |      -----
 |      Faster than ``.sort_values(ascending=False).head(n)`` for small `n`
 |      relative to the size of the ``Series`` object.
 |      
 |      See Also
 |      --------
 |      Series.nsmallest
 |      
 |      Examples
 |      --------
 |      >>> import pandas as pd
 |      >>> import numpy as np
 |      >>> s = pd.Series(np.random.randn(1e6))
 |      >>> s.nlargest(10)  # only sorts up to the N requested
 |  
 |  nsmallest(self, n=5, keep='first')
 |      Return the smallest `n` elements.
 |      
 |      Parameters
 |      ----------
 |      n : int
 |          Return this many ascending sorted values
 |      keep : {'first', 'last', False}, default 'first'
 |          Where there are duplicate values:
 |          - ``first`` : take the first occurrence.
 |          - ``last`` : take the last occurrence.
 |      take_last : deprecated
 |      
 |      Returns
 |      -------
 |      bottom_n : Series
 |          The n smallest values in the Series, in sorted order
 |      
 |      Notes
 |      -----
 |      Faster than ``.sort_values().head(n)`` for small `n` relative to
 |      the size of the ``Series`` object.
 |      
 |      See Also
 |      --------
 |      Series.nlargest
 |      
 |      Examples
 |      --------
 |      >>> import pandas as pd
 |      >>> import numpy as np
 |      >>> s = pd.Series(np.random.randn(1e6))
 |      >>> s.nsmallest(10)  # only sorts up to the N requested
 |  
 |  nunique(self, dropna=True)
 |      Returns number of unique elements in the group
 |  
 |  transform(self, func, *args, **kwargs)
 |      Call function producing a like-indexed Series on each group and return
 |      a Series with the transformed values
 |      
 |      Parameters
 |      ----------
 |      func : function
 |          To apply to each group. Should return a Series with the same index
 |      
 |      Examples
 |      --------
 |      >>> grouped.transform(lambda x: (x - x.mean()) / x.std())
 |      
 |      Returns
 |      -------
 |      transformed : Series
 |  
 |  value_counts(self, normalize=False, sort=True, ascending=False, bins=None, dropna=True)
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |  
 |  all
 |      
 |      Return whether all elements are True over requested axis
 |      
 |      Parameters
 |      ----------
 |      axis : {index (0)}
 |      skipna : boolean, default True
 |          Exclude NA/null values. If an entire row/column is NA, the result
 |          will be NA
 |      level : int or level name, default None
 |          If the axis is a MultiIndex (hierarchical), count along a
 |          particular level, collapsing into a scalar
 |      bool_only : boolean, default None
 |          Include only boolean columns. If None, will attempt to use everything,
 |          then use only boolean data. Not implemented for Series.
 |      
 |      Returns
 |      -------
 |      all : scalar or Series (if level specified)
 |  
 |  any
 |      
 |      Return whether any element is True over requested axis
 |      
 |      Parameters
 |      ----------
 |      axis : {index (0)}
 |      skipna : boolean, default True
 |          Exclude NA/null values. If an entire row/column is NA, the result
 |          will be NA
 |      level : int or level name, default None
 |          If the axis is a MultiIndex (hierarchical), count along a
 |          particular level, collapsing into a scalar
 |      bool_only : boolean, default None
 |          Include only boolean columns. If None, will attempt to use everything,
 |          then use only boolean data. Not implemented for Series.
 |      
 |      Returns
 |      -------
 |      any : scalar or Series (if level specified)
 |  
 |  corr
 |      Compute correlation with `other` Series, excluding missing values
 |      
 |      Parameters
 |      ----------
 |      other : Series
 |      method : {'pearson', 'kendall', 'spearman'}
 |          * pearson : standard correlation coefficient
 |          * kendall : Kendall Tau correlation coefficient
 |          * spearman : Spearman rank correlation
 |      min_periods : int, optional
 |          Minimum number of observations needed to have a valid result
 |      
 |      
 |      Returns
 |      -------
 |      correlation : float
 |  
 |  cov
 |      Compute covariance with Series, excluding missing values
 |      
 |      Parameters
 |      ----------
 |      other : Series
 |      min_periods : int, optional
 |          Minimum number of observations needed to have a valid result
 |      
 |      Returns
 |      -------
 |      covariance : float
 |      
 |      Normalized by N-1 (unbiased estimator).
 |  
 |  cummax
 |          Return cumulative max over requested axis.
 |      
 |      Parameters
 |      ----------
 |      axis : {index (0)}
 |      skipna : boolean, default True
 |          Exclude NA/null values. If an entire row/column is NA, the result
 |          will be NA
 |      
 |      Returns
 |      -------
 |      cummax : scalar
 |  
 |  cummin
 |          Return cumulative minimum over requested axis.
 |      
 |      Parameters
 |      ----------
 |      axis : {index (0)}
 |      skipna : boolean, default True
 |          Exclude NA/null values. If an entire row/column is NA, the result
 |          will be NA
 |      
 |      Returns
 |      -------
 |      cummin : scalar
 |  
 |  describe
 |      Generate various summary statistics, excluding NaN values.
 |      
 |      Parameters
 |      ----------
 |      percentiles : array-like, optional
 |          The percentiles to include in the output. Should all
 |          be in the interval [0, 1]. By default `percentiles` is
 |          [.25, .5, .75], returning the 25th, 50th, and 75th percentiles.
 |      include, exclude : list-like, 'all', or None (default)
 |          Specify the form of the returned result. Either:
 |      
 |          - None to both (default). The result will include only
 |            numeric-typed columns or, if none are, only categorical columns.
 |          - A list of dtypes or strings to be included/excluded.
 |            To select all numeric types use numpy numpy.number. To select
 |            categorical objects use type object. See also the select_dtypes
 |            documentation. eg. df.describe(include=['O'])
 |          - If include is the string 'all', the output column-set will
 |            match the input one.
 |      
 |      Returns
 |      -------
 |      summary: NDFrame of summary statistics
 |      
 |      Notes
 |      -----
 |      The output DataFrame index depends on the requested dtypes:
 |      
 |      For numeric dtypes, it will include: count, mean, std, min,
 |      max, and lower, 50, and upper percentiles.
 |      
 |      For object dtypes (e.g. timestamps or strings), the index
 |      will include the count, unique, most common, and frequency of the
 |      most common. Timestamps also include the first and last items.
 |      
 |      For mixed dtypes, the index will be the union of the corresponding
 |      output types. Non-applicable entries will be filled with NaN.
 |      Note that mixed-dtype outputs can only be returned from mixed-dtype
 |      inputs and appropriate use of the include/exclude arguments.
 |      
 |      If multiple values have the highest count, then the
 |      `count` and `most common` pair will be arbitrarily chosen from
 |      among those with the highest count.
 |      
 |      The include, exclude arguments are ignored for Series.
 |      
 |      See Also
 |      --------
 |      DataFrame.select_dtypes
 |  
 |  diff
 |      1st discrete difference of object
 |      
 |      Parameters
 |      ----------
 |      periods : int, default 1
 |          Periods to shift for forming difference
 |      
 |      Returns
 |      -------
 |      diffed : Series
 |  
 |  dtype
 |      return the dtype object of the underlying data
 |  
 |  fillna
 |      Fill NA/NaN values using the specified method
 |      
 |      Parameters
 |      ----------
 |      value : scalar, dict, Series, or DataFrame
 |          Value to use to fill holes (e.g. 0), alternately a
 |          dict/Series/DataFrame of values specifying which value to use for
 |          each index (for a Series) or column (for a DataFrame). (values not
 |          in the dict/Series/DataFrame will not be filled). This value cannot
 |          be a list.
 |      method : {'backfill', 'bfill', 'pad', 'ffill', None}, default None
 |          Method to use for filling holes in reindexed Series
 |          pad / ffill: propagate last valid observation forward to next valid
 |          backfill / bfill: use NEXT valid observation to fill gap
 |      axis : {0, 'index'}
 |      inplace : boolean, default False
 |          If True, fill in place. Note: this will modify any
 |          other views on this object, (e.g. a no-copy slice for a column in a
 |          DataFrame).
 |      limit : int, default None
 |          If method is specified, this is the maximum number of consecutive
 |          NaN values to forward/backward fill. In other words, if there is
 |          a gap with more than this number of consecutive NaNs, it will only
 |          be partially filled. If method is not specified, this is the
 |          maximum number of entries along the entire axis where NaNs will be
 |          filled.
 |      downcast : dict, default is None
 |          a dict of item->dtype of what to downcast if possible,
 |          or the string 'infer' which will try to downcast to an appropriate
 |          equal type (e.g. float64 to int64 if possible)
 |      
 |      See Also
 |      --------
 |      reindex, asfreq
 |      
 |      Returns
 |      -------
 |      filled : Series
 |  
 |  hist
 |      Draw histogram of the input series using matplotlib
 |      
 |      Parameters
 |      ----------
 |      by : object, optional
 |          If passed, then used to form histograms for separate groups
 |      ax : matplotlib axis object
 |          If not passed, uses gca()
 |      grid : boolean, default True
 |          Whether to show axis grid lines
 |      xlabelsize : int, default None
 |          If specified changes the x-axis label size
 |      xrot : float, default None
 |          rotation of x axis labels
 |      ylabelsize : int, default None
 |          If specified changes the y-axis label size
 |      yrot : float, default None
 |          rotation of y axis labels
 |      figsize : tuple, default None
 |          figure size in inches by default
 |      bins: integer, default 10
 |          Number of histogram bins to be used
 |      kwds : keywords
 |          To be passed to the actual plotting function
 |      
 |      Notes
 |      -----
 |      See matplotlib documentation online for more on this
 |  
 |  idxmax
 |      Index of first occurrence of maximum of values.
 |      
 |      Parameters
 |      ----------
 |      skipna : boolean, default True
 |          Exclude NA/null values
 |      
 |      Returns
 |      -------
 |      idxmax : Index of maximum of values
 |      
 |      Notes
 |      -----
 |      This method is the Series version of ``ndarray.argmax``.
 |      
 |      See Also
 |      --------
 |      DataFrame.idxmax
 |      numpy.ndarray.argmax
 |  
 |  idxmin
 |      Index of first occurrence of minimum of values.
 |      
 |      Parameters
 |      ----------
 |      skipna : boolean, default True
 |          Exclude NA/null values
 |      
 |      Returns
 |      -------
 |      idxmin : Index of minimum of values
 |      
 |      Notes
 |      -----
 |      This method is the Series version of ``ndarray.argmin``.
 |      
 |      See Also
 |      --------
 |      DataFrame.idxmin
 |      numpy.ndarray.argmin
 |  
 |  mad
 |      
 |      Return the mean absolute deviation of the values for the requested axis
 |      
 |      Parameters
 |      ----------
 |      axis : {index (0)}
 |      skipna : boolean, default True
 |          Exclude NA/null values. If an entire row/column is NA, the result
 |          will be NA
 |      level : int or level name, default None
 |          If the axis is a MultiIndex (hierarchical), count along a
 |          particular level, collapsing into a scalar
 |      numeric_only : boolean, default None
 |          Include only float, int, boolean columns. If None, will attempt to use
 |          everything, then use only numeric data. Not implemented for Series.
 |      
 |      Returns
 |      -------
 |      mad : scalar or Series (if level specified)
 |  
 |  name
 |      since we are a series, we by definition only have
 |      a single name, but may be the result of a selection or
 |      the name of our object
 |  
 |  pct_change
 |      Percent change over given number of periods.
 |      
 |      Parameters
 |      ----------
 |      periods : int, default 1
 |          Periods to shift for forming percent change
 |      fill_method : str, default 'pad'
 |          How to handle NAs before computing percent changes
 |      limit : int, default None
 |          The number of consecutive NAs to fill before stopping
 |      freq : DateOffset, timedelta, or offset alias string, optional
 |          Increment to use from time series API (e.g. 'M' or BDay())
 |      
 |      Returns
 |      -------
 |      chg : NDFrame
 |      
 |      Notes
 |      -----
 |      
 |      By default, the percentage change is calculated along the stat
 |      axis: 0, or ``Index``, for ``DataFrame`` and 1, or ``minor`` for
 |      ``Panel``. You can change this with the ``axis`` keyword argument.
 |  
 |  quantile
 |      Return value at the given quantile, a la numpy.percentile.
 |      
 |      Parameters
 |      ----------
 |      q : float or array-like, default 0.5 (50% quantile)
 |          0 <= q <= 1, the quantile(s) to compute
 |      interpolation : {'linear', 'lower', 'higher', 'midpoint', 'nearest'}
 |          .. versionadded:: 0.18.0
 |      
 |          This optional parameter specifies the interpolation method to use,
 |          when the desired quantile lies between two data points `i` and `j`:
 |      
 |              * linear: `i + (j - i) * fraction`, where `fraction` is the
 |                fractional part of the index surrounded by `i` and `j`.
 |              * lower: `i`.
 |              * higher: `j`.
 |              * nearest: `i` or `j` whichever is nearest.
 |              * midpoint: (`i` + `j`) / 2.
 |      
 |      Returns
 |      -------
 |      quantile : float or Series
 |          if ``q`` is an array, a Series will be returned where the
 |          index is ``q`` and the values are the quantiles.
 |      
 |      Examples
 |      --------
 |      >>> s = Series([1, 2, 3, 4])
 |      >>> s.quantile(.5)
 |      2.5
 |      >>> s.quantile([.25, .5, .75])
 |      0.25    1.75
 |      0.50    2.50
 |      0.75    3.25
 |      dtype: float64
 |  
 |  rank
 |      Compute numerical data ranks (1 through n) along axis. Equal values are
 |      assigned a rank that is the average of the ranks of those values
 |      
 |      Parameters
 |      ----------
 |      axis: {0 or 'index', 1 or 'columns'}, default 0
 |          index to direct ranking
 |      method : {'average', 'min', 'max', 'first', 'dense'}
 |          * average: average rank of group
 |          * min: lowest rank in group
 |          * max: highest rank in group
 |          * first: ranks assigned in order they appear in the array
 |          * dense: like 'min', but rank always increases by 1 between groups
 |      numeric_only : boolean, default None
 |          Include only float, int, boolean data. Valid only for DataFrame or
 |          Panel objects
 |      na_option : {'keep', 'top', 'bottom'}
 |          * keep: leave NA values where they are
 |          * top: smallest rank if ascending
 |          * bottom: smallest rank if descending
 |      ascending : boolean, default True
 |          False for ranks by high (1) to low (N)
 |      pct : boolean, default False
 |          Computes percentage rank of data
 |      
 |      Returns
 |      -------
 |      ranks : same type as caller
 |  
 |  skew
 |      
 |      Return unbiased skew over requested axis
 |      Normalized by N-1
 |      
 |      Parameters
 |      ----------
 |      axis : {index (0)}
 |      skipna : boolean, default True
 |          Exclude NA/null values. If an entire row/column is NA, the result
 |          will be NA
 |      level : int or level name, default None
 |          If the axis is a MultiIndex (hierarchical), count along a
 |          particular level, collapsing into a scalar
 |      numeric_only : boolean, default None
 |          Include only float, int, boolean columns. If None, will attempt to use
 |          everything, then use only numeric data. Not implemented for Series.
 |      
 |      Returns
 |      -------
 |      skew : scalar or Series (if level specified)
 |  
 |  take
 |      return Series corresponding to requested indices
 |      
 |      Parameters
 |      ----------
 |      indices : list / array of ints
 |      convert : translate negative to positive indices (default)
 |      
 |      Returns
 |      -------
 |      taken : Series
 |      
 |      See also
 |      --------
 |      numpy.ndarray.take
 |  
 |  tshift
 |      Shift the time index, using the index's frequency if available.
 |      
 |      Parameters
 |      ----------
 |      periods : int
 |          Number of periods to move, can be positive or negative
 |      freq : DateOffset, timedelta, or time rule string, default None
 |          Increment to use from the tseries module or time rule (e.g. 'EOM')
 |      axis : int or basestring
 |          Corresponds to the axis that contains the Index
 |      
 |      Notes
 |      -----
 |      If freq is not specified then tries to use the freq or inferred_freq
 |      attributes of the index. If neither of those attributes exist, a
 |      ValueError is thrown
 |      
 |      Returns
 |      -------
 |      shifted : NDFrame
 |  
 |  unique
 |      Return np.ndarray of unique values in the object.
 |      Significantly faster than numpy.unique. Includes NA values.
 |      The order of the original is preserved.
 |      
 |      Returns
 |      -------
 |      uniques : np.ndarray
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from GroupBy:
 |  
 |  backfill(self, limit=None)
 |      Backward fill the values
 |      
 |      Parameters
 |      ----------
 |      limit : integer, optional
 |          limit of how many values to fill
 |      
 |      See Also
 |      --------
 |      Series.fillna
 |      DataFrame.fillna
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  bfill = backfill(self, limit=None)
 |  
 |  cumcount(self, ascending=True)
 |      Number each item in each group from 0 to the length of that group - 1.
 |      
 |      Essentially this is equivalent to
 |      
 |      >>> self.apply(lambda x: Series(np.arange(len(x)), x.index))
 |      
 |      Parameters
 |      ----------
 |      ascending : bool, default True
 |          If False, number in reverse, from length of group - 1 to 0.
 |      
 |      Examples
 |      --------
 |      
 |      >>> df = pd.DataFrame([['a'], ['a'], ['a'], ['b'], ['b'], ['a']],
 |      ...                   columns=['A'])
 |      >>> df
 |         A
 |      0  a
 |      1  a
 |      2  a
 |      3  b
 |      4  b
 |      5  a
 |      >>> df.groupby('A').cumcount()
 |      0    0
 |      1    1
 |      2    2
 |      3    0
 |      4    1
 |      5    3
 |      dtype: int64
 |      >>> df.groupby('A').cumcount(ascending=False)
 |      0    3
 |      1    2
 |      2    1
 |      3    1
 |      4    0
 |      5    0
 |      dtype: int64
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  cumprod(self, axis=0, *args, **kwargs)
 |      Cumulative product for each group
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  cumsum(self, axis=0, *args, **kwargs)
 |      Cumulative sum for each group
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  expanding(self, *args, **kwargs)
 |      Return an expanding grouper, providing expanding
 |      functionaility per group
 |      
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  ffill = pad(self, limit=None)
 |  
 |  first = f(self)
 |      Compute first of group values
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  head(self, n=5)
 |      Returns first n rows of each group.
 |      
 |      Essentially equivalent to ``.apply(lambda x: x.head(n))``,
 |      except ignores as_index flag.
 |      
 |      Examples
 |      --------
 |      
 |      >>> df = DataFrame([[1, 2], [1, 4], [5, 6]],
 |                         columns=['A', 'B'])
 |      >>> df.groupby('A', as_index=False).head(1)
 |         A  B
 |      0  1  2
 |      2  5  6
 |      >>> df.groupby('A').head(1)
 |         A  B
 |      0  1  2
 |      2  5  6
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  irow(self, i)
 |      DEPRECATED. Use ``.nth(i)`` instead
 |  
 |  last = f(self)
 |      Compute last of group values
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  max = f(self)
 |      Compute max of group values
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  mean(self, *args, **kwargs)
 |      Compute mean of groups, excluding missing values
 |      
 |      For multiple groupings, the result index will be a MultiIndex
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  median(self)
 |      Compute median of groups, excluding missing values
 |      
 |      For multiple groupings, the result index will be a MultiIndex
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  min = f(self)
 |      Compute min of group values
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  nth(self, n, dropna=None)
 |      Take the nth row from each group if n is an int, or a subset of rows
 |      if n is a list of ints.
 |      
 |      If dropna, will take the nth non-null row, dropna is either
 |      Truthy (if a Series) or 'all', 'any' (if a DataFrame);
 |      this is equivalent to calling dropna(how=dropna) before the
 |      groupby.
 |      
 |      Parameters
 |      ----------
 |      n : int or list of ints
 |          a single nth value for the row or a list of nth values
 |      dropna : None or str, optional
 |          apply the specified dropna operation before counting which row is
 |          the nth row. Needs to be None, 'any' or 'all'
 |      
 |      Examples
 |      --------
 |      
 |      >>> df = pd.DataFrame({'A': [1, 1, 2, 1, 2],
 |      ...                    'B': [np.nan, 2, 3, 4, 5]}, columns=['A', 'B'])
 |      >>> g = df.groupby('A')
 |      >>> g.nth(0)
 |           B
 |      A
 |      1  NaN
 |      2  3.0
 |      >>> g.nth(1)
 |           B
 |      A
 |      1  2.0
 |      2  5.0
 |      >>> g.nth(-1)
 |           B
 |      A
 |      1  4.0
 |      2  5.0
 |      >>> g.nth([0, 1])
 |           B
 |      A
 |      1  NaN
 |      1  2.0
 |      2  3.0
 |      2  5.0
 |      
 |      Specifying ``dropna`` allows count ignoring NaN
 |      
 |      >>> g.nth(0, dropna='any')
 |           B
 |      A
 |      1  2.0
 |      2  3.0
 |      
 |      NaNs denote group exhausted when using dropna
 |      
 |      >>> g.nth(3, dropna='any')
 |          B
 |      A
 |      1 NaN
 |      2 NaN
 |      
 |      Specifying ``as_index=False`` in ``groupby`` keeps the original index.
 |      
 |      >>> df.groupby('A', as_index=False).nth(1)
 |         A    B
 |      1  1  2.0
 |      4  2  5.0
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  ohlc(self)
 |      Compute sum of values, excluding missing values
 |      For multiple groupings, the result index will be a MultiIndex
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  pad(self, limit=None)
 |      Forward fill the values
 |      
 |      Parameters
 |      ----------
 |      limit : integer, optional
 |          limit of how many values to fill
 |      
 |      See Also
 |      --------
 |      Series.fillna
 |      DataFrame.fillna
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  prod = f(self)
 |      Compute prod of group values
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  resample(self, rule, *args, **kwargs)
 |      Provide resampling when using a TimeGrouper
 |      Return a new grouper with our resampler appended
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  rolling(self, *args, **kwargs)
 |      Return a rolling grouper, providing rolling
 |      functionaility per group
 |      
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  sem(self, ddof=1)
 |      Compute standard error of the mean of groups, excluding missing values
 |      
 |      For multiple groupings, the result index will be a MultiIndex
 |      
 |      Parameters
 |      ----------
 |      ddof : integer, default 1
 |          degrees of freedom
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  shift(self, periods=1, freq=None, axis=0)
 |      Shift each group by periods observations
 |      
 |      Parameters
 |      ----------
 |      periods : integer, default 1
 |          number of periods to shift
 |      freq : frequency string
 |      axis : axis to shift, default 0
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  size(self)
 |      Compute group sizes
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  std(self, ddof=1, *args, **kwargs)
 |      Compute standard deviation of groups, excluding missing values
 |      
 |      For multiple groupings, the result index will be a MultiIndex
 |      
 |      Parameters
 |      ----------
 |      ddof : integer, default 1
 |          degrees of freedom
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  sum = f(self)
 |      Compute sum of group values
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  tail(self, n=5)
 |      Returns last n rows of each group
 |      
 |      Essentially equivalent to ``.apply(lambda x: x.tail(n))``,
 |      except ignores as_index flag.
 |      
 |      Examples
 |      --------
 |      
 |      >>> df = DataFrame([['a', 1], ['a', 2], ['b', 1], ['b', 2]],
 |                         columns=['A', 'B'])
 |      >>> df.groupby('A').tail(1)
 |         A  B
 |      1  a  2
 |      3  b  2
 |      >>> df.groupby('A').head(1)
 |         A  B
 |      0  a  1
 |      2  b  1
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  var(self, ddof=1, *args, **kwargs)
 |      Compute variance of groups, excluding missing values
 |      
 |      For multiple groupings, the result index will be a MultiIndex
 |      
 |      Parameters
 |      ----------
 |      ddof : integer, default 1
 |          degrees of freedom
 |      
 |      
 |      See also
 |      --------
 |      pandas.Series.groupby
 |      pandas.DataFrame.groupby
 |      pandas.Panel.groupby
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from _GroupBy:
 |  
 |  __getattr__(self, attr)
 |  
 |  __init__(self, obj, keys=None, axis=0, level=None, grouper=None, exclusions=None, selection=None, as_index=True, sort=True, group_keys=True, squeeze=False, **kwargs)
 |      Initialize self.  See help(type(self)) for accurate signature.
 |  
 |  __iter__(self)
 |      Groupby iterator
 |      
 |      Returns
 |      -------
 |      Generator yielding sequence of (name, subsetted object)
 |      for each group
 |  
 |  __len__(self)
 |  
 |  __unicode__(self)
 |      Return a string representation for a particular object.
 |      
 |      Invoked by unicode(obj) in py2 only. Yields a Unicode String in both
 |      py2/py3.
 |  
 |  apply(self, func, *args, **kwargs)
 |      Apply function and combine results together in an intelligent way. The
 |      split-apply-combine combination rules attempt to be as common sense
 |      based as possible. For example:
 |      
 |      case 1:
 |      group DataFrame
 |      apply aggregation function (f(chunk) -> Series)
 |      yield DataFrame, with group axis having group labels
 |      
 |      case 2:
 |      group DataFrame
 |      apply transform function ((f(chunk) -> DataFrame with same indexes)
 |      yield DataFrame with resulting chunks glued together
 |      
 |      case 3:
 |      group Series
 |      apply function with f(chunk) -> DataFrame
 |      yield DataFrame with result of chunks glued together
 |      
 |      Parameters
 |      ----------
 |      func : function
 |      
 |      Notes
 |      -----
 |      See online documentation for full exposition on how to use apply.
 |      
 |      In the current implementation apply calls func twice on the
 |      first group to decide whether it can take a fast or slow code
 |      path. This can lead to unexpected behavior if func has
 |      side-effects, as they will take effect twice for the first
 |      group.
 |      
 |      
 |      See also
 |      --------
 |      aggregate, transform
 |  
 |  get_group(self, name, obj=None)
 |      Constructs NDFrame from group with provided name
 |      
 |      Parameters
 |      ----------
 |      name : object
 |          the name of the group to get as a DataFrame
 |      obj : NDFrame, default None
 |          the NDFrame to take the DataFrame out of.  If
 |          it is None, the object groupby was called on will
 |          be used
 |      
 |      Returns
 |      -------
 |      group : type of obj
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from _GroupBy:
 |  
 |  groups
 |      dict {group name -> group labels}
 |  
 |  indices
 |      dict {group name -> group indices}
 |  
 |  ngroups
 |  
 |  plot
 |      Class implementing the .plot attribute for groupby objects
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from pandas.core.base.PandasObject:
 |  
 |  __dir__(self)
 |      Provide method name lookup and completion
 |      Only provide 'public' methods
 |  
 |  __sizeof__(self)
 |      Generates the total memory usage for a object that returns
 |      either a value or Series of values
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from pandas.core.base.StringMixin:
 |  
 |  __bytes__(self)
 |      Return a string representation for a particular object.
 |      
 |      Invoked by bytes(obj) in py3 only.
 |      Yields a bytestring in both py2/py3.
 |  
 |  __repr__(self)
 |      Return a string representation for a particular object.
 |      
 |      Yields Bytestring in Py2, Unicode String in py3.
 |  
 |  __str__(self)
 |      Return a string representation for a particular Object
 |      
 |      Invoked by str(df) in both py2/py3.
 |      Yields Bytestring in Py2, Unicode String in py3.
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from pandas.core.base.StringMixin:
 |  
 |  __dict__
 |      dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |      list of weak references to the object (if defined)
 |  
 |  ----------------------------------------------------------------------
 |  Methods inherited from pandas.core.base.SelectionMixin:
 |  
 |  __getitem__(self, key)
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from pandas.core.base.SelectionMixin:
 |  
 |  ndim

grouped.mean()

key1
a   -0.198323
b   -0.613947
Name: data1, dtype: float64

df['data1'].groupby(['a','a','b','b','a']).mean()

a   -0.198323
b   -0.613947
Name: data1, dtype: float64

p=df['data1'];p

0   -0.288087
1    0.804061
2   -1.672009
3    0.444115
4   -1.110942
Name: data1, dtype: float64

p.groupby(df['key1']).mean()

key1
a   -0.198323
b   -0.613947
Name: data1, dtype: float64

The important thing above is that the data(Series) has been aggregated according to the group key,producing a new Series that is now indexed by the unique values in the key1 column.

The result index has the name 'key1' because the DataFrame column df['key1'] did!

If instead we had passed multiple arrays as a list,we'd get something different:

means=df['data1'].groupby([df['key1'],df['key2']]).mean();means

key1  key2
a     one    -0.699514
      two     0.804061
b     one    -1.672009
      two     0.444115
Name: data1, dtype: float64

df['data1'].groupby([df['key2'],df['key1']]).mean()

key2  key1
one   a      -0.699514
      b      -1.672009
two   a       0.804061
      b       0.444115
Name: data1, dtype: float64

Here we grouped the data using two keys, and the resulting Series now has a hierarchical index consisting of the unique pairs of keys observed:

means.unstack()

In this example,the group keys are all Series,though they could be any arrays of the right length:

states=np.array(['Ohio','Califonia','Ohio','Ohio','Califonia'])

years=np.array([2005,2005,2006,2005,2006])

df['data1'].groupby([states,years]).mean()

Califonia  2005    0.804061
           2006   -1.110942
Ohio       2005    0.078014
           2006   -1.672009
Name: data1, dtype: float64

• The second method:
  Frequently the grouping information is found in the same DataFrame as the data you want to work on.In this case,you can pass column names(whether those are strings,numbers or other python objects) as the group keys:

df.groupby('key1').mean()

df.groupby(['key1','key2']).mean()

You may have noticed in the first case of df.groupby('key').mean() that there is no key2 column in the result.**Because df['key2'] is not numeric data,it is said to be a nuisance column,which is therefor excluded from the result.By default,all of teh numeric columns are aggregated,though it is possible to filter down to a subset.

help(df.groupby(['key1','key2']).size)

Help on method size in module pandas.core.groupby:

size() method of pandas.core.groupby.DataFrameGroupBy instance
    Compute group sizes
    
    See also
    --------
    pandas.Series.groupby
    pandas.DataFrame.groupby
    pandas.Panel.groupby

A generally useful Groupby method is size,which returns a Series containing group sizes:

df.groupby(['key1','key2']).size()

key1  key2
a     one     2
      two     1
b     one     1
      two     1
dtype: int64

df

We can see from above,the number of ('a','one') is 2 indeed...

Iterating over groups

The Groupby object supports iteration,generating a sequence of 2-tuples containning the group name along with the chunk of data.

for name,group in df.groupby('key1'):
    print(name)
    print(group)

a
      data1     data2 key1 key2
0 -0.288087 -1.355619    a  one
1  0.804061  0.654039    a  two
4 -1.110942  0.581883    a  one
b
      data1     data2 key1 key2
2 -1.672009  1.780028    b  one
3  0.444115  0.988676    b  two

In the case of multiple keys,the first element in the tuple will be a tuple of key values.

for (k1,k2),group in df.groupby(['key1','key2']):
    print((k1,k2))
    print(group)

('a', 'one')
      data1     data2 key1 key2
0 -0.288087 -1.355619    a  one
4 -1.110942  0.581883    a  one
('a', 'two')
      data1     data2 key1 key2
1  0.804061  0.654039    a  two
('b', 'one')
      data1     data2 key1 key2
2 -1.672009  1.780028    b  one
('b', 'two')
      data1     data2 key1 key2
3  0.444115  0.988676    b  two

df

A useful recipe to get the pieces of data through computing a dict of the data pieces as a one-liner:

pieces=dict(list(df.groupby('key1')))

pieces

{'a':       data1     data2 key1 key2
 0 -0.288087 -1.355619    a  one
 1  0.804061  0.654039    a  two
 4 -1.110942  0.581883    a  one, 'b':       data1     data2 key1 key2
 2 -1.672009  1.780028    b  one
 3  0.444115  0.988676    b  two}

pieces['b']

list(df.groupby('key1'))

[('a',       data1     data2 key1 key2
  0 -0.288087 -1.355619    a  one
  1  0.804061  0.654039    a  two
  4 -1.110942  0.581883    a  one), ('b',       data1     data2 key1 key2
  2 -1.672009  1.780028    b  one
  3  0.444115  0.988676    b  two)]

dict([('a',1),('b',2)])

{'a': 1, 'b': 2}

dict([('a',1)])

{'a': 1}

dict([(1,2)])

{1: 2}

Above: dict's built-in creation.

df

df.dtypes

data1    float64
data2    float64
key1      object
key2      object
dtype: object

grouped=df.groupby(df.dtypes,axis=1)  # notice here,axis=1,df is divided into 2 DataFrame accroding `dtype`.

for dtype,group in grouped:
    print(dtype)
    print(group)

float64
      data1     data2
0 -0.288087 -1.355619
1  0.804061  0.654039
2 -1.672009  1.780028
3  0.444115  0.988676
4 -1.110942  0.581883
object
  key1 key2
0    a  one
1    a  two
2    b  one
3    b  two
4    a  one

pp=dict(list(grouped))

pp

{dtype('float64'):       data1     data2
 0 -0.288087 -1.355619
 1  0.804061  0.654039
 2 -1.672009  1.780028
 3  0.444115  0.988676
 4 -1.110942  0.581883, dtype('O'):   key1 key2
 0    a  one
 1    a  two
 2    b  one
 3    b  two
 4    a  one}

i,j=pp.keys()

pp[i]

pp[j]

type(i)

numpy.dtype

pp[np.dtype('object')]

Selecting a column or subset of columns

a=list(df.groupby('key1')['data1']);a

[('a', 0   -0.288087
  1    0.804061
  4   -1.110942
  Name: data1, dtype: float64), ('b', 2   -1.672009
  3    0.444115
  Name: data1, dtype: float64)]

list(df['data1'].groupby(df['key1']))

[('a', 0   -0.288087
  1    0.804061
  4   -1.110942
  Name: data1, dtype: float64), ('b', 2   -1.672009
  3    0.444115
  Name: data1, dtype: float64)]

df

df.groupby('key1').size()

key1
a    3
b    2
dtype: int64

a=df.groupby('key1')['data1'];a

<pandas.core.groupby.SeriesGroupBy object at 0x000001DDB5670F60>

b=df.groupby('key1')[['data1']];a

<pandas.core.groupby.SeriesGroupBy object at 0x000001DDB5670F60>

aa=df['data1'].groupby(df['key1']);aa

<pandas.core.groupby.SeriesGroupBy object at 0x000001DDB56707B8>

bb=df[['data1']].groupby(df['key1']);bb

<pandas.core.groupby.DataFrameGroupBy object at 0x000001DDB56703C8>

Indexing a groupby object created from a DataFrame with a column name or array of column names has the efffect of column subsetting for aggregation.

df

list(df.groupby(['key1','key2'])['data2'])

[(('a', 'one'), 0   -1.355619
  4    0.581883
  Name: data2, dtype: float64), (('a', 'two'), 1    0.654039
  Name: data2, dtype: float64), (('b', 'one'), 2    1.780028
  Name: data2, dtype: float64), (('b', 'two'), 3    0.988676
  Name: data2, dtype: float64)]

df.groupby(['key1','key2'])[['data2']].mean()

df[['data1','data2']].mean()

data1   -0.364572
data2    0.529801
dtype: float64

Grouping with Dicts and Series

Grouping information may exist in a form other than an array.

people=pd.DataFrame(np.random.randn(5,5),columns=list('abcde'),index=['Joe','Steve','Wes','Jim','Travis']);people

Now suppose we have a group correspondence for the columns and want to sum together the columns by group.

mapping={'a':'red','b':'red',
        'c':'blue','d':'blue',
        'e':'red','f':'orange'}

by_column=people.groupby(mapping,axis=1)

by_column.sum()

The same functionality holds for Series,which can be viewed as a fixed-size mapping.

map_series=pd.Series(mapping);map_series

a       red
b       red
c      blue
d      blue
e       red
f    orange
dtype: object

people.groupby(map_series,axis=1).count()

people.groupby(['red','red','blue','blue','red']).mean()

people.groupby(['red','red','blue','blue','red'],axis=1).mean()

Grouping with function

Using python functions is a more generic way of defining a group mappingg compared with a dict or Series.Any function passed as a group key will be called once per index value,with the return values being used as the group names.

Suppose you want to group by the length of the names;while you compute an array of string lengths,it's simpler to just pass the len function.

people.groupby(len).sum()

Mixing functions with arrays,dicts,or Series is not a problem as everything gets converted to arrays internaly:

key_list=['one','one','one','two','two']

key_test=[1,1,1,1,1]

people.groupby([len,key_list]).min()

people.groupby([len,key_test]).min()

Grouping by Index levels

A final vonvenience for hierarchically indexed datasets is the ability to aggregate using one of the levels of an index.

columns=pd.MultiIndex.from_arrays([['US','US','US','JP','JP'],[1,3,5,1,3,]],names=['City','tenor'])

hier_df=pd.DataFrame(np.random.randn(4,5),columns=columns);hier_df

To group by level,pass the level number or name using level keyword.

hier_df.groupby(level='City',axis=1).count()

Data aggregation

Aggregations refer to any data transformation that produces scalar values from arrays.

description=['Number of non-NAN values in the group',
            'Sum of non-NA values',
            'Mean of non-NA values',
            'Arithmetic median off non-NAN values',
            'Unbiased(n-1 denominator)standard deviation and variance',
            'Minimun and maximum of non-NA values',
            'Product of non-NA values',
            'First and last non-NA value']

Tips:Custom aggregation functions are generally much slower than the optimized functions found in above table.This is because there is some extra overhead(function calls,data rearrangement) in constructing the intermeediate chunks.

index=pd.Index(['count','sum','mean','median','std,var','min,max','prod','first,last'],names='Function name')

pd.DataFrame(description,index=index)

To use your own aggregation functions,pass any function that aggregates an array to the aggregate or agg method.

grouped=df.groupby('key1')

def peak_to_peak(arr):
    return arr.max()-arr.min()

grouped.agg(peak_to_peak)

df

def length(arr):
    l=0
    for i in arr:
        l+=i**2
    return l**0.5
        

grouped.agg(length)

grouped.describe()

You may notice some methods like describe also work,even though they are not aggregations.

Column-wise and multiple function application

tips=pd.read_csv(r'./pydata-book-2nd-edition/examples/tips.csv')

tips.head()

tips['tip_pct']=tips['tip']/tips['total_bill']

As we have seen,aggregating a Series or all of the columns of a DataFrame is a matter of using aggregae with the desired function or calling a method like mean or std.However,you may want to aggregate using a different function depending on the column,or multiple functions at once.

grouped=tips.groupby(['day','smoker'])

group_pct=grouped['tip_pct']

grouped.size()

day   smoker
Fri   No         4
      Yes       15
Sat   No        45
      Yes       42
Sun   No        57
      Yes       19
Thur  No        45
      Yes       17
dtype: int64

group_pct.agg('mean')

day   smoker
Fri   No        0.151650
      Yes       0.174783
Sat   No        0.158048
      Yes       0.147906
Sun   No        0.160113
      Yes       0.187250
Thur  No        0.160298
      Yes       0.163863
Name: tip_pct, dtype: float64

If you pass a list of functions or function names instead,you get back a DataFrame with column names taken from the functions:

group_pct.agg(['mean','std',peak_to_peak])

df=pd.DataFrame(np.arange(12).reshape(3,4),index=['one','two','three'],columns=['o','t','th','f']);df

list(df.groupby([0,0,1])['f'])

[(0, one    3
  two    7
  Name: f, dtype: int32), (1, three    11
  Name: f, dtype: int32)]

If you pass a list of (name,function) tuples,the first element of each tuple will be used as the DataFrame column names.

group_pct.agg([('foo','mean'),('bar',np.std)])

With a DataFrame you have more options,as you can specify a list of functions to apply to all of the columns or different functions per column.To start ,suppose we wanted to compute the same three statistics for the tip_pct and total_bill columns:

functions=['count','mean','max']

result=grouped['tip_pct','total_bill'].agg(functions);result

As you can see,the resulting DataFrame has a hierarchical columns,the same as you would get aggregating each column separately and using concat to glue the results together using the column names as the keys argument

As before,a list of tuples with custom names can be passed:

ftuples=[('Durchs','mean'),('adkfd',np.var)]

grouped['tip_pct','total_bill'].agg(ftuples)

Now,suppose you want to apply potentially different functions to one or more of the columns.To do this,pass a dict to agg that contains a mappingg of column names to any of the function specifications listed so far:

grouped.agg({'tip':np.max,'size':'sum'})

grouped.agg(ftuples)

grouped.agg({'tip_pct':['min','max','mean','std'],'size':'sum'})

A DataFrame will have hierarchical columns only if multiple functions are applied to at least one column.

Returning Aggregated data without row index

In all of the examples up until now,the aggregated data comes back with an index,potentially hierarchical,composed from the unique group key combinations.Since this is not always desirable,you can disable this behaviour in most cases by passing as_index=False to groupby.

tips.groupby(['day','smoker'],as_index=False).mean()

tips.groupby(['day','smoker']).mean()

Apply:General split-apply-combine

The most general-purpose GroupBy method is apply,which is the subject of the rest of this section.

tips.head()

Returning to the tipping dataset from before,suppose you want to select the top five tip_pct values by group.First,write a function that selects the rows with the largest values in a particular column:

def top(df,n=5,column='tip_pct'):
    return df.sort_values(by=column)[-n:]

tips.sort_values(by='tip_pct')[-5:]

top(tips,n=6)

Now,if we group by smoker,and call apply with this function.we get:

tips.groupby('smoker').apply(top)

If you pass a function to apply that takes other arguments or keywords,you can pass these after the function:

tips.groupby(['smoker','day']).apply(top,n=2,column='total_bill')

Suppressing the Group keys

In the preceding examples,you see that the resulting object has a hierarchical index formed from the group keys along the indexes of each piece of the original object.You can disable this by passing group_keys=False to groupby.

tips.groupby('smoker',group_keys=False).apply(top)

**Notice the difference between agg and apply:agg aggregates GroupBy object's every column into one value,while apply apply function to every column of GroupBy object and the result may not to be just one value like agg.

tips.groupby('smoker').agg('max')

tips.groupby('smoker').apply(max)

The above two methods return just the same result.To some extent,aggg is a specific version of apply.

Quantile and bucket analysis

Combining qcut or cut with groupby makes it convenient to perform bucket or quantile analysis on a dataset.

frame=pd.DataFrame({'data1':np.random.randn(1000),
                   'data2':np.random.randn(1000)})

quantiles=pd.cut(frame.data1,4)

quantiles[:10]

0    (0.00174, 1.893]
1    (0.00174, 1.893]
2    (0.00174, 1.893]
3     (-3.788, -1.89]
4    (0.00174, 1.893]
5    (0.00174, 1.893]
6    (0.00174, 1.893]
7     (-3.788, -1.89]
8    (0.00174, 1.893]
9    (0.00174, 1.893]
Name: data1, dtype: category
Categories (4, object): [(-3.788, -1.89] < (-1.89, 0.00174] < (0.00174, 1.893] < (1.893, 3.784]]

help(pd.cut)

Help on function cut in module pandas.tools.tile:

cut(x, bins, right=True, labels=None, retbins=False, precision=3, include_lowest=False)
    Return indices of half-open bins to which each value of `x` belongs.
    
    Parameters
    ----------
    x : array-like
        Input array to be binned. It has to be 1-dimensional.
    bins : int or sequence of scalars
        If `bins` is an int, it defines the number of equal-width bins in the
        range of `x`. However, in this case, the range of `x` is extended
        by .1% on each side to include the min or max values of `x`. If
        `bins` is a sequence it defines the bin edges allowing for
        non-uniform bin width. No extension of the range of `x` is done in
        this case.
    right : bool, optional
        Indicates whether the bins include the rightmost edge or not. If
        right == True (the default), then the bins [1,2,3,4] indicate
        (1,2], (2,3], (3,4].
    labels : array or boolean, default None
        Used as labels for the resulting bins. Must be of the same length as
        the resulting bins. If False, return only integer indicators of the
        bins.
    retbins : bool, optional
        Whether to return the bins or not. Can be useful if bins is given
        as a scalar.
    precision : int
        The precision at which to store and display the bins labels
    include_lowest : bool
        Whether the first interval should be left-inclusive or not.
    
    Returns
    -------
    out : Categorical or Series or array of integers if labels is False
        The return type (Categorical or Series) depends on the input: a Series
        of type category if input is a Series else Categorical. Bins are
        represented as categories when categorical data is returned.
    bins : ndarray of floats
        Returned only if `retbins` is True.
    
    Notes
    -----
    The `cut` function can be useful for going from a continuous variable to
    a categorical variable. For example, `cut` could convert ages to groups
    of age ranges.
    
    Any NA values will be NA in the result.  Out of bounds values will be NA in
    the resulting Categorical object
    
    
    Examples
    --------
    >>> pd.cut(np.array([.2, 1.4, 2.5, 6.2, 9.7, 2.1]), 3, retbins=True)
    ([(0.191, 3.367], (0.191, 3.367], (0.191, 3.367], (3.367, 6.533],
      (6.533, 9.7], (0.191, 3.367]]
    Categories (3, object): [(0.191, 3.367] < (3.367, 6.533] < (6.533, 9.7]],
    array([ 0.1905    ,  3.36666667,  6.53333333,  9.7       ]))
    >>> pd.cut(np.array([.2, 1.4, 2.5, 6.2, 9.7, 2.1]), 3,
               labels=["good","medium","bad"])
    [good, good, good, medium, bad, good]
    Categories (3, object): [good < medium < bad]
    >>> pd.cut(np.ones(5), 4, labels=False)
    array([1, 1, 1, 1, 1], dtype=int64)

help(pd.qcut)

Help on function qcut in module pandas.tools.tile:

qcut(x, q, labels=None, retbins=False, precision=3)
    Quantile-based discretization function. Discretize variable into
    equal-sized buckets based on rank or based on sample quantiles. For example
    1000 values for 10 quantiles would produce a Categorical object indicating
    quantile membership for each data point.
    
    Parameters
    ----------
    x : ndarray or Series
    q : integer or array of quantiles
        Number of quantiles. 10 for deciles, 4 for quartiles, etc. Alternately
        array of quantiles, e.g. [0, .25, .5, .75, 1.] for quartiles
    labels : array or boolean, default None
        Used as labels for the resulting bins. Must be of the same length as
        the resulting bins. If False, return only integer indicators of the
        bins.
    retbins : bool, optional
        Whether to return the bins or not. Can be useful if bins is given
        as a scalar.
    precision : int
        The precision at which to store and display the bins labels
    
    Returns
    -------
    out : Categorical or Series or array of integers if labels is False
        The return type (Categorical or Series) depends on the input: a Series
        of type category if input is a Series else Categorical. Bins are
        represented as categories when categorical data is returned.
    bins : ndarray of floats
        Returned only if `retbins` is True.
    
    Notes
    -----
    Out of bounds values will be NA in the resulting Categorical object
    
    Examples
    --------
    >>> pd.qcut(range(5), 4)
    [[0, 1], [0, 1], (1, 2], (2, 3], (3, 4]]
    Categories (4, object): [[0, 1] < (1, 2] < (2, 3] < (3, 4]]
    >>> pd.qcut(range(5), 3, labels=["good","medium","bad"])
    [good, good, medium, bad, bad]
    Categories (3, object): [good < medium < bad]
    >>> pd.qcut(range(5), 4, labels=False)
    array([0, 0, 1, 2, 3], dtype=int64)

The Categorical object returned by cut can be passed directly to groupby.So we could compute a set of statistics for the data2 like so:

def get_status(group):
    return {'min':group.min(),'max':group.max(),
           'count':group.count(),'mean':group.mean()}

grouped=frame.data2.groupby(quantiles)

all(frame.data2==frame['data2'])

True

df=pd.Series([1,2,3,5])

pd.cut(df,2)

0    (0.996, 3]
1    (0.996, 3]
2    (0.996, 3]
3        (3, 5]
dtype: category
Categories (2, object): [(0.996, 3] < (3, 5]]

pd.qcut(df,2)

0    [1, 2.5]
1    [1, 2.5]
2    (2.5, 5]
3    (2.5, 5]
dtype: category
Categories (2, object): [[1, 2.5] < (2.5, 5]]

fig=plt.figure()

df.plot()

<matplotlib.axes._subplots.AxesSubplot at 0x1ddb56f2908>

grouped=frame.data2.groupby(quantiles)

grouped.size()

data1
(-3.788, -1.89]      34
(-1.89, 0.00174]    443
(0.00174, 1.893]    492
(1.893, 3.784]       31
dtype: int64

Here,pd.cut(frame.data1,4) returns 1000 rows of Categorical object dividing data1 into 4 groups

grouped.apply(get_status)

data1                  
(-3.788, -1.89]   count     34.000000
                  max        1.935178
                  mean      -0.147920
                  min       -1.843208
(-1.89, 0.00174]  count    443.000000
                  max        2.547017
                  mean      -0.020845
                  min       -2.779875
(0.00174, 1.893]  count    492.000000
                  max        2.729491
                  mean      -0.066996
                  min       -3.854003
(1.893, 3.784]    count     31.000000
                  max        1.533001
                  mean      -0.070277
                  min       -1.639600
Name: data2, dtype: float64

grouped.apply(get_status).unstack()

There were equal-length buckets;to compute equal-size buckets based on sample quantiles,use qcut.We will pass labels=False to jsut get quantile numbers:

grouping=pd.qcut(frame.data1,10,labels=False)

grouped=frame.data2.groupby(grouping)

grouped.size()

data1
0    100
1    100
2    100
3    100
4    100
5    100
6    100
7    100
8    100
9    100
dtype: int64

grouped.apply(get_status).unstack()

Example:Filling missing values with group-specific values

When cleaning up missing data,in some cases you will replace data observations using dropna,but in others you may want to impute(fill in) the null(NA) values using a fixed value or some value derived from the data.fillna is the right tool to use:

s=pd.Series(np.random.randn(6))

s[::2]=np.nan

s

0         NaN
1   -0.875438
2         NaN
3    1.106316
4         NaN
5    0.020065
dtype: float64

s.fillna(s.mean())

0    0.083648
1   -0.875438
2    0.083648
3    1.106316
4    0.083648
5    0.020065
dtype: float64

Suppose you need the fill value to vary by group.One way to do this is to group the data and use apply with a function that calls fillna on each data chunk.

states=['Ohio','New york','Vermont','Florida','Oregon','Nevada','California','Idaho']

group_key=['East']*4+['West']*4

data=pd.Series(np.random.randn(8),index=states)

data[['Vermont','Nevada','Idaho']]=np.nan

data.groupby(group_key).mean()

East    0.199033
West   -0.704578
dtype: float64

We can fill the NA values using the group means like so:

fill_mean=lambda g : g.fillna(g.mean())

data.groupby(group_key).apply(fill_mean)

Ohio         -0.953675
New york      0.531946
Vermont       0.199033
Florida       1.018829
Oregon       -0.539930
Nevada       -0.704578
California   -0.869226
Idaho        -0.704578
dtype: float64

In another case,you might have predefined fill values in your code that vary by group.

fill_values={'East':0.5,'West':-1}

fill_func=lambda g:g.fillna(fill_values[g.name])

data.groupby(group_key).apply(fill_func)

Ohio         -0.953675
New york      0.531946
Vermont       0.500000
Florida       1.018829
Oregon       -0.539930
Nevada       -1.000000
California   -0.869226
Idaho        -1.000000
dtype: float64

Example: Random sampling and permutation

Suppose you wanted to draw a random sample(with or without replacement) from a large dataset for Monte Carlo simulation purpose or some other application.There are a number of ways to perform the 'draws':Here we use sample method for Series.

suits=list('HSCD')

card_val=(list(range(1,11))+[10]*3)*4

base_names=['A']+list(range(2,11))+['J','K','Q'];base_names

['A', 2, 3, 4, 5, 6, 7, 8, 9, 10, 'J', 'K', 'Q']

cards=[]

for suit in ['H','S','C','D']:
    cards.extend(str(num)+suit for num in base_names)

deck=pd.Series(card_val,index=cards)

deck

AH      1
2H      2
3H      3
4H      4
5H      5
6H      6
7H      7
8H      8
9H      9
10H    10
JH     10
KH     10
QH     10
AS      1
2S      2
3S      3
4S      4
5S      5
6S      6
7S      7
8S      8
9S      9
10S    10
JS     10
KS     10
QS     10
AC      1
2C      2
3C      3
4C      4
5C      5
6C      6
7C      7
8C      8
9C      9
10C    10
JC     10
KC     10
QC     10
AD      1
2D      2
3D      3
4D      4
5D      5
6D      6
7D      7
8D      8
9D      9
10D    10
JD     10
KD     10
QD     10
dtype: int64

def draw(deck,n=5):
    return deck.sample(n)

draw(deck)

4H     4
AD     1
8S     8
3C     3
KH    10
dtype: int64

Suppose you want two random cards from each suit.Because the suit is the last character of each card name,we can group based on this and use apply.

get_suit=lambda card:card[-1]

deck.groupby(get_suit).apply(draw,n=2)

C  KC     10
   8C      8
D  5D      5
   8D      8
H  QH     10
   5H      5
S  4S      4
   10S    10
dtype: int64

Alternatively,we could write:

deck.groupby(get_suit,group_keys=False).apply(draw,2)

8C      8
KC     10
8D      8
9D      9
8H      8
QH     10
10S    10
4S      4
dtype: int64

Example:Group weighted average and correlation

Under the split-apply-combine paradigm of groupby operations between columns in a DataFrame or two Series,such as a group weighted average,are possible.

df=pd.DataFrame({'category':['a','a','a','a','b','b','b','b'],
                'data':np.random.randn(8),'weights':np.random.randn(8)});df

The group weighted average by category would then be:

grouped=df.groupby('category')

get_wavg=lambda g:np.average(g['data'],weights=g['weights'])

list(grouped)

[('a',   category      data   weights
  0        a  0.902349  1.730949
  1        a  1.287855  0.768677
  2        a -0.751881 -0.571836
  3        a  1.208645  0.229177), ('b',   category      data   weights
  4        b  0.986127 -0.262788
  5        b  1.566069 -0.252579
  6        b  0.285959  0.184662
  7        b  1.059962  1.804769)]

grouped.apply(get_wavg)

category
a    1.510830
b    0.889441
dtype: float64

close_px=pd.read_csv('./pydata-book-2nd-edition/examples/stock_px_2.csv',parse_dates=True,index_col=0)

close_px.head()

help(pd.read_csv)

Help on function read_csv in module pandas.io.parsers:

read_csv(filepath_or_buffer, sep=',', delimiter=None, header='infer', names=None, index_col=None, usecols=None, squeeze=False, prefix=None, mangle_dupe_cols=True, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skipinitialspace=False, skiprows=None, nrows=None, na_values=None, keep_default_na=True, na_filter=True, verbose=False, skip_blank_lines=True, parse_dates=False, infer_datetime_format=False, keep_date_col=False, date_parser=None, dayfirst=False, iterator=False, chunksize=None, compression='infer', thousands=None, decimal=b'.', lineterminator=None, quotechar='"', quoting=0, escapechar=None, comment=None, encoding=None, dialect=None, tupleize_cols=False, error_bad_lines=True, warn_bad_lines=True, skipfooter=0, skip_footer=0, doublequote=True, delim_whitespace=False, as_recarray=False, compact_ints=False, use_unsigned=False, low_memory=True, buffer_lines=None, memory_map=False, float_precision=None)
    Read CSV (comma-separated) file into DataFrame
    
    Also supports optionally iterating or breaking of the file
    into chunks.
    
    Additional help can be found in the `online docs for IO Tools
    <http://pandas.pydata.org/pandas-docs/stable/io.html>`_.
    
    Parameters
    ----------
    filepath_or_buffer : str, pathlib.Path, py._path.local.LocalPath or any object with a read() method (such as a file handle or StringIO)
        The string could be a URL. Valid URL schemes include http, ftp, s3, and
        file. For file URLs, a host is expected. For instance, a local file could
        be file ://localhost/path/to/table.csv
    sep : str, default ','
        Delimiter to use. If sep is None, will try to automatically determine
        this. Separators longer than 1 character and different from ``'\s+'`` will
        be interpreted as regular expressions, will force use of the python parsing
        engine and will ignore quotes in the data. Regex example: ``'\r\t'``
    delimiter : str, default ``None``
        Alternative argument name for sep.
    delim_whitespace : boolean, default False
        Specifies whether or not whitespace (e.g. ``' '`` or ``'    '``) will be
        used as the sep. Equivalent to setting ``sep='\s+'``. If this option
        is set to True, nothing should be passed in for the ``delimiter``
        parameter.
    
        .. versionadded:: 0.18.1 support for the Python parser.
    
    header : int or list of ints, default 'infer'
        Row number(s) to use as the column names, and the start of the data.
        Default behavior is as if set to 0 if no ``names`` passed, otherwise
        ``None``. Explicitly pass ``header=0`` to be able to replace existing
        names. The header can be a list of integers that specify row locations for
        a multi-index on the columns e.g. [0,1,3]. Intervening rows that are not
        specified will be skipped (e.g. 2 in this example is skipped). Note that
        this parameter ignores commented lines and empty lines if
        ``skip_blank_lines=True``, so header=0 denotes the first line of data
        rather than the first line of the file.
    names : array-like, default None
        List of column names to use. If file contains no header row, then you
        should explicitly pass header=None. Duplicates in this list are not
        allowed unless mangle_dupe_cols=True, which is the default.
    index_col : int or sequence or False, default None
        Column to use as the row labels of the DataFrame. If a sequence is given, a
        MultiIndex is used. If you have a malformed file with delimiters at the end
        of each line, you might consider index_col=False to force pandas to _not_
        use the first column as the index (row names)
    usecols : array-like, default None
        Return a subset of the columns. All elements in this array must either
        be positional (i.e. integer indices into the document columns) or strings
        that correspond to column names provided either by the user in `names` or
        inferred from the document header row(s). For example, a valid `usecols`
        parameter would be [0, 1, 2] or ['foo', 'bar', 'baz']. Using this parameter
        results in much faster parsing time and lower memory usage.
    as_recarray : boolean, default False
        DEPRECATED: this argument will be removed in a future version. Please call
        `pd.read_csv(...).to_records()` instead.
    
        Return a NumPy recarray instead of a DataFrame after parsing the data.
        If set to True, this option takes precedence over the `squeeze` parameter.
        In addition, as row indices are not available in such a format, the
        `index_col` parameter will be ignored.
    squeeze : boolean, default False
        If the parsed data only contains one column then return a Series
    prefix : str, default None
        Prefix to add to column numbers when no header, e.g. 'X' for X0, X1, ...
    mangle_dupe_cols : boolean, default True
        Duplicate columns will be specified as 'X.0'...'X.N', rather than
        'X'...'X'. Passing in False will cause data to be overwritten if there
        are duplicate names in the columns.
    dtype : Type name or dict of column -> type, default None
        Data type for data or columns. E.g. {'a': np.float64, 'b': np.int32}
        (Unsupported with engine='python'). Use `str` or `object` to preserve and
        not interpret dtype.
    engine : {'c', 'python'}, optional
        Parser engine to use. The C engine is faster while the python engine is
        currently more feature-complete.
    converters : dict, default None
        Dict of functions for converting values in certain columns. Keys can either
        be integers or column labels
    true_values : list, default None
        Values to consider as True
    false_values : list, default None
        Values to consider as False
    skipinitialspace : boolean, default False
        Skip spaces after delimiter.
    skiprows : list-like or integer, default None
        Line numbers to skip (0-indexed) or number of lines to skip (int)
        at the start of the file
    skipfooter : int, default 0
        Number of lines at bottom of file to skip (Unsupported with engine='c')
    skip_footer : int, default 0
        DEPRECATED: use the `skipfooter` parameter instead, as they are identical
    nrows : int, default None
        Number of rows of file to read. Useful for reading pieces of large files
    na_values : scalar, str, list-like, or dict, default None
        Additional strings to recognize as NA/NaN. If dict passed, specific
        per-column NA values.  By default the following values are interpreted as
        NaN: '', '#N/A', '#N/A N/A', '#NA', '-1.#IND', '-1.#QNAN', '-NaN', '-nan',
    '1.#IND', '1.#QNAN', 'N/A', 'NA', 'NULL', 'NaN', 'nan'`.
    keep_default_na : bool, default True
        If na_values are specified and keep_default_na is False the default NaN
        values are overridden, otherwise they're appended to.
    na_filter : boolean, default True
        Detect missing value markers (empty strings and the value of na_values). In
        data without any NAs, passing na_filter=False can improve the performance
        of reading a large file
    verbose : boolean, default False
        Indicate number of NA values placed in non-numeric columns
    skip_blank_lines : boolean, default True
        If True, skip over blank lines rather than interpreting as NaN values
    parse_dates : boolean or list of ints or names or list of lists or dict, default False
    
        * boolean. If True -> try parsing the index.
        * list of ints or names. e.g. If [1, 2, 3] -> try parsing columns 1, 2, 3
          each as a separate date column.
        * list of lists. e.g.  If [[1, 3]] -> combine columns 1 and 3 and parse as
            a single date column.
        * dict, e.g. {'foo' : [1, 3]} -> parse columns 1, 3 as date and call result
          'foo'
    
        Note: A fast-path exists for iso8601-formatted dates.
    infer_datetime_format : boolean, default False
        If True and parse_dates is enabled, pandas will attempt to infer the format
        of the datetime strings in the columns, and if it can be inferred, switch
        to a faster method of parsing them. In some cases this can increase the
        parsing speed by ~5-10x.
    keep_date_col : boolean, default False
        If True and parse_dates specifies combining multiple columns then
        keep the original columns.
    date_parser : function, default None
        Function to use for converting a sequence of string columns to an array of
        datetime instances. The default uses ``dateutil.parser.parser`` to do the
        conversion. Pandas will try to call date_parser in three different ways,
        advancing to the next if an exception occurs: 1) Pass one or more arrays
        (as defined by parse_dates) as arguments; 2) concatenate (row-wise) the
        string values from the columns defined by parse_dates into a single array
        and pass that; and 3) call date_parser once for each row using one or more
        strings (corresponding to the columns defined by parse_dates) as arguments.
    dayfirst : boolean, default False
        DD/MM format dates, international and European format
    iterator : boolean, default False
        Return TextFileReader object for iteration or getting chunks with
        ``get_chunk()``.
    chunksize : int, default None
        Return TextFileReader object for iteration. `See IO Tools docs for more
        information
        <http://pandas.pydata.org/pandas-docs/stable/io.html#io-chunking>`_ on
        ``iterator`` and ``chunksize``.
    compression : {'infer', 'gzip', 'bz2', 'zip', 'xz', None}, default 'infer'
        For on-the-fly decompression of on-disk data. If 'infer', then use gzip,
        bz2, zip or xz if filepath_or_buffer is a string ending in '.gz', '.bz2',
        '.zip', or 'xz', respectively, and no decompression otherwise. If using
        'zip', the ZIP file must contain only one data file to be read in.
        Set to None for no decompression.
    
        .. versionadded:: 0.18.1 support for 'zip' and 'xz' compression.
    
    thousands : str, default None
        Thousands separator
    decimal : str, default '.'
        Character to recognize as decimal point (e.g. use ',' for European data).
    float_precision : string, default None
        Specifies which converter the C engine should use for floating-point
        values. The options are `None` for the ordinary converter,
        `high` for the high-precision converter, and `round_trip` for the
        round-trip converter.
    lineterminator : str (length 1), default None
        Character to break file into lines. Only valid with C parser.
    quotechar : str (length 1), optional
        The character used to denote the start and end of a quoted item. Quoted
        items can include the delimiter and it will be ignored.
    quoting : int or csv.QUOTE_* instance, default 0
        Control field quoting behavior per ``csv.QUOTE_*`` constants. Use one of
        QUOTE_MINIMAL (0), QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or QUOTE_NONE (3).
    doublequote : boolean, default ``True``
       When quotechar is specified and quoting is not ``QUOTE_NONE``, indicate
       whether or not to interpret two consecutive quotechar elements INSIDE a
       field as a single ``quotechar`` element.
    escapechar : str (length 1), default None
        One-character string used to escape delimiter when quoting is QUOTE_NONE.
    comment : str, default None
        Indicates remainder of line should not be parsed. If found at the beginning
        of a line, the line will be ignored altogether. This parameter must be a
        single character. Like empty lines (as long as ``skip_blank_lines=True``),
        fully commented lines are ignored by the parameter `header` but not by
        `skiprows`. For example, if comment='#', parsing '#empty\na,b,c\n1,2,3'
        with `header=0` will result in 'a,b,c' being
        treated as the header.
    encoding : str, default None
        Encoding to use for UTF when reading/writing (ex. 'utf-8'). `List of Python
        standard encodings
        <https://docs.python.org/3/library/codecs.html#standard-encodings>`_
    dialect : str or csv.Dialect instance, default None
        If None defaults to Excel dialect. Ignored if sep longer than 1 char
        See csv.Dialect documentation for more details
    tupleize_cols : boolean, default False
        Leave a list of tuples on columns as is (default is to convert to
        a Multi Index on the columns)
    error_bad_lines : boolean, default True
        Lines with too many fields (e.g. a csv line with too many commas) will by
        default cause an exception to be raised, and no DataFrame will be returned.
        If False, then these "bad lines" will dropped from the DataFrame that is
        returned. (Only valid with C parser)
    warn_bad_lines : boolean, default True
        If error_bad_lines is False, and warn_bad_lines is True, a warning for each
        "bad line" will be output. (Only valid with C parser).
    low_memory : boolean, default True
        Internally process the file in chunks, resulting in lower memory use
        while parsing, but possibly mixed type inference.  To ensure no mixed
        types either set False, or specify the type with the `dtype` parameter.
        Note that the entire file is read into a single DataFrame regardless,
        use the `chunksize` or `iterator` parameter to return the data in chunks.
        (Only valid with C parser)
    buffer_lines : int, default None
        DEPRECATED: this argument will be removed in a future version because its
        value is not respected by the parser
    compact_ints : boolean, default False
        DEPRECATED: this argument will be removed in a future version
    
        If compact_ints is True, then for any column that is of integer dtype,
        the parser will attempt to cast it as the smallest integer dtype possible,
        either signed or unsigned depending on the specification from the
        `use_unsigned` parameter.
    use_unsigned : boolean, default False
        DEPRECATED: this argument will be removed in a future version
    
        If integer columns are being compacted (i.e. `compact_ints=True`), specify
        whether the column should be compacted to the smallest signed or unsigned
        integer dtype.
    memory_map : boolean, default False
        If a filepath is provided for `filepath_or_buffer`, map the file object
        directly onto memory and access the data directly from there. Using this
        option can improve performance because there is no longer any I/O overhead.
    
    Returns
    -------
    result : DataFrame or TextParser

close_px.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 2214 entries, 2003-01-02 to 2011-10-14
Data columns (total 4 columns):
AAPL    2214 non-null float64
MSFT    2214 non-null float64
XOM     2214 non-null float64
SPX     2214 non-null float64
dtypes: float64(4)
memory usage: 86.5 KB

close_px[-4:]

help(pd.DataFrame.info)

Help on function info in module pandas.core.frame:

info(self, verbose=None, buf=None, max_cols=None, memory_usage=None, null_counts=None)
    Concise summary of a DataFrame.
    
    Parameters
    ----------
    verbose : {None, True, False}, optional
        Whether to print the full summary.
        None follows the `display.max_info_columns` setting.
        True or False overrides the `display.max_info_columns` setting.
    buf : writable buffer, defaults to sys.stdout
    max_cols : int, default None
        Determines whether full summary or short summary is printed.
        None follows the `display.max_info_columns` setting.
    memory_usage : boolean/string, default None
        Specifies whether total memory usage of the DataFrame
        elements (including index) should be displayed. None follows
        the `display.memory_usage` setting. True or False overrides
        the `display.memory_usage` setting. A value of 'deep' is equivalent
        of True, with deep introspection. Memory usage is shown in
        human-readable units (base-2 representation).
    null_counts : boolean, default None
        Whether to show the non-null counts
    
        - If None, then only show if the frame is smaller than
          max_info_rows and max_info_columns.
        - If True, always show counts.
        - If False, never show counts.

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 8 entries, 0 to 7
Data columns (total 3 columns):
category    8 non-null object
data        8 non-null float64
weights     8 non-null float64
dtypes: float64(2), object(1)
memory usage: 272.0+ bytes

One task of interest might be to compute a DataFrame consisting of the yearly correlations of daily returns(computed from the percent changes) with SPX. As one way to do this,we first create a function that computes the pairwise correlation of each column with SPX column:

spx_corr=lambda x:x.corrwith(x['SPX'])

help(pd.DataFrame.pct_change)

Help on function pct_change in module pandas.core.generic:

pct_change(self, periods=1, fill_method='pad', limit=None, freq=None, **kwargs)
    Percent change over given number of periods.
    
    Parameters
    ----------
    periods : int, default 1
        Periods to shift for forming percent change
    fill_method : str, default 'pad'
        How to handle NAs before computing percent changes
    limit : int, default None
        The number of consecutive NAs to fill before stopping
    freq : DateOffset, timedelta, or offset alias string, optional
        Increment to use from time series API (e.g. 'M' or BDay())
    
    Returns
    -------
    chg : NDFrame
    
    Notes
    -----
    
    By default, the percentage change is calculated along the stat
    axis: 0, or ``Index``, for ``DataFrame`` and 1, or ``minor`` for
    ``Panel``. You can change this with the ``axis`` keyword argument.

df=pd.DataFrame(np.arange(12).reshape((3,4)));df

df.pct_change()

Above
(7-3)/3=1.33333,(6-2)/2=2,(5-1)/1=4,
(4-0)/0=inf,(8-4)/4=1,(9-5)/5=0.6667,(11-7)/7=0.57143

rets=close_px.pct_change().dropna()

rets.head()

Lastly,we group these percent changed by year,which can be extracted from each row label with a one-line function that returns the year attribute of each datetime laebl:

get_year=lambda x:x.year

by_year=rets.groupby(get_year) # get_year will be applied to rets.index

list(by_year)

[(2003,                 AAPL      MSFT       XOM       SPX
  2003-01-03  0.006757  0.001421  0.000684 -0.000484
  2003-01-06  0.000000  0.017975  0.024624  0.022474
  2003-01-07 -0.002685  0.019052 -0.033712 -0.006545
  2003-01-08 -0.020188 -0.028272 -0.004145 -0.014086
  2003-01-09  0.008242  0.029094  0.021159  0.019386
  2003-01-10  0.002725  0.001824 -0.013927  0.000000
  2003-01-13 -0.005435  0.008648 -0.004134 -0.001412
  2003-01-14 -0.002732  0.010379  0.008993  0.005830
  2003-01-15 -0.010959 -0.012506 -0.013713 -0.014426
  2003-01-16  0.012465 -0.016282  0.004519 -0.003942
  2003-01-17 -0.035568 -0.070345 -0.010381 -0.014017
  2003-01-21 -0.005674 -0.002473 -0.023077 -0.015702
  2003-01-22 -0.009986 -0.006445 -0.012885 -0.010432
  2003-01-23  0.021614  0.024950 -0.002175  0.010224
  2003-01-24 -0.026798 -0.046251 -0.021439 -0.029233
  2003-01-27  0.024638 -0.013783 -0.026736 -0.016160
  2003-01-28  0.031117 -0.007246  0.026326  0.013050
  2003-01-29  0.024691  0.022419  0.036431  0.006779
  2003-01-30 -0.041499 -0.033656 -0.018293 -0.022849
  2003-01-31  0.002793 -0.015831  0.027768  0.013130
  2003-02-03  0.020891  0.023056  0.013864  0.005399
  2003-02-04 -0.004093 -0.025681  0.000000 -0.014088
  2003-02-05 -0.010959 -0.007531 -0.014376 -0.005435
  2003-02-06  0.000000  0.009756 -0.008538 -0.006449
  2003-02-07 -0.020776 -0.017713 -0.007535 -0.010094
  2003-02-10  0.015559  0.017486  0.007592  0.007569
  2003-02-11  0.000000 -0.019871 -0.007176 -0.008098
  2003-02-12  0.002786  0.000000 -0.019877 -0.012687
  2003-02-13  0.009722  0.011507  0.012906 -0.001600
  2003-02-14  0.009629  0.028169  0.009465  0.021435
  ...              ...       ...       ...       ...
  2003-11-18 -0.034091  0.000000 -0.012604 -0.009084
  2003-11-19  0.000980  0.008024  0.008062  0.008016
  2003-11-20 -0.001959 -0.009950 -0.003665 -0.008432
  2003-11-21 -0.004907  0.000503  0.001003  0.001577
  2003-11-24  0.042406  0.024611  0.006014  0.016227
  2003-11-25 -0.021760 -0.012745  0.007307  0.001720
  2003-11-26  0.001934  0.001986  0.012859  0.004327
  2003-11-28  0.008687  0.009911 -0.002279 -0.000236
  2003-12-01  0.038278  0.005397  0.005546  0.011264
  2003-12-02 -0.007373 -0.007321  0.002920 -0.003271
  2003-12-03 -0.023213  0.000492  0.000971 -0.001772
  2003-12-04  0.004753  0.020639  0.013251  0.004687
  2003-12-05 -0.013245 -0.008185 -0.006380 -0.007684
  2003-12-08  0.008629  0.009709  0.016693  0.007348
  2003-12-09 -0.027567  0.005288  0.006315 -0.008529
  2003-12-10 -0.003910  0.008130  0.004079 -0.001066
  2003-12-11  0.040236  0.000949  0.003125  0.011482
  2003-12-12 -0.015094  0.001422  0.004050  0.002735
  2003-12-15 -0.033525  0.003313 -0.006826 -0.005679
  2003-12-16 -0.002973  0.011792  0.017807  0.006638
  2003-12-17 -0.011928 -0.000466  0.010436  0.001256
  2003-12-18  0.008048  0.013060  0.013670  0.011798
  2003-12-19 -0.016966 -0.001381  0.004795 -0.000477
  2003-12-22  0.008122 -0.006455 -0.003281  0.003931
  2003-12-23 -0.003021 -0.001392  0.007181  0.002818
  2003-12-24  0.030303 -0.003717  0.002080 -0.001807
  2003-12-26  0.018627  0.006063  0.005336  0.001691
  2003-12-29  0.017324  0.009272  0.013270  0.012401
  2003-12-30  0.006623  0.002297  0.002619  0.000144
  2003-12-31  0.004699 -0.005500  0.007837  0.002055
  
  [251 rows x 4 columns]),
 (2004,                 AAPL      MSFT       XOM       SPX
  2004-01-02 -0.004677  0.002765 -0.008929 -0.003094
  2004-01-05  0.042293  0.025276  0.023249  0.012395
  2004-01-06 -0.003607  0.003586 -0.006816  0.001292
  2004-01-07  0.022624 -0.001340 -0.007149  0.002367
  2004-01-08  0.033628 -0.001342 -0.002592  0.004963
  2004-01-09 -0.015411 -0.017913 -0.014727 -0.008888
  2004-01-12  0.031304 -0.003192  0.014068  0.004787
  2004-01-13  0.016863 -0.005032 -0.004335 -0.005332
  2004-01-14  0.003317  0.009655  0.000000  0.008295
  2004-01-15 -0.055372 -0.005920 -0.009869  0.001353
  2004-01-16 -0.006124  0.010078  0.005570  0.006872
  2004-01-20  0.000000  0.010431  0.009329 -0.000930
  2004-01-21 -0.004401  0.007181  0.013287  0.007772
  2004-01-22 -0.019452 -0.010250 -0.003421 -0.003207
  2004-01-23  0.017133  0.016659 -0.006007 -0.002089
  2004-01-26  0.020390  0.011072  0.012374  0.012106
  2004-01-27  0.001738 -0.018835 -0.008528 -0.009798
  2004-01-28 -0.023417 -0.019196 -0.009174 -0.013610
  2004-01-29  0.007105  0.007283  0.016204  0.004989
  2004-01-30 -0.005291 -0.009489 -0.016515 -0.002628
  2004-02-02 -0.010638 -0.009124 -0.002316  0.003651
  2004-02-03 -0.002688 -0.003683 -0.006965  0.000678
  2004-02-04 -0.020665 -0.010628 -0.001461 -0.008371
  2004-02-05  0.028440 -0.001868  0.000585  0.001838
  2004-02-06  0.012489  0.004679  0.003217  0.012555
  2004-02-09 -0.000881 -0.006521  0.011662 -0.002581
  2004-02-10  0.013228  0.004219  0.008934  0.005027
  2004-02-11  0.035683  0.004669  0.020280  0.010667
  2004-02-12 -0.003361 -0.006970 -0.000280 -0.004880
  2004-02-13 -0.030354 -0.013570  0.000000 -0.005468
  ...              ...       ...       ...       ...
  2004-11-18  0.008743 -0.003738  0.000922  0.001362
  2004-11-19 -0.003973 -0.007920  0.006680 -0.011161
  2004-11-22  0.112038 -0.007563  0.010526  0.005896
  2004-11-23 -0.000978 -0.004657  0.005661 -0.000255
  2004-11-24  0.045366  0.004254  0.005629  0.004095
  2004-11-26  0.007805 -0.001694  0.003135  0.000753
  2004-11-29  0.060099  0.006364 -0.005357 -0.003450
  2004-11-30 -0.020164  0.001686 -0.002469 -0.004030
  2004-12-01  0.011035  0.016414 -0.001800  0.014951
  2004-12-02 -0.038348 -0.005797 -0.019157 -0.000873
  2004-12-03 -0.038650  0.004998  0.001838  0.000706
  2004-12-06  0.049458  0.003730 -0.002523 -0.000772
  2004-12-07 -0.044086 -0.009496 -0.008738 -0.011073
  2004-12-08  0.006361  0.010838  0.006495  0.004877
  2004-12-09  0.011378 -0.004948  0.005992  0.005436
  2004-12-10  0.018125 -0.005387 -0.003436 -0.001043
  2004-12-13 -0.003683  0.006250  0.010345  0.008990
  2004-12-14  0.005853 -0.000828 -0.000228  0.003921
  2004-12-15 -0.000613 -0.004144 -0.002959  0.001945
  2004-12-16  0.020533  0.001665 -0.007989 -0.002082
  2004-12-17 -0.024324 -0.007478 -0.005983 -0.007488
  2004-12-20 -0.034780 -0.000419  0.025231  0.000377
  2004-12-21  0.015306  0.004606  0.012192  0.009040
  2004-12-22  0.001256 -0.003752 -0.007584  0.003418
  2004-12-23  0.004078  0.001674  0.013261  0.000463
  2004-12-27 -0.013433 -0.006266 -0.021074 -0.004305
  2004-12-28  0.016149  0.003783  0.003399  0.007154
  2004-12-29  0.004051 -0.001675  0.001581 -0.000074
  2004-12-30  0.005587 -0.005034 -0.002029  0.000082
  2004-12-31 -0.006173 -0.001686  0.004519 -0.001343
  
  [252 rows x 4 columns]),
 (2005,                 AAPL      MSFT       XOM       SPX
  2005-01-03 -0.017081  0.000845 -0.022717 -0.008119
  2005-01-04  0.010111  0.003797 -0.006904 -0.011671
  2005-01-05  0.008758 -0.002522 -0.005098 -0.003628
  2005-01-06  0.000930 -0.000843  0.012579  0.003506
  2005-01-07  0.072491 -0.002952 -0.006441 -0.001431
  2005-01-10 -0.004044  0.004653  0.003705  0.003423
  2005-01-11 -0.063805 -0.002526 -0.002537 -0.006100
  2005-01-12  0.013941  0.001688  0.014801  0.003981
  2005-01-13  0.066300 -0.018963 -0.000684 -0.008630
  2005-01-14  0.005731 -0.005584  0.010262  0.006005
  2005-01-18  0.006553  0.007775  0.008352  0.009675
  2005-01-19 -0.011039 -0.013288 -0.008731 -0.009490
  2005-01-20  0.008300 -0.004344 -0.006549 -0.007783
  2005-01-21  0.000284 -0.008290 -0.005456 -0.006415
  2005-01-24  0.003973  0.000880  0.013714 -0.003528
  2005-01-25  0.018372  0.013626  0.003833  0.004004
  2005-01-26  0.002498 -0.000434  0.007188  0.004844
  2005-01-27  0.005537  0.003905  0.001115  0.000409
  2005-01-28  0.018447  0.002593 -0.009356 -0.002716
  2005-01-31  0.039470  0.003879  0.006521  0.008460
  2005-02-01  0.008062  0.004294  0.032395  0.006891
  2005-02-02  0.027090  0.002565  0.012335  0.003178
  2005-02-03 -0.022859 -0.010661  0.010902 -0.002766
  2005-02-04  0.013368  0.005603  0.014168  0.011043
  2005-02-07  0.001268 -0.006429  0.001877 -0.001089
  2005-02-08  0.024829  0.003020  0.011863  0.000483
  2005-02-09 -0.026700 -0.006452 -0.004319 -0.008575
  2005-02-10 -0.004826 -0.000433  0.014460  0.004211
  2005-02-11  0.036243 -0.003032 -0.004072  0.006926
  2005-02-14  0.042118  0.001303  0.012676  0.000697
  ...              ...       ...       ...       ...
  2005-11-17 -0.006620  0.008441  0.003360  0.009414
  2005-11-18  0.000620  0.003587  0.015166  0.004401
  2005-11-21  0.006196  0.003177  0.019208  0.005271
  2005-11-22  0.024015 -0.008709  0.004950  0.005084
  2005-11-23  0.008870  0.000399  0.003599  0.003473
  2005-11-25  0.033229 -0.005988  0.003964  0.002086
  2005-11-28  0.004615 -0.000402 -0.022749 -0.008508
  2005-11-29 -0.022394 -0.002411 -0.006926  0.000016
  2005-11-30 -0.004112  0.000000 -0.005231 -0.006362
  2005-12-01  0.055736  0.007652  0.022785  0.012157
  2005-12-02  0.014385  0.004396 -0.004760  0.000324
  2005-12-05 -0.011152 -0.005969  0.007461 -0.002363
  2005-12-06  0.031050 -0.005604  0.003038  0.001276
  2005-12-07 -0.001350  0.002013 -0.011359 -0.005009
  2005-12-08  0.001758 -0.002009  0.006894 -0.001217
  2005-12-09  0.003375  0.000805 -0.015595  0.002811
  2005-12-12  0.007803 -0.009654  0.006182  0.000842
  2005-12-13  0.000934 -0.011373  0.003456  0.005554
  2005-12-14 -0.039611 -0.001643  0.013586  0.004190
  2005-12-15  0.002361 -0.006173 -0.006230 -0.001414
  2005-12-16 -0.014824 -0.000828 -0.024126 -0.002848
  2005-12-19  0.003797 -0.002487 -0.006035 -0.005839
  2005-12-20  0.010227  0.000831  0.003917 -0.000238
  2005-12-21  0.019276 -0.004566 -0.005657  0.002517
  2005-12-22  0.007075 -0.005421 -0.008633  0.004221
  2005-12-23 -0.009052  0.002096  0.000000  0.000426
  2005-12-27  0.011997 -0.006695 -0.021571 -0.009553
  2005-12-28 -0.008891 -0.002949  0.006675  0.001297
  2005-12-29 -0.028816 -0.004225 -0.000201 -0.002981
  2005-12-30  0.006158 -0.004667 -0.001206 -0.004887
  
  [252 rows x 4 columns]),
 (2006,                 AAPL      MSFT       XOM       SPX
  2006-01-03  0.039783  0.026428  0.041046  0.016430
  2006-01-04  0.002943  0.004568  0.001739  0.003673
  2006-01-05 -0.007870  0.000827 -0.005016  0.000016
  2006-01-06  0.025813 -0.002891  0.019779  0.009399
  2006-01-09 -0.003277 -0.002071 -0.000570  0.003656
  2006-01-10  0.063248  0.005396  0.007801 -0.000357
  2006-01-11  0.037596  0.010735  0.006796  0.003481
  2006-01-12  0.004648 -0.005310 -0.010501 -0.006274
  2006-01-13  0.015423  0.001643  0.022361  0.001205
  2006-01-17 -0.010282 -0.007380  0.009268 -0.003557
  2006-01-18 -0.026207 -0.005783 -0.013958 -0.003975
  2006-01-19 -0.041823  0.007063  0.013597  0.005564
  2006-01-20 -0.037323 -0.022690 -0.015803 -0.018326
  2006-01-23  0.020765 -0.002111  0.011016  0.001847
  2006-01-24 -0.020986 -0.002961 -0.003878  0.002405
  2006-01-25 -0.024198  0.004667 -0.012236 -0.001721
  2006-01-26 -0.025202  0.003801 -0.004317  0.007235
  2006-01-27 -0.004148  0.048801  0.022243  0.007764
  2006-01-30  0.041233  0.007621  0.029688  0.001145
  2006-01-31  0.006800  0.005175 -0.005731 -0.003976
  2006-02-01 -0.001192 -0.003960 -0.012608  0.001859
  2006-02-02 -0.044020 -0.012724  0.000000 -0.009061
  2006-02-03 -0.003467 -0.005236 -0.009121 -0.005359
  2006-02-06 -0.063326 -0.013360  0.009389  0.000783
  2006-02-07  0.004458 -0.008207 -0.022798 -0.008095
  2006-02-08  0.017899 -0.001241  0.002613  0.008663
  2006-02-09 -0.056096 -0.009114 -0.007818 -0.001478
  2006-02-10  0.036336  0.000836 -0.008068  0.002540
  2006-02-13 -0.038627 -0.011278  0.002837 -0.003260
  2006-02-14  0.045279  0.010139 -0.000943  0.010033
  ...              ...       ...       ...       ...
  2006-11-16  0.018560  0.012075 -0.028457  0.002284
  2006-11-17  0.002803 -0.002610  0.005645  0.001029
  2006-11-20  0.007222  0.016822 -0.008799 -0.000500
  2006-11-21  0.024633  0.001103  0.013162  0.001649
  2006-11-22  0.019300  0.000000 -0.006495  0.002338
  2006-11-24  0.014616 -0.005509 -0.007298 -0.003656
  2006-11-27 -0.022809 -0.009232  0.001225 -0.013555
  2006-11-28  0.025352 -0.002982  0.023405  0.003444
  2006-11-29 -0.000109  0.005981  0.025112  0.009202
  2006-11-30 -0.001525 -0.007061  0.010353  0.000822
  2006-12-01 -0.003709 -0.008234  0.005051 -0.002799
  2006-12-04 -0.002190  0.007170  0.007323  0.008885
  2006-12-05  0.001646 -0.006744  0.003849  0.004002
  2006-12-06 -0.015777 -0.004904 -0.022437 -0.001315
  2006-12-07 -0.031059 -0.004928 -0.007844 -0.003971
  2006-12-08  0.014017  0.019048 -0.002782  0.001812
  2006-12-11  0.005552  0.004860 -0.001909  0.002270
  2006-12-12 -0.029408 -0.003720  0.011768 -0.001047
  2006-12-13  0.033782  0.004108  0.014539  0.001169
  2006-12-14 -0.005615  0.017479  0.017770  0.008689
  2006-12-15 -0.009373  0.004020 -0.018164  0.001122
  2006-12-18 -0.025650 -0.009829 -0.023089 -0.003230
  2006-12-19  0.009828  0.003309  0.019524  0.002158
  2006-12-20 -0.017959  0.003298 -0.011807 -0.001417
  2006-12-21 -0.021944 -0.003652 -0.002768 -0.003674
  2006-12-22 -0.008444 -0.011364 -0.005991 -0.005316
  2006-12-26 -0.008394  0.011865  0.008379  0.004352
  2006-12-27  0.000123  0.001099  0.011953  0.007015
  2006-12-28 -0.007974 -0.001464  0.004898 -0.001479
  2006-12-29  0.049091 -0.004032 -0.009031 -0.004513
  
  [251 rows x 4 columns]),
 (2007,                 AAPL      MSFT       XOM       SPX
  2007-01-03 -0.012258  0.000000 -0.032981 -0.001199
  2007-01-04  0.022196 -0.001472 -0.018699  0.001228
  2007-01-05 -0.007121 -0.005898  0.007165 -0.006085
  2007-01-08  0.004938  0.010011 -0.008022  0.002220
  2007-01-09  0.083070  0.000734 -0.007782 -0.000517
  2007-01-10  0.047856 -0.009905 -0.015224  0.001940
  2007-01-11 -0.012371  0.035198 -0.000156  0.006340
  2007-01-12 -0.012317  0.016464  0.023583  0.004853
  2007-01-16  0.026210 -0.001408 -0.014037  0.000818
  2007-01-17 -0.022142 -0.002116  0.011452 -0.000894
  2007-01-18 -0.061927 -0.003180 -0.006885 -0.002971
  2007-01-19 -0.006399  0.003545  0.021876  0.002895
  2007-01-22 -0.019322 -0.012363 -0.008593 -0.005278
  2007-01-23 -0.012559  0.000358  0.021898  0.003542
  2007-01-24  0.011669  0.011441  0.005506  0.008501
  2007-01-25 -0.005190 -0.020502 -0.018499 -0.011270
  2007-01-26 -0.010087  0.005052  0.001206 -0.001208
  2007-01-29  0.006559 -0.002513 -0.005572 -0.001097
  2007-01-30 -0.004538 -0.001440  0.016356  0.005772
  2007-01-31  0.002104  0.012257 -0.004023  0.006593
  2007-02-01 -0.011548 -0.009615  0.013315  0.005354
  2007-02-02  0.000118 -0.012226  0.006053  0.001694
  2007-02-05 -0.009558 -0.018930  0.001761 -0.000967
  2007-02-06  0.002502 -0.003711 -0.002783  0.000698
  2007-02-07  0.023767 -0.004469 -0.004701  0.001395
  2007-02-08  0.000348 -0.003741  0.009004 -0.001179
  2007-02-09 -0.033767 -0.009763 -0.003218 -0.007077
  2007-02-12  0.019335 -0.001138 -0.008218 -0.003261
  2007-02-13 -0.002121  0.005695  0.011394  0.007597
  2007-02-14  0.007084  0.013590  0.002048  0.007644
  ...              ...       ...       ...       ...
  2007-11-16  0.012721  0.009634  0.007228  0.005230
  2007-11-19 -0.014664 -0.003817 -0.011661 -0.017460
  2007-11-20  0.029887  0.018519  0.044081  0.004486
  2007-11-21 -0.002310 -0.010345 -0.008817 -0.015927
  2007-11-23  0.018283 -0.003484  0.014282  0.016891
  2007-11-26  0.005830 -0.033376 -0.029521 -0.023239
  2007-11-27  0.013156  0.002631  0.008146  0.014930
  2007-11-28  0.030948  0.019351  0.017801  0.028560
  2007-11-29  0.022584 -0.003218  0.007690  0.000477
  2007-11-30 -0.011232  0.000323  0.006401  0.007770
  2007-12-03 -0.018439 -0.020329 -0.003425 -0.005887
  2007-12-04  0.005311 -0.004282 -0.008223 -0.006540
  2007-12-05  0.031645  0.042011  0.020418  0.015190
  2007-12-06  0.023989  0.011746  0.016857  0.015037
  2007-12-07  0.022901 -0.000628  0.000716 -0.001778
  2007-12-10 -0.000463  0.006593  0.005720  0.007510
  2007-12-11 -0.029195 -0.019027 -0.018960 -0.025271
  2007-12-12  0.012305  0.010811  0.018118  0.006050
  2007-12-13  0.005082  0.022020  0.008779  0.001224
  2007-12-14 -0.007507  0.002462 -0.016700 -0.013746
  2007-12-17 -0.031462 -0.026098 -0.014113 -0.015021
  2007-12-18 -0.007701  0.010088  0.016984  0.006280
  2007-12-19  0.000765  0.001561  0.000119 -0.001361
  2007-12-20  0.022335  0.020879  0.007634  0.004900
  2007-12-21  0.035789  0.015263  0.014205  0.016670
  2007-12-24  0.025218  0.014432  0.002451  0.008077
  2007-12-26  0.000755  0.000889  0.012225  0.000809
  2007-12-27 -0.001910 -0.017471 -0.011962 -0.014282
  2007-12-28  0.006345  0.004219  0.014203  0.001504
  2007-12-31 -0.008757 -0.014406 -0.013889 -0.006852
  
  [251 rows x 4 columns]),
 (2008,                 AAPL      MSFT       XOM       SPX
  2008-01-02 -0.016357 -0.010658 -0.001862 -0.014438
  2008-01-03  0.000462  0.004001  0.003382  0.000000
  2008-01-04 -0.076335 -0.027897 -0.018596 -0.024552
  2008-01-07 -0.013385  0.006623 -0.009356  0.003223
  2008-01-08 -0.035972 -0.033521 -0.012791 -0.018352
  2008-01-09  0.047591  0.029822  0.016711  0.013624
  2008-01-10 -0.007692 -0.003148  0.001072  0.007948
  2008-01-11 -0.029940 -0.012314 -0.014753 -0.013595
  2008-01-14  0.035266  0.014066  0.005796  0.010871
  2008-01-15 -0.054480 -0.011349 -0.019930 -0.024925
  2008-01-16 -0.055608 -0.022640 -0.027931 -0.005612
  2008-01-17  0.007830 -0.003589 -0.030246 -0.029093
  2008-01-18  0.002921 -0.002947  0.013905 -0.006045
  2008-01-22 -0.035449 -0.031856 -0.030890 -0.011085
  2008-01-23 -0.106464 -0.001018  0.012035  0.021442
  2008-01-24 -0.024951  0.041426  0.030580  0.010063
  2008-01-25 -0.041224 -0.009455 -0.023967 -0.015872
  2008-01-28  0.000000 -0.006583  0.014291  0.017548
  2008-01-29  0.011768 -0.003645 -0.000640  0.006160
  2008-01-30  0.004865 -0.012305  0.002307 -0.004764
  2008-01-31  0.024058  0.012458  0.004987  0.016772
  2008-02-01 -0.011894 -0.065846  0.002927  0.012237
  2008-02-04 -0.015701 -0.008544 -0.005963 -0.010463
  2008-02-05 -0.017395 -0.037343 -0.038928 -0.031995
  2008-02-06 -0.056895 -0.018650 -0.008234 -0.007624
  2008-02-07 -0.006230 -0.014063  0.009909  0.007886
  2008-02-08  0.034972  0.015420 -0.002254 -0.004204
  2008-02-11  0.031639 -0.012149  0.018472  0.005889
  2008-02-12 -0.035458  0.004612  0.013961  0.007266
  2008-02-13  0.036361  0.021806  0.013126  0.013604
  ...              ...       ...       ...       ...
  2008-11-18  0.020082  0.022185  0.040222  0.009838
  2008-11-19 -0.040262 -0.067824 -0.038106 -0.061156
  2008-11-20 -0.067215 -0.041327 -0.066851 -0.067123
  2008-11-21  0.025966  0.122647  0.106446  0.063248
  2008-11-24  0.125575  0.051379  0.039498  0.064723
  2008-11-25 -0.023131 -0.033951 -0.008685  0.006551
  2008-11-26  0.046256  0.025027  0.035455  0.035328
  2008-11-28 -0.024526 -0.013506 -0.009122  0.009643
  2008-12-01 -0.040358 -0.079516 -0.072849 -0.089295
  2008-12-02  0.039807  0.029176  0.044467  0.039941
  2008-12-03  0.037093  0.037799  0.016947  0.025836
  2008-12-04 -0.046820 -0.038565 -0.033735 -0.029308
  2008-12-05  0.028334  0.040111  0.004347  0.036499
  2008-12-08  0.060851  0.057311  0.039229  0.038387
  2008-12-09  0.003410 -0.019757 -0.017598 -0.023118
  2008-12-10 -0.018489  0.000517  0.023930  0.011894
  2008-12-11 -0.032685 -0.056302 -0.000668 -0.028524
  2008-12-12  0.034421 -0.004379  0.005345  0.007028
  2008-12-15 -0.035820 -0.016493 -0.006115 -0.012686
  2008-12-16  0.007177  0.055897  0.039856  0.051360
  2008-12-17 -0.065703 -0.022234 -0.025080 -0.009593
  2008-12-18  0.003028 -0.018408 -0.050000 -0.021163
  2008-12-19  0.006374 -0.009377 -0.025691  0.002937
  2008-12-22 -0.047333  0.003341 -0.001568 -0.018302
  2008-12-23  0.007464  0.004994  0.002570 -0.009717
  2008-12-24 -0.015513 -0.005522  0.009113  0.005781
  2008-12-26  0.009055 -0.002221  0.018485  0.005356
  2008-12-29  0.009323 -0.008904  0.010806 -0.003873
  2008-12-30 -0.003695  0.020213  0.007264  0.024407
  2008-12-31 -0.010893  0.004953  0.015784  0.014158
  
  [253 rows x 4 columns]),
 (2009,                 AAPL      MSFT       XOM       SPX
  2009-01-02  0.063269  0.046002  0.022773  0.031608
  2009-01-05  0.042204  0.009424 -0.000131 -0.004668
  2009-01-06 -0.016494  0.011411 -0.016374  0.007817
  2009-01-07 -0.021608 -0.060000 -0.025436 -0.030010
  2009-01-08  0.018569  0.031097  0.010659  0.003397
  2009-01-09 -0.022869 -0.029630 -0.019200 -0.021303
  2009-01-12 -0.021197 -0.002726 -0.013234 -0.022564
  2009-01-13 -0.010715  0.018043  0.018022  0.001758
  2009-01-14 -0.027135 -0.037057 -0.036229 -0.033460
  2009-01-15 -0.022852  0.007808  0.020789  0.001329
  2009-01-16 -0.012593  0.024903  0.018831  0.007562
  2009-01-20 -0.050164 -0.062635 -0.023275 -0.052816
  2009-01-21  0.059207  0.048963  0.038968  0.043491
  2009-01-22  0.066763 -0.117518 -0.012952 -0.015162
  2009-01-23  0.000000  0.005600 -0.002460  0.005378
  2009-01-26  0.014486  0.024752  0.007125  0.005553
  2009-01-27  0.012160  0.001812  0.004082  0.010926
  2009-01-28  0.038245  0.021700  0.004201  0.033558
  2009-01-29 -0.012739 -0.025369 -0.028336 -0.033120
  2009-01-30 -0.030860 -0.027845 -0.006805 -0.022789
  2009-02-02  0.015311  0.042964  0.002796 -0.000533
  2009-02-03  0.016064  0.037612  0.018544  0.015834
  2009-02-04  0.006130  0.006904 -0.005886 -0.007489
  2009-02-05  0.031106  0.022286  0.027403  0.016366
  2009-02-06  0.033796  0.032420  0.012063  0.026896
  2009-02-09  0.027978 -0.011370 -0.010727  0.001485
  2009-02-10 -0.045654 -0.032859 -0.042035 -0.049121
  2009-02-11 -0.010324  0.022084 -0.020402  0.007955
  2009-02-12  0.025305  0.002216  0.008559  0.001739
  2009-02-13 -0.001108 -0.008845 -0.008487 -0.009998
  ...              ...       ...       ...       ...
  2009-11-18 -0.005024  0.003814  0.003204 -0.000468
  2009-11-19 -0.026461 -0.011054 -0.008331 -0.013426
  2009-11-20 -0.002942 -0.005239 -0.003500 -0.003215
  2009-11-23  0.029812  0.010885  0.017704  0.013616
  2009-11-24 -0.006994 -0.001042  0.003590 -0.000533
  2009-11-25 -0.001223 -0.004172  0.006603  0.004504
  2009-11-27 -0.017631 -0.019204 -0.021047 -0.017233
  2009-11-30 -0.003390  0.006764  0.002792  0.003793
  2009-12-01 -0.014707  0.020156  0.012808  0.012075
  2009-12-02 -0.003757 -0.007626 -0.003162  0.000343
  2009-12-03  0.001274  0.001746 -0.010618 -0.008402
  2009-12-04 -0.016083  0.004881 -0.009895  0.005509
  2009-12-07 -0.022605 -0.006246 -0.006475 -0.002468
  2009-12-08  0.004869 -0.007332 -0.011051 -0.010252
  2009-12-09  0.041765  0.004573 -0.002292  0.003672
  2009-12-10 -0.006926  0.005602 -0.005313  0.005840
  2009-12-11 -0.008960 -0.000696  0.005919  0.003683
  2009-12-14  0.011866  0.008711 -0.043054  0.006959
  2009-12-15 -0.014265 -0.003109 -0.007499 -0.005547
  2009-12-16  0.004429  0.002772 -0.010728  0.001128
  2009-12-17 -0.016254 -0.016586 -0.003055 -0.011811
  2009-12-18  0.018607  0.025650 -0.000153  0.005830
  2009-12-21  0.014327  0.005139  0.004444  0.010504
  2009-12-22  0.010745  0.009884  0.000915  0.003564
  2009-12-23  0.008684  0.003375 -0.004572  0.002299
  2009-12-24  0.034339  0.002355  0.005818  0.005256
  2009-12-28  0.012294  0.005705  0.006241  0.001154
  2009-12-29 -0.011861  0.007007 -0.003480 -0.001401
  2009-12-30  0.012147 -0.013585 -0.001063  0.000195
  2009-12-31 -0.004300 -0.015788 -0.008511 -0.010050
  
  [252 rows x 4 columns]),
 (2010,                 AAPL      MSFT       XOM       SPX
  2010-01-04  0.015565  0.015700  0.014102  0.016043
  2010-01-05  0.001729  0.000336  0.003930  0.003116
  2010-01-06 -0.015906 -0.006382  0.008582  0.000546
  2010-01-07 -0.001849 -0.010142 -0.003135  0.004001
  2010-01-08  0.006648  0.006831 -0.003893  0.002882
  2010-01-11 -0.008822 -0.012890  0.011124  0.001747
  2010-01-12 -0.011375 -0.006529 -0.004906 -0.009381
  2010-01-13  0.014106  0.009339 -0.004034  0.008326
  2010-01-14 -0.005792  0.020219  0.000150  0.002427
  2010-01-15 -0.016712 -0.003359 -0.008250 -0.010823
  2010-01-19  0.044238  0.007752  0.002420  0.012500
  2010-01-20 -0.015392 -0.016388 -0.017954 -0.010598
  2010-01-21 -0.017286 -0.019041 -0.019511 -0.018945
  2010-01-22 -0.049599 -0.035009 -0.009088 -0.022141
  2010-01-25  0.026903  0.012572 -0.003795  0.004598
  2010-01-26  0.014133  0.006031  0.001111 -0.004203
  2010-01-27  0.009420  0.005994 -0.005708  0.004880
  2010-01-28 -0.041322 -0.017175 -0.008930 -0.011818
  2010-01-29 -0.036279 -0.033880 -0.008045 -0.009829
  2010-02-01  0.013902  0.008121  0.027088  0.014266
  2010-02-02  0.005803  0.001831  0.011845  0.012973
  2010-02-03  0.017206  0.006213 -0.005463 -0.005474
  2010-02-04 -0.036039 -0.027606 -0.028249 -0.031141
  2010-02-05  0.017756  0.006350  0.001292  0.002897
  2010-02-08 -0.006856 -0.010765 -0.000484 -0.008863
  2010-02-09  0.010664  0.010507  0.013232  0.013040
  2010-02-10 -0.005454 -0.000743 -0.005415 -0.002233
  2010-02-11  0.018194  0.004831  0.006085  0.009680
  2010-02-12  0.008607 -0.007027 -0.006844 -0.002745
  2010-02-16  0.015071  0.019739  0.022917  0.018001
  ...              ...       ...       ...       ...
  2010-11-18  0.026389  0.010359  0.018876  0.015357
  2010-11-19 -0.005512 -0.005915  0.003329  0.002540
  2010-11-22  0.021615  0.001587 -0.005049 -0.001575
  2010-11-23 -0.014775 -0.023762 -0.017254 -0.014284
  2010-11-24  0.019661  0.010142  0.013131  0.014923
  2010-11-26  0.000635 -0.004819 -0.009320 -0.007469
  2010-11-29  0.005937  0.002421  0.003234 -0.001379
  2010-11-30 -0.018052 -0.002013  0.001465 -0.006070
  2010-12-01  0.016873  0.031061  0.025457  0.021617
  2010-12-02  0.005531  0.032473  0.002140  0.012818
  2010-12-03 -0.002232  0.004926 -0.003986  0.002603
  2010-12-06  0.008537 -0.006787  0.001572 -0.001298
  2010-12-07 -0.006060  0.001139  0.002141  0.000515
  2010-12-08  0.008799  0.013652  0.005412  0.003702
  2010-12-09 -0.003894 -0.005612  0.002125  0.003843
  2010-12-10  0.002502  0.009406  0.002544  0.006002
  2010-12-13  0.003463 -0.003354  0.002538  0.000048
  2010-12-14 -0.004290  0.013837 -0.002531  0.000911
  2010-12-15  0.000219  0.008115 -0.004652 -0.005122
  2010-12-16  0.002778  0.005123  0.005241  0.006185
  2010-12-17 -0.001992 -0.003276 -0.000705  0.000837
  2010-12-20  0.004990 -0.003287  0.000705  0.002548
  2010-12-21  0.006176  0.009527  0.006904  0.006030
  2010-12-22  0.002961  0.004356  0.001120  0.003380
  2010-12-23 -0.004798  0.003975  0.005451 -0.001644
  2010-12-27  0.003337 -0.008279 -0.002641  0.000613
  2010-12-28  0.002433 -0.002178  0.005715  0.000771
  2010-12-29 -0.000553 -0.001455 -0.000693  0.001009
  2010-12-30 -0.005011 -0.004372 -0.000139 -0.001508
  2010-12-31 -0.003399  0.002195 -0.003329 -0.000191
  
  [252 rows x 4 columns]),
 (2011,                 AAPL      MSFT       XOM       SPX
  2011-01-03  0.021732  0.002556  0.019624  0.011315
  2011-01-04  0.005219  0.004006  0.004641 -0.001313
  2011-01-05  0.008180 -0.003264 -0.002717  0.005007
  2011-01-06 -0.000808  0.029476  0.006540 -0.002123
  2011-01-07  0.007161 -0.007777  0.005414 -0.001845
  2011-01-10  0.018833 -0.013181 -0.006058 -0.001376
  2011-01-11 -0.002365 -0.003971  0.007450  0.003725
  2011-01-12  0.008137  0.015585  0.011697  0.009008
  2011-01-13  0.003658 -0.012491  0.001728 -0.001711
  2011-01-14  0.008100  0.003975  0.014725  0.007385
  2011-01-18 -0.022469  0.012599  0.011243  0.001376
  2011-01-19 -0.005313 -0.006754 -0.006076 -0.010116
  2011-01-20 -0.018180 -0.004295 -0.006243 -0.001295
  2011-01-21 -0.017915 -0.011503  0.015838  0.002414
  2011-01-24  0.032842  0.012727 -0.004896  0.005836
  2011-01-25  0.011705  0.002513  0.001165  0.000263
  2011-01-26  0.007176  0.011819  0.012416  0.004221
  2011-01-27 -0.001861  0.002832  0.002683  0.002244
  2011-01-28 -0.020716 -0.038475 -0.011084 -0.017852
  2011-01-31  0.009580 -0.000734  0.021386  0.007663
  2011-02-01  0.016828  0.009184  0.039985  0.016694
  2011-02-02 -0.002058 -0.001820 -0.005943 -0.002723
  2011-02-03 -0.002556 -0.010212  0.000366  0.002354
  2011-02-04  0.008910  0.004422 -0.001829  0.002884
  2011-02-07  0.015527  0.015407  0.007698  0.006240
  2011-02-08  0.009435  0.002890 -0.006063  0.004185
  2011-02-09  0.008333 -0.011167 -0.005124 -0.002786
  2011-02-10 -0.010107 -0.016758  0.007848  0.000750
  2011-02-11  0.006515 -0.009263 -0.004623  0.005507
  2011-02-14  0.006529 -0.000374  0.025303  0.002385
  ...              ...       ...       ...       ...
  2011-09-02 -0.018319 -0.015643 -0.018370 -0.025282
  2011-09-06  0.015212 -0.011240 -0.013723 -0.007436
  2011-09-07  0.011034  0.019208  0.035137  0.028646
  2011-09-08  0.000547  0.008462 -0.011270 -0.010612
  2011-09-09 -0.017337 -0.018307 -0.024856 -0.026705
  2011-09-12  0.006517  0.005828  0.011688  0.006966
  2011-09-13  0.012318  0.005794 -0.002645  0.009120
  2011-09-14  0.012168  0.017665  0.013817  0.013480
  2011-09-15  0.009401  0.018491  0.018860  0.017187
  2011-09-16  0.019188  0.004817  0.007296  0.005707
  2011-09-19  0.027790  0.003319 -0.011402 -0.009803
  2011-09-20  0.004421 -0.008453  0.004206 -0.001661
  2011-09-21 -0.003168 -0.036694 -0.027564 -0.029390
  2011-09-22 -0.025040 -0.035783 -0.037932 -0.031883
  2011-09-23  0.006172  0.000000  0.001011  0.006082
  2011-09-26 -0.002795  0.015164  0.034771  0.023336
  2011-09-27 -0.009698  0.009041  0.016592  0.010688
  2011-09-28 -0.005635 -0.003506 -0.011521 -0.020691
  2011-09-29 -0.016221 -0.005082  0.025114  0.008114
  2011-09-30 -0.023683 -0.022004 -0.016919 -0.024974
  2011-10-03 -0.017623 -0.014464 -0.020377 -0.028451
  2011-10-04 -0.005606  0.033021  0.023612  0.022488
  2011-10-05  0.015436  0.021705  0.015378  0.017866
  2011-10-06 -0.002327  0.017381 -0.000811  0.018304
  2011-10-07 -0.020060 -0.003417 -0.004466 -0.008163
  2011-10-10  0.051406  0.026286  0.036977  0.034125
  2011-10-11  0.029526  0.002227 -0.000131  0.000544
  2011-10-12  0.004747 -0.001481  0.011669  0.009795
  2011-10-13  0.015515  0.008160 -0.010238 -0.002974
  2011-10-14  0.033225  0.003311  0.022784  0.017380
  
  [199 rows x 4 columns])]

rets.index[0].year

2003

by_year.apply(spx_corr)

We could also compute inter-column correlations.Here we compute the annual correlation between Apple and Microsoft.

rets['AAPL'].corr(rets['MSFT'])

0.44469678861732237

df

df[0].corr(df[1])

1.0

help(pd.Series.corr)

Help on function corr in module pandas.core.series:

corr(self, other, method='pearson', min_periods=None)
    Compute correlation with `other` Series, excluding missing values
    
    Parameters
    ----------
    other : Series
    method : {'pearson', 'kendall', 'spearman'}
        * pearson : standard correlation coefficient
        * kendall : Kendall Tau correlation coefficient
        * spearman : Spearman rank correlation
    min_periods : int, optional
        Minimum number of observations needed to have a valid result
    
    
    Returns
    -------
    correlation : float

Example:Group-wise Linear regression

You can use groupby to perform more complex group-wise statistical analysis,as long as the function returns a pandas object or scalar value.For example,I can define the following regress function(using the statsmodels econometrics libraby),which executes an ordinary least quares(OLS) regression on each chunk of data:

import statsmodels.api as sm

def regress(data,yvar,xvars):
    Y=data[yvar]
    X=data[xvars]
    X['intercept']=1
    result=sm.OLS(Y,X).fit()
    return result.params

by_year.apply(regress,'AAPL',['SPX'])

Pivot tables and cross-tabulation

help(pd.DataFrame.pivot_table)

Help on function pivot_table in module pandas.tools.pivot:

pivot_table(data, values=None, index=None, columns=None, aggfunc='mean', fill_value=None, margins=False, dropna=True, margins_name='All')
    Create a spreadsheet-style pivot table as a DataFrame. The levels in the
    pivot table will be stored in MultiIndex objects (hierarchical indexes) on
    the index and columns of the result DataFrame
    
    Parameters
    ----------
    data : DataFrame
    values : column to aggregate, optional
    index : column, Grouper, array, or list of the previous
        If an array is passed, it must be the same length as the data. The list
        can contain any of the other types (except list).
        Keys to group by on the pivot table index.  If an array is passed, it
        is being used as the same manner as column values.
    columns : column, Grouper, array, or list of the previous
        If an array is passed, it must be the same length as the data. The list
        can contain any of the other types (except list).
        Keys to group by on the pivot table column.  If an array is passed, it
        is being used as the same manner as column values.
    aggfunc : function or list of functions, default numpy.mean
        If list of functions passed, the resulting pivot table will have
        hierarchical columns whose top level are the function names (inferred
        from the function objects themselves)
    fill_value : scalar, default None
        Value to replace missing values with
    margins : boolean, default False
        Add all row / columns (e.g. for subtotal / grand totals)
    dropna : boolean, default True
        Do not include columns whose entries are all NaN
    margins_name : string, default 'All'
        Name of the row / column that will contain the totals
        when margins is True.
    
    Examples
    --------
    >>> df
       A   B   C      D
    0  foo one small  1
    1  foo one large  2
    2  foo one large  2
    3  foo two small  3
    4  foo two small  3
    5  bar one large  4
    6  bar one small  5
    7  bar two small  6
    8  bar two large  7
    
    >>> table = pivot_table(df, values='D', index=['A', 'B'],
    ...                     columns=['C'], aggfunc=np.sum)
    >>> table
              small  large
    foo  one  1      4
         two  6      NaN
    bar  one  5      4
         two  6      7
    
    Returns
    -------
    table : DataFrame

ex1=pd.read_csv('ex1.csv')

ex1

ex1.pivot_table(index='sex',aggfunc=[np.mean,np.sum,np.min,np.max],margins=True)

ex1.pivot_table(index='sex')

Attention :min and max will discard non-number column 'province`.

df=pd.DataFrame(np.arange(4).reshape((2,2)),index=['a','b'],columns=['one','two']);df

df.pivot_table(index='one',aggfunc=np.sum,margins=True)

tips.head()

tips.pivot_table(index=['day','smoker'])

tips.pivot_table(values=['tip_pct','size'],index=['time','day'],columns='smoker')

tips.pivot_table(['tip_pct','size'],index=['time','day'],columns='smoker') #The same with above

We could augment this table to include partial totals by passing margins=True.This has the effect of adding All row and column labels,with corresponding values being the group statistics for all the data within a single tier:

tips.pivot_table(['tip_pct','size'],index=['time','day'],columns='smoker',margins=True)

Here,the All values are means without taking into account smoker versus non-smoker or any of the two levels of grouping on the rows.

if some combinations are empty,you may wish to pass a fill_value.

tips.pivot_table('tip_pct',index=['time','size','smoker'],columns='day',aggfunc='mean',fill_value=0)

data

Ohio         -0.953675
New york      0.531946
Vermont            NaN
Florida       1.018829
Oregon       -0.539930
Nevada             NaN
California   -0.869226
Idaho              NaN
dtype: float64

Cross-tabulation:Crosstab

A cross-tabulation is a special case of a pivot table that computes group frequencies.

data=pd.read_csv('ex2.csv')

data

pd.crosstab(data.Nationality,data.Handeness,margins=True)

pd.crosstab([tips.time,tips.day],tips.smoker,margins=True)

help(pd.crosstab)

Help on function crosstab in module pandas.tools.pivot:

crosstab(index, columns, values=None, rownames=None, colnames=None, aggfunc=None, margins=False, dropna=True, normalize=False)
    Compute a simple cross-tabulation of two (or more) factors. By default
    computes a frequency table of the factors unless an array of values and an
    aggregation function are passed
    
    Parameters
    ----------
    index : array-like, Series, or list of arrays/Series
        Values to group by in the rows
    columns : array-like, Series, or list of arrays/Series
        Values to group by in the columns
    values : array-like, optional
        Array of values to aggregate according to the factors.
        Requires `aggfunc` be specified.
    aggfunc : function, optional
        If specified, requires `values` be specified as well
    rownames : sequence, default None
        If passed, must match number of row arrays passed
    colnames : sequence, default None
        If passed, must match number of column arrays passed
    margins : boolean, default False
        Add row/column margins (subtotals)
    dropna : boolean, default True
        Do not include columns whose entries are all NaN
    normalize : boolean, {'all', 'index', 'columns'}, or {0,1}, default False
        Normalize by dividing all values by the sum of values.
    
        - If passed 'all' or `True`, will normalize over all values.
        - If passed 'index' will normalize over each row.
        - If passed 'columns' will normalize over each column.
        - If margins is `True`, will also normalize margin values.
    
        .. versionadded:: 0.18.1
    
    
    Notes
    -----
    Any Series passed will have their name attributes used unless row or column
    names for the cross-tabulation are specified.
    
    Any input passed containing Categorical data will have **all** of its
    categories included in the cross-tabulation, even if the actual data does
    not contain any instances of a particular category.
    
    In the event that there aren't overlapping indexes an empty DataFrame will
    be returned.
    
    Examples
    --------
    >>> a
    array([foo, foo, foo, foo, bar, bar,
           bar, bar, foo, foo, foo], dtype=object)
    >>> b
    array([one, one, one, two, one, one,
           one, two, two, two, one], dtype=object)
    >>> c
    array([dull, dull, shiny, dull, dull, shiny,
           shiny, dull, shiny, shiny, shiny], dtype=object)
    
    >>> crosstab(a, [b, c], rownames=['a'], colnames=['b', 'c'])
    b    one          two
    c    dull  shiny  dull  shiny
    a
    bar  1     2      1     0
    foo  2     2      1     2
    
    >>> foo = pd.Categorical(['a', 'b'], categories=['a', 'b', 'c'])
    >>> bar = pd.Categorical(['d', 'e'], categories=['d', 'e', 'f'])
    >>> crosstab(foo, bar)  # 'c' and 'f' are not represented in the data,
                            # but they still will be counted in the output
    col_0  d  e  f
    row_0
    a      1  0  0
    b      0  1  0
    c      0  0  0
    
    Returns
    -------
    crosstab : DataFrame

原文地址：https://www.cnblogs.com/johnyang/p/12793520.html