Skip to main content
`Ctrl`+`K`

# Version 0.18.0 (March 13, 2016)

This is a major release from 0.17.1 and includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version.

Warning

pandas >= 0.18.0 no longer supports compatibility with Python version 2.6 and 3.3 (GH 7718, GH 11273)

Warning

`numexpr` version 2.4.4 will now show a warning and not be used as a computation back-end for pandas because of some buggy behavior. This does not affect other versions (>= 2.1 and >= 2.4.6). (GH 12489)

Highlights include:

- Moving and expanding window functions are now methods on Series and DataFrame, similar to `.groupby`, see here.

- Adding support for a `RangeIndex` as a specialized form of the `Int64Index` for memory savings, see here.

- API breaking change to the `.resample` method to make it more `.groupby` like, see here.

- Removal of support for positional indexing with floats, which was deprecated since 0.14.0. This will now raise a `TypeError`, see here.

- The `.to_xarray()` function has been added for compatibility with the xarray package, see here.

- The `read_sas` function has been enhanced to read `sas7bdat` files, see here.

- Addition of the .str.extractall() method, and API changes to the .str.extract() method and .str.cat() method.

- `pd.test()` top-level nose test runner is available (GH 4327).

Check the API Changes and deprecations before updating.

## New features

### Window functions are now methods

Window functions have been refactored to be methods on `Series/DataFrame` objects, rather than top-level functions, which are now deprecated. This allows these window-type functions, to have a similar API to that of `.groupby`. See the full documentation here (GH 11603, GH 12373)

    In [1]: np.random.seed(1234)

    In [2]: df = pd.DataFrame({'A': range(10), 'B': np.random.randn(10)})

    In [3]: df
    Out[3]:
       A         B
    0  0  0.471435
    1  1 -1.190976
    2  2  1.432707
    3  3 -0.312652
    4  4 -0.720589
    5  5  0.887163
    6  6  0.859588
    7  7 -0.636524
    8  8  0.015696
    9  9 -2.242685

    [10 rows x 2 columns]

Previous behavior:

    In [8]: pd.rolling_mean(df, window=3)
            FutureWarning: pd.rolling_mean is deprecated for DataFrame and will be removed in a future version, replace with
                           DataFrame.rolling(window=3,center=False).mean()
    Out[8]:
        A         B
    0 NaN       NaN
    1 NaN       NaN
    2   1  0.237722
    3   2 -0.023640
    4   3  0.133155
    5   4 -0.048693
    6   5  0.342054
    7   6  0.370076
    8   7  0.079587
    9   8 -0.954504

New behavior:

    In [4]: r = df.rolling(window=3)

These show a descriptive repr

    In [5]: r
    Out[5]: Rolling [window=3,center=False,method=single]

with tab-completion of available methods and properties.

    In [9]: r.<TAB>  # noqa E225, E999
    r.A           r.agg         r.apply       r.count       r.exclusions  r.max         r.median      r.name        r.skew        r.sum
    r.B           r.aggregate   r.corr        r.cov         r.kurt        r.mean        r.min         r.quantile    r.std         r.var

The methods operate on the `Rolling` object itself

    In [6]: r.mean()
    Out[6]:
         A         B
    0  NaN       NaN
    1  NaN       NaN
    2  1.0  0.237722
    3  2.0 -0.023640
    4  3.0  0.133155
    5  4.0 -0.048693
    6  5.0  0.342054
    7  6.0  0.370076
    8  7.0  0.079587
    9  8.0 -0.954504

    [10 rows x 2 columns]

They provide getitem accessors

    In [7]: r['A'].mean()
    Out[7]:
    0    NaN
    1    NaN
    2    1.0
    3    2.0
    4    3.0
    5    4.0
    6    5.0
    7    6.0
    8    7.0
    9    8.0
    Name: A, Length: 10, dtype: float64

And multiple aggregations

    In [8]: r.agg({'A': ['mean', 'std'],
       ...:        'B': ['mean', 'std']})
       ...:
    Out[8]:
         A              B
      mean  std      mean       std
    0  NaN  NaN       NaN       NaN
    1  NaN  NaN       NaN       NaN
    2  1.0  1.0  0.237722  1.327364
    3  2.0  1.0 -0.023640  1.335505
    4  3.0  1.0  0.133155  1.143778
    5  4.0  1.0 -0.048693  0.835747
    6  5.0  1.0  0.342054  0.920379
    7  6.0  1.0  0.370076  0.871850
    8  7.0  1.0  0.079587  0.750099
    9  8.0  1.0 -0.954504  1.162285

    [10 rows x 4 columns]

### Changes to rename

`Series.rename` and `NDFrame.rename_axis` can now take a scalar or list-like argument for altering the Series or axis name, in addition to their old behaviors of altering labels. (GH 9494, GH 11965)

    In [9]: s = pd.Series(np.random.randn(5))

    In [10]: s.rename('newname')
    Out[10]:
    0    1.150036
    1    0.991946
    2    0.953324
    3   -2.021255
    4   -0.334077
    Name: newname, Length: 5, dtype: float64

    In [11]: df = pd.DataFrame(np.random.randn(5, 2))

    In [12]: (df.rename_axis("indexname")
       ....:    .rename_axis("columns_name", axis="columns"))
       ....:
    Out[12]:
    columns_name         0         1
    indexname
    0             0.002118  0.405453
    1             0.289092  1.321158
    2            -1.546906 -0.202646
    3            -0.655969  0.193421
    4             0.553439  1.318152

    [5 rows x 2 columns]

The new functionality works well in method chains. Previously these methods only accepted functions or dicts mapping a label to a new label. This continues to work as before for function or dict-like values.

### Range Index

A `RangeIndex` has been added to the `Int64Index` sub-classes to support a memory saving alternative for common use cases. This has a similar implementation to the python `range` object (`xrange` in python 2), in that it only stores the start, stop, and step values for the index. It will transparently interact with the user API, converting to `Int64Index` if needed.

This will now be the default constructed index for `NDFrame` objects, rather than previous an `Int64Index`. (GH 939, GH 12070, GH 12071, GH 12109, GH 12888)

Previous behavior:

    In [3]: s = pd.Series(range(1000))

    In [4]: s.index
    Out[4]:
    Int64Index([  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,
                ...
                990, 991, 992, 993, 994, 995, 996, 997, 998, 999], dtype='int64', length=1000)

    In [6]: s.index.nbytes
    Out[6]: 8000

New behavior:

    In [13]: s = pd.Series(range(1000))

    In [14]: s.index
    Out[14]: RangeIndex(start=0, stop=1000, step=1)

    In [15]: s.index.nbytes
    Out[15]: 132

### Changes to str.extract

The .str.extract method takes a regular expression with capture groups, finds the first match in each subject string, and returns the contents of the capture groups (GH 11386).

In v0.18.0, the `expand` argument was added to `extract`.

- `expand=False`: it returns a `Series`, `Index`, or `DataFrame`, depending on the subject and regular expression pattern (same behavior as pre-0.18.0).

- `expand=True`: it always returns a `DataFrame`, which is more consistent and less confusing from the perspective of a user.

Currently the default is `expand=None` which gives a `FutureWarning` and uses `expand=False`. To avoid this warning, please explicitly specify `expand`.

    In [1]: pd.Series(['a1', 'b2', 'c3']).str.extract(r'[ab](\d)', expand=None)
    FutureWarning: currently extract(expand=None) means expand=False (return Index/Series/DataFrame)
    but in a future version of pandas this will be changed to expand=True (return DataFrame)

    Out[1]:
    0      1
    1      2
    2    NaN
    dtype: object

Extracting a regular expression with one group returns a Series if `expand=False`.

    In [16]: pd.Series(['a1', 'b2', 'c3']).str.extract(r'[ab](\d)', expand=False)
    Out[16]:
    0      1
    1      2
    2    NaN
    Length: 3, dtype: str

It returns a `DataFrame` with one column if `expand=True`.

    In [17]: pd.Series(['a1', 'b2', 'c3']).str.extract(r'[ab](\d)', expand=True)
    Out[17]:
         0
    0    1
    1    2
    2  NaN

    [3 rows x 1 columns]

Calling on an `Index` with a regex with exactly one capture group returns an `Index` if `expand=False`.

    In [18]: s = pd.Series(["a1", "b2", "c3"], ["A11", "B22", "C33"])

    In [19]: s.index
    Out[19]: Index(['A11', 'B22', 'C33'], dtype='str')

    In [20]: s.index.str.extract("(?P<letter>[a-zA-Z])", expand=False)
    Out[20]: Index(['A', 'B', 'C'], dtype='str', name='letter')

It returns a `DataFrame` with one column if `expand=True`.

    In [21]: s.index.str.extract("(?P<letter>[a-zA-Z])", expand=True)
    Out[21]:
      letter
    0      A
    1      B
    2      C

    [3 rows x 1 columns]

Calling on an `Index` with a regex with more than one capture group raises `ValueError` if `expand=False`.

    >>> s.index.str.extract("(?P<letter>[a-zA-Z])([0-9]+)", expand=False)
    ValueError: only one regex group is supported with Index

It returns a `DataFrame` if `expand=True`.

    In [22]: s.index.str.extract("(?P<letter>[a-zA-Z])([0-9]+)", expand=True)
    Out[22]:
      letter   1
    0      A  11
    1      B  22
    2      C  33

    [3 rows x 2 columns]

In summary, `extract(expand=True)` always returns a `DataFrame` with a row for every subject string, and a column for every capture group.

### Addition of str.extractall

The .str.extractall method was added (GH 11386). Unlike `extract`, which returns only the first match.

    In [23]: s = pd.Series(["a1a2", "b1", "c1"], ["A", "B", "C"])

    In [24]: s
    Out[24]:
    A    a1a2
    B      b1
    C      c1
    Length: 3, dtype: str

    In [25]: s.str.extract(r"(?P<letter>[ab])(?P<digit>\d)", expand=False)
    Out[25]:
      letter digit
    A      a     1
    B      b     1
    C    NaN   NaN

    [3 rows x 2 columns]

The `extractall` method returns all matches.

    In [26]: s.str.extractall(r"(?P<letter>[ab])(?P<digit>\d)")
    Out[26]:
            letter digit
      match
    A 0          a     1
      1          a     2
    B 0          b     1

    [3 rows x 2 columns]

### Changes to str.cat

The method `.str.cat()` concatenates the members of a `Series`. Before, if `NaN` values were present in the Series, calling `.str.cat()` on it would return `NaN`, unlike the rest of the `Series.str.*` API. This behavior has been amended to ignore `NaN` values by default. (GH 11435).

A new, friendlier `ValueError` is added to protect against the mistake of supplying the `sep` as an arg, rather than as a kwarg. (GH 11334).

    In [27]: pd.Series(['a', 'b', np.nan, 'c']).str.cat(sep=' ')
    Out[27]: 'a b c'

    In [28]: pd.Series(['a', 'b', np.nan, 'c']).str.cat(sep=' ', na_rep='?')
    Out[28]: 'a b ? c'

    In [2]: pd.Series(['a', 'b', np.nan, 'c']).str.cat(' ')
    ValueError: Did you mean to supply a ``sep`` keyword?

### Datetimelike rounding

`DatetimeIndex`, `Timestamp`, `TimedeltaIndex`, `Timedelta` have gained the `.round()`, `.floor()` and `.ceil()` method for datetimelike rounding, flooring and ceiling. (GH 4314, GH 11963)

Naive datetimes

    In [29]: dr = pd.date_range('20130101 09:12:56.1234', periods=3)

    In [30]: dr
    Out[30]:
    DatetimeIndex(['2013-01-01 09:12:56.123400', '2013-01-02 09:12:56.123400',
                   '2013-01-03 09:12:56.123400'],
                  dtype='datetime64[us]', freq='D')

    In [31]: dr.round('s')
    Out[31]:
    DatetimeIndex(['2013-01-01 09:12:56', '2013-01-02 09:12:56',
                   '2013-01-03 09:12:56'],
                  dtype='datetime64[us]', freq=None)

    # Timestamp scalar
    In [32]: dr[0]
    Out[32]: Timestamp('2013-01-01 09:12:56.123400')

    In [33]: dr[0].round('10s')
    Out[33]: Timestamp('2013-01-01 09:13:00')

Tz-aware are rounded, floored and ceiled in local times

    In [34]: dr = dr.tz_localize('US/Eastern')

    In [35]: dr
    Out[35]:
    DatetimeIndex(['2013-01-01 09:12:56.123400-05:00',
                   '2013-01-02 09:12:56.123400-05:00',
                   '2013-01-03 09:12:56.123400-05:00'],
                  dtype='datetime64[us, US/Eastern]', freq=None)

    In [36]: dr.round('s')
    Out[36]:
    DatetimeIndex(['2013-01-01 09:12:56-05:00', '2013-01-02 09:12:56-05:00',
                   '2013-01-03 09:12:56-05:00'],
                  dtype='datetime64[us, US/Eastern]', freq=None)

Timedeltas

    In [37]: t = pd.timedelta_range('1 days 2 hr 13 min 45 us', periods=3, freq='d')

    In [38]: t
    Out[38]:
    TimedeltaIndex(['1 days 02:13:00.000045', '2 days 02:13:00.000045',
                    '3 days 02:13:00.000045'],
                   dtype='timedelta64[ns]', freq='D')

    In [39]: t.round('10min')
    Out[39]:
    TimedeltaIndex(['1 days 02:10:00', '2 days 02:10:00',
                    '3 days 02:10:00'],
                   dtype='timedelta64[ns]', freq=None)

    # Timedelta scalar
    In [40]: t[0]
    Out[40]: Timedelta('1 days 02:13:00.000045')

    In [41]: t[0].round('2h')
    Out[41]: Timedelta('1 days 02:00:00')

In addition, `.round()`, `.floor()` and `.ceil()` will be available through the `.dt` accessor of `Series`.

    In [37]: s = pd.Series(dr)

    In [38]: s
    Out[38]:
    0   2013-01-01 09:12:56.123400-05:00
    1   2013-01-02 09:12:56.123400-05:00
    2   2013-01-03 09:12:56.123400-05:00
    Length: 3, dtype: datetime64[us, US/Eastern]

    In [39]: s.dt.round('D')
    Out[39]:
    0   2013-01-01 00:00:00-05:00
    1   2013-01-02 00:00:00-05:00
    2   2013-01-03 00:00:00-05:00
    Length: 3, dtype: datetime64[us, US/Eastern]

### Formatting of integers in FloatIndex

Integers in `FloatIndex`, e.g. 1., are now formatted with a decimal point and a `0` digit, e.g. `1.0` (GH 11713) This change not only affects the display to the console, but also the output of IO methods like `.to_csv` or `.to_html`.

Previous behavior:

    In [2]: s = pd.Series([1, 2, 3], index=np.arange(3.))

    In [3]: s
    Out[3]:
    0    1
    1    2
    2    3
    dtype: int64

    In [4]: s.index
    Out[4]: Float64Index([0.0, 1.0, 2.0], dtype='float64')

    In [5]: print(s.to_csv(path=None))
    0,1
    1,2
    2,3

New behavior:

    In [40]: s = pd.Series([1, 2, 3], index=np.arange(3.))

    In [41]: s
    Out[41]:
    0.0    1
    1.0    2
    2.0    3
    Length: 3, dtype: int64

    In [42]: s.index
    Out[42]: Index([0.0, 1.0, 2.0], dtype='float64')

    In [43]: print(s.to_csv(path_or_buf=None, header=False))
    0.0,1
    1.0,2
    2.0,3

### Changes to dtype assignment behaviors

When a DataFrame’s slice is updated with a new slice of the same dtype, the dtype of the DataFrame will now remain the same. (GH 10503)

Previous behavior:

    In [5]: df = pd.DataFrame({'a': [0, 1, 1],
                               'b': pd.Series([100, 200, 300], dtype='uint32')})

    In [7]: df.dtypes
    Out[7]:
    a     int64
    b    uint32
    dtype: object

    In [8]: ix = df['a'] == 1

    In [9]: df.loc[ix, 'b'] = df.loc[ix, 'b']

    In [11]: df.dtypes
    Out[11]:
    a    int64
    b    int64
    dtype: object

New behavior:

    In [44]: df = pd.DataFrame({'a': [0, 1, 1],
       ....:                    'b': pd.Series([100, 200, 300], dtype='uint32')})
       ....:

    In [45]: df.dtypes
    Out[45]:
    a     int64
    b    uint32
    Length: 2, dtype: object

    In [46]: ix = df['a'] == 1

    In [47]: df.loc[ix, 'b'] = df.loc[ix, 'b']

    In [48]: df.dtypes
    Out[48]:
    a     int64
    b    uint32
    Length: 2, dtype: object

When a DataFrame’s integer slice is partially updated with a new slice of floats that could potentially be down-casted to integer without losing precision, the dtype of the slice will be set to float instead of integer.

Previous behavior:

    In [4]: df = pd.DataFrame(np.array(range(1,10)).reshape(3,3),
                              columns=list('abc'),
                              index=[[4,4,8], [8,10,12]])

    In [5]: df
    Out[5]:
          a  b  c
    4 8   1  2  3
      10  4  5  6
    8 12  7  8  9

    In [7]: df.ix[4, 'c'] = np.array([0., 1.])

    In [8]: df
    Out[8]:
          a  b  c
    4 8   1  2  0
      10  4  5  1
    8 12  7  8  9

New behavior:

    In [49]: df = pd.DataFrame(np.array(range(1,10)).reshape(3,3),
       ....:                   columns=list('abc'),
       ....:                   index=[[4,4,8], [8,10,12]])
       ....:

    In [50]: df
    Out[50]:
          a  b  c
    4 8   1  2  3
      10  4  5  6
    8 12  7  8  9

    [3 rows x 3 columns]

    In [51]: df.loc[4, 'c'] = np.array([0., 1.])

    In [52]: df
    Out[52]:
          a  b  c
    4 8   1  2  0
      10  4  5  1
    8 12  7  8  9

    [3 rows x 3 columns]

### Method to_xarray

In a future version of pandas, we will be deprecating `Panel` and other > 2 ndim objects. In order to provide for continuity, all `NDFrame` objects have gained the `.to_xarray()` method in order to convert to `xarray` objects, which has a pandas-like interface for > 2 ndim. (GH 11972)

See the xarray full-documentation here.

    In [1]: p = Panel(np.arange(2*3*4).reshape(2,3,4))

    In [2]: p.to_xarray()
    Out[2]:
    <xarray.DataArray (items: 2, major_axis: 3, minor_axis: 4)>
    array([[[ 0,  1,  2,  3],
            [ 4,  5,  6,  7],
            [ 8,  9, 10, 11]],

           [[12, 13, 14, 15],
            [16, 17, 18, 19],
            [20, 21, 22, 23]]])
    Coordinates:
      * items       (items) int64 0 1
      * major_axis  (major_axis) int64 0 1 2
      * minor_axis  (minor_axis) int64 0 1 2 3

### Latex representation

`DataFrame` has gained a `._repr_latex_()` method in order to allow for conversion to latex in a ipython/jupyter notebook using nbconvert. (GH 11778)

Note that this must be activated by setting the option `pd.display.latex.repr=True` (GH 12182)

For example, if you have a jupyter notebook you plan to convert to latex using nbconvert, place the statement `pd.display.latex.repr=True` in the first cell to have the contained DataFrame output also stored as latex.

The options `display.latex.escape` and `display.latex.longtable` have also been added to the configuration and are used automatically by the `to_latex` method. See the available options docs for more info.

### `pd.read_sas()` changes

`read_sas` has gained the ability to read SAS7BDAT files, including compressed files. The files can be read in entirety, or incrementally. For full details see here. (GH 4052)

### Other enhancements

- Handle truncated floats in SAS xport files (GH 11713)

- Added option to hide index in `Series.to_string` (GH 11729)

- `read_excel` now supports s3 urls of the format `s3://bucketname/filename` (GH 11447)

- add support for `AWS_S3_HOST` env variable when reading from s3 (GH 12198)

- A simple version of `Panel.round()` is now implemented (GH 11763)

- For Python 3.x, `round(DataFrame)`, `round(Series)`, `round(Panel)` will work (GH 11763)

- `sys.getsizeof(obj)` returns the memory usage of a pandas object, including the values it contains (GH 11597)

- `Series` gained an `is_unique` attribute (GH 11946)

- `DataFrame.quantile` and `Series.quantile` now accept `interpolation` keyword (GH 10174).

- Added `DataFrame.style.format` for more flexible formatting of cell values (GH 11692)

- `DataFrame.select_dtypes` now allows the `np.float16` type code (GH 11990)

- `pivot_table()` now accepts most iterables for the `values` parameter (GH 12017)

- Added Google `BigQuery` service account authentication support, which enables authentication on remote servers. (GH 11881, GH 12572). For further details see here

- `HDFStore` is now iterable: `for k in store` is equivalent to `for k in store.keys()` (GH 12221).

- Add missing methods/fields to `.dt` for `Period` (GH 8848)

- The entire code base has been `PEP`-ified (GH 12096)

## Backwards incompatible API changes

- the leading white spaces have been removed from the output of `.to_string(index=False)` method (GH 11833)

- the `out` parameter has been removed from the `Series.round()` method. (GH 11763)

- `DataFrame.round()` leaves non-numeric columns unchanged in its return, rather than raises. (GH 11885)

- `DataFrame.head(0)` and `DataFrame.tail(0)` return empty frames, rather than `self`. (GH 11937)

- `Series.head(0)` and `Series.tail(0)` return empty series, rather than `self`. (GH 11937)

- `to_msgpack` and `read_msgpack` encoding now defaults to `'utf-8'`. (GH 12170)

- the order of keyword arguments to text file parsing functions (`.read_csv()`, `.read_table()`, `.read_fwf()`) changed to group related arguments. (GH 11555)

- `NaTType.isoformat` now returns the string `'NaT` to allow the result to be passed to the constructor of `Timestamp`. (GH 12300)

### NaT and Timedelta operations

`NaT` and `Timedelta` have expanded arithmetic operations, which are extended to `Series` arithmetic where applicable. Operations defined for `datetime64[ns]` or `timedelta64[ns]` are now also defined for `NaT` (GH 11564).

`NaT` now supports arithmetic operations with integers and floats.

    In [53]: pd.NaT * 1
    Out[53]: NaT

    In [54]: pd.NaT * 1.5
    Out[54]: NaT

    In [55]: pd.NaT / 2
    Out[55]: NaT

    In [56]: pd.NaT * np.nan
    Out[56]: NaT

`NaT` defines more arithmetic operations with `datetime64[ns]` and `timedelta64[ns]`.

    In [57]: pd.NaT / pd.NaT
    Out[57]: nan

    In [58]: pd.Timedelta('1s') / pd.NaT
    Out[58]: nan

`NaT` may represent either a `datetime64[ns]` null or a `timedelta64[ns]` null. Given the ambiguity, it is treated as a `timedelta64[ns]`, which allows more operations to succeed.

    In [59]: pd.NaT + pd.NaT
    Out[59]: NaT

    # same as
    In [60]: pd.Timedelta('1s') + pd.Timedelta('1s')
    Out[60]: Timedelta('0 days 00:00:02')

as opposed to

    In [3]: pd.Timestamp('19900315') + pd.Timestamp('19900315')
    TypeError: unsupported operand type(s) for +: 'Timestamp' and 'Timestamp'

However, when wrapped in a `Series` whose `dtype` is `datetime64[ns]` or `timedelta64[ns]`, the `dtype` information is respected.

    In [1]: pd.Series([pd.NaT], dtype='<M8[ns]') + pd.Series([pd.NaT], dtype='<M8[ns]')
    TypeError: can only operate on a datetimes for subtraction,
               but the operator [__add__] was passed

    In [61]: pd.Series([pd.NaT], dtype='<m8[ns]') + pd.Series([pd.NaT], dtype='<m8[ns]')
    Out[61]:
    0   NaT
    Length: 1, dtype: timedelta64[ns]

`Timedelta` division by `floats` now works.

    In [62]: pd.Timedelta('1s') / 2.0
    Out[62]: Timedelta('0 days 00:00:00.500000')

Subtraction by `Timedelta` in a `Series` by a `Timestamp` works (GH 11925)

    In [63]: ser = pd.Series(pd.timedelta_range('1 day', periods=3))

    In [64]: ser
    Out[64]:
    0   1 days
    1   2 days
    2   3 days
    Length: 3, dtype: timedelta64[us]

    In [65]: pd.Timestamp('2012-01-01') - ser
    Out[65]:
    0   2011-12-31
    1   2011-12-30
    2   2011-12-29
    Length: 3, dtype: datetime64[us]

`NaT.isoformat()` now returns `'NaT'`. This change allows `pd.Timestamp` to rehydrate any timestamp like object from its isoformat (GH 12300).

### Changes to msgpack

Forward incompatible changes in `msgpack` writing format were made over 0.17.0 and 0.18.0; older versions of pandas cannot read files packed by newer versions (GH 12129, GH 10527)

Bugs in `to_msgpack` and `read_msgpack` introduced in 0.17.0 and fixed in 0.18.0, caused files packed in Python 2 unreadable by Python 3 (GH 12142). The following table describes the backward and forward compat of msgpacks.

Warning

Packed with | Can be unpacked with
--- | ---
pre-0.17 / Python 2 | any
pre-0.17 / Python 3 | any
0.17 / Python 2 | ==0.17 / Python 2 >=0.18 / any Python
0.17 / Python 3 | >=0.18 / any Python
0.18 | >= 0.18

0.18.0 is backward-compatible for reading files packed by older versions, except for files packed with 0.17 in Python 2, in which case only they can only be unpacked in Python 2.

### Signature change for .rank

`Series.rank` and `DataFrame.rank` now have the same signature (GH 11759)

Previous signature

    In [3]: pd.Series([0,1]).rank(method='average', na_option='keep',
                                  ascending=True, pct=False)
    Out[3]:
    0    1
    1    2
    dtype: float64

    In [4]: pd.DataFrame([0,1]).rank(axis=0, numeric_only=None,
                                     method='average', na_option='keep',
                                     ascending=True, pct=False)
    Out[4]:
       0
    0  1
    1  2

New signature

    In [66]: pd.Series([0,1]).rank(axis=0, method='average', numeric_only=False,
       ....:                       na_option='keep', ascending=True, pct=False)
       ....:
    Out[66]:
    0    1.0
    1    2.0
    Length: 2, dtype: float64

    In [67]: pd.DataFrame([0,1]).rank(axis=0, method='average', numeric_only=False,
       ....:                          na_option='keep', ascending=True, pct=False)
       ....:
    Out[67]:
         0
    0  1.0
    1  2.0

    [2 rows x 1 columns]

### Bug in QuarterBegin with n=0

In previous versions, the behavior of the QuarterBegin offset was inconsistent depending on the date when the `n` parameter was 0. (GH 11406)

The general semantics of anchored offsets for `n=0` is to not move the date when it is an anchor point (e.g., a quarter start date), and otherwise roll forward to the next anchor point.

    In [68]: d = pd.Timestamp('2014-02-01')

    In [69]: d
    Out[69]: Timestamp('2014-02-01 00:00:00')

    In [70]: d + pd.offsets.QuarterBegin(n=0, startingMonth=2)
    Out[70]: Timestamp('2014-02-01 00:00:00')

    In [71]: d + pd.offsets.QuarterBegin(n=0, startingMonth=1)
    Out[71]: Timestamp('2014-04-01 00:00:00')

For the `QuarterBegin` offset in previous versions, the date would be rolled backwards if date was in the same month as the quarter start date.

    In [3]: d = pd.Timestamp('2014-02-15')

    In [4]: d + pd.offsets.QuarterBegin(n=0, startingMonth=2)
    Out[4]: Timestamp('2014-02-01 00:00:00')

This behavior has been corrected in version 0.18.0, which is consistent with other anchored offsets like `MonthBegin` and `YearBegin`.

    In [72]: d = pd.Timestamp('2014-02-15')

    In [73]: d + pd.offsets.QuarterBegin(n=0, startingMonth=2)
    Out[73]: Timestamp('2014-05-01 00:00:00')

### Resample API

Like the change in the window functions API above, `.resample(...)` is changing to have a more groupby-like API. (GH 11732, GH 12702, GH 12202, GH 12332, GH 12334, GH 12348, GH 12448).

    In [74]: np.random.seed(1234)

    In [75]: df = pd.DataFrame(np.random.rand(10,4),
       ....:                   columns=list('ABCD'),
       ....:                   index=pd.date_range('2010-01-01 09:00:00',
       ....:                                       periods=10, freq='s'))
       ....:

    In [76]: df
    Out[76]:
                                A         B         C         D
    2010-01-01 09:00:00  0.191519  0.622109  0.437728  0.785359
    2010-01-01 09:00:01  0.779976  0.272593  0.276464  0.801872
    2010-01-01 09:00:02  0.958139  0.875933  0.357817  0.500995
    2010-01-01 09:00:03  0.683463  0.712702  0.370251  0.561196
    2010-01-01 09:00:04  0.503083  0.013768  0.772827  0.882641
    2010-01-01 09:00:05  0.364886  0.615396  0.075381  0.368824
    2010-01-01 09:00:06  0.933140  0.651378  0.397203  0.788730
    2010-01-01 09:00:07  0.316836  0.568099  0.869127  0.436173
    2010-01-01 09:00:08  0.802148  0.143767  0.704261  0.704581
    2010-01-01 09:00:09  0.218792  0.924868  0.442141  0.909316

    [10 rows x 4 columns]

Previous API:

You would write a resampling operation that immediately evaluates. If a `how` parameter was not provided, it would default to `how='mean'`.

    In [6]: df.resample('2s')
    Out[6]:
                             A         B         C         D
    2010-01-01 09:00:00  0.485748  0.447351  0.357096  0.793615
    2010-01-01 09:00:02  0.820801  0.794317  0.364034  0.531096
    2010-01-01 09:00:04  0.433985  0.314582  0.424104  0.625733
    2010-01-01 09:00:06  0.624988  0.609738  0.633165  0.612452
    2010-01-01 09:00:08  0.510470  0.534317  0.573201  0.806949

You could also specify a `how` directly

    In [7]: df.resample('2s', how='sum')
    Out[7]:
                             A         B         C         D
    2010-01-01 09:00:00  0.971495  0.894701  0.714192  1.587231
    2010-01-01 09:00:02  1.641602  1.588635  0.728068  1.062191
    2010-01-01 09:00:04  0.867969  0.629165  0.848208  1.251465
    2010-01-01 09:00:06  1.249976  1.219477  1.266330  1.224904
    2010-01-01 09:00:08  1.020940  1.068634  1.146402  1.613897

New API:

Now, you can write `.resample(..)` as a 2-stage operation like `.groupby(...)`, which yields a `Resampler`.

    In [77]: r = df.resample('2s')

    In [78]: r
    Out[78]: <pandas.core.resample.DatetimeIndexResampler object at 0x7f2be69479d0>

#### Downsampling

You can then use this object to perform operations. These are downsampling operations (going from a higher frequency to a lower one).

    In [79]: r.mean()
    Out[79]:
                                A         B         C         D
    2010-01-01 09:00:00  0.485748  0.447351  0.357096  0.793615
    2010-01-01 09:00:02  0.820801  0.794317  0.364034  0.531096
    2010-01-01 09:00:04  0.433985  0.314582  0.424104  0.625733
    2010-01-01 09:00:06  0.624988  0.609738  0.633165  0.612452
    2010-01-01 09:00:08  0.510470  0.534317  0.573201  0.806949

    [5 rows x 4 columns]

    In [80]: r.sum()
    Out[80]:
                                A         B         C         D
    2010-01-01 09:00:00  0.971495  0.894701  0.714192  1.587231
    2010-01-01 09:00:02  1.641602  1.588635  0.728068  1.062191
    2010-01-01 09:00:04  0.867969  0.629165  0.848208  1.251465
    2010-01-01 09:00:06  1.249976  1.219477  1.266330  1.224904
    2010-01-01 09:00:08  1.020940  1.068634  1.146402  1.613897

    [5 rows x 4 columns]

Furthermore, resample now supports `getitem` operations to perform the resample on specific columns.

    In [81]: r[['A','C']].mean()
    Out[81]:
                                A         C
    2010-01-01 09:00:00  0.485748  0.357096
    2010-01-01 09:00:02  0.820801  0.364034
    2010-01-01 09:00:04  0.433985  0.424104
    2010-01-01 09:00:06  0.624988  0.633165
    2010-01-01 09:00:08  0.510470  0.573201

    [5 rows x 2 columns]

and `.aggregate` type operations.

    In [82]: r.agg({'A' : 'mean', 'B' : 'sum'})
    Out[82]:
                                A         B
    2010-01-01 09:00:00  0.485748  0.894701
    2010-01-01 09:00:02  0.820801  1.588635
    2010-01-01 09:00:04  0.433985  0.629165
    2010-01-01 09:00:06  0.624988  1.219477
    2010-01-01 09:00:08  0.510470  1.068634

    [5 rows x 2 columns]

These accessors can of course, be combined

    In [83]: r[['A','B']].agg(['mean','sum'])
    Out[83]:
                                A                   B
                             mean       sum      mean       sum
    2010-01-01 09:00:00  0.485748  0.971495  0.447351  0.894701
    2010-01-01 09:00:02  0.820801  1.641602  0.794317  1.588635
    2010-01-01 09:00:04  0.433985  0.867969  0.314582  0.629165
    2010-01-01 09:00:06  0.624988  1.249976  0.609738  1.219477
    2010-01-01 09:00:08  0.510470  1.020940  0.534317  1.068634

    [5 rows x 4 columns]

#### Upsampling

Upsampling operations take you from a lower frequency to a higher frequency. These are now performed with the `Resampler` objects with `backfill()`, `ffill()`, `fillna()` and `asfreq()` methods.

    In [89]: s = pd.Series(np.arange(5, dtype='int64'),
                  index=pd.date_range('2010-01-01', periods=5, freq='Q'))

    In [90]: s
    Out[90]:
    2010-03-31    0
    2010-06-30    1
    2010-09-30    2
    2010-12-31    3
    2011-03-31    4
    Freq: Q-DEC, Length: 5, dtype: int64

Previously

    In [6]: s.resample('M', fill_method='ffill')
    Out[6]:
    2010-03-31    0
    2010-04-30    0
    2010-05-31    0
    2010-06-30    1
    2010-07-31    1
    2010-08-31    1
    2010-09-30    2
    2010-10-31    2
    2010-11-30    2
    2010-12-31    3
    2011-01-31    3
    2011-02-28    3
    2011-03-31    4
    Freq: M, dtype: int64

New API

    In [91]: s.resample('M').ffill()
    Out[91]:
    2010-03-31    0
    2010-04-30    0
    2010-05-31    0
    2010-06-30    1
    2010-07-31    1
    2010-08-31    1
    2010-09-30    2
    2010-10-31    2
    2010-11-30    2
    2010-12-31    3
    2011-01-31    3
    2011-02-28    3
    2011-03-31    4
    Freq: M, Length: 13, dtype: int64

Note

In the new API, you can either downsample OR upsample. The prior implementation would allow you to pass an aggregator function (like `mean`) even though you were upsampling, providing a bit of confusion.

#### Previous API will work but with deprecations

Warning

This new API for resample includes some internal changes for the prior-to-0.18.0 API, to work with a deprecation warning in most cases, as the resample operation returns a deferred object. We can intercept operations and just do what the (pre 0.18.0) API did (with a warning). Here is a typical use case:

    In [4]: r = df.resample('2s')

    In [6]: r*10
    pandas/tseries/resample.py:80: FutureWarning: .resample() is now a deferred operation
    use .resample(...).mean() instead of .resample(...)

    Out[6]:
                          A         B         C         D
    2010-01-01 09:00:00  4.857476  4.473507  3.570960  7.936154
    2010-01-01 09:00:02  8.208011  7.943173  3.640340  5.310957
    2010-01-01 09:00:04  4.339846  3.145823  4.241039  6.257326
    2010-01-01 09:00:06  6.249881  6.097384  6.331650  6.124518
    2010-01-01 09:00:08  5.104699  5.343172  5.732009  8.069486

However, getting and assignment operations directly on a `Resampler` will raise a `ValueError`:

    In [7]: r.iloc[0] = 5
    ValueError: .resample() is now a deferred operation
    use .resample(...).mean() instead of .resample(...)

There is a situation where the new API can not perform all the operations when using original code. This code is intending to resample every 2s, take the `mean` AND then take the `min` of those results.

    In [4]: df.resample('2s').min()
    Out[4]:
    A    0.433985
    B    0.314582
    C    0.357096
    D    0.531096
    dtype: float64

The new API will:

    In [84]: df.resample('2s').min()
    Out[84]:
                                A         B         C         D
    2010-01-01 09:00:00  0.191519  0.272593  0.276464  0.785359
    2010-01-01 09:00:02  0.683463  0.712702  0.357817  0.500995
    2010-01-01 09:00:04  0.364886  0.013768  0.075381  0.368824
    2010-01-01 09:00:06  0.316836  0.568099  0.397203  0.436173
    2010-01-01 09:00:08  0.218792  0.143767  0.442141  0.704581

    [5 rows x 4 columns]

The good news is the return dimensions will differ between the new API and the old API, so this should loudly raise an exception.

To replicate the original operation

    In [85]: df.resample('2s').mean().min()
    Out[85]:
    A    0.433985
    B    0.314582
    C    0.357096
    D    0.531096
    Length: 4, dtype: float64

### Changes to eval

In prior versions, new columns assignments in an `eval` expression resulted in an inplace change to the `DataFrame`. (GH 9297, GH 8664, GH 10486)

    In [86]: df = pd.DataFrame({'a': np.linspace(0, 10, 5), 'b': range(5)})

    In [87]: df
    Out[87]:
          a  b
    0   0.0  0
    1   2.5  1
    2   5.0  2
    3   7.5  3
    4  10.0  4

    [5 rows x 2 columns]

    In [12]: df.eval('c = a + b')
    FutureWarning: eval expressions containing an assignment currentlydefault to operating inplace.
    This will change in a future version of pandas, use inplace=True to avoid this warning.

    In [13]: df
    Out[13]:
          a  b     c
    0   0.0  0   0.0
    1   2.5  1   3.5
    2   5.0  2   7.0
    3   7.5  3  10.5
    4  10.0  4  14.0

In version 0.18.0, a new `inplace` keyword was added to choose whether the assignment should be done inplace or return a copy.

    In [88]: df
    Out[88]:
          a  b     c
    0   0.0  0   0.0
    1   2.5  1   3.5
    2   5.0  2   7.0
    3   7.5  3  10.5
    4  10.0  4  14.0

    [5 rows x 3 columns]

    In [89]: df.eval('d = c - b', inplace=False)
    Out[89]:
          a  b     c     d
    0   0.0  0   0.0   0.0
    1   2.5  1   3.5   2.5
    2   5.0  2   7.0   5.0
    3   7.5  3  10.5   7.5
    4  10.0  4  14.0  10.0

    [5 rows x 4 columns]

    In [90]: df
    Out[90]:
          a  b     c
    0   0.0  0   0.0
    1   2.5  1   3.5
    2   5.0  2   7.0
    3   7.5  3  10.5
    4  10.0  4  14.0

    [5 rows x 3 columns]

    In [91]: df.eval('d = c - b', inplace=True)

    In [92]: df
    Out[92]:
          a  b     c     d
    0   0.0  0   0.0   0.0
    1   2.5  1   3.5   2.5
    2   5.0  2   7.0   5.0
    3   7.5  3  10.5   7.5
    4  10.0  4  14.0  10.0

    [5 rows x 4 columns]

Warning

For backwards compatibility, `inplace` defaults to `True` if not specified. This will change in a future version of pandas. If your code depends on an inplace assignment you should update to explicitly set `inplace=True`

The `inplace` keyword parameter was also added the `query` method.

    In [93]: df.query('a > 5')
    Out[93]:
          a  b     c     d
    3   7.5  3  10.5   7.5
    4  10.0  4  14.0  10.0

    [2 rows x 4 columns]

    In [94]: df.query('a > 5', inplace=True)

    In [95]: df
    Out[95]:
          a  b     c     d
    3   7.5  3  10.5   7.5
    4  10.0  4  14.0  10.0

    [2 rows x 4 columns]

Warning

Note that the default value for `inplace` in a `query` is `False`, which is consistent with prior versions.

`eval` has also been updated to allow multi-line expressions for multiple assignments. These expressions will be evaluated one at a time in order. Only assignments are valid for multi-line expressions.

    In [96]: df
    Out[96]:
          a  b     c     d
    3   7.5  3  10.5   7.5
    4  10.0  4  14.0  10.0

    [2 rows x 4 columns]

    In [97]: df.eval("""
       ....: e = d + a
       ....: f = e - 22
       ....: g = f / 2.0""", inplace=True)
       ....:

    In [98]: df
    Out[98]:
          a  b     c     d     e    f    g
    3   7.5  3  10.5   7.5  15.0 -7.0 -3.5
    4  10.0  4  14.0  10.0  20.0 -2.0 -1.0

    [2 rows x 7 columns]

### Other API changes

- `DataFrame.between_time` and `Series.between_time` now only parse a fixed set of time strings. Parsing of date strings is no longer supported and raises a `ValueError`. (GH 11818)

      In [107]: s = pd.Series(range(10), pd.date_range('2015-01-01', freq='H', periods=10))

      In [108]: s.between_time("7:00am", "9:00am")
      Out[108]:
      2015-01-01 07:00:00    7
      2015-01-01 08:00:00    8
      2015-01-01 09:00:00    9
      Freq: H, Length: 3, dtype: int64

  This will now raise.

      In [2]: s.between_time('20150101 07:00:00','20150101 09:00:00')
      ValueError: Cannot convert arg ['20150101 07:00:00'] to a time.

- `.memory_usage()` now includes values in the index, as does memory_usage in `.info()` (GH 11597)

- `DataFrame.to_latex()` now supports non-ascii encodings (eg `utf-8`) in Python 2 with the parameter `encoding` (GH 7061)

- `pandas.merge()` and `DataFrame.merge()` will show a specific error message when trying to merge with an object that is not of type `DataFrame` or a subclass (GH 12081)

- `DataFrame.unstack` and `Series.unstack` now take `fill_value` keyword to allow direct replacement of missing values when an unstack results in missing values in the resulting `DataFrame`. As an added benefit, specifying `fill_value` will preserve the data type of the original stacked data. (GH 9746)

- As part of the new API for window functions and resampling, aggregation functions have been clarified, raising more informative error messages on invalid aggregations. (GH 9052). A full set of examples are presented in groupby.

- Statistical functions for `NDFrame` objects (like `sum(), mean(), min()`) will now raise if non-numpy-compatible arguments are passed in for `**kwargs` (GH 12301)

- `.to_latex` and `.to_html` gain a `decimal` parameter like `.to_csv`; the default is `'.'` (GH 12031)

- More helpful error message when constructing a `DataFrame` with empty data but with indices (GH 8020)

- `.describe()` will now properly handle bool dtype as a categorical (GH 6625)

- More helpful error message with an invalid `.transform` with user defined input (GH 10165)

- Exponentially weighted functions now allow specifying alpha directly (GH 10789) and raise `ValueError` if parameters violate `0 < alpha <= 1` (GH 12492)

### Deprecations

- The functions `pd.rolling_*`, `pd.expanding_*`, and `pd.ewm*` are deprecated and replaced by the corresponding method call. Note that the new suggested syntax includes all of the arguments (even if default) (GH 11603)

      In [1]: s = pd.Series(range(3))

      In [2]: pd.rolling_mean(s,window=2,min_periods=1)
              FutureWarning: pd.rolling_mean is deprecated for Series and
                   will be removed in a future version, replace with
                   Series.rolling(min_periods=1,window=2,center=False).mean()
      Out[2]:
              0    0.0
              1    0.5
              2    1.5
              dtype: float64

      In [3]: pd.rolling_cov(s, s, window=2)
              FutureWarning: pd.rolling_cov is deprecated for Series and
                   will be removed in a future version, replace with
                   Series.rolling(window=2).cov(other=<Series>)
      Out[3]:
              0    NaN
              1    0.5
              2    0.5
              dtype: float64

- The `freq` and `how` arguments to the `.rolling`, `.expanding`, and `.ewm` (new) functions are deprecated, and will be removed in a future version. You can simply resample the input prior to creating a window function. (GH 11603).

  For example, instead of `s.rolling(window=5,freq='D').max()` to get the max value on a rolling 5 Day window, one could use `s.resample('D').mean().rolling(window=5).max()`, which first resamples the data to daily data, then provides a rolling 5 day window.

- `pd.tseries.frequencies.get_offset_name` function is deprecated. Use offset’s `.freqstr` property as alternative (GH 11192)

- `pandas.stats.fama_macbeth` routines are deprecated and will be removed in a future version (GH 6077)

- `pandas.stats.ols`, `pandas.stats.plm` and `pandas.stats.var` routines are deprecated and will be removed in a future version (GH 6077)

- show a `FutureWarning` rather than a `DeprecationWarning` on using long-time deprecated syntax in `HDFStore.select`, where the `where` clause is not a string-like (GH 12027)

- The `pandas.options.display.mpl_style` configuration has been deprecated and will be removed in a future version of pandas. This functionality is better handled by matplotlib’s style sheets (GH 11783).

### Removal of deprecated float indexers

In GH 4892 indexing with floating point numbers on a non-`Float64Index` was deprecated (in version 0.14.0). In 0.18.0, this deprecation warning is removed and these will now raise a `TypeError`. (GH 12165, GH 12333)

    In [99]: s = pd.Series([1, 2, 3], index=[4, 5, 6])

    In [100]: s
    Out[100]:
    4    1
    5    2
    6    3
    Length: 3, dtype: int64

    In [101]: s2 = pd.Series([1, 2, 3], index=list('abc'))

    In [102]: s2
    Out[102]:
    a    1
    b    2
    c    3
    Length: 3, dtype: int64

Previous behavior:

    # this is label indexing
    In [2]: s[5.0]
    FutureWarning: scalar indexers for index type Int64Index should be integers and not floating point
    Out[2]: 2

    # this is positional indexing
    In [3]: s.iloc[1.0]
    FutureWarning: scalar indexers for index type Int64Index should be integers and not floating point
    Out[3]: 2

    # this is label indexing
    In [4]: s.loc[5.0]
    FutureWarning: scalar indexers for index type Int64Index should be integers and not floating point
    Out[4]: 2

    # .ix would coerce 1.0 to the positional 1, and index
    In [5]: s2.ix[1.0] = 10
    FutureWarning: scalar indexers for index type Index should be integers and not floating point

    In [6]: s2
    Out[6]:
    a     1
    b    10
    c     3
    dtype: int64

New behavior:

For iloc, getting & setting via a float scalar will always raise.

    In [3]: s.iloc[2.0]
    TypeError: cannot do label indexing on <class 'pandas.indexes.numeric.Int64Index'> with these indexers [2.0] of <type 'float'>

Other indexers will coerce to a like integer for both getting and setting. The `FutureWarning` has been dropped for `.loc`, `.ix` and `[]`.

    In [103]: s[5.0]
    Out[103]: np.int64(2)

    In [104]: s.loc[5.0]
    Out[104]: np.int64(2)

and setting

    In [105]: s_copy = s.copy()

    In [106]: s_copy[5.0] = 10

    In [107]: s_copy
    Out[107]:
    4     1
    5    10
    6     3
    Length: 3, dtype: int64

    In [108]: s_copy = s.copy()

    In [109]: s_copy.loc[5.0] = 10

    In [110]: s_copy
    Out[110]:
    4     1
    5    10
    6     3
    Length: 3, dtype: int64

Positional setting with `.ix` and a float indexer will ADD this value to the index, rather than previously setting the value by position.

    In [3]: s2.ix[1.0] = 10
    In [4]: s2
    Out[4]:
    a       1
    b       2
    c       3
    1.0    10
    dtype: int64

Slicing will also coerce integer-like floats to integers for a non-`Float64Index`.

    In [111]: s.loc[5.0:6]
    Out[111]:
    5    2
    6    3
    Length: 2, dtype: int64

Note that for floats that are NOT coercible to ints, the label based bounds will be excluded

    In [112]: s.loc[5.1:6]
    Out[112]:
    6    3
    Length: 1, dtype: int64

Float indexing on a `Float64Index` is unchanged.

    In [113]: s = pd.Series([1, 2, 3], index=np.arange(3.))

    In [114]: s[1.0]
    Out[114]: np.int64(2)

    In [115]: s[1.0:2.5]
    Out[115]:
    1.0    2
    2.0    3
    Length: 2, dtype: int64

### Removal of prior version deprecations/changes

- Removal of `rolling_corr_pairwise` in favor of `.rolling().corr(pairwise=True)` (GH 4950)

- Removal of `expanding_corr_pairwise` in favor of `.expanding().corr(pairwise=True)` (GH 4950)

- Removal of `DataMatrix` module. This was not imported into the pandas namespace in any event (GH 12111)

- Removal of `cols` keyword in favor of `subset` in `DataFrame.duplicated()` and `DataFrame.drop_duplicates()` (GH 6680)

- Removal of the `read_frame` and `frame_query` (both aliases for `pd.read_sql`) and `write_frame` (alias of `to_sql`) functions in the `pd.io.sql` namespace, deprecated since 0.14.0 (GH 6292).

- Removal of the `order` keyword from `.factorize()` (GH 6930)

## Performance improvements

- Improved performance of `andrews_curves` (GH 11534)

- Improved huge `DatetimeIndex`, `PeriodIndex` and `TimedeltaIndex`’s ops performance including `NaT` (GH 10277)

- Improved performance of `pandas.concat` (GH 11958)

- Improved performance of `StataReader` (GH 11591)

- Improved performance in construction of `Categoricals` with `Series` of datetimes containing `NaT` (GH 12077)

- Improved performance of ISO 8601 date parsing for dates without separators (GH 11899), leading zeros (GH 11871) and with white space preceding the time zone (GH 9714)

## Bug fixes

- Bug in `GroupBy.size` when data-frame is empty. (GH 11699)

- Bug in `Period.end_time` when a multiple of time period is requested (GH 11738)

- Regression in `.clip` with tz-aware datetimes (GH 11838)

- Bug in `date_range` when the boundaries fell on the frequency (GH 11804, GH 12409)

- Bug in consistency of passing nested dicts to `.groupby(...).agg(...)` (GH 9052)

- Accept unicode in `Timedelta` constructor (GH 11995)

- Bug in value label reading for `StataReader` when reading incrementally (GH 12014)

- Bug in vectorized `DateOffset` when `n` parameter is `0` (GH 11370)

- Compat for numpy 1.11 w.r.t. `NaT` comparison changes (GH 12049)

- Bug in `read_csv` when reading from a `StringIO` in threads (GH 11790)

- Bug in not treating `NaT` as a missing value in datetimelikes when factorizing & with `Categoricals` (GH 12077)

- Bug in getitem when the values of a `Series` were tz-aware (GH 12089)

- Bug in `Series.str.get_dummies` when one of the variables was ‘name’ (GH 12180)

- Bug in `pd.concat` while concatenating tz-aware NaT series. (GH 11693, GH 11755, GH 12217)

- Bug in `pd.read_stata` with version <= 108 files (GH 12232)

- Bug in `Series.resample` using a frequency of `Nano` when the index is a `DatetimeIndex` and contains non-zero nanosecond parts (GH 12037)

- Bug in resampling with `.nunique` and a sparse index (GH 12352)

- Removed some compiler warnings (GH 12471)

- Work around compat issues with `boto` in python 3.5 (GH 11915)

- Bug in `NaT` subtraction from `Timestamp` or `DatetimeIndex` with timezones (GH 11718)

- Bug in subtraction of `Series` of a single tz-aware `Timestamp` (GH 12290)

- Use compat iterators in PY2 to support `.next()` (GH 12299)

- Bug in `Timedelta.round` with negative values (GH 11690)

- Bug in `.loc` against `CategoricalIndex` may result in normal `Index` (GH 11586)

- Bug in `DataFrame.info` when duplicated column names exist (GH 11761)

- Bug in `.copy` of datetime tz-aware objects (GH 11794)

- Bug in `Series.apply` and `Series.map` where `timedelta64` was not boxed (GH 11349)

- Bug in `DataFrame.set_index()` with tz-aware `Series` (GH 12358)

- Bug in subclasses of `DataFrame` where `AttributeError` did not propagate (GH 11808)

- Bug groupby on tz-aware data where selection not returning `Timestamp` (GH 11616)

- Bug in `pd.read_clipboard` and `pd.to_clipboard` functions not supporting Unicode; upgrade included `pyperclip` to v1.5.15 (GH 9263)

- Bug in `DataFrame.query` containing an assignment (GH 8664)

- Bug in `from_msgpack` where `__contains__()` fails for columns of the unpacked `DataFrame`, if the `DataFrame` has object columns. (GH 11880)

- Bug in `.resample` on categorical data with `TimedeltaIndex` (GH 12169)

- Bug in timezone info lost when broadcasting scalar datetime to `DataFrame` (GH 11682)

- Bug in `Index` creation from `Timestamp` with mixed tz coerces to UTC (GH 11488)

- Bug in `to_numeric` where it does not raise if input is more than one dimension (GH 11776)

- Bug in parsing timezone offset strings with non-zero minutes (GH 11708)

- Bug in `df.plot` using incorrect colors for bar plots under matplotlib 1.5+ (GH 11614)

- Bug in the `groupby` `plot` method when using keyword arguments (GH 11805).

- Bug in `DataFrame.duplicated` and `drop_duplicates` causing spurious matches when setting `keep=False` (GH 11864)

- Bug in `.loc` result with duplicated key may have `Index` with incorrect dtype (GH 11497)

- Bug in `pd.rolling_median` where memory allocation failed even with sufficient memory (GH 11696)

- Bug in `DataFrame.style` with spurious zeros (GH 12134)

- Bug in `DataFrame.style` with integer columns not starting at 0 (GH 12125)

- Bug in `.style.bar` may not rendered properly using specific browser (GH 11678)

- Bug in rich comparison of `Timedelta` with a `numpy.array` of `Timedelta` that caused an infinite recursion (GH 11835)

- Bug in `DataFrame.round` dropping column index name (GH 11986)

- Bug in `df.replace` while replacing value in mixed dtype `Dataframe` (GH 11698)

- Bug in `Index` prevents copying name of passed `Index`, when a new name is not provided (GH 11193)

- Bug in `read_excel` failing to read any non-empty sheets when empty sheets exist and `sheetname=None` (GH 11711)

- Bug in `read_excel` failing to raise `NotImplemented` error when keywords `parse_dates` and `date_parser` are provided (GH 11544)

- Bug in `read_sql` with `pymysql` connections failing to return chunked data (GH 11522)

- Bug in `.to_csv` ignoring formatting parameters `decimal`, `na_rep`, `float_format` for float indexes (GH 11553)

- Bug in `Int64Index` and `Float64Index` preventing the use of the modulo operator (GH 9244)

- Bug in `MultiIndex.drop` for not lexsorted MultiIndexes (GH 12078)

- Bug in `DataFrame` when masking an empty `DataFrame` (GH 11859)

- Bug in `.plot` potentially modifying the `colors` input when the number of columns didn’t match the number of series provided (GH 12039).

- Bug in `Series.plot` failing when index has a `CustomBusinessDay` frequency (GH 7222).

- Bug in `.to_sql` for `datetime.time` values with sqlite fallback (GH 8341)

- Bug in `read_excel` failing to read data with one column when `squeeze=True` (GH 12157)

- Bug in `read_excel` failing to read one empty column (GH 12292, GH 9002)

- Bug in `.groupby` where a `KeyError` was not raised for a wrong column if there was only one row in the dataframe (GH 11741)

- Bug in `.read_csv` with dtype specified on empty data producing an error (GH 12048)

- Bug in `.read_csv` where strings like `'2E'` are treated as valid floats (GH 12237)

- Bug in building pandas with debugging symbols (GH 12123)

- Removed `millisecond` property of `DatetimeIndex`. This would always raise a `ValueError` (GH 12019).

- Bug in `Series` constructor with read-only data (GH 11502)

- Removed `pandas._testing.choice()`. Should use `np.random.choice()`, instead. (GH 12386)

- Bug in `.loc` setitem indexer preventing the use of a TZ-aware DatetimeIndex (GH 12050)

- Bug in `.style` indexes and MultiIndexes not appearing (GH 11655)

- Bug in `to_msgpack` and `from_msgpack` which did not correctly serialize or deserialize `NaT` (GH 12307).

- Bug in `.skew` and `.kurt` due to roundoff error for highly similar values (GH 11974)

- Bug in `Timestamp` constructor where microsecond resolution was lost if HHMMSS were not separated with ‘:’ (GH 10041)

- Bug in `buffer_rd_bytes` src->buffer could be freed more than once if reading failed, causing a segfault (GH 12098)

- Bug in `crosstab` where arguments with non-overlapping indexes would return a `KeyError` (GH 10291)

- Bug in `DataFrame.apply` in which reduction was not being prevented for cases in which `dtype` was not a numpy dtype (GH 12244)

- Bug when initializing categorical series with a scalar value. (GH 12336)

- Bug when specifying a UTC `DatetimeIndex` by setting `utc=True` in `.to_datetime` (GH 11934)

- Bug when increasing the buffer size of CSV reader in `read_csv` (GH 12494)

- Bug when setting columns of a `DataFrame` with duplicate column names (GH 12344)

## Contributors

A total of 101 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.

- ARF +

- Alex Alekseyev +

- Andrew McPherson +

- Andrew Rosenfeld

- Andy Hayden

- Anthonios Partheniou

- Anton I. Sipos

- Ben +

- Ben North +

- Bran Yang +

- Chris

- Chris Carroux +

- Christopher C. Aycock +

- Christopher Scanlin +

- Cody +

- Da Wang +

- Daniel Grady +

- Dorozhko Anton +

- Dr-Irv +

- Erik M. Bray +

- Evan Wright

- Francis T. O’Donovan +

- Frank Cleary +

- Gianluca Rossi

- Graham Jeffries +

- Guillaume Horel

- Henry Hammond +

- Isaac Schwabacher +

- Jean-Mathieu Deschenes

- Jeff Reback

- Joe Jevnik +

- John Freeman +

- John Fremlin +

- Jonas Hoersch +

- Joris Van den Bossche

- Joris Vankerschaver

- Justin Lecher

- Justin Lin +

- Ka Wo Chen

- Keming Zhang +

- Kerby Shedden

- Kyle +

- Marco Farrugia +

- MasonGallo +

- MattRijk +

- Matthew Lurie +

- Maximilian Roos

- Mayank Asthana +

- Mortada Mehyar

- Moussa Taifi +

- Navreet Gill +

- Nicolas Bonnotte

- Paul Reiners +

- Philip Gura +

- Pietro Battiston

- RahulHP +

- Randy Carnevale

- Rinoc Johnson

- Rishipuri +

- Sangmin Park +

- Scott E Lasley

- Sereger13 +

- Shannon Wang +

- Skipper Seabold

- Thierry Moisan

- Thomas A Caswell

- Toby Dylan Hocking +

- Tom Augspurger

- Travis +

- Trent Hauck

- Tux1

- Varun

- Wes McKinney

- Will Thompson +

- Yoav Ram

- Yoong Kang Lim +

- Yoshiki Vázquez Baeza

- Young Joong Kim +

- Younggun Kim

- Yuval Langer +

- alex argunov +

- behzad nouri

- boombard +

- brian-pantano +

- chromy +

- daniel +

- dgram0 +

- gfyoung +

- hack-c +

- hcontrast +

- jfoo +

- kaustuv deolal +

- llllllllll

- ranarag +

- rockg

- scls19fr

- seales +

- sinhrks

- srib +

- surveymedia.ca +

- tworec +