Skip to main content
`Ctrl`+`K`

# Version 0.19.0 (October 2, 2016)

This is a major release from 0.18.1 and includes number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. We recommend that all users upgrade to this version.

Highlights include:

- `merge_asof()` for asof-style time-series joining, see here

- `.rolling()` is now time-series aware, see here

- `read_csv()` now supports parsing `Categorical` data, see here

- A function `union_categorical()` has been added for combining categoricals, see here

- `PeriodIndex` now has its own `period` dtype, and changed to be more consistent with other `Index` classes. See here

- Sparse data structures gained enhanced support of `int` and `bool` dtypes, see here

- Comparison operations with `Series` no longer ignores the index, see here for an overview of the API changes.

- Introduction of a pandas development API for utility functions, see here.

- Deprecation of `Panel4D` and `PanelND`. We recommend to represent these types of n-dimensional data with the xarray package.

- Removal of the previously deprecated modules `pandas.io.data`, `pandas.io.wb`, `pandas.tools.rplot`.

Warning

pandas >= 0.19.0 will no longer silence numpy ufunc warnings upon import, see here.

## New features

### Function `merge_asof` for asof-style time-series joining

A long-time requested feature has been added through the `merge_asof()` function, to support asof style joining of time-series (GH 1870, GH 13695, GH 13709, GH 13902). Full documentation is here.

The `merge_asof()` performs an asof merge, which is similar to a left-join except that we match on nearest key rather than equal keys.

    In [1]: left = pd.DataFrame({"a": [1, 5, 10], "left_val": ["a", "b", "c"]})

    In [2]: right = pd.DataFrame({"a": [1, 2, 3, 6, 7], "right_val": [1, 2, 3, 6, 7]})

    In [3]: left
    Out[3]:
        a left_val
    0   1        a
    1   5        b
    2  10        c

    [3 rows x 2 columns]

    In [4]: right
    Out[4]:
       a  right_val
    0  1          1
    1  2          2
    2  3          3
    3  6          6
    4  7          7

    [5 rows x 2 columns]

We typically want to match exactly when possible, and use the most recent value otherwise.

    In [5]: pd.merge_asof(left, right, on="a")
    Out[5]:
        a left_val  right_val
    0   1        a          1
    1   5        b          3
    2  10        c          7

    [3 rows x 3 columns]

We can also match rows ONLY with prior data, and not an exact match.

    In [6]: pd.merge_asof(left, right, on="a", allow_exact_matches=False)
    Out[6]:
        a left_val  right_val
    0   1        a        NaN
    1   5        b        3.0
    2  10        c        7.0

    [3 rows x 3 columns]

In a typical time-series example, we have `trades` and `quotes` and we want to `asof-join` them. This also illustrates using the `by` parameter to group data before merging.

    In [7]: trades = pd.DataFrame(
       ...:     {
       ...:         "time": pd.to_datetime(
       ...:             [
       ...:                 "20160525 13:30:00.023",
       ...:                 "20160525 13:30:00.038",
       ...:                 "20160525 13:30:00.048",
       ...:                 "20160525 13:30:00.048",
       ...:                 "20160525 13:30:00.048",
       ...:             ]
       ...:         ),
       ...:         "ticker": ["MSFT", "MSFT", "GOOG", "GOOG", "AAPL"],
       ...:         "price": [51.95, 51.95, 720.77, 720.92, 98.00],
       ...:         "quantity": [75, 155, 100, 100, 100],
       ...:     },
       ...:     columns=["time", "ticker", "price", "quantity"],
       ...: )
       ...:

    In [8]: quotes = pd.DataFrame(
       ...:     {
       ...:         "time": pd.to_datetime(
       ...:             [
       ...:                 "20160525 13:30:00.023",
       ...:                 "20160525 13:30:00.023",
       ...:                 "20160525 13:30:00.030",
       ...:                 "20160525 13:30:00.041",
       ...:                 "20160525 13:30:00.048",
       ...:                 "20160525 13:30:00.049",
       ...:                 "20160525 13:30:00.072",
       ...:                 "20160525 13:30:00.075",
       ...:             ]
       ...:         ),
       ...:         "ticker": ["GOOG", "MSFT", "MSFT", "MSFT", "GOOG", "AAPL", "GOOG", "MSFT"],
       ...:         "bid": [720.50, 51.95, 51.97, 51.99, 720.50, 97.99, 720.50, 52.01],
       ...:         "ask": [720.93, 51.96, 51.98, 52.00, 720.93, 98.01, 720.88, 52.03],
       ...:     },
       ...:     columns=["time", "ticker", "bid", "ask"],
       ...: )
       ...:

    In [9]: trades
    Out[9]:
                         time ticker   price  quantity
    0 2016-05-25 13:30:00.023   MSFT   51.95        75
    1 2016-05-25 13:30:00.038   MSFT   51.95       155
    2 2016-05-25 13:30:00.048   GOOG  720.77       100
    3 2016-05-25 13:30:00.048   GOOG  720.92       100
    4 2016-05-25 13:30:00.048   AAPL   98.00       100

    [5 rows x 4 columns]

    In [10]: quotes
    Out[10]:
                         time ticker     bid     ask
    0 2016-05-25 13:30:00.023   GOOG  720.50  720.93
    1 2016-05-25 13:30:00.023   MSFT   51.95   51.96
    2 2016-05-25 13:30:00.030   MSFT   51.97   51.98
    3 2016-05-25 13:30:00.041   MSFT   51.99   52.00
    4 2016-05-25 13:30:00.048   GOOG  720.50  720.93
    5 2016-05-25 13:30:00.049   AAPL   97.99   98.01
    6 2016-05-25 13:30:00.072   GOOG  720.50  720.88
    7 2016-05-25 13:30:00.075   MSFT   52.01   52.03

    [8 rows x 4 columns]

An asof merge joins on the `on`, typically a datetimelike field, which is ordered, and in this case we are using a grouper in the `by` field. This is like a left-outer join, except that forward filling happens automatically taking the most recent non-NaN value.

    In [11]: pd.merge_asof(trades, quotes, on="time", by="ticker")
    Out[11]:
                         time ticker   price  quantity     bid     ask
    0 2016-05-25 13:30:00.023   MSFT   51.95        75   51.95   51.96
    1 2016-05-25 13:30:00.038   MSFT   51.95       155   51.97   51.98
    2 2016-05-25 13:30:00.048   GOOG  720.77       100  720.50  720.93
    3 2016-05-25 13:30:00.048   GOOG  720.92       100  720.50  720.93
    4 2016-05-25 13:30:00.048   AAPL   98.00       100     NaN     NaN

    [5 rows x 6 columns]

This returns a merged DataFrame with the entries in the same order as the original left passed DataFrame (`trades` in this case), with the fields of the `quotes` merged.

### Method `.rolling()` is now time-series aware

`.rolling()` objects are now time-series aware and can accept a time-series offset (or convertible) for the `window` argument (GH 13327, GH 12995). See the full documentation here.

    In [12]: dft = pd.DataFrame(
       ....:     {"B": [0, 1, 2, np.nan, 4]},
       ....:     index=pd.date_range("20130101 09:00:00", periods=5, freq="s"),
       ....: )
       ....:

    In [13]: dft
    Out[13]:
                           B
    2013-01-01 09:00:00  0.0
    2013-01-01 09:00:01  1.0
    2013-01-01 09:00:02  2.0
    2013-01-01 09:00:03  NaN
    2013-01-01 09:00:04  4.0

    [5 rows x 1 columns]

This is a regular frequency index. Using an integer window parameter works to roll along the window frequency.

    In [14]: dft.rolling(2).sum()
    Out[14]:
                           B
    2013-01-01 09:00:00  NaN
    2013-01-01 09:00:01  1.0
    2013-01-01 09:00:02  3.0
    2013-01-01 09:00:03  NaN
    2013-01-01 09:00:04  NaN

    [5 rows x 1 columns]

    In [15]: dft.rolling(2, min_periods=1).sum()
    Out[15]:
                           B
    2013-01-01 09:00:00  0.0
    2013-01-01 09:00:01  1.0
    2013-01-01 09:00:02  3.0
    2013-01-01 09:00:03  2.0
    2013-01-01 09:00:04  4.0

    [5 rows x 1 columns]

Specifying an offset allows a more intuitive specification of the rolling frequency.

    In [16]: dft.rolling("2s").sum()
    Out[16]:
                           B
    2013-01-01 09:00:00  0.0
    2013-01-01 09:00:01  1.0
    2013-01-01 09:00:02  3.0
    2013-01-01 09:00:03  2.0
    2013-01-01 09:00:04  4.0

    [5 rows x 1 columns]

Using a non-regular, but still monotonic index, rolling with an integer window does not impart any special calculation.

    In [17]: dft = pd.DataFrame(
       ....:     {"B": [0, 1, 2, np.nan, 4]},
       ....:     index=pd.Index(
       ....:         [
       ....:             pd.Timestamp("20130101 09:00:00"),
       ....:             pd.Timestamp("20130101 09:00:02"),
       ....:             pd.Timestamp("20130101 09:00:03"),
       ....:             pd.Timestamp("20130101 09:00:05"),
       ....:             pd.Timestamp("20130101 09:00:06"),
       ....:         ],
       ....:         name="foo",
       ....:     ),
       ....: )
       ....:

    In [18]: dft
    Out[18]:
                           B
    foo
    2013-01-01 09:00:00  0.0
    2013-01-01 09:00:02  1.0
    2013-01-01 09:00:03  2.0
    2013-01-01 09:00:05  NaN
    2013-01-01 09:00:06  4.0

    [5 rows x 1 columns]

    In [19]: dft.rolling(2).sum()
    Out[19]:
                           B
    foo
    2013-01-01 09:00:00  NaN
    2013-01-01 09:00:02  1.0
    2013-01-01 09:00:03  3.0
    2013-01-01 09:00:05  NaN
    2013-01-01 09:00:06  NaN

    [5 rows x 1 columns]

Using the time-specification generates variable windows for this sparse data.

    In [20]: dft.rolling("2s").sum()
    Out[20]:
                           B
    foo
    2013-01-01 09:00:00  0.0
    2013-01-01 09:00:02  1.0
    2013-01-01 09:00:03  3.0
    2013-01-01 09:00:05  NaN
    2013-01-01 09:00:06  4.0

    [5 rows x 1 columns]

Furthermore, we now allow an optional `on` parameter to specify a column (rather than the default of the index) in a DataFrame.

    In [21]: dft = dft.reset_index()

    In [22]: dft
    Out[22]:
                      foo    B
    0 2013-01-01 09:00:00  0.0
    1 2013-01-01 09:00:02  1.0
    2 2013-01-01 09:00:03  2.0
    3 2013-01-01 09:00:05  NaN
    4 2013-01-01 09:00:06  4.0

    [5 rows x 2 columns]

    In [23]: dft.rolling("2s", on="foo").sum()
    Out[23]:
                      foo    B
    0 2013-01-01 09:00:00  0.0
    1 2013-01-01 09:00:02  1.0
    2 2013-01-01 09:00:03  3.0
    3 2013-01-01 09:00:05  NaN
    4 2013-01-01 09:00:06  4.0

    [5 rows x 2 columns]

### Method `read_csv` has improved support for duplicate column names

Duplicate column names are now supported in `read_csv()` whether they are in the file or passed in as the `names` parameter (GH 7160, GH 9424)

    In [24]: data = "0,1,2\n3,4,5"

    In [25]: names = ["a", "b", "a"]

Previous behavior:

    In [2]: pd.read_csv(StringIO(data), names=names)
    Out[2]:
       a  b  a
    0  2  1  2
    1  5  4  5

The first `a` column contained the same data as the second `a` column, when it should have contained the values `[0, 3]`.

New behavior:

    In [26]: pd.read_csv(StringIO(data), names=names)
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    Cell In[26], line 1
    ----> 1 pd.read_csv(StringIO(data), names=names)

    File ~/work/pandas/pandas/pandas/io/parsers/readers.py:873, in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, skip_blank_lines, parse_dates, date_format, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, on_bad_lines, low_memory, memory_map, float_precision, storage_options, dtype_backend)
        861 kwds_defaults = _refine_defaults_read(
        862     dialect,
        863     delimiter,
       (...)    869     dtype_backend=dtype_backend,
        870 )
        871 kwds.update(kwds_defaults)
    --> 873 return _read(filepath_or_buffer, kwds)

    File ~/work/pandas/pandas/pandas/io/parsers/readers.py:297, in _read(filepath_or_buffer, kwds)
        294 nrows = kwds.get("nrows", None)
        296 # Check for duplicates in names.
    --> 297 _validate_names(kwds.get("names", None))
        299 # Create the parser.
        300 parser = TextFileReader(filepath_or_buffer, **kwds)

    File ~/work/pandas/pandas/pandas/io/parsers/readers.py:251, in _validate_names(names)
        249 if names is not None:
        250     if len(names) != len(set(names)):
    --> 251         raise ValueError("Duplicate names are not allowed.")
        252     if not (
        253         is_list_like(names, allow_sets=False) or isinstance(names, abc.KeysView)
        254     ):
        255         raise ValueError("Names should be an ordered collection.")

    ValueError: Duplicate names are not allowed.

### Method `read_csv` supports parsing `Categorical` directly

The `read_csv()` function now supports parsing a `Categorical` column when specified as a dtype (GH 10153). Depending on the structure of the data, this can result in a faster parse time and lower memory usage compared to converting to `Categorical` after parsing. See the io docs here.

    In [27]: data = """
       ....: col1,col2,col3
       ....: a,b,1
       ....: a,b,2
       ....: c,d,3
       ....: """
       ....:

    In [28]: pd.read_csv(StringIO(data))
    Out[28]:
      col1 col2  col3
    0    a    b     1
    1    a    b     2
    2    c    d     3

    [3 rows x 3 columns]

    In [29]: pd.read_csv(StringIO(data)).dtypes
    Out[29]:
    col1      str
    col2      str
    col3    int64
    Length: 3, dtype: object

    In [30]: pd.read_csv(StringIO(data), dtype="category").dtypes
    Out[30]:
    col1    category
    col2    category
    col3    category
    Length: 3, dtype: object

Individual columns can be parsed as a `Categorical` using a dict specification

    In [31]: pd.read_csv(StringIO(data), dtype={"col1": "category"}).dtypes
    Out[31]:
    col1    category
    col2         str
    col3       int64
    Length: 3, dtype: object

Note

The resulting categories will always be parsed as strings (object dtype). If the categories are numeric they can be converted using the `to_numeric()` function, or as appropriate, another converter such as `to_datetime()`.

    In [32]: df = pd.read_csv(StringIO(data), dtype="category")

    In [33]: df.dtypes
    Out[33]:
    col1    category
    col2    category
    col3    category
    Length: 3, dtype: object

    In [34]: df["col3"]
    Out[34]:
    0    1
    1    2
    2    3
    Name: col3, Length: 3, dtype: category
    Categories (3, str): ['1', '2', '3']

    In [35]: new_categories = pd.to_numeric(df["col3"].cat.categories)

    In [36]: df["col3"] = df["col3"].cat.rename_categories(new_categories)

    In [37]: df["col3"]
    Out[37]:
    0    1
    1    2
    2    3
    Name: col3, Length: 3, dtype: category
    Categories (3, int64): [1, 2, 3]

### Categorical concatenation

- A function `union_categoricals()` has been added for combining categoricals, see Unioning Categoricals (GH 13361, GH 13763, GH 13846, GH 14173)

      In [38]: from pandas.api.types import union_categoricals

      In [39]: a = pd.Categorical(["b", "c"])

      In [40]: b = pd.Categorical(["a", "b"])

      In [41]: union_categoricals([a, b])
      Out[41]:
      ['b', 'c', 'a', 'b']
      Categories (3, str): ['b', 'c', 'a']

- `concat` and `append` now can concat `category` dtypes with different `categories` as `object` dtype (GH 13524)

      In [42]: s1 = pd.Series(["a", "b"], dtype="category")

      In [43]: s2 = pd.Series(["b", "c"], dtype="category")

Previous behavior:

    In [1]: pd.concat([s1, s2])
    ValueError: incompatible categories in categorical concat

New behavior:

    In [44]: pd.concat([s1, s2])
    Out[44]:
    0    a
    1    b
    0    b
    1    c
    Length: 4, dtype: str

### Semi-month offsets

pandas has gained new frequency offsets, `SemiMonthEnd` (‘SM’) and `SemiMonthBegin` (‘SMS’). These provide date offsets anchored (by default) to the 15th and end of month, and 15th and 1st of month respectively. (GH 1543)

    In [45]: from pandas.tseries.offsets import SemiMonthEnd, SemiMonthBegin

SemiMonthEnd:

    In [46]: pd.Timestamp("2016-01-01") + SemiMonthEnd()
    Out[46]: Timestamp('2016-01-15 00:00:00')

    In [47]: pd.date_range("2015-01-01", freq="SM", periods=4)
    Out[47]: DatetimeIndex(['2015-01-15', '2015-01-31', '2015-02-15', '2015-02-28'], dtype='datetime64[ns]', freq='SM-15')

SemiMonthBegin:

    In [46]: pd.Timestamp("2016-01-01") + SemiMonthBegin()
    Out[46]: Timestamp('2016-01-15 00:00:00')

    In [47]: pd.date_range("2015-01-01", freq="SMS", periods=4)
    Out[47]: DatetimeIndex(['2015-01-01', '2015-01-15', '2015-02-01', '2015-02-15'], dtype='datetime64[us]', freq='SMS-15')

Using the anchoring suffix, you can also specify the day of month to use instead of the 15th.

    In [50]: pd.date_range("2015-01-01", freq="SMS-16", periods=4)
    Out[50]: DatetimeIndex(['2015-01-01', '2015-01-16', '2015-02-01', '2015-02-16'], dtype='datetime64[ns]', freq='SMS-16')

    In [51]: pd.date_range("2015-01-01", freq="SM-14", periods=4)
    Out[51]: DatetimeIndex(['2015-01-14', '2015-01-31', '2015-02-14', '2015-02-28'], dtype='datetime64[ns]', freq='SM-14')

### New Index methods

The following methods and options are added to `Index`, to be more consistent with the `Series` and `DataFrame` API.

`Index` now supports the `.where()` function for same shape indexing (GH 13170)

    In [48]: idx = pd.Index(["a", "b", "c"])

    In [49]: idx.where([True, False, True])
    Out[49]: Index(['a', nan, 'c'], dtype='str')

`Index` now supports `.dropna()` to exclude missing values (GH 6194)

    In [50]: idx = pd.Index([1, 2, np.nan, 4])

    In [51]: idx.dropna()
    Out[51]: Index([1.0, 2.0, 4.0], dtype='float64')

For `MultiIndex`, values are dropped if any level is missing by default. Specifying `how='all'` only drops values where all levels are missing.

    In [52]: midx = pd.MultiIndex.from_arrays([[1, 2, np.nan, 4], [1, 2, np.nan, np.nan]])

    In [53]: midx
    Out[53]:
    MultiIndex([(1.0, 1.0),
                (2.0, 2.0),
                (nan, nan),
                (4.0, nan)],
               )

    In [54]: midx.dropna()
    Out[54]:
    MultiIndex([(1, 1),
                (2, 2)],
               )

    In [55]: midx.dropna(how="all")
    Out[55]:
    MultiIndex([(1, 1.0),
                (2, 2.0),
                (4, nan)],
               )

`Index` now supports `.str.extractall()` which returns a `DataFrame`, see the docs here (GH 10008, GH 13156)

    In [56]: idx = pd.Index(["a1a2", "b1", "c1"])

    In [57]: idx.str.extractall(r"[ab](?P<digit>\d)")
    Out[57]:
            digit
      match
    0 0         1
      1         2
    1 0         1

    [3 rows x 1 columns]

`Index.astype()` now accepts an optional boolean argument `copy`, which allows optional copying if the requirements on dtype are satisfied (GH 13209)

### Google BigQuery enhancements

- The `read_gbq()` method has gained the `dialect` argument to allow users to specify whether to use BigQuery’s legacy SQL or BigQuery’s standard SQL. See the docs for more details (GH 13615).

- The `to_gbq()` method now allows the DataFrame column order to differ from the destination table schema (GH 11359).

### Fine-grained NumPy errstate

Previous versions of pandas would permanently silence numpy’s ufunc error handling when `pandas` was imported. pandas did this in order to silence the warnings that would arise from using numpy ufuncs on missing data, which are usually represented as `NaN` s. Unfortunately, this silenced legitimate warnings arising in non-pandas code in the application. Starting with 0.19.0, pandas will use the `numpy.errstate` context manager to silence these warnings in a more fine-grained manner, only around where these operations are actually used in the pandas code base. (GH 13109, GH 13145)

After upgrading pandas, you may see new `RuntimeWarnings` being issued from your code. These are likely legitimate, and the underlying cause likely existed in the code when using previous versions of pandas that simply silenced the warning. Use numpy.errstate around the source of the `RuntimeWarning` to control how these conditions are handled.

### Method `get_dummies` now returns integer dtypes

The `pd.get_dummies` function now returns dummy-encoded columns as small integers, rather than floats (GH 8725). This should provide an improved memory footprint.

Previous behavior:

    In [1]: pd.get_dummies(['a', 'b', 'a', 'c']).dtypes

    Out[1]:
    a    float64
    b    float64
    c    float64
    dtype: object

New behavior:

    In [58]: pd.get_dummies(["a", "b", "a", "c"]).dtypes
    Out[58]:
    a    bool
    b    bool
    c    bool
    Length: 3, dtype: object

### Downcast values to smallest possible dtype in `to_numeric`

`pd.to_numeric()` now accepts a `downcast` parameter, which will downcast the data if possible to smallest specified numerical dtype (GH 13352)

    In [59]: s = ["1", 2, 3]

    In [60]: pd.to_numeric(s, downcast="unsigned")
    Out[60]: array([1, 2, 3], dtype=uint8)

    In [61]: pd.to_numeric(s, downcast="integer")
    Out[61]: array([1, 2, 3], dtype=int8)

### pandas development API

As part of making pandas API more uniform and accessible in the future, we have created a standard sub-package of pandas, `pandas.api` to hold public API’s. We are starting by exposing type introspection functions in `pandas.api.types`. More sub-packages and officially sanctioned API’s will be published in future versions of pandas (GH 13147, GH 13634)

The following are now part of this API:

    In [62]: import pprint

    In [63]: from pandas.api import types

    In [64]: funcs = [f for f in dir(types) if not f.startswith("_")]

    In [65]: pprint.pprint(funcs)
    ['CategoricalDtype',
     'DatetimeTZDtype',
     'IntervalDtype',
     'PeriodDtype',
     'infer_dtype',
     'is_any_real_numeric_dtype',
     'is_array_like',
     'is_bool',
     'is_bool_dtype',
     'is_categorical_dtype',
     'is_complex',
     'is_complex_dtype',
     'is_datetime64_any_dtype',
     'is_datetime64_dtype',
     'is_datetime64_ns_dtype',
     'is_datetime64tz_dtype',
     'is_dict_like',
     'is_dtype_equal',
     'is_extension_array_dtype',
     'is_file_like',
     'is_float',
     'is_float_dtype',
     'is_hashable',
     'is_int64_dtype',
     'is_integer',
     'is_integer_dtype',
     'is_interval_dtype',
     'is_iterator',
     'is_list_like',
     'is_named_tuple',
     'is_number',
     'is_numeric_dtype',
     'is_object_dtype',
     'is_period_dtype',
     'is_re',
     'is_re_compilable',
     'is_scalar',
     'is_signed_integer_dtype',
     'is_sparse',
     'is_string_dtype',
     'is_timedelta64_dtype',
     'is_timedelta64_ns_dtype',
     'is_unsigned_integer_dtype',
     'pandas_dtype',
     'union_categoricals']

Note

Calling these functions from the internal module `pandas.core.common` will now show a `DeprecationWarning` (GH 13990)

### Other enhancements

- `Timestamp` can now accept positional and keyword parameters similar to `datetime.datetime()` (GH 10758, GH 11630)

      In [66]: pd.Timestamp(2012, 1, 1)
      Out[66]: Timestamp('2012-01-01 00:00:00')

      In [67]: pd.Timestamp(year=2012, month=1, day=1, hour=8, minute=30)
      Out[67]: Timestamp('2012-01-01 08:30:00')

- The `.resample()` function now accepts a `on=` or `level=` parameter for resampling on a datetimelike column or `MultiIndex` level (GH 13500)

      In [68]: df = pd.DataFrame(
         ....:     {"date": pd.date_range("2015-01-01", freq="W", periods=5), "a": np.arange(5)},
         ....:     index=pd.MultiIndex.from_arrays(
         ....:         [[1, 2, 3, 4, 5], pd.date_range("2015-01-01", freq="W", periods=5)],
         ....:         names=["v", "d"],
         ....:     ),
         ....: )
         ....:

      In [69]: df
      Out[69]:
                         date  a
      v d
      1 2015-01-04 2015-01-04  0
      2 2015-01-11 2015-01-11  1
      3 2015-01-18 2015-01-18  2
      4 2015-01-25 2015-01-25  3
      5 2015-02-01 2015-02-01  4

      [5 rows x 2 columns]

      In [74]: df.resample("M", on="date")[["a"]].sum()
      Out[74]:
                  a
      date
      2015-01-31  6
      2015-02-28  4

      [2 rows x 1 columns]

      In [75]: df.resample("M", level="d")[["a"]].sum()
      Out[75]:
                  a
      d
      2015-01-31  6
      2015-02-28  4

      [2 rows x 1 columns]

- The `.get_credentials()` method of `GbqConnector` can now first try to fetch the application default credentials. See the docs for more details (GH 13577).

- The `.tz_localize()` method of `DatetimeIndex` and `Timestamp` has gained the `errors` keyword, so you can potentially coerce nonexistent timestamps to `NaT`. The default behavior remains to raising a `NonExistentTimeError` (GH 13057)

- `.to_hdf/read_hdf()` now accept path objects (e.g. `pathlib.Path`, `py.path.local`) for the file path (GH 11773)

- The `pd.read_csv()` with `engine='python'` has gained support for the `decimal` (GH 12933), `na_filter` (GH 13321) and the `memory_map` option (GH 13381).

- Consistent with the Python API, `pd.read_csv()` will now interpret `+inf` as positive infinity (GH 13274)

- The `pd.read_html()` has gained support for the `na_values`, `converters`, `keep_default_na` options (GH 13461)

- `Categorical.astype()` now accepts an optional boolean argument `copy`, effective when dtype is categorical (GH 13209)

- `DataFrame` has gained the `.asof()` method to return the last non-NaN values according to the selected subset (GH 13358)

- The `DataFrame` constructor will now respect key ordering if a list of `OrderedDict` objects are passed in (GH 13304)

- `pd.read_html()` has gained support for the `decimal` option (GH 12907)

- `Series` has gained the properties `.is_monotonic`, `.is_monotonic_increasing`, `.is_monotonic_decreasing`, similar to `Index` (GH 13336)

- `DataFrame.to_sql()` now allows a single value as the SQL type for all columns (GH 11886).

- `Series.append` now supports the `ignore_index` option (GH 13677)

- `.to_stata()` and `StataWriter` can now write variable labels to Stata dta files using a dictionary to make column names to labels (GH 13535, GH 13536)

- `.to_stata()` and `StataWriter` will automatically convert `datetime64[ns]` columns to Stata format `%tc`, rather than raising a `ValueError` (GH 12259)

- `read_stata()` and `StataReader` raise with a more explicit error message when reading Stata files with repeated value labels when `convert_categoricals=True` (GH 13923)

- `DataFrame.style` will now render sparsified MultiIndexes (GH 11655)

- `DataFrame.style` will now show column level names (e.g. `DataFrame.columns.names`) (GH 13775)

- `DataFrame` has gained support to re-order the columns based on the values in a row using `df.sort_values(by='...', axis=1)` (GH 10806)

      In [70]: df = pd.DataFrame({"A": [2, 7], "B": [3, 5], "C": [4, 8]}, index=["row1", "row2"])

      In [71]: df
      Out[71]:
            A  B  C
      row1  2  3  4
      row2  7  5  8

      [2 rows x 3 columns]

      In [72]: df.sort_values(by="row2", axis=1)
      Out[72]:
            B  A  C
      row1  3  2  4
      row2  5  7  8

      [2 rows x 3 columns]

- Added documentation to I/O regarding the perils of reading in columns with mixed dtypes and how to handle it (GH 13746)

- `to_html()` now has a `border` argument to control the value in the opening `<table>` tag. The default is the value of the `html.border` option, which defaults to 1. This also affects the notebook HTML repr, but since Jupyter’s CSS includes a border-width attribute, the visual effect is the same. (GH 11563).

- Raise `ImportError` in the sql functions when `sqlalchemy` is not installed and a connection string is used (GH 11920).

- Compatibility with matplotlib 2.0. Older versions of pandas should also work with matplotlib 2.0 (GH 13333)

- `Timestamp`, `Period`, `DatetimeIndex`, `PeriodIndex` and `.dt` accessor have gained a `.is_leap_year` property to check whether the date belongs to a leap year. (GH 13727)

- `astype()` will now accept a dict of column name to data types mapping as the `dtype` argument. (GH 12086)

- The `pd.read_json` and `DataFrame.to_json` has gained support for reading and writing json lines with `lines` option see Line delimited json (GH 9180)

- `read_excel()` now supports the true_values and false_values keyword arguments (GH 13347)

- `groupby()` will now accept a scalar and a single-element list for specifying `level` on a non-`MultiIndex` grouper. (GH 13907)

- Non-convertible dates in an excel date column will be returned without conversion and the column will be `object` dtype, rather than raising an exception (GH 10001).

- `pd.Timedelta(None)` is now accepted and will return `NaT`, mirroring `pd.Timestamp` (GH 13687)

- `pd.read_stata()` can now handle some format 111 files, which are produced by SAS when generating Stata dta files (GH 11526)

- `Series` and `Index` now support `divmod` which will return a tuple of series or indices. This behaves like a standard binary operator with regards to broadcasting rules (GH 14208).

## API changes

### `Series.tolist()` will now return Python types

`Series.tolist()` will now return Python types in the output, mimicking NumPy `.tolist()` behavior (GH 10904)

    In [73]: s = pd.Series([1, 2, 3])

Previous behavior:

    In [7]: type(s.tolist()[0])
    Out[7]:
     <class 'numpy.int64'>

New behavior:

    In [74]: type(s.tolist()[0])
    Out[74]: int

### `Series` operators for different indexes

Following `Series` operators have been changed to make all operators consistent, including `DataFrame` (GH 1134, GH 4581, GH 13538)

- `Series` comparison operators now raise `ValueError` when `index` are different.

- `Series` logical operators align both `index` of left and right hand side.

Warning

Until 0.18.1, comparing `Series` with the same length, would succeed even if the `.index` are different (the result ignores `.index`). As of 0.19.0, this will raises `ValueError` to be more strict. This section also describes how to keep previous behavior or align different indexes, using the flexible comparison methods like `.eq`.

As a result, `Series` and `DataFrame` operators behave as below:

#### Arithmetic operators

Arithmetic operators align both `index` (no changes).

    In [75]: s1 = pd.Series([1, 2, 3], index=list("ABC"))

    In [76]: s2 = pd.Series([2, 2, 2], index=list("ABD"))

    In [77]: s1 + s2
    Out[77]:
    A    3.0
    B    4.0
    C    NaN
    D    NaN
    Length: 4, dtype: float64

    In [78]: df1 = pd.DataFrame([1, 2, 3], index=list("ABC"))

    In [79]: df2 = pd.DataFrame([2, 2, 2], index=list("ABD"))

    In [80]: df1 + df2
    Out[80]:
         0
    A  3.0
    B  4.0
    C  NaN
    D  NaN

    [4 rows x 1 columns]

#### Comparison operators

Comparison operators raise `ValueError` when `.index` are different.

Previous behavior (`Series`):

`Series` compared values ignoring the `.index` as long as both had the same length:

    In [1]: s1 == s2
    Out[1]:
    A    False
    B     True
    C    False
    dtype: bool

New behavior (`Series`):

    In [2]: s1 == s2
    Out[2]:
    ValueError: Can only compare identically-labeled Series objects

Note

To achieve the same result as previous versions (compare values based on locations ignoring `.index`), compare both `.values`.

    In [81]: s1.values == s2.values
    Out[81]: array([False,  True, False])

If you want to compare `Series` aligning its `.index`, see flexible comparison methods section below:

    In [82]: s1.eq(s2)
    Out[82]:
    A    False
    B     True
    C    False
    D    False
    Length: 4, dtype: bool

Current behavior (`DataFrame`, no change):

    In [3]: df1 == df2
    Out[3]:
    ValueError: Can only compare identically-labeled DataFrame objects

#### Logical operators

Logical operators align both `.index` of left and right hand side.

Previous behavior (`Series`), only left hand side `index` was kept:

    In [4]: s1 = pd.Series([True, False, True], index=list('ABC'))
    In [5]: s2 = pd.Series([True, True, True], index=list('ABD'))
    In [6]: s1 & s2
    Out[6]:
    A     True
    B    False
    C    False
    dtype: bool

New behavior (`Series`):

    In [83]: s1 = pd.Series([True, False, True], index=list("ABC"))

    In [84]: s2 = pd.Series([True, True, True], index=list("ABD"))

    In [85]: s1 & s2
    Out[85]:
    A     True
    B    False
    C    False
    D    False
    Length: 4, dtype: bool

Note

`Series` logical operators fill a `NaN` result with `False`.

Note

To achieve the same result as previous versions (compare values based on only left hand side index), you can use `reindex_like`:

    In [86]: s1 & s2.reindex_like(s1)
    Out[86]:
    A     True
    B    False
    C    False
    Length: 3, dtype: bool

Current behavior (`DataFrame`, no change):

    In [87]: df1 = pd.DataFrame([True, False, True], index=list("ABC"))

    In [88]: df2 = pd.DataFrame([True, True, True], index=list("ABD"))

    In [89]: df1 & df2
    Out[89]:
           0
    A   True
    B  False
    C  False
    D  False

    [4 rows x 1 columns]

#### Flexible comparison methods

`Series` flexible comparison methods like `eq`, `ne`, `le`, `lt`, `ge` and `gt` now align both `index`. Use these operators if you want to compare two `Series` which has the different `index`.

    In [90]: s1 = pd.Series([1, 2, 3], index=["a", "b", "c"])

    In [91]: s2 = pd.Series([2, 2, 2], index=["b", "c", "d"])

    In [92]: s1.eq(s2)
    Out[92]:
    a    False
    b     True
    c    False
    d    False
    Length: 4, dtype: bool

    In [93]: s1.ge(s2)
    Out[93]:
    a    False
    b     True
    c     True
    d    False
    Length: 4, dtype: bool

Previously, this worked the same as comparison operators (see above).

### `Series` type promotion on assignment

A `Series` will now correctly promote its dtype for assignment with incompat values to the current dtype (GH 13234)

    In [94]: s = pd.Series()

Previous behavior:

    In [2]: s["a"] = pd.Timestamp("2016-01-01")

    In [3]: s["b"] = 3.0
    TypeError: invalid type promotion

New behavior:

    In [95]: s["a"] = pd.Timestamp("2016-01-01")

    In [96]: s["b"] = 3.0

    In [97]: s
    Out[97]:
    a    2016-01-01 00:00:00
    b                    3.0
    Length: 2, dtype: object

    In [98]: s.dtype
    Out[98]: dtype('O')

### Function `.to_datetime()` changes

Previously if `.to_datetime()` encountered mixed integers/floats and strings, but no datetimes with `errors='coerce'` it would convert all to `NaT`.

Previous behavior:

    In [2]: pd.to_datetime([1, 'foo'], errors='coerce')
    Out[2]: DatetimeIndex(['NaT', 'NaT'], dtype='datetime64[ns]', freq=None)

Current behavior:

This will now convert integers/floats with the default unit of `ns`.

    In [99]: pd.to_datetime([1, "foo"], errors="coerce")
    Out[99]: DatetimeIndex(['1970-01-01 00:00:00.000000001', 'NaT'], dtype='datetime64[ns]', freq=None)

Bug fixes related to `.to_datetime()`:

- Bug in `pd.to_datetime()` when passing integers or floats, and no `unit` and `errors='coerce'` (GH 13180).

- Bug in `pd.to_datetime()` when passing invalid data types (e.g. bool); will now respect the `errors` keyword (GH 13176)

- Bug in `pd.to_datetime()` which overflowed on `int8`, and `int16` dtypes (GH 13451)

- Bug in `pd.to_datetime()` raise `AttributeError` with `NaN` and the other string is not valid when `errors='ignore'` (GH 12424)

- Bug in `pd.to_datetime()` did not cast floats correctly when `unit` was specified, resulting in truncated datetime (GH 13834)

### Merging changes

Merging will now preserve the dtype of the join keys (GH 8596)

    In [100]: df1 = pd.DataFrame({"key": [1], "v1": [10]})

    In [101]: df1
    Out[101]:
       key  v1
    0    1  10

    [1 rows x 2 columns]

    In [102]: df2 = pd.DataFrame({"key": [1, 2], "v1": [20, 30]})

    In [103]: df2
    Out[103]:
       key  v1
    0    1  20
    1    2  30

    [2 rows x 2 columns]

Previous behavior:

    In [5]: pd.merge(df1, df2, how='outer')
    Out[5]:
       key    v1
    0  1.0  10.0
    1  1.0  20.0
    2  2.0  30.0

    In [6]: pd.merge(df1, df2, how='outer').dtypes
    Out[6]:
    key    float64
    v1     float64
    dtype: object

New behavior:

We are able to preserve the join keys

    In [104]: pd.merge(df1, df2, how="outer")
    Out[104]:
       key  v1
    0    1  10
    1    1  20
    2    2  30

    [3 rows x 2 columns]

    In [105]: pd.merge(df1, df2, how="outer").dtypes
    Out[105]:
    key    int64
    v1     int64
    Length: 2, dtype: object

Of course if you have missing values that are introduced, then the resulting dtype will be upcast, which is unchanged from previous.

    In [106]: pd.merge(df1, df2, how="outer", on="key")
    Out[106]:
       key  v1_x  v1_y
    0    1  10.0    20
    1    2   NaN    30

    [2 rows x 3 columns]

    In [107]: pd.merge(df1, df2, how="outer", on="key").dtypes
    Out[107]:
    key       int64
    v1_x    float64
    v1_y      int64
    Length: 3, dtype: object

### Method `.describe()` changes

Percentile identifiers in the index of a `.describe()` output will now be rounded to the least precision that keeps them distinct (GH 13104)

    In [108]: s = pd.Series([0, 1, 2, 3, 4])

    In [109]: df = pd.DataFrame([0, 1, 2, 3, 4])

Previous behavior:

The percentiles were rounded to at most one decimal place, which could raise `ValueError` for a data frame if the percentiles were duplicated.

    In [3]: s.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
    Out[3]:
    count     5.000000
    mean      2.000000
    std       1.581139
    min       0.000000
    0.0%      0.000400
    0.1%      0.002000
    0.1%      0.004000
    50%       2.000000
    99.9%     3.996000
    100.0%    3.998000
    100.0%    3.999600
    max       4.000000
    dtype: float64

    In [4]: df.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
    Out[4]:
    ...
    ValueError: cannot reindex from a duplicate axis

New behavior:

    In [110]: s.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
    Out[110]:
    count     5.000000
    mean      2.000000
    std       1.581139
    min       0.000000
    0.01%     0.000400
    0.05%     0.002000
    0.1%      0.004000
    99.9%     3.996000
    99.95%    3.998000
    99.99%    3.999600
    max       4.000000
    Length: 11, dtype: float64

    In [111]: df.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.9999])
    Out[111]:
                   0
    count   5.000000
    mean    2.000000
    std     1.581139
    min     0.000000
    0.01%   0.000400
    0.05%   0.002000
    0.1%    0.004000
    99.9%   3.996000
    99.95%  3.998000
    99.99%  3.999600
    max     4.000000

    [11 rows x 1 columns]

Furthermore:

- Passing duplicated `percentiles` will now raise a `ValueError`.

- Bug in `.describe()` on a DataFrame with a mixed-dtype column index, which would previously raise a `TypeError` (GH 13288)

### `Period` changes

#### The `PeriodIndex` now has `period` dtype

`PeriodIndex` now has its own `period` dtype. The `period` dtype is a pandas extension dtype like `category` or the timezone aware dtype (`datetime64[ns, tz]`) (GH 13941). As a consequence of this change, `PeriodIndex` no longer has an integer dtype:

Previous behavior:

    In [1]: pi = pd.PeriodIndex(['2016-08-01'], freq='D')

    In [2]: pi
    Out[2]: PeriodIndex(['2016-08-01'], dtype='int64', freq='D')

    In [3]: pd.api.types.is_integer_dtype(pi)
    Out[3]: True

    In [4]: pi.dtype
    Out[4]: dtype('int64')

New behavior:

    In [112]: pi = pd.PeriodIndex(["2016-08-01"], freq="D")

    In [113]: pi
    Out[113]: PeriodIndex(['2016-08-01'], dtype='period[D]')

    In [114]: pd.api.types.is_integer_dtype(pi)
    Out[114]: False

    In [115]: pd.api.types.is_period_dtype(pi)
    Out[115]: True

    In [116]: pi.dtype
    Out[116]: period[D]

    In [117]: type(pi.dtype)
    Out[117]: pandas.PeriodDtype

#### `Period('NaT')` now returns `pd.NaT`

Previously, `Period` has its own `Period('NaT')` representation different from `pd.NaT`. Now `Period('NaT')` has been changed to return `pd.NaT`. (GH 12759, GH 13582)

Previous behavior:

    In [5]: pd.Period('NaT', freq='D')
    Out[5]: Period('NaT', 'D')

New behavior:

These result in `pd.NaT` without providing `freq` option.

    In [118]: pd.Period("NaT")
    Out[118]: NaT

    In [119]: pd.Period(None)
    Out[119]: NaT

To be compatible with `Period` addition and subtraction, `pd.NaT` now supports addition and subtraction with `int`. Previously it raised `ValueError`.

Previous behavior:

    In [5]: pd.NaT + 1
    ...
    ValueError: Cannot add integral value to Timestamp without freq.

New behavior:

    In [120]: pd.NaT + 1
    Out[120]: NaT

    In [121]: pd.NaT - 1
    Out[121]: NaT

#### `PeriodIndex.values` now returns array of `Period` object

`.values` is changed to return an array of `Period` objects, rather than an array of integers (GH 13988).

Previous behavior:

    In [6]: pi = pd.PeriodIndex(['2011-01', '2011-02'], freq='M')
    In [7]: pi.values
    Out[7]: array([492, 493])

New behavior:

    In [122]: pi = pd.PeriodIndex(["2011-01", "2011-02"], freq="M")

    In [123]: pi.values
    Out[123]: array([Period('2011-01', 'M'), Period('2011-02', 'M')], dtype=object)

### Index `+` / `-` no longer used for set operations

Addition and subtraction of the base Index type and of DatetimeIndex (not the numeric index types) previously performed set operations (set union and difference). This behavior was already deprecated since 0.15.0 (in favor using the specific `.union()` and `.difference()` methods), and is now disabled. When possible, `+` and `-` are now used for element-wise operations, for example for concatenating strings or subtracting datetimes (GH 8227, GH 14127).

Previous behavior:

    In [1]: pd.Index(['a', 'b']) + pd.Index(['a', 'c'])
    FutureWarning: using '+' to provide set union with Indexes is deprecated, use '|' or .union()
    Out[1]: Index(['a', 'b', 'c'], dtype='object')

New behavior: the same operation will now perform element-wise addition:

    In [124]: pd.Index(["a", "b"]) + pd.Index(["a", "c"])
    Out[124]: Index(['aa', 'bc'], dtype='str')

Note that numeric Index objects already performed element-wise operations. For example, the behavior of adding two integer Indexes is unchanged. The base `Index` is now made consistent with this behavior.

    In [125]: pd.Index([1, 2, 3]) + pd.Index([2, 3, 4])
    Out[125]: Index([3, 5, 7], dtype='int64')

Further, because of this change, it is now possible to subtract two DatetimeIndex objects resulting in a TimedeltaIndex:

Previous behavior:

    In [1]: (pd.DatetimeIndex(['2016-01-01', '2016-01-02'])
       ...:  - pd.DatetimeIndex(['2016-01-02', '2016-01-03']))
    FutureWarning: using '-' to provide set differences with datetimelike Indexes is deprecated, use .difference()
    Out[1]: DatetimeIndex(['2016-01-01'], dtype='datetime64[ns]', freq=None)

New behavior:

    In [126]: (
       .....:     pd.DatetimeIndex(["2016-01-01", "2016-01-02"])
       .....:     - pd.DatetimeIndex(["2016-01-02", "2016-01-03"])
       .....: )
       .....:
    Out[126]: TimedeltaIndex(['-1 days', '-1 days'], dtype='timedelta64[us]', freq=None)

### `Index.difference` and `.symmetric_difference` changes

`Index.difference` and `Index.symmetric_difference` will now, more consistently, treat `NaN` values as any other values. (GH 13514)

    In [127]: idx1 = pd.Index([1, 2, 3, np.nan])

    In [128]: idx2 = pd.Index([0, 1, np.nan])

Previous behavior:

    In [3]: idx1.difference(idx2)
    Out[3]: Float64Index([nan, 2.0, 3.0], dtype='float64')

    In [4]: idx1.symmetric_difference(idx2)
    Out[4]: Float64Index([0.0, nan, 2.0, 3.0], dtype='float64')

New behavior:

    In [129]: idx1.difference(idx2)
    Out[129]: Index([2.0, 3.0], dtype='float64')

    In [130]: idx1.symmetric_difference(idx2)
    Out[130]: Index([0.0, 2.0, 3.0], dtype='float64')

### `Index.unique` consistently returns `Index`

`Index.unique()` now returns unique values as an `Index` of the appropriate `dtype`. (GH 13395). Previously, most `Index` classes returned `np.ndarray`, and `DatetimeIndex`, `TimedeltaIndex` and `PeriodIndex` returned `Index` to keep metadata like timezone.

Previous behavior:

    In [1]: pd.Index([1, 2, 3]).unique()
    Out[1]: array([1, 2, 3])

    In [2]: pd.DatetimeIndex(['2011-01-01', '2011-01-02',
       ...:                   '2011-01-03'], tz='Asia/Tokyo').unique()
    Out[2]:
    DatetimeIndex(['2011-01-01 00:00:00+09:00', '2011-01-02 00:00:00+09:00',
                   '2011-01-03 00:00:00+09:00'],
                  dtype='datetime64[ns, Asia/Tokyo]', freq=None)

New behavior:

    In [131]: pd.Index([1, 2, 3]).unique()
    Out[131]: Index([1, 2, 3], dtype='int64')

    In [132]: pd.DatetimeIndex(
       .....:     ["2011-01-01", "2011-01-02", "2011-01-03"], tz="Asia/Tokyo"
       .....: ).unique()
       .....:
    Out[132]:
    DatetimeIndex(['2011-01-01 00:00:00+09:00', '2011-01-02 00:00:00+09:00',
                   '2011-01-03 00:00:00+09:00'],
                  dtype='datetime64[us, Asia/Tokyo]', freq=None)

### `MultiIndex` constructors, `groupby` and `set_index` preserve categorical dtypes

`MultiIndex.from_arrays` and `MultiIndex.from_product` will now preserve categorical dtype in `MultiIndex` levels (GH 13743, GH 13854).

    In [133]: cat = pd.Categorical(["a", "b"], categories=list("bac"))

    In [134]: lvl1 = ["foo", "bar"]

    In [135]: midx = pd.MultiIndex.from_arrays([cat, lvl1])

    In [136]: midx
    Out[136]:
    MultiIndex([('a', 'foo'),
                ('b', 'bar')],
               )

Previous behavior:

    In [4]: midx.levels[0]
    Out[4]: Index(['b', 'a', 'c'], dtype='object')

    In [5]: midx.get_level_values[0]
    Out[5]: Index(['a', 'b'], dtype='object')

New behavior: the single level is now a `CategoricalIndex`:

    In [137]: midx.levels[0]
    Out[137]: CategoricalIndex(['b', 'a', 'c'], categories=['b', 'a', 'c'], ordered=False, dtype='category')

    In [138]: midx.get_level_values(0)
    Out[138]: CategoricalIndex(['a', 'b'], categories=['b', 'a', 'c'], ordered=False, dtype='category')

An analogous change has been made to `MultiIndex.from_product`. As a consequence, `groupby` and `set_index` also preserve categorical dtypes in indexes

    In [139]: df = pd.DataFrame({"A": [0, 1], "B": [10, 11], "C": cat})

    In [140]: df_grouped = df.groupby(by=["A", "C"], observed=False).first()

    In [141]: df_set_idx = df.set_index(["A", "C"])

Previous behavior:

    In [11]: df_grouped.index.levels[1]
    Out[11]: Index(['b', 'a', 'c'], dtype='object', name='C')
    In [12]: df_grouped.reset_index().dtypes
    Out[12]:
    A      int64
    C     object
    B    float64
    dtype: object

    In [13]: df_set_idx.index.levels[1]
    Out[13]: Index(['b', 'a', 'c'], dtype='object', name='C')
    In [14]: df_set_idx.reset_index().dtypes
    Out[14]:
    A      int64
    C     object
    B      int64
    dtype: object

New behavior:

    In [142]: df_grouped.index.levels[1]
    Out[142]: CategoricalIndex(['b', 'a', 'c'], categories=['b', 'a', 'c'], ordered=False, dtype='category', name='C')

    In [143]: df_grouped.reset_index().dtypes
    Out[143]:
    A       int64
    C    category
    B     float64
    Length: 3, dtype: object

    In [144]: df_set_idx.index.levels[1]
    Out[144]: CategoricalIndex(['b', 'a', 'c'], categories=['b', 'a', 'c'], ordered=False, dtype='category', name='C')

    In [145]: df_set_idx.reset_index().dtypes
    Out[145]:
    A       int64
    C    category
    B       int64
    Length: 3, dtype: object

### Function `read_csv` will progressively enumerate chunks

When `read_csv()` is called with `chunksize=n` and without specifying an index, each chunk used to have an independently generated index from `0` to `n-1`. They are now given instead a progressive index, starting from `0` for the first chunk, from `n` for the second, and so on, so that, when concatenated, they are identical to the result of calling `read_csv()` without the `chunksize=` argument (GH 12185).

    In [146]: data = "A,B\n0,1\n2,3\n4,5\n6,7"

Previous behavior:

    In [2]: pd.concat(pd.read_csv(StringIO(data), chunksize=2))
    Out[2]:
       A  B
    0  0  1
    1  2  3
    0  4  5
    1  6  7

New behavior:

    In [147]: pd.concat(pd.read_csv(StringIO(data), chunksize=2))
    Out[147]:
       A  B
    0  0  1
    1  2  3
    2  4  5
    3  6  7

    [4 rows x 2 columns]

### Sparse changes

These changes allow pandas to handle sparse data with more dtypes, and for work to make a smoother experience with data handling.

#### Types `int64` and `bool` support enhancements

Sparse data structures now gained enhanced support of `int64` and `bool` `dtype` (GH 667, GH 13849).

Previously, sparse data were `float64` dtype by default, even if all inputs were of `int` or `bool` dtype. You had to specify `dtype` explicitly to create sparse data with `int64` dtype. Also, `fill_value` had to be specified explicitly because the default was `np.nan` which doesn’t appear in `int64` or `bool` data.

    In [1]: pd.SparseArray([1, 2, 0, 0])
    Out[1]:
    [1.0, 2.0, 0.0, 0.0]
    Fill: nan
    IntIndex
    Indices: array([0, 1, 2, 3], dtype=int32)

    # specifying int64 dtype, but all values are stored in sp_values because
    # fill_value default is np.nan
    In [2]: pd.SparseArray([1, 2, 0, 0], dtype=np.int64)
    Out[2]:
    [1, 2, 0, 0]
    Fill: nan
    IntIndex
    Indices: array([0, 1, 2, 3], dtype=int32)

    In [3]: pd.SparseArray([1, 2, 0, 0], dtype=np.int64, fill_value=0)
    Out[3]:
    [1, 2, 0, 0]
    Fill: 0
    IntIndex
    Indices: array([0, 1], dtype=int32)

As of v0.19.0, sparse data keeps the input dtype, and uses more appropriate `fill_value` defaults (`0` for `int64` dtype, `False` for `bool` dtype).

    In [148]: pd.arrays.SparseArray([1, 2, 0, 0], dtype=np.int64)
    Out[148]:
    <SparseArray>
    [1, 2, 0, 0]
    Length: 4, dtype: Sparse[int64, 0]

    In [149]: pd.arrays.SparseArray([True, False, False, False])
    Out[149]:
    <SparseArray>
    [True, False, False, False]
    Length: 4, dtype: Sparse[bool, False]

See the docs for more details.

#### Operators now preserve dtypes

- Sparse data structure now can preserve `dtype` after arithmetic ops (GH 13848)

    s = pd.SparseSeries([0, 2, 0, 1], fill_value=0, dtype=np.int64)
    s.dtype

    s + 1

- Sparse data structure now support `astype` to convert internal `dtype` (GH 13900)

    s = pd.SparseSeries([1.0, 0.0, 2.0, 0.0], fill_value=0)
    s
    s.astype(np.int64)

`astype` fails if data contains values which cannot be converted to specified `dtype`. Note that the limitation is applied to `fill_value` which default is `np.nan`.

    In [7]: pd.SparseSeries([1., np.nan, 2., np.nan], fill_value=np.nan).astype(np.int64)
    Out[7]:
    ValueError: unable to coerce current fill_value nan to int64 dtype

#### Other sparse fixes

- Subclassed `SparseDataFrame` and `SparseSeries` now preserve class types when slicing or transposing. (GH 13787)

- `SparseArray` with `bool` dtype now supports logical (bool) operators (GH 14000)

- Bug in `SparseSeries` with `MultiIndex` `[]` indexing may raise `IndexError` (GH 13144)

- Bug in `SparseSeries` with `MultiIndex` `[]` indexing result may have normal `Index` (GH 13144)

- Bug in `SparseDataFrame` in which `axis=None` did not default to `axis=0` (GH 13048)

- Bug in `SparseSeries` and `SparseDataFrame` creation with `object` dtype may raise `TypeError` (GH 11633)

- Bug in `SparseDataFrame` doesn’t respect passed `SparseArray` or `SparseSeries` ‘s dtype and `fill_value` (GH 13866)

- Bug in `SparseArray` and `SparseSeries` don’t apply ufunc to `fill_value` (GH 13853)

- Bug in `SparseSeries.abs` incorrectly keeps negative `fill_value` (GH 13853)

- Bug in single row slicing on multi-type `SparseDataFrame` s, types were previously forced to float (GH 13917)

- Bug in `SparseSeries` slicing changes integer dtype to float (GH 8292)

- Bug in `SparseDataFarme` comparison ops may raise `TypeError` (GH 13001)

- Bug in `SparseDataFarme.isnull` raises `ValueError` (GH 8276)

- Bug in `SparseSeries` representation with `bool` dtype may raise `IndexError` (GH 13110)

- Bug in `SparseSeries` and `SparseDataFrame` of `bool` or `int64` dtype may display its values like `float64` dtype (GH 13110)

- Bug in sparse indexing using `SparseArray` with `bool` dtype may return incorrect result (GH 13985)

- Bug in `SparseArray` created from `SparseSeries` may lose `dtype` (GH 13999)

- Bug in `SparseSeries` comparison with dense returns normal `Series` rather than `SparseSeries` (GH 13999)

### Indexer dtype changes

Note

This change only affects 64 bit python running on Windows, and only affects relatively advanced indexing operations

Methods such as `Index.get_indexer` that return an indexer array, coerce that array to a “platform int”, so that it can be directly used in 3rd party library operations like `numpy.take`. Previously, a platform int was defined as `np.int_` which corresponds to a C integer, but the correct type, and what is being used now, is `np.intp`, which corresponds to the C integer size that can hold a pointer (GH 3033, GH 13972).

These types are the same on many platform, but for 64 bit python on Windows, `np.int_` is 32 bits, and `np.intp` is 64 bits. Changing this behavior improves performance for many operations on that platform.

Previous behavior:

    In [1]: i = pd.Index(['a', 'b', 'c'])

    In [2]: i.get_indexer(['b', 'b', 'c']).dtype
    Out[2]: dtype('int32')

New behavior:

    In [1]: i = pd.Index(['a', 'b', 'c'])

    In [2]: i.get_indexer(['b', 'b', 'c']).dtype
    Out[2]: dtype('int64')

### Other API changes

- `Timestamp.to_pydatetime` will issue a `UserWarning` when `warn=True`, and the instance has a non-zero number of nanoseconds, previously this would print a message to stdout (GH 14101).

- `Series.unique()` with datetime and timezone now returns return array of `Timestamp` with timezone (GH 13565).

- `Panel.to_sparse()` will raise a `NotImplementedError` exception when called (GH 13778).

- `Index.reshape()` will raise a `NotImplementedError` exception when called (GH 12882).

- `.filter()` enforces mutual exclusion of the keyword arguments (GH 12399).

- `eval`’s upcasting rules for `float32` types have been updated to be more consistent with NumPy’s rules. New behavior will not upcast to `float64` if you multiply a pandas `float32` object by a scalar float64 (GH 12388).

- An `UnsupportedFunctionCall` error is now raised if NumPy ufuncs like `np.mean` are called on groupby or resample objects (GH 12811).

- `__setitem__` will no longer apply a callable rhs as a function instead of storing it. Call `where` directly to get the previous behavior (GH 13299).

- Calls to `.sample()` will respect the random seed set via `numpy.random.seed(n)` (GH 13161)

- `Styler.apply` is now more strict about the outputs your function must return. For `axis=0` or `axis=1`, the output shape must be identical. For `axis=None`, the output must be a DataFrame with identical columns and index labels (GH 13222).

- `Float64Index.astype(int)` will now raise `ValueError` if `Float64Index` contains `NaN` values (GH 13149)

- `TimedeltaIndex.astype(int)` and `DatetimeIndex.astype(int)` will now return `Int64Index` instead of `np.array` (GH 13209)

- Passing `Period` with multiple frequencies to normal `Index` now returns `Index` with `object` dtype (GH 13664)

- `PeriodIndex.fillna` with `Period` has different freq now coerces to `object` dtype (GH 13664)

- Faceted boxplots from `DataFrame.boxplot(by=col)` now return a `Series` when `return_type` is not None. Previously these returned an `OrderedDict`. Note that when `return_type=None`, the default, these still return a 2-D NumPy array (GH 12216, GH 7096).

- `pd.read_hdf` will now raise a `ValueError` instead of `KeyError`, if a mode other than `r`, `r+` and `a` is supplied. (GH 13623)

- `pd.read_csv()`, `pd.read_table()`, and `pd.read_hdf()` raise the builtin `FileNotFoundError` exception for Python 3.x when called on a nonexistent file; this is back-ported as `IOError` in Python 2.x (GH 14086)

- More informative exceptions are passed through the csv parser. The exception type would now be the original exception type instead of `CParserError` (GH 13652).

- `pd.read_csv()` in the C engine will now issue a `ParserWarning` or raise a `ValueError` when `sep` encoded is more than one character long (GH 14065)

- `DataFrame.values` will now return `float64` with a `DataFrame` of mixed `int64` and `uint64` dtypes, conforming to `np.find_common_type` (GH 10364, GH 13917)

- `.groupby.groups` will now return a dictionary of `Index` objects, rather than a dictionary of `np.ndarray` or `lists` (GH 14293)

## Deprecations

- `Series.reshape` and `Categorical.reshape` have been deprecated and will be removed in a subsequent release (GH 12882, GH 12882)

- `PeriodIndex.to_datetime` has been deprecated in favor of `PeriodIndex.to_timestamp` (GH 8254)

- `Timestamp.to_datetime` has been deprecated in favor of `Timestamp.to_pydatetime` (GH 8254)

- `Index.to_datetime` and `DatetimeIndex.to_datetime` have been deprecated in favor of `pd.to_datetime` (GH 8254)

- `pandas.core.datetools` module has been deprecated and will be removed in a subsequent release (GH 14094)

- `SparseList` has been deprecated and will be removed in a future version (GH 13784)

- `DataFrame.to_html()` and `DataFrame.to_latex()` have dropped the `colSpace` parameter in favor of `col_space` (GH 13857)

- `DataFrame.to_sql()` has deprecated the `flavor` parameter, as it is superfluous when SQLAlchemy is not installed (GH 13611)

- Deprecated `read_csv` keywords:

  - `compact_ints` and `use_unsigned` have been deprecated and will be removed in a future version (GH 13320)

  - `buffer_lines` has been deprecated and will be removed in a future version (GH 13360)

  - `as_recarray` has been deprecated and will be removed in a future version (GH 13373)

  - `skip_footer` has been deprecated in favor of `skipfooter` and will be removed in a future version (GH 13349)

- top-level `pd.ordered_merge()` has been renamed to `pd.merge_ordered()` and the original name will be removed in a future version (GH 13358)

- `Timestamp.offset` property (and named arg in the constructor), has been deprecated in favor of `freq` (GH 12160)

- `pd.tseries.util.pivot_annual` is deprecated. Use `pivot_table` as alternative, an example is here (GH 736)

- `pd.tseries.util.isleapyear` has been deprecated and will be removed in a subsequent release. Datetime-likes now have a `.is_leap_year` property (GH 13727)

- `Panel4D` and `PanelND` constructors are deprecated and will be removed in a future version. The recommended way to represent these types of n-dimensional data are with the xarray package. pandas provides a `to_xarray()` method to automate this conversion (GH 13564).

- `pandas.tseries.frequencies.get_standard_freq` is deprecated. Use `pandas.tseries.frequencies.to_offset(freq).rule_code` instead (GH 13874)

- `pandas.tseries.frequencies.to_offset`’s `freqstr` keyword is deprecated in favor of `freq` (GH 13874)

- `Categorical.from_array` has been deprecated and will be removed in a future version (GH 13854)

## Removal of prior version deprecations/changes

- The `SparsePanel` class has been removed (GH 13778)

- The `pd.sandbox` module has been removed in favor of the external library `pandas-qt` (GH 13670)

- The `pandas.io.data` and `pandas.io.wb` modules are removed in favor of the pandas-datareader package (GH 13724).

- The `pandas.tools.rplot` module has been removed in favor of the seaborn package (GH 13855)

- `DataFrame.to_csv()` has dropped the `engine` parameter, as was deprecated in 0.17.1 (GH 11274, GH 13419)

- `DataFrame.to_dict()` has dropped the `outtype` parameter in favor of `orient` (GH 13627, GH 8486)

- `pd.Categorical` has dropped setting of the `ordered` attribute directly in favor of the `set_ordered` method (GH 13671)

- `pd.Categorical` has dropped the `levels` attribute in favor of `categories` (GH 8376)

- `DataFrame.to_sql()` has dropped the `mysql` option for the `flavor` parameter (GH 13611)

- `Panel.shift()` has dropped the `lags` parameter in favor of `periods` (GH 14041)

- `pd.Index` has dropped the `diff` method in favor of `difference` (GH 13669)

- `pd.DataFrame` has dropped the `to_wide` method in favor of `to_panel` (GH 14039)

- `Series.to_csv` has dropped the `nanRep` parameter in favor of `na_rep` (GH 13804)

- `Series.xs`, `DataFrame.xs`, `Panel.xs`, `Panel.major_xs`, and `Panel.minor_xs` have dropped the `copy` parameter (GH 13781)

- `str.split` has dropped the `return_type` parameter in favor of `expand` (GH 13701)

- Removal of the legacy time rules (offset aliases), deprecated since 0.17.0 (this has been alias since 0.8.0) (GH 13590, GH 13868). Now legacy time rules raises `ValueError`. For the list of currently supported offsets, see here.

- The default value for the `return_type` parameter for `DataFrame.plot.box` and `DataFrame.boxplot` changed from `None` to `"axes"`. These methods will now return a matplotlib axes by default instead of a dictionary of artists. See here (GH 6581).

- The `tquery` and `uquery` functions in the `pandas.io.sql` module are removed (GH 5950).

## Performance improvements

- Improved performance of sparse `IntIndex.intersect` (GH 13082)

- Improved performance of sparse arithmetic with `BlockIndex` when the number of blocks are large, though recommended to use `IntIndex` in such cases (GH 13082)

- Improved performance of `DataFrame.quantile()` as it now operates per-block (GH 11623)

- Improved performance of float64 hash table operations, fixing some very slow indexing and groupby operations in python 3 (GH 13166, GH 13334)

- Improved performance of `DataFrameGroupBy.transform` (GH 12737)

- Improved performance of `Index` and `Series` `.duplicated` (GH 10235)

- Improved performance of `Index.difference` (GH 12044)

- Improved performance of `RangeIndex.is_monotonic_increasing` and `is_monotonic_decreasing` (GH 13749)

- Improved performance of datetime string parsing in `DatetimeIndex` (GH 13692)

- Improved performance of hashing `Period` (GH 12817)

- Improved performance of `factorize` of datetime with timezone (GH 13750)

- Improved performance of by lazily creating indexing hashtables on larger Indexes (GH 14266)

- Improved performance of `groupby.groups` (GH 14293)

- Unnecessary materializing of a MultiIndex when introspecting for memory usage (GH 14308)

## Bug fixes

- Bug in `groupby().shift()`, which could cause a segfault or corruption in rare circumstances when grouping by columns with missing values (GH 13813)

- Bug in `groupby().cumsum()` calculating `cumprod` when `axis=1`. (GH 13994)

- Bug in `pd.to_timedelta()` in which the `errors` parameter was not being respected (GH 13613)

- Bug in `io.json.json_normalize()`, where non-ascii keys raised an exception (GH 13213)

- Bug when passing a not-default-indexed `Series` as `xerr` or `yerr` in `.plot()` (GH 11858)

- Bug in area plot draws legend incorrectly if subplot is enabled or legend is moved after plot (matplotlib 1.5.0 is required to draw area plot legend properly) (GH 9161, GH 13544)

- Bug in `DataFrame` assignment with an object-dtyped `Index` where the resultant column is mutable to the original object. (GH 13522)

- Bug in matplotlib `AutoDataFormatter`; this restores the second scaled formatting and re-adds micro-second scaled formatting (GH 13131)

- Bug in selection from a `HDFStore` with a fixed format and `start` and/or `stop` specified will now return the selected range (GH 8287)

- Bug in `Categorical.from_codes()` where an unhelpful error was raised when an invalid `ordered` parameter was passed in (GH 14058)

- Bug in `Series` construction from a tuple of integers on windows not returning default dtype (int64) (GH 13646)

- Bug in `TimedeltaIndex` addition with a Datetime-like object where addition overflow was not being caught (GH 14068)

- Bug in `.groupby(..).resample(..)` when the same object is called multiple times (GH 13174)

- Bug in `.to_records()` when index name is a unicode string (GH 13172)

- Bug in calling `.memory_usage()` on object which doesn’t implement (GH 12924)

- Regression in `Series.quantile` with nans (also shows up in `.median()` and `.describe()` ); furthermore now names the `Series` with the quantile (GH 13098, GH 13146)

- Bug in `SeriesGroupBy.transform` with datetime values and missing groups (GH 13191)

- Bug where empty `Series` were incorrectly coerced in datetime-like numeric operations (GH 13844)

- Bug in `Categorical` constructor when passed a `Categorical` containing datetimes with timezones (GH 14190)

- Bug in `Series.str.extractall()` with `str` index raises `ValueError` (GH 13156)

- Bug in `Series.str.extractall()` with single group and quantifier (GH 13382)

- Bug in `DatetimeIndex` and `Period` subtraction raises `ValueError` or `AttributeError` rather than `TypeError` (GH 13078)

- Bug in `Index` and `Series` created with `NaN` and `NaT` mixed data may not have `datetime64` dtype (GH 13324)

- Bug in `Index` and `Series` may ignore `np.datetime64('nat')` and `np.timdelta64('nat')` to infer dtype (GH 13324)

- Bug in `PeriodIndex` and `Period` subtraction raises `AttributeError` (GH 13071)

- Bug in `PeriodIndex` construction returning a `float64` index in some circumstances (GH 13067)

- Bug in `.resample(..)` with a `PeriodIndex` not changing its `freq` appropriately when empty (GH 13067)

- Bug in `.resample(..)` with a `PeriodIndex` not retaining its type or name with an empty `DataFrame` appropriately when empty (GH 13212)

- Bug in `groupby(..).apply(..)` when the passed function returns scalar values per group (GH 13468).

- Bug in `groupby(..).resample(..)` where passing some keywords would raise an exception (GH 13235)

- Bug in `.tz_convert` on a tz-aware `DateTimeIndex` that relied on index being sorted for correct results (GH 13306)

- Bug in `.tz_localize` with `dateutil.tz.tzlocal` may return incorrect result (GH 13583)

- Bug in `DatetimeTZDtype` dtype with `dateutil.tz.tzlocal` cannot be regarded as valid dtype (GH 13583)

- Bug in `pd.read_hdf()` where attempting to load an HDF file with a single dataset, that had one or more categorical columns, failed unless the key argument was set to the name of the dataset. (GH 13231)

- Bug in `.rolling()` that allowed a negative integer window in construction of the `Rolling()` object, but would later fail on aggregation (GH 13383)

- Bug in `Series` indexing with tuple-valued data and a numeric index (GH 13509)

- Bug in printing `pd.DataFrame` where unusual elements with the `object` dtype were causing segfaults (GH 13717)

- Bug in ranking `Series` which could result in segfaults (GH 13445)

- Bug in various index types, which did not propagate the name of passed index (GH 12309)

- Bug in `DatetimeIndex`, which did not honour the `copy=True` (GH 13205)

- Bug in `DatetimeIndex.is_normalized` returns incorrectly for normalized date_range in case of local timezones (GH 13459)

- Bug in `pd.concat` and `.append` may coerces `datetime64` and `timedelta` to `object` dtype containing python built-in `datetime` or `timedelta` rather than `Timestamp` or `Timedelta` (GH 13626)

- Bug in `PeriodIndex.append` may raises `AttributeError` when the result is `object` dtype (GH 13221)

- Bug in `CategoricalIndex.append` may accept normal `list` (GH 13626)

- Bug in `pd.concat` and `.append` with the same timezone get reset to UTC (GH 7795)

- Bug in `Series` and `DataFrame` `.append` raises `AmbiguousTimeError` if data contains datetime near DST boundary (GH 13626)

- Bug in `DataFrame.to_csv()` in which float values were being quoted even though quotations were specified for non-numeric values only (GH 12922, GH 13259)

- Bug in `DataFrame.describe()` raising `ValueError` with only boolean columns (GH 13898)

- Bug in `MultiIndex` slicing where extra elements were returned when level is non-unique (GH 12896)

- Bug in `.str.replace` does not raise `TypeError` for invalid replacement (GH 13438)

- Bug in `MultiIndex.from_arrays` which didn’t check for input array lengths matching (GH 13599)

- Bug in `cartesian_product` and `MultiIndex.from_product` which may raise with empty input arrays (GH 12258)

- Bug in `pd.read_csv()` which may cause a segfault or corruption when iterating in large chunks over a stream/file under rare circumstances (GH 13703)

- Bug in `pd.read_csv()` which caused errors to be raised when a dictionary containing scalars is passed in for `na_values` (GH 12224)

- Bug in `pd.read_csv()` which caused BOM files to be incorrectly parsed by not ignoring the BOM (GH 4793)

- Bug in `pd.read_csv()` with `engine='python'` which raised errors when a numpy array was passed in for `usecols` (GH 12546)

- Bug in `pd.read_csv()` where the index columns were being incorrectly parsed when parsed as dates with a `thousands` parameter (GH 14066)

- Bug in `pd.read_csv()` with `engine='python'` in which `NaN` values weren’t being detected after data was converted to numeric values (GH 13314)

- Bug in `pd.read_csv()` in which the `nrows` argument was not properly validated for both engines (GH 10476)

- Bug in `pd.read_csv()` with `engine='python'` in which infinities of mixed-case forms were not being interpreted properly (GH 13274)

- Bug in `pd.read_csv()` with `engine='python'` in which trailing `NaN` values were not being parsed (GH 13320)

- Bug in `pd.read_csv()` with `engine='python'` when reading from a `tempfile.TemporaryFile` on Windows with Python 3 (GH 13398)

- Bug in `pd.read_csv()` that prevents `usecols` kwarg from accepting single-byte unicode strings (GH 13219)

- Bug in `pd.read_csv()` that prevents `usecols` from being an empty set (GH 13402)

- Bug in `pd.read_csv()` in the C engine where the NULL character was not being parsed as NULL (GH 14012)

- Bug in `pd.read_csv()` with `engine='c'` in which NULL `quotechar` was not accepted even though `quoting` was specified as `None` (GH 13411)

- Bug in `pd.read_csv()` with `engine='c'` in which fields were not properly cast to float when quoting was specified as non-numeric (GH 13411)

- Bug in `pd.read_csv()` in Python 2.x with non-UTF8 encoded, multi-character separated data (GH 3404)

- Bug in `pd.read_csv()`, where aliases for utf-xx (e.g. UTF-xx, UTF_xx, utf_xx) raised UnicodeDecodeError (GH 13549)

- Bug in `pd.read_csv`, `pd.read_table`, `pd.read_fwf`, `pd.read_stata` and `pd.read_sas` where files were opened by parsers but not closed if both `chunksize` and `iterator` were `None`. (GH 13940)

- Bug in `StataReader`, `StataWriter`, `XportReader` and `SAS7BDATReader` where a file was not properly closed when an error was raised. (GH 13940)

- Bug in `pd.pivot_table()` where `margins_name` is ignored when `aggfunc` is a list (GH 13354)

- Bug in `pd.Series.str.zfill`, `center`, `ljust`, `rjust`, and `pad` when passing non-integers, did not raise `TypeError` (GH 13598)

- Bug in checking for any null objects in a `TimedeltaIndex`, which always returned `True` (GH 13603)

- Bug in `Series` arithmetic raises `TypeError` if it contains datetime-like as `object` dtype (GH 13043)

- Bug `Series.isnull()` and `Series.notnull()` ignore `Period('NaT')` (GH 13737)

- Bug `Series.fillna()` and `Series.dropna()` don’t affect to `Period('NaT')` (GH 13737)

- Bug in `.fillna(value=np.nan)` incorrectly raises `KeyError` on a `category` dtyped `Series` (GH 14021)

- Bug in extension dtype creation where the created types were not is/identical (GH 13285)

- Bug in `.resample(..)` where incorrect warnings were triggered by IPython introspection (GH 13618)

- Bug in `NaT` - `Period` raises `AttributeError` (GH 13071)

- Bug in `Series` comparison may output incorrect result if rhs contains `NaT` (GH 9005)

- Bug in `Series` and `Index` comparison may output incorrect result if it contains `NaT` with `object` dtype (GH 13592)

- Bug in `Period` addition raises `TypeError` if `Period` is on right hand side (GH 13069)

- Bug in `Period` and `Series` or `Index` comparison raises `TypeError` (GH 13200)

- Bug in `pd.set_eng_float_format()` that would prevent NaN and Inf from formatting (GH 11981)

- Bug in `.unstack` with `Categorical` dtype resets `.ordered` to `True` (GH 13249)

- Clean some compile time warnings in datetime parsing (GH 13607)

- Bug in `factorize` raises `AmbiguousTimeError` if data contains datetime near DST boundary (GH 13750)

- Bug in `.set_index` raises `AmbiguousTimeError` if new index contains DST boundary and multi levels (GH 12920)

- Bug in `.shift` raises `AmbiguousTimeError` if data contains datetime near DST boundary (GH 13926)

- Bug in `pd.read_hdf()` returns incorrect result when a `DataFrame` with a `categorical` column and a query which doesn’t match any values (GH 13792)

- Bug in `.iloc` when indexing with a non lexsorted MultiIndex (GH 13797)

- Bug in `.loc` when indexing with date strings in a reverse sorted `DatetimeIndex` (GH 14316)

- Bug in `Series` comparison operators when dealing with zero dim NumPy arrays (GH 13006)

- Bug in `.combine_first` may return incorrect `dtype` (GH 7630, GH 10567)

- Bug in `groupby` where `apply` returns different result depending on whether first result is `None` or not (GH 12824)

- Bug in `groupby(..).nth()` where the group key is included inconsistently if called after `.head()/.tail()` (GH 12839)

- Bug in `.to_html`, `.to_latex` and `.to_string` silently ignore custom datetime formatter passed through the `formatters` key word (GH 10690)

- Bug in `DataFrame.iterrows()`, not yielding a `Series` subclasse if defined (GH 13977)

- Bug in `pd.to_numeric` when `errors='coerce'` and input contains non-hashable objects (GH 13324)

- Bug in invalid `Timedelta` arithmetic and comparison may raise `ValueError` rather than `TypeError` (GH 13624)

- Bug in invalid datetime parsing in `to_datetime` and `DatetimeIndex` may raise `TypeError` rather than `ValueError` (GH 11169, GH 11287)

- Bug in `Index` created with tz-aware `Timestamp` and mismatched `tz` option incorrectly coerces timezone (GH 13692)

- Bug in `DatetimeIndex` with nanosecond frequency does not include timestamp specified with `end` (GH 13672)

- Bug in `Series` when setting a slice with a `np.timedelta64` (GH 14155)

- Bug in `Index` raises `OutOfBoundsDatetime` if `datetime` exceeds `datetime64[ns]` bounds, rather than coercing to `object` dtype (GH 13663)

- Bug in `Index` may ignore specified `datetime64` or `timedelta64` passed as `dtype` (GH 13981)

- Bug in `RangeIndex` can be created without no arguments rather than raises `TypeError` (GH 13793)

- Bug in `.value_counts()` raises `OutOfBoundsDatetime` if data exceeds `datetime64[ns]` bounds (GH 13663)

- Bug in `DatetimeIndex` may raise `OutOfBoundsDatetime` if input `np.datetime64` has other unit than `ns` (GH 9114)

- Bug in `Series` creation with `np.datetime64` which has other unit than `ns` as `object` dtype results in incorrect values (GH 13876)

- Bug in `resample` with timedelta data where data was casted to float (GH 13119).

- Bug in `pd.isnull()` `pd.notnull()` raise `TypeError` if input datetime-like has other unit than `ns` (GH 13389)

- Bug in `pd.merge()` may raise `TypeError` if input datetime-like has other unit than `ns` (GH 13389)

- Bug in `HDFStore`/`read_hdf()` discarded `DatetimeIndex.name` if `tz` was set (GH 13884)

- Bug in `Categorical.remove_unused_categories()` changes `.codes` dtype to platform int (GH 13261)

- Bug in `groupby` with `as_index=False` returns all NaN’s when grouping on multiple columns including a categorical one (GH 13204)

- Bug in `df.groupby(...)[...]` where getitem with `Int64Index` raised an error (GH 13731)

- Bug in the CSS classes assigned to `DataFrame.style` for index names. Previously they were assigned `"col_heading level<n> col<c>"` where `n` was the number of levels + 1. Now they are assigned `"index_name level<n>"`, where `n` is the correct level for that MultiIndex.

- Bug where `pd.read_gbq()` could throw `ImportError: No module named discovery` as a result of a naming conflict with another python package called apiclient (GH 13454)

- Bug in `Index.union` returns an incorrect result with a named empty index (GH 13432)

- Bugs in `Index.difference` and `DataFrame.join` raise in Python3 when using mixed-integer indexes (GH 13432, GH 12814)

- Bug in subtract tz-aware `datetime.datetime` from tz-aware `datetime64` series (GH 14088)

- Bug in `.to_excel()` when DataFrame contains a MultiIndex which contains a label with a NaN value (GH 13511)

- Bug in invalid frequency offset string like “D1”, “-2-3H” may not raise `ValueError` (GH 13930)

- Bug in `concat` and `groupby` for hierarchical frames with `RangeIndex` levels (GH 13542).

- Bug in `Series.str.contains()` for Series containing only `NaN` values of `object` dtype (GH 14171)

- Bug in `agg()` function on groupby dataframe changes dtype of `datetime64[ns]` column to `float64` (GH 12821)

- Bug in using NumPy ufunc with `PeriodIndex` to add or subtract integer raise `IncompatibleFrequency`. Note that using standard operator like `+` or `-` is recommended, because standard operators use more efficient path (GH 13980)

- Bug in operations on `NaT` returning `float` instead of `datetime64[ns]` (GH 12941)

- Bug in `Series` flexible arithmetic methods (like `.add()`) raises `ValueError` when `axis=None` (GH 13894)

- Bug in `DataFrame.to_csv()` with `MultiIndex` columns in which a stray empty line was added (GH 6618)

- Bug in `DatetimeIndex`, `TimedeltaIndex` and `PeriodIndex.equals()` may return `True` when input isn’t `Index` but contains the same values (GH 13107)

- Bug in assignment against datetime with timezone may not work if it contains datetime near DST boundary (GH 14146)

- Bug in `pd.eval()` and `HDFStore` query truncating long float literals with python 2 (GH 14241)

- Bug in `Index` raises `KeyError` displaying incorrect column when column is not in the df and columns contains duplicate values (GH 13822)

- Bug in `Period` and `PeriodIndex` creating wrong dates when frequency has combined offset aliases (GH 13874)

- Bug in `.to_string()` when called with an integer `line_width` and `index=False` raises an UnboundLocalError exception because `idx` referenced before assignment.

- Bug in `eval()` where the `resolvers` argument would not accept a list (GH 14095)

- Bugs in `stack`, `get_dummies`, `make_axis_dummies` which don’t preserve categorical dtypes in (multi)indexes (GH 13854)

- `PeriodIndex` can now accept `list` and `array` which contains `pd.NaT` (GH 13430)

- Bug in `df.groupby` where `.median()` returns arbitrary values if grouped dataframe contains empty bins (GH 13629)

- Bug in `Index.copy()` where `name` parameter was ignored (GH 14302)

## Contributors

A total of 117 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time.

- Adrien Emery +

- Alex Alekseyev

- Alex Vig +

- Allen Riddell +

- Amol +

- Amol Agrawal +

- Andy R. Terrel +

- Anthonios Partheniou

- Ben Kandel +

- Bob Baxley +

- Brett Rosen +

- Camilo Cota +

- Chris

- Chris Grinolds

- Chris Warth

- Christian Hudon

- Christopher C. Aycock

- Daniel Siladji +

- Douglas McNeil

- Drewrey Lupton +

- Eduardo Blancas Reyes +

- Elliot Marsden +

- Evan Wright

- Felix Marczinowski +

- Francis T. O’Donovan

- Geraint Duck +

- Giacomo Ferroni +

- Grant Roch +

- Gábor Lipták

- Haleemur Ali +

- Hassan Shamim +

- Iulius Curt +

- Ivan Nazarov +

- Jeff Reback

- Jeffrey Gerard +

- Jenn Olsen +

- Jim Crist

- Joe Jevnik

- John Evans +

- John Freeman

- John Liekezer +

- John W. O’Brien

- John Zwinck +

- Johnny Gill +

- Jordan Erenrich +

- Joris Van den Bossche

- Josh Howes +

- Jozef Brandys +

- Ka Wo Chen

- Kamil Sindi +

- Kerby Shedden

- Kernc +

- Kevin Sheppard

- Matthieu Brucher +

- Maximilian Roos

- Michael Scherer +

- Mike Graham +

- Mortada Mehyar

- Muhammad Haseeb Tariq +

- Nate George +

- Neil Parley +

- Nicolas Bonnotte

- OXPHOS

- Pan Deng / Zora +

- Paul +

- Paul Mestemaker +

- Pauli Virtanen

- Pawel Kordek +

- Pietro Battiston

- Piotr Jucha +

- Ravi Kumar Nimmi +

- Robert Gieseke

- Robert Kern +

- Roger Thomas

- Roy Keyes +

- Russell Smith +

- Sahil Dua +

- Sanjiv Lobo +

- Sašo Stanovnik +

- Shawn Heide +

- Sinhrks

- Stephen Kappel +

- Steve Choi +

- Stewart Henderson +

- Sudarshan Konge +

- Thomas A Caswell

- Tom Augspurger

- Tom Bird +

- Uwe Hoffmann +

- WillAyd +

- Xiang Zhang +

- YG-Riku +

- Yadunandan +

- Yaroslav Halchenko

- Yuichiro Kaneko +

- adneu

- agraboso +

- babakkeyvani +

- c123w +

- chris-b1

- cmazzullo +

- conquistador1492 +

- cr3 +

- dsm054

- gfyoung

- harshul1610 +

- iamsimha +

- jackieleng +

- mpuels +

- pijucha +

- priyankjain +

- sinhrks

- wcwagner +

- yui-knk +

- zhangjinjie +

- znmean +

- 颜发才（Yan Facai） +