API: to_datetime(ints, unit) give requested unit #63347
Conversation
jorisvandenbossche
added
Non-Nano
| # Without this as_unit cast, we would fail to overflow | ||
| # and get much-too-large dates | ||
| return to_datetime(new_data, errors="raise", unit=date_unit).dt.as_unit( | ||
| "ns" |
There was a problem hiding this comment.
I am not directly understanding that comment. new_data are integers here? Why does the return unit of this function need to be nanoseconds? (to preserve current functionality?) Why would this give (wrong?) too large dates?
There was a problem hiding this comment.
This is inside a block that tries large units and if they overflow then tries smaller units. This PR makes the large units not-overflow in cases where this piece of code expects them to. Without this edit, e.g. pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_date_unit fails with
left = DatetimeIndex(['30004724859-08-03', '30007462766-08-06', '30010200673-08-08',
'30012938580-08-10', '300... '30106027418-11-06', '30108765325-11-08', '30111503232-11-10'],
dtype='datetime64[s]', freq=None)
right = DatetimeIndex(['2000-01-03', '2000-01-04', '2000-01-05', '2000-01-06',
'2000-01-07', '2000-01-10', '200...2000-02-08', '2000-02-09',
'2000-02-10', '2000-02-11'],
dtype='datetime64[ns]', freq=None)
|
|
||
| result = read_json(StringIO(json), typ="series") | ||
| expected = ts.copy() | ||
| expected = ts.copy().dt.as_unit("ns") |
There was a problem hiding this comment.
Not for this PR, but so this is another case where we currently return ns unit but could change to use us by default?
|
|
||
| index = pd.MultiIndex( | ||
| levels=[[1, 2, 3], [pd.to_datetime("2000-01-01", unit="ns")]], | ||
| levels=[[1, 2, 3], [pd.to_datetime("2000-01-01", unit="ns").as_unit("ns")]], |
There was a problem hiding this comment.
Did this PR change that? (that this no longer returns nanoseconds)
There was a problem hiding this comment.
Yes, but i didn't realize it until I just checked. I thought this PR only affected integer cases. I also didn't think on main that the unit keyword would have any effect in this case. So there's at least two things I need to look into.
2 participants
to_datetime analogue of #63303.