feat: Accept kwargs in {DataFrame,Series}.to_pandas

[dangotbanned]

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Related issues

• Related issue enh: Passing arguments to pa.Table.to_pandas #2123
• Originated from refactor: Simplify compliant Series.hist #2839 (comment)
• Extra context (enh: Passing arguments to pa.Table.to_pandas #2123 (comment))

Description

Note

Changed scope a bit from the original PR, see (#2879 (review)) and the original description for details

Show original description

Description

While I was looking into (#2839 (comment)) I noticed that the default (and only) behavior for nw.DataFrame.to_pandas when backed by either pa.Table, pl.DataFrame isn't great for nested datatypes:

import pyarrow as pa

data = {"breakpoint": [2, 3], "count": [3, 2]}

>>> pa.table(data).to_struct_array().to_pandas()
0    {'breakpoint': 2, 'count': 3}
1    {'breakpoint': 3, 'count': 2}
dtype: object

We can do better than this, and currently this PR will instead give us something that works with pd.Series.struct:

import pyarrow as pa
import narwhals as nw

data = {"breakpoint": [2, 3], "count": [3, 2]}

>>> nw.from_native(pa.table(data).to_struct_array(), series_only=True).to_pandas()
0    {'breakpoint': 2, 'count': 3}
1    {'breakpoint': 3, 'count': 2}
Name: , dtype: struct<breakpoint: int64, count: int64>[pyarrow]

Questions

1. Would we want this for any other DTypes e.g. List?
2. Would it be worthwhile to expose this as the to_pandas(use_pyarrow_extension_array=...) parameter from polars?
   i. Note that this defaults to False, which is equivalent to our current behavior and pyarrow's

Adds optional keyword-only arguments to both of DataFrame.to_pandas, Series.to_pandas.

This aligns us with the same methods found in polars and pyarrow:

• polars.Series.to_pandas
• pyarrow.ChunkedArray.to_pandas
• polars.DataFrame.to_pandas
• pyarrow.Table.to_pandas

As mentioned in (#2123), this can reduce the memory overhead if configured correctly.

But, the main benefit I'm excited about is that we can preserve pyarrow data types (nulls, nested data) see (#2123 (comment)) - which were previously lost unconditionally

Example

import pyarrow as pa

import narwhals as nw

data = {"breakpoint": [2, 3], "count": [3, 2], "what": [None, 1]}
native = pa.table(data).to_struct_array()
series = nw.from_native(native, series_only=True)

Before

>>> series.to_pandas()
0    {'breakpoint': 2, 'count': 3, 'what': None}
1     {'breakpoint': 3, 'count': 2, 'what': 1.0}
Name: , dtype: object

After

>>> series.to_pandas(use_pyarrow_extension_array=True)
0    {'breakpoint': 2, 'count': 3, 'what': None}
1     {'breakpoint': 3, 'count': 2, 'what': 1.0}
Name: , dtype: struct<breakpoint: int64, count: int64, what: int64>[pyarrow]

Tasks

• Update Compliant* signatures
• Update Compliant* runtime behavior
  • Arrow(DataFrame|Series)
  • Polars(DataFrame|Series)
  • PandasLike(DataFrame|Series)
    • https://docs.rapids.ai/api/cudf/stable/user_guide/api_docs/api/cudf.dataframe.to_pandas/#cudf.DataFrame.to_pandas
    • https://docs.rapids.ai/api/cudf/stable/user_guide/api_docs/api/cudf.series.to_pandas/#cudf.Series.to_pandas
    • feat: Partially utilize for cuDF
• Update DataFrame.to_pandas
• Update Series.to_pandas
  • Signature, pass down to compliant
  • Doc
    • Description (49cb5c6)
    • Example
• Replace earlier tests (0951477), (9e3d295)
  • test: Update DataFrame.to_pandas
  • Series.to_pandas
  • Find which pyarrow added Table.to_struct_array
    • https://github.com/narwhals-dev/narwhals/actions/runs/16556200639/job/46817960571?pr=2879
    • fix: Add to_struct_array compat for pyarrow<15
• Test polars and pandas as well
  • Series
  • DataFrame
• Update PR description to match the updated PR since (feat: Accept kwargs in {DataFrame,Series}.to_pandas #2879 (review))
• Investigate PandasLike alternatives (feat: Accept kwargs in {DataFrame,Series}.to_pandas #2879 (comment))

[FBruzzesi]

Thanks for spotting this @dangotbanned , it seems reasonable to me!
Yet if we do it, I would rather keep it consistent for all the dtypes that would only be supported with ArrowDtype's. WDYT?

[dangotbanned]

Hey @MarcoGorelli, would you be able to give me some boxes to tick in order to get this one over the line please? 🙏

It seems like we've been in agreement from the start on use_pyarrow_extension_array 🙂

But it doesn't seem like the changes I've made in response to (#2879 (comment)) and (#2879 (comment)) have gotten us closer

Really appreciate you bearing with me on this!