Skip to content

[SPARK-58470][PS] Use native expressions for NumPy trunc - #57673

Draft
zhengruifeng wants to merge 2 commits into
apache:masterfrom
zhengruifeng:pandas-native-trunc-dev2
Draft

[SPARK-58470][PS] Use native expressions for NumPy trunc#57673
zhengruifeng wants to merge 2 commits into
apache:masterfrom
zhengruifeng:pandas-native-trunc-dev2

Conversation

@zhengruifeng

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Replace the pandas UDF mapping for NumPy trunc on pandas-on-Spark objects with native Spark expressions. The implementation uses the remainder of the absolute double value and reapplies the sign, preserving nulls, NaN, infinities, and signed zero. Add coverage for these values and fractional positive and negative inputs.

Why are the changes needed?

Spark does not provide a numeric trunc function, but the operation can be evaluated with native expressions without crossing the Python worker boundary.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

  • Added parity coverage in NumPyCompatTests.test_np_math_functions.
  • Manually verified the native expression for signed zero, fractional values, infinities, NaN, and large doubles.
  • Passed Ruff format and lint checks for the changed Python files.
  • Focused PySpark tests were not run locally.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)

@zhengruifeng
zhengruifeng marked this pull request as draft July 31, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant