[SPARK-58464][PS] Use native functions for NumPy bitwise shifts

[zhengruifeng]

What changes were proposed in this pull request?

Replace the scalar pandas UDF mappings for NumPy left_shift and right_shift on pandas-on-Spark objects with native Spark SQL shiftleft and shiftright functions through call_function. The public PySpark helpers accept only literal shift counts, while NumPy ufunc dispatch can supply column-valued counts.

Native conditional expressions preserve NumPy behavior for negative and out-of-range int64 shift counts. Add pandas-on-Spark parity coverage for signed int64 boundary values and shift counts below, at, and above the bit width.

Why are the changes needed?

Spark provides native bit-shift functions, so these mappings no longer need to cross the Python worker boundary.

Does this PR introduce any user-facing change?

Yes. NumPy bitwise shifts now execute natively and preserve the Spark integral type of the left operand instead of always returning a long result from the pandas UDF. Their values remain NumPy-compatible, including out-of-range shift counts.

How was this patch tested?

• Added pandas-on-Spark parity coverage for NumPy bitwise shifts with int64 boundary values and out-of-range counts.
• Ran JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 SPARK_TESTING=1 SPARK_PREPEND_CLASSES=1 python/run-tests --testnames pyspark.pandas.tests.test_numpy_compat.
• Ran git diff --check.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)