Skip to content

[SPARK-45900][SQL][PYTHON] Add xxh3_64 and xxh3_128 functions - #57690

Open
SreeramaYeshwanthGowd wants to merge 1 commit into
apache:masterfrom
SreeramaYeshwanthGowd:add-xxh3-hash-functions
Open

[SPARK-45900][SQL][PYTHON] Add xxh3_64 and xxh3_128 functions#57690
SreeramaYeshwanthGowd wants to merge 1 commit into
apache:masterfrom
SreeramaYeshwanthGowd:add-xxh3-hash-functions

Conversation

@SreeramaYeshwanthGowd

Copy link
Copy Markdown

What changes were proposed in this pull request?

Add two built in scalar SQL functions that hash their argument with the XXH3 algorithm:

  • xxh3_64(expr) returns a 64-bit hash as a BIGINT.
  • xxh3_128(expr) returns a 128-bit hash as a 32-character lowercase hex STRING.
> SELECT xxh3_64('Spark');
 80997306238743657
> SELECT xxh3_128('Spark');
 7d57dd84c60c86ca1f4e82ab91a12b5e

API surface added:

  • SQL: xxh3_64, xxh3_128
  • Scala DataFrame: functions.xxh3_64(col), functions.xxh3_128(col)
  • PySpark, classic and Spark Connect: pyspark.sql.functions.xxh3_64 / xxh3_128

Implementation notes:

  • A new XXH3 Java class ports the XXH3 64-bit and 128-bit hashes from the reference implementation (github.com/Cyan4973/xxHash, per doc/xxhash_spec.md). It is a self contained port with no new dependency, mirroring the existing XXH64 helper.
  • The output is byte compatible with the reference implementation, using the default seed 0 and the standard canonical serialization for the 128-bit result (high 64 bits then low 64 bits, big endian).
  • The two expressions live next to the other digest functions in hash.scala and follow the md5 / crc32 pattern: single argument, input implicitly cast to BinaryType, nullIntolerant, with xxh3_64 returning LongType (like crc32) and xxh3_128 returning a hex string (like md5).

Why are the changes needed?

Spark's existing xxhash64 returns 64 bits, which collides too often for large scale surrogate keys or deterministic sampling (a birthday collision becomes likely in the billions of rows). XXH3 is the modern successor to XXH64: xxh3_128 provides a collision resistant 128-bit digest, and xxh3_64 is a faster 64-bit alternative. This is a natural extension of the existing hash function family (md5, crc32, xxhash64).

Design decisions (worth an explicit review, as this is a permanent public API):

  • These are single argument digests that hash the raw bytes of one value with the default seed 0, so the output is byte compatible with the reference XXH3 and interoperates with the reference tools (for example xxh128sum). This differs from xxhash64, which is a variadic, structural hash with an internal seed (not reference compatible). The 128-bit result is the canonical XXH3 hex; the 64-bit result is the two's complement view of the unsigned XXH3 value, so its hex matches the reference while the signed decimal representation may differ.
  • Two functions with static return types are provided instead of one xxhash3(expr, bits) function, because a Spark expression needs a statically known result type.
  • xxh3_128 returns a hex STRING (like md5), matching the reference tool's canonical hex output, rather than raw BINARY. A seeded variant could be added as a follow up.

Does this PR introduce any user-facing change?

Yes. It adds two new built in SQL functions and their Scala and PySpark DataFrame API entries. No existing behavior changes.

How was this patch tested?

  • XXH3Suite validates the port against the reference implementation's known-answer vectors, covering every length branch (0, 1-3, 4-8, 9-16, 17-128, 129-240, and the long path) and block boundaries with both a zero and a non-zero seed, for both the 64-bit and 128-bit outputs.
  • HashExpressionsSuite checks the expressions' values, null handling, and interpreted vs codegen consistency.
  • SQL golden tests in misc-functions.sql (string and binary input, and null propagation), and regenerated sql-expression-schema.md.
  • PySpark doctests.

Was this patch authored or co-authored using generative AI tooling? No

return hash128Hex(input, 0L);
}

/** Returns the XXH3 128-bit hash as a 32-character lowercase hex string (canonical big-endian). */

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like > 100 chars, please fix to avoid lint issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants