gh-138407: avoid useless join operation when calculate the hash vaule for pathlib.Path

[Zheaoli]

Signed-off-by: Manjusaka me@manjusaka.me<!--

Try to fix #138407

The benchmark:

Before this patch

╰─ python -m pyperf timeit -s 'from pathlib import Path' 'p = hash(Path("/tmp/foo/bar/12345"))'
.....................
WARNING: the benchmark result may be unstable
* Not enough samples to get a stable result (95% certainly of less than 1% variation)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

Mean +- std dev: 3.76 us +- 0.11 us

After

╰─ python -m pyperf timeit -s 'from pathlib import Path' 'p = hash(Path("/tmp/foo/bar/12345"))'
.....................
Mean +- std dev: 2.47 us +- 0.04 us

• Issue: Performance regression of pathlib.Path hashing #138407

[picnixz]

Please check.

• that instantiation is not too slow
• that joining two paths is not too slow
• the hashing of a cached instantiated value is faster (but be sure that we always clean the cache after we call hash). This will need a real benchmark function where you should only measure the time to do hash(x) but avoid measuring the time of doing delattr(...).

Benchmark this with paths of various lengths (very short and very long) and with many arguments to joinpath and something like a / b / c / d / e. For that, use a benchmark script but be very careful when dealing with cached attributes. It will be a long work to do because there are a lot of cases.

[Zheaoli]

Please check.

• that instantiation is not too slow
• that joining two paths is not too slow
• the hashing of a cached instantiated value is faster (but be sure that we always clean the cache after we call hash). This will need a real benchmark function where you should only measure the time to do hash(x) but avoid measuring the time of doing delattr(...).

Benchmark this with paths of various lengths (very short and very long) and with many arguments to joinpath and something like a / b / c / d / e. For that, use a benchmark script but be very careful when dealing with cached attributes. It will be a long work to do because there are a lot of cases.

Make sense, I will update benchmark later(I will fix failed CI first)

[barneygale]

FWIW the tests are failing because you need to case-normalize the path on Windows.

This is difficult to optimize because it depends on the whether the user calls str() too. I'd guess that this PR is:

• Faster for hash(p)
• Equal for str(p)
• Slower for str(p); hash(p)
• Slower for hash(p); str(p)

[barneygale]

To add another wrinkle: sometimes paths are instantiated with their normalized string representation already set, e.g. this is true for the results of Path.iterdir(). I suspect your patch will be slower for something like set(big_dir.iterdir()) (but I might be wrong).

[Zheaoli]

I have made a full benchmark test

The following is the result

🧪 Available Tests

• Short Single-Level Path Benchmark: Tests hash performance on simple paths like "/abc"

  • bench_single_level_path.py - Original pathlib implementation
  • bench_single_level_path_new.py - Optimized pathlib implementation

• Long Single-Level Path Benchmark: Tests hash performance on 512-character paths

  • bench_long_single_level_path.py - Original pathlib implementation
  • bench_long_single_level_path_new.py - Optimized pathlib implementation

• Deep Path Benchmark: Tests hash performance on 512-level deep paths (/a/b/c/...)

  • bench_deep_path.py - Original pathlib implementation
  • bench_deep_path_new.py - Optimized pathlib implementation

• Deep Long Path Benchmark: Tests hash performance on 512-level deep paths with 10-character names (/abcdefghij/klmnopqrst/...)

  • bench_deep_long_path.py - Original pathlib implementation
  • bench_deep_long_path_new.py - Optimized pathlib implementation

=� Benchmark Details

The benchmark tests hash performance on single-level paths ("/abc") by:

• Creating 100,000 Path objects in a loop
• Computing hash for each Path object
• Using pyperf for statistical accuracy

=� Performance Results

Local Benchmark Results

+-----------+-------------------+-----------------------+
| Benchmark | single_level_path | single_level_path_new |
+===========+===================+=======================+
| bench     | 197 ms            | 137 ms: 1.44x faster  |
+-----------+-------------------+-----------------------+

Summary: The optimized implementation shows a 1.44x performance improvement (44% faster) for short single-level path hashing operations.

Long Path Benchmark Results

+-----------+------------------------+----------------------------+
| Benchmark | long_single_level_path | long_single_level_path_new |
+===========+========================+============================+
| bench     | 241 ms                 | 180 ms: 1.34x faster       |
+-----------+------------------------+----------------------------+

Summary: The optimized implementation shows a 1.34x performance improvement (34% faster) for long single-level path hashing operations.

Deep Path Benchmark Results

+-----------+-----------+------------------------+
| Benchmark | deep_path | deep_path_new          |
+===========+===========+========================+
| bench     | 1.19 sec  | 1.07 sec: 1.12x faster |
+-----------+-----------+------------------------+

Summary: The optimized implementation shows a 1.12x performance improvement (12% faster) for 512-level deep path hashing operations.

Deep Long Path Benchmark Results

+-----------+----------------+------------------------+
| Benchmark | deep_long_path | deep_long_path_new     |
+===========+================+========================+
| bench     | 1.90 sec       | 2.02 sec: 1.06x slower |
+-----------+----------------+------------------------+

Summary: The optimized implementation shows a 1.06x performance regression (6% slower) for 512-level deep paths with 10-character component names.

Operation Pattern Benchmark Results

Testing different operation patterns on short single-level paths (/abc):

1. hash(p) only - 1.44x faster ✅

+-----------+-------------------+-----------------------+
| Benchmark | single_level_path | single_level_path_new |
+===========+===================+=======================+
| bench     | 197 ms            | 137 ms: 1.44x faster  |
+-----------+-------------------+-----------------------+

2. str(p) only - Equal performance ✅

Benchmark hidden because not significant (1): bench

3. str(p); hash(p) - 1.12x faster ❌ (Expected: Slower)

+-----------+----------+----------------------+
| Benchmark | str_hash | str_hash_new         |
+===========+==========+======================+
| bench     | 213 ms   | 190 ms: 1.12x faster |
+-----------+----------+----------------------+

4. hash(p); str(p) - 1.06x faster ❌ (Expected: Slower)

+-----------+----------+----------------------+
| Benchmark | hash_str | hash_str_new         |
+===========+==========+======================+
| bench     | 201 ms   | 189 ms: 1.06x faster |
+-----------+----------+----------------------+

[Zheaoli]

Here's full benchmark repo https://github.com/Zheaoli/pathlib-benchmark

[barneygale]

I'm struggling to understand why the new code is faster for str(p); hash(p) - the first part is unchanged, and the second part was effectively hash(known_string.lower()) in the old code, whereas the new code seems to do much more. Is it the attribute lookup miss on _str_normcase?

[barneygale]

I'm not sure if it's faster, but this is the simplest way I can think of to implement __hash__() without using str(self) / self._str_normcase:

    def __hash__(self):
        try:
            return self._hash
        except AttributeError:
            if self.parser is posixpath:
                self._hash = hash((self.root, tuple(self._tail)))
            else:
                self._hash = hash((self.drive.lower(),
                                   self.root.lower(),
                                   tuple([part.lower() for part in self._tail])))
            return self._hash

Would you mind running that through your benchmarks please?

[Zheaoli]

Would you mind running that through your benchmarks please?

Sorry for reply late. I'm working on PyCon China 2025 last week. I will update this PR ASAP I can

[Zheaoli]

@barneygale Here's new benchmark

📊 Benchmark Details

The benchmark tests hash performance on single-level paths ("/abc") by:

• Creating 100,000 Path objects in a loop
• Computing hash for each Path object
• Using pyperf for statistical accuracy

📈 Performance Results

Short Single-Level Path Results

+-----------+-------------------+-----------------------+
| Benchmark | single_level_path | single_level_path_new |
+===========+===================+=======================+
| bench     | 268 ms            | 173 ms: 1.55x faster  |
+-----------+-------------------+-----------------------+

Summary: The optimized implementation shows a 1.55x performance improvement (55% faster) for short single-level path hashing operations.

Long Single-Level Path Results

+-----------+------------------------+----------------------------+
| Benchmark | long_single_level_path | long_single_level_path_new |
+===========+========================+============================+
| bench     | 314 ms                 | 214 ms: 1.46x faster       |
+-----------+------------------------+----------------------------+

Summary: The optimized implementation shows a 1.46x performance improvement (46% faster) for long single-level path hashing operations.

Deep Path Results

+-----------+-----------+------------------------+
| Benchmark | deep_path | deep_path_new          |
+===========+===========+========================+
| bench     | 1.19 sec  | 1.06 sec: 1.13x faster |
+-----------+-----------+------------------------+

Summary: The optimized implementation shows a 1.13x performance improvement (13% faster) for 512-level deep path hashing operations.

Deep Long Path Results

+-----------+----------------+------------------------+
| Benchmark | deep_long_path | deep_long_path_new     |
+===========+================+========================+
| bench     | 2.09 sec       | 2.13 sec: 1.02x slower |
+-----------+----------------+------------------------+

Summary: The optimized implementation shows a 1.02x performance regression (2% slower) for 512-level deep paths with 10-character component names.

Operation Pattern Results

Testing different operation patterns on short single-level paths (/abc):

1. hash(p) only - 1.55x faster ✅

+-----------+-------------------+-----------------------+
| Benchmark | single_level_path | single_level_path_new |
+===========+===================+=======================+
| bench     | 268 ms            | 173 ms: 1.55x faster  |
+-----------+-------------------+-----------------------+

2. str(p) only - Equal performance ✅

Benchmark hidden because not significant (1): bench

3. str(p); hash(p) - 1.18x faster ✅

+-----------+----------+----------------------+
| Benchmark | str_hash | str_hash_new         |
+===========+==========+======================+
| bench     | 279 ms   | 236 ms: 1.18x faster |
+-----------+----------+----------------------+

4. hash(p); str(p) - 1.15x faster ✅

+-----------+----------+----------------------+
| Benchmark | hash_str | hash_str_new         |
+===========+==========+======================+
| bench     | 272 ms   | 237 ms: 1.15x faster |
+-----------+----------+----------------------+

[Zheaoli]

@barneygale PTAL when you have time

[github-actions]

This PR is stale because it has been open for 30 days with no activity.