Restore ROCm job for Ubuntu

[superm1]

Overview

Restore the Ubuntu 22 ROCm job with a working ccache.

Additional information

ccache stopped working for two reasons:

1. We enabled more architectures and were living on the edge and blowing the size away and thus only had 500MB of cache hits.
2. ccache on mtime doesn't work with ROCm delivered from pip wheels (there is a new mtime on clang for each build). We need to change it to content.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES

Used to do experiments and identify the root cause.

[superm1]

CC @jimw567
CC @CISC
CC @IMbackK

[ggerganov]

2GB of the cache seems like a lot for this job - this is 20% of our total budget (10GB). Btw, how long does the build take now?

Anyway, I think we have to start migrating our linux CI to HF-hosted infrastructure. We are starting to hit some hard limits here.

[jimw567]

2GB of the cache seems like a lot for this job - this is 20% of our total budget (10GB). Btw, how long does the build take now?

Anyway, I think we have to start migrating our linux CI to HF-hosted infrastructure. We are starting to hit some hard limits here.

Here is the data from mtime and content based warm runs on the default github runner (4 vCPUs). As shown in the table the ccache is 431MB. We could live with 1GB, I guess. I am also setting up some runners with more CPU cores to speed up cold run as well. As a reference, my regular machine has 16 cores and the build takes about 30mins.

[CISC]

The main issue was probably mtime, upping the size to 1G should be more than enough and will likely never be fully used.

[ggerganov]

Ok, let's change to 1 GB and we can look to merge. Though we should likely wait for the Release CI to flush a bit - we can probably give it a try over the weekend.