clusterloader2: create_to_schedule etc are based on second-granularity measurements

[justinsb]

What happened:

When I run tests, I see e.g. create_to_schedule has zero values:

    {
      "data": {
        "Perc50": 0,
        "Perc90": 0,
        "Perc99": 0
      },
      "unit": "ms",
      "labels": {
        "Metric": "create_to_schedule"
      }
    },

What you expected to happen:

Some measurements that we can work from.

How to reproduce it (as minimally and precisely as possible):

kind create cluster
go run ./cmd/clusterloader.go --nodes=1 --provider=kind --report-dir=artifacts/ --testconfig=testing/node-throughput/config.yaml --testoverrides=./testing/overrides/node_containerd.yaml --kubeconfig=${HOME}/.kube/config --v=2

Anything else we need to know?:

[justinsb]

So I believe the issue here is that createPhase is based on metadata.creationTimestamp, and schedulePhase is based on the event timestamp. Both have per-second granularity. This means that for small measurements (like scheduling) the values are dominated by randomness around whether we "cross a second".

I think the fix is to record "precise" values based on when we see them in the watch stream. It's not perfect (there's an extra variable which is watch latency), but we don't have to deal with the second-rounding problem.

I'm happy to send a PR to fix this, just wanted to anchor it in an issue.