Handle case when nvidia-smi fails

[brotchie]

Sometimes a LambdaLabs instance comes up that doesn't have packages correctly installed.

The fix is:

$ sudo apt update && sudo apt upgrade

Should try running nvidia-smi and if it fails, then run the above command.

[brotchie]

Above fix doesn't seem to work. So simply add a diagnostics check when waiting for instance to be available:

1. Could be pinging the IP,
2. Trying to simply open a socket to port 22 (e.g. is it actually listening on port 22)?
3. Running nvidia-smi to check it returns status code 0 (or even parse the output and check which GPU is connected)