cuda_10.1.105_418.39 & cudnn-10.1-linux-x64-v7.5.0.56 の両方をインストール(アップグレード)してみました。
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.39 Driver Version: 418.39 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:18:00.0 Off | N/A |
| 23% 26C P8 8W / 250W | 10MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... Off | 00000000:3B:00.0 Off | N/A |
| 23% 24C P8 8W / 250W | 10MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 108... Off | 00000000:86:00.0 Off | N/A |
| 23% 25C P8 8W / 250W | 10MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX 108... Off | 00000000:AF:00.0 Off | N/A |
| 49% 82C P2 254W / 250W | 1485MiB / 11178MiB | 100% Default |
+-------------------------------+----------------------+----------------------+
$ time python ../imagenet/train_imagenet.py -a alex -g 3 -E 50 train.txt test.txt
epoch iteration main/loss validation/main/loss main/accuracy validation/main/accuracy lr
4 1000 3.32066 0.291875 0.01
9 2000 1.94515 0.522844 0.01
13 3000 1.26534 0.665781 0.01
18 4000 0.833074 0.767656 0.01
23 5000 0.562213 0.839125 0.01
27 6000 0.390792 0.88625 0.01
32 7000 0.31295 0.909281 0.01
37 8000 0.250312 0.927938 0.01
41 9000 0.195863 0.943937 0.01
46 10000 0.194644 0.946906 0.01
real 8m27.097s
user 57m21.980s
sys 2m56.775s
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.39 Driver Version: 418.39 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:18:00.0 Off | N/A |
| 23% 26C P8 8W / 250W | 10MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... Off | 00000000:3B:00.0 Off | N/A |
| 23% 24C P8 8W / 250W | 10MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 108... Off | 00000000:86:00.0 Off | N/A |
| 23% 25C P8 8W / 250W | 10MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 GeForce GTX 108... Off | 00000000:AF:00.0 Off | N/A |
| 49% 82C P2 254W / 250W | 1485MiB / 11178MiB | 100% Default |
+-------------------------------+----------------------+----------------------+
$ time python ../imagenet/train_imagenet.py -a alex -g 3 -E 50 train.txt test.txt
epoch iteration main/loss validation/main/loss main/accuracy validation/main/accuracy lr
4 1000 3.32066 0.291875 0.01
9 2000 1.94515 0.522844 0.01
13 3000 1.26534 0.665781 0.01
18 4000 0.833074 0.767656 0.01
23 5000 0.562213 0.839125 0.01
27 6000 0.390792 0.88625 0.01
32 7000 0.31295 0.909281 0.01
37 8000 0.250312 0.927938 0.01
41 9000 0.195863 0.943937 0.01
46 10000 0.194644 0.946906 0.01
real 8m27.097s
user 57m21.980s
sys 2m56.775s