Tesla V100 のクロックを下げる

Tesla V100 搭載のマシンが熱暴走気味なので、クロックを下げてみました (SM クロック 1245MHz-> 1102 MHz)

# nvidia-smi
Mon Aug 5 22:40:26 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro K620 On | 00000000:02:00.0 Off | N/A |
| 34% 33C P8 1W / 30W | 0MiB / 2000MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-PCIE... On | 00000000:82:00.0 Off | 0 |
| N/A 41C P0 26W / 250W | 0MiB / 16130MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

# nvidia-smi -pm 1
# nvidia-smi -q -d SUPPORTED_CLOCKS

==============NVSMI LOG==============

Timestamp : Mon Aug 5 22:41:26 2019
Driver Version : 418.67
CUDA Version : 10.1

Attached GPUs : 2
GPU 00000000:02:00.0
Supported Clocks
Memory : 900 MHz
Graphics : 1124 MHz
Graphics : 1058 MHz
Graphics : 1006 MHz
Memory : 405 MHz
Graphics : 405 MHz

GPU 00000000:82:00.0
Supported Clocks
Memory : 877 MHz
Graphics : 1380 MHz
Graphics : 1372 MHz
Graphics : 1365 MHz
Graphics : 1357 MHz
Graphics : 1350 MHz
Graphics : 1342 MHz
Graphics : 1335 MHz
Graphics : 1327 MHz
Graphics : 1320 MHz
Graphics : 1312 MHz
Graphics : 1305 MHz
Graphics : 1297 MHz
Graphics : 1290 MHz
Graphics : 1282 MHz
Graphics : 1275 MHz
Graphics : 1267 MHz
Graphics : 1260 MHz
Graphics : 1252 MHz
Graphics : 1245 MHz
Graphics : 1237 MHz
Graphics : 1230 MHz
Graphics : 1222 MHz
Graphics : 1215 MHz
Graphics : 1207 MHz
Graphics : 1200 MHz
Graphics : 1192 MHz
Graphics : 1185 MHz
Graphics : 1177 MHz
Graphics : 1170 MHz
Graphics : 1162 MHz
Graphics : 1155 MHz
Graphics : 1147 MHz
Graphics : 1140 MHz
Graphics : 1132 MHz
Graphics : 1125 MHz
Graphics : 1117 MHz
Graphics : 1110 MHz
Graphics : 1102 MHz
Graphics : 1095 MHz
Graphics : 1087 MHz
Graphics : 1080 MHz
Graphics : 1072 MHz
Graphics : 1065 MHz
Graphics : 1057 MHz
Graphics : 1050 MHz
Graphics : 1042 MHz
Graphics : 1035 MHz
Graphics : 1027 MHz
Graphics : 1020 MHz
Graphics : 1012 MHz
Graphics : 1005 MHz
Graphics : 997 MHz
Graphics : 990 MHz
Graphics : 982 MHz
Graphics : 975 MHz
Graphics : 967 MHz
Graphics : 960 MHz
Graphics : 952 MHz
Graphics : 945 MHz
Graphics : 937 MHz
Graphics : 930 MHz
Graphics : 922 MHz
Graphics : 915 MHz
Graphics : 907 MHz
Graphics : 900 MHz
Graphics : 892 MHz
Graphics : 885 MHz
Graphics : 877 MHz
Graphics : 870 MHz
Graphics : 862 MHz
Graphics : 855 MHz
Graphics : 847 MHz
Graphics : 840 MHz
Graphics : 832 MHz
Graphics : 825 MHz
Graphics : 817 MHz
Graphics : 810 MHz
Graphics : 802 MHz
Graphics : 795 MHz
Graphics : 787 MHz
Graphics : 780 MHz
Graphics : 772 MHz
Graphics : 765 MHz
Graphics : 757 MHz
Graphics : 750 MHz
Graphics : 742 MHz
Graphics : 735 MHz
Graphics : 727 MHz
Graphics : 720 MHz
Graphics : 712 MHz
Graphics : 705 MHz
Graphics : 697 MHz
Graphics : 690 MHz
Graphics : 682 MHz
Graphics : 675 MHz
Graphics : 667 MHz
Graphics : 660 MHz
Graphics : 652 MHz
Graphics : 645 MHz
Graphics : 637 MHz
Graphics : 630 MHz
Graphics : 622 MHz
Graphics : 615 MHz
Graphics : 607 MHz
Graphics : 600 MHz
Graphics : 592 MHz
Graphics : 585 MHz
Graphics : 577 MHz
Graphics : 570 MHz
Graphics : 562 MHz
Graphics : 555 MHz
Graphics : 547 MHz
Graphics : 540 MHz
Graphics : 532 MHz
Graphics : 525 MHz
Graphics : 517 MHz
Graphics : 510 MHz
Graphics : 502 MHz
Graphics : 495 MHz
Graphics : 487 MHz
Graphics : 480 MHz
Graphics : 472 MHz
Graphics : 465 MHz
Graphics : 457 MHz
Graphics : 450 MHz
Graphics : 442 MHz
Graphics : 435 MHz
Graphics : 427 MHz
Graphics : 420 MHz
Graphics : 412 MHz
Graphics : 405 MHz
Graphics : 397 MHz
Graphics : 390 MHz
Graphics : 382 MHz
Graphics : 375 MHz
Graphics : 367 MHz
Graphics : 360 MHz
Graphics : 352 MHz
Graphics : 345 MHz
Graphics : 337 MHz
Graphics : 330 MHz
Graphics : 322 MHz
Graphics : 315 MHz
Graphics : 307 MHz
Graphics : 300 MHz
Graphics : 292 MHz
Graphics : 285 MHz
Graphics : 277 MHz
Graphics : 270 MHz
Graphics : 262 MHz
Graphics : 255 MHz
Graphics : 247 MHz
Graphics : 240 MHz
Graphics : 232 MHz
Graphics : 225 MHz
Graphics : 217 MHz
Graphics : 210 MHz
Graphics : 202 MHz
Graphics : 195 MHz
Graphics : 187 MHz
Graphics : 180 MHz
Graphics : 172 MHz
Graphics : 165 MHz
Graphics : 157 MHz
Graphics : 150 MHz
Graphics : 142 MHz
Graphics : 135 MHz

# nvidia-smi -i 1 -ac 877,1102
Applications clocks set to "(MEM 877, SM 1102)" for GPU 00000000:82:00.0
All done.

# nvidia-smi -q -d CLOCK

==============NVSMI LOG==============

Timestamp : Mon Aug 5 22:42:13 2019
Driver Version : 418.67
CUDA Version : 10.1

Attached GPUs : 2
GPU 00000000:02:00.0
Clocks
Graphics : 405 MHz
SM : 405 MHz
Memory : 405 MHz
Video : 405 MHz
Applications Clocks
Graphics : 1006 MHz
Memory : 900 MHz
Default Applications Clocks
Graphics : 1058 MHz
Memory : 900 MHz
Max Clocks
Graphics : 1124 MHz
SM : 1124 MHz
Memory : 900 MHz
Video : 1011 MHz
Max Customer Boost Clocks
Graphics : N/A
SM Clock Samples
Duration : 216.51 sec
Number of Samples : 5
Max : 1058 MHz
Min : 405 MHz
Avg : 405 MHz
Memory Clock Samples
Duration : 216.50 sec
Number of Samples : 5
Max : 900 MHz
Min : 405 MHz
Avg : 405 MHz
Clock Policy
Auto Boost : On
Auto Boost Default : On

GPU 00000000:82:00.0
Clocks
Graphics : 135 MHz
SM : 135 MHz
Memory : 877 MHz
Video : 555 MHz
Applications Clocks
Graphics : 1102 MHz
Memory : 877 MHz
Default Applications Clocks
Graphics : 1245 MHz
Memory : 877 MHz
Max Clocks
Graphics : 1380 MHz
SM : 1380 MHz
Memory : 877 MHz
Video : 1237 MHz
Max Customer Boost Clocks
Graphics : 1380 MHz
SM Clock Samples
Duration : 310.29 sec
Number of Samples : 14
Max : 1245 MHz
Min : 135 MHz
Avg : 174 MHz
Memory Clock Samples
Duration : 310.28 sec
Number of Samples : 14
Max : 877 MHz
Min : 877 MHz
Avg : 877 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A