Hardware components | ||||||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
Introduction
Hardware Upgrade #2
Estimating the Electrical Cost
Read moreLast year, I delivered a webinar series where we trained a custom model in the cloud using Kaggle, and created a fun robotics application.
- [Webinar Series] How to Control Robots with Gestures
- [Hackster.io] ASL Classification with Vitis-AI
- [Hackster.io] Controlling Robots with ZUBoard
These webinars were given in collaboration with AMD, and at that time we reflected that it would be interesting to do the training locally with an AMD GPU, using their ROCm stack, directly in the Vitis-AI docker containers.
Fast-forward to now, and AMD has sent me a Radeon Pro W7900 GPU card on loan. How cool is that ?
AMD Radeon Pro W7900
- [AMD] Radeon Pro W7900 Product Page
- [AMD] Radeon Pro W7900 DataSheet
- [AMD] Machine Learning Development on Radeon Pro W7900
Coincidently, the day after I received the GPU card, AMD just posted their "Advancing AI" presentation, which I definitely recommend watching:
After watching this presentation, I would have chosen an Instinct GPU, but this Radeon Pro W7900 GPU is very well appreciated :)
Installing the BeastAs crazy as this may seem, I never owned a GPU card, so installing one of these is all new to me...
The first thing to consider are the hardware system requirements for this GPU card, which are:
- one (1) PCI Express 4.0 x16 slot (backward compatible with 3.0)
- triple slot, full height
- two (2) 8pin power connectors
- 295W total board power
I have a HP Z4 G4 workstation, which has a PCIe 3.0 x16 slot, with spacing for a triple slot full height board.
I have a 750W power supply, but only two 6pin power connectors.
1 / 2 • two (2) 6pin connectors on my 750W power supply
Although there are adapter cables that allow pairing up 8pin and 6pin connectors, the opinions on whether it is OK to do this are shared. Some even mention danger of fire, which convinced me not to take this route.
It turns out that my workstation had a 1000W power supply option that included the required 6+2pin connectors for additional power.
1 / 2 • 6+2pin connectors
Although this represents a $150 purchase, it is well worth the investment in order to properly power the Radeon Pro W7900 GPU. I have ordered this replacement PSU, so while I wait for it to arrive...
Installing the SoftwareSince my ultimate goal is to use this GPU card with the ROCm stack in the Vitis-AI docker containers, this all starts with the Vitis-AI documentation:
The following chart indicates the currently supported OS and software:
Vitis-AI 3.5 Requirements
In summary, Vitis-AI 3.5 requires the following:
- Ubuntu 20.04
- ROCm v5.5
Which is really too bad, since I have upgraded to the following a while back:
- Ubuntu 22.04.3
Before taking the drastic step of reverting back to Ubuntu 20.04, I will see where I get with Ubuntu 22.04, and report my findings.
My first attempt at installing the amdgpu driver and ROCm stack was with the following ROCm v5.5 installer for Ubuntu 22.04:
Radeon Pro Software (ROCm) v5.5 for Ubuntu 22.04.3
This failed to build the "amdgpu" kernel driver, which is a known issue reported here:
My second attempt was with the following ROCm v5.7 installer for Ubuntu 22.04:
Radeon Pro Software (ROCm) v5.7 for Ubuntu 22.04.3
Although this installation completed successfully, I now have two unknowns:
- Will the ROCm-enabled Vitis-AI docker containers work with ROCm v5.7 ?
- Will the ROCm-enabled Vitis-AI docker containers recognize the AMD Radeon Pro W7900 GPU ?
I finally received the 1000W power supply that included the required 6+2pin connectors for additional power.
HP Z4 G4 1000W Power Supply
After the open-heart surgery, I installed the GPU card on the available x16 PCIe slot, and booted the workstation.
HP Z4 G4 - Available PCIe X16 slot (triple spacing, full height)
The workstation detected this as a special configuration that required an optional Front Chassis Fan.
POST Error - missing Front Chassis Fan
Looking back at the options for my HP Z4 G4 workstation, I realize that the 1000W power supply upgrade was paired with an additional Front Chassis fan ... don't know how I missed this.
I ordered this as well, so while I wait for it to arrive I am keeping the tower cover off to maximize airflow ...
Detecting the GPU CardAfter the hardware upgrades, the linux machine detected the PCI card.
albertabeef@albertabeef-HP-Z4-G4-Workstation:~/ROCm$ lspci | grep AMD
15:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev 10)
16:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch (rev 10)
17:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 7448
17:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device ab30
albertabeef@albertabeef-HP-Z4-G4-Workstation:~/ROCm$ rocm-smi
========================= ROCm System Management Interface =========================
=================================== Concise Info ===================================
GPU Temp (DieEdge) AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU%
0 29.0c 11.0W 5Mhz 96Mhz 20.0% auto 241.0W 1% 0%
====================================================================================
=============================== End of ROCm SMI Log ================================In addition to the "rocm-smi" utility, there is also a "rocminfo" utility, which provides more detailed information:
root@albertabeef-HP-Z4-G4-Workstation:/dockerx# rocminfo
ROCk module is loaded
=====================
HSA System Attributes
=====================
Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
DMAbuf Support: YES
==========
HSA Agents
==========
*******
Agent 1
*******
Name: Intel(R) Xeon(R) W-2223 CPU @ 3.60GHz
Uuid: CPU-XX
Marketing Name: Intel(R) Xeon(R) W-2223 CPU @ 3.60GHz
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3900
BDFID: 0
Internal Node ID: 0
Compute Unit: 8
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 65579576(0x3e8aa38) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 65579576(0x3e8aa38) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 65579576(0x3e8aa38) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:
*******
Agent 2
*******
Name: gfx1100
Uuid: GPU-4f5da502742a6f3f
Marketing Name: AMD Radeon PRO W7900
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 1
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 6144(0x1800) KB
L3: 98304(0x18000) KB
Chip ID: 29768(0x7448)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 1760
BDFID: 5888
Internal Node ID: 1
Compute Unit: 96
SIMDs per CU: 2
Shader Engines: 6
Shader Arrs. per Eng.: 2
WatchPts on Addr. Ranges:4
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 32(0x20)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 528
SDMA engine uCode:: 19
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 47169536(0x2cfc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS:
Size: 47169536(0x2cfc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx1100
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***This is the test I am most eager to try, running the latest Vitis-AI docker containers (3.5) with the GPU card.
Both the pytorch and tensorflow2 docker containers recognize the ROCm v5.7 stack and Radeon Pro W7900 GPU card, as shown below:
albertabeef@albertabeef-HP-Z4-G4-Workstation1$ ./docker_run.sh xilinx/vitis-ai-pytorch-rocm
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: server misbehaving
Setting up albertabeef 's environment in the Docker container...
usermod: UID '1000' already exists
groupmod: GID '1000' already exists
Running as vitis-ai-user with ID 0 and group 0
==========================================
__ ___ _ _ _____
\ \ / (_) | (_) /\ |_ _|
\ \ / / _| |_ _ ___ ______ / \ | |
\ \/ / | | __| / __|______/ /\ \ | |
\ / | | |_| \__ \ / ____ \ _| |_
\/ |_|\__|_|___/ /_/ \_\_____|
==========================================
Docker Image Version: ubuntu2004-3.5.0.306 (ROCM)
Vitis AI Git Hash: 6a9757a
Build Date: 2023-06-26
WorkFlow: pytorch
vitis-ai-user@albertabeef-HP-Z4-G4-Workstation:/workspace$ rocm-smi
======================= ROCm System Management Interface =======================
================================= Concise Info =================================
GPU Temp (DieEdge) AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU%
0 28.0c 10.0W 0Mhz 96Mhz 20.0% auto 241.0W 1% 0%
================================================================================
============================= End of ROCm SMI Log ==============================
vitis-ai-user@albertabeef-HP-Z4-G4-Workstation:/workspace$ exit
exit
albertabeef@albertabeef-HP-Z4-G4-Workstation$ ./docker_run.sh xilinx/vitis-ai-tensorflow2-rocm
Using default tag: latest
latest: Pulling from xilinx/vitis-ai-tensorflow2-rocm
Digest: sha256:45305a785fcaee6ec41e278aee342f984c9a7a078b46ad0d71c9329e252d25e2
Status: Image is up to date for xilinx/vitis-ai-tensorflow2-rocm:latest
docker.io/xilinx/vitis-ai-tensorflow2-rocm:latest
Setting up albertabeef 's environment in the Docker container...
usermod: UID '1000' already exists
groupmod: GID '1000' already exists
Running as vitis-ai-user with ID 0 and group 0
==========================================
__ ___ _ _ _____
\ \ / (_) | (_) /\ |_ _|
\ \ / / _| |_ _ ___ ______ / \ | |
\ \/ / | | __| / __|______/ /\ \ | |
\ / | | |_| \__ \ / ____ \ _| |_
\/ |_|\__|_|___/ /_/ \_\_____|
==========================================
Docker Image Version: ubuntu2004-3.5.0.301 (ROCM)
Vitis AI Git Hash: 6a9757a
Build Date: 2023-06-27
WorkFlow: tf2
vitis-ai-user@albertabeef-HP-Z4-G4-Workstation:/workspace$ rocm-smi
======================= ROCm System Management Interface =======================
================================= Concise Info =================================
GPU Temp (DieEdge) AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU%
0 28.0c 16.0W 12Mhz 96Mhz 20.0% auto 241.0W 1% 1%
================================================================================
============================= End of ROCm SMI Log ==============================
vitis-ai-user@albertabeef-HP-Z4-G4-Workstation:/workspace$ conda activate vitis-ai-tensorflow2
(vitis-ai-tensorflow2) vitis-ai-user@albertabeef-HP-Z4-G4-Workstation:/workspace$Despite this, however, attempting to run the following system configuration code:
def system_config():
# Get list of GPUs.
gpu_devices = tf.config.list_physical_devices('GPU')
print(gpu_devices)
if len(gpu_devices) > 0:
print('Using GPU')
else:
print('Using CPU')
system_config()Fails to recognize the GPU, and returns the following:
[]
Using CPU
2023-12-19 09:08:59.971003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1990] Ignoring visible gpu device (device: 0, name: , pci bus id: 0000:17:00.0) with AMDGPU version : gfx1100. The supported AMDGPU versions are gfx1030, gfx900, gfx906, gfx908, gfx90a.That is really too bad... I will have to wait for the next Vitis-AI version and hope that this GPU card is supported at that time.
Testing the ROCm docker containersBut not all is lost...
AMD provides docker containers for ROCm v5.7, which I have installed and working:
1 / 2 • AMD docker container for PyTorch framework
Strangely, only Pytorch is available as a docker container for Ubuntu 22.04...
The following quick start page, confirms that the AMD Radeon Pro W7900 is supported:
https://hub.docker.com/r/rocm/pytorch
Starting from ROCm5.7, the "latest" tag docker images also contain Navi31 (gfx1100) support for AMD Radeon RX 7900 XTX and AMD Radeon PRO W7900.
Here we go !
albertabeef@albertabeef-HP-Z4-G4-Workstation$ docker pull rocm/pytorch:rocm5.7_ubuntu22.04_py3.10_pytorch_2.0.1
albertabeef@albertabeef-HP-Z4-G4-Workstation$ alias drun='sudo docker run -it --network=host --device=/dev/kfd --device=/dev/dri --group-add=video --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --shm-size 8G -v $HOME/dockerx:/dockerx -w /dockerx'
root@albertabeef-HP-Z4-G4-Workstation:/dockerx# $ drun rocm/pytorch:rocm5.7_ubuntu22.04_py3.10_pytorch_2.0.1A quick sanity check is via the "torch" package in python:
root@albertabeef-HP-Z4-G4-Workstation:/dockerx# $ python3
>>> import torch
>>> torch.cuda.is_available()
True
>>> print(torch.cuda.get_device_name(0))
AMD Radeon PRO W7900
>>> exit()
root@albertabeef-HP-Z4-G4-Workstation:/dockerx# $A more thorough sanity check is to run the pytorch examples available on github:
root@albertabeef-HP-Z4-G4-Workstation:/dockerx# git clone https://github.com/pytorch/examplesThe PyTorch examples come with the classic MNIST classification example:
root@albertabeef-HP-Z4-G4-Workstation:/dockerx# cd examples/mnist
root@albertabeef-HP-Z4-G4-Workstation:/dockerx/examples/mnist# python3 main.py
Train Epoch: 1 [0/60000 (0%)] Loss: 2.279416
Train Epoch: 1 [640/60000 (1%)] Loss: 1.650533
Train Epoch: 1 [1280/60000 (2%)] Loss: 0.765778
Train Epoch: 1 [1920/60000 (3%)] Loss: 0.614472
Train Epoch: 1 [2560/60000 (4%)] Loss: 0.484266
Train Epoch: 1 [3200/60000 (5%)] Loss: 0.482448
Train Epoch: 1 [3840/60000 (6%)] Loss: 0.221892
Train Epoch: 1 [4480/60000 (7%)] Loss: 0.549753
Train Epoch: 1 [5120/60000 (9%)] Loss: 0.247153
Train Epoch: 1 [5760/60000 (10%)] Loss: 0.201950
Train Epoch: 1 [6400/60000 (11%)] Loss: 0.363852
Train Epoch: 1 [7040/60000 (12%)] Loss: 0.272119
Train Epoch: 1 [7680/60000 (13%)] Loss: 0.239990
Train Epoch: 1 [8320/60000 (14%)] Loss: 0.252087
Train Epoch: 1 [8960/60000 (15%)] Loss: 0.267706
Train Epoch: 1 [9600/60000 (16%)] Loss: 0.150306
Train Epoch: 1 [10240/60000 (17%)] Loss: 0.135036
Train Epoch: 1 [10880/60000 (18%)] Loss: 0.174162
Train Epoch: 1 [11520/60000 (19%)] Loss: 0.176854
Train Epoch: 1 [12160/60000 (20%)] Loss: 0.146198
Train Epoch: 1 [12800/60000 (21%)] Loss: 0.171615
Train Epoch: 1 [13440/60000 (22%)] Loss: 0.116368
Train Epoch: 1 [14080/60000 (23%)] Loss: 0.221513
Train Epoch: 1 [14720/60000 (25%)] Loss: 0.072705
Train Epoch: 1 [15360/60000 (26%)] Loss: 0.308204
Train Epoch: 1 [16000/60000 (27%)] Loss: 0.300730
Train Epoch: 1 [16640/60000 (28%)] Loss: 0.092501
Train Epoch: 1 [17280/60000 (29%)] Loss: 0.134204
Train Epoch: 1 [17920/60000 (30%)] Loss: 0.180102
Train Epoch: 1 [18560/60000 (31%)] Loss: 0.198876
Train Epoch: 1 [19200/60000 (32%)] Loss: 0.207271
Train Epoch: 1 [19840/60000 (33%)] Loss: 0.117478
Train Epoch: 1 [20480/60000 (34%)] Loss: 0.215317
Train Epoch: 1 [21120/60000 (35%)] Loss: 0.144949
Train Epoch: 1 [21760/60000 (36%)] Loss: 0.306057
Train Epoch: 1 [22400/60000 (37%)] Loss: 0.136936
Train Epoch: 1 [23040/60000 (38%)] Loss: 0.188730
Train Epoch: 1 [23680/60000 (39%)] Loss: 0.186152
Train Epoch: 1 [24320/60000 (41%)] Loss: 0.110216
Train Epoch: 1 [24960/60000 (42%)] Loss: 0.069326
Train Epoch: 1 [25600/60000 (43%)] Loss: 0.126163
Train Epoch: 1 [26240/60000 (44%)] Loss: 0.258545
Train Epoch: 1 [26880/60000 (45%)] Loss: 0.283125
Train Epoch: 1 [27520/60000 (46%)] Loss: 0.110059
Train Epoch: 1 [28160/60000 (47%)] Loss: 0.127521
Train Epoch: 1 [28800/60000 (48%)] Loss: 0.031571
Train Epoch: 1 [29440/60000 (49%)] Loss: 0.118762
Train Epoch: 1 [30080/60000 (50%)] Loss: 0.122128
Train Epoch: 1 [30720/60000 (51%)] Loss: 0.140337
Train Epoch: 1 [31360/60000 (52%)] Loss: 0.171650
Train Epoch: 1 [32000/60000 (53%)] Loss: 0.225133
Train Epoch: 1 [32640/60000 (54%)] Loss: 0.200294
Train Epoch: 1 [33280/60000 (55%)] Loss: 0.100042
Train Epoch: 1 [33920/60000 (57%)] Loss: 0.266923
Train Epoch: 1 [34560/60000 (58%)] Loss: 0.112025
Train Epoch: 1 [35200/60000 (59%)] Loss: 0.073423
Train Epoch: 1 [35840/60000 (60%)] Loss: 0.143871
Train Epoch: 1 [36480/60000 (61%)] Loss: 0.211315
Train Epoch: 1 [37120/60000 (62%)] Loss: 0.093414
Train Epoch: 1 [37760/60000 (63%)] Loss: 0.052994
Train Epoch: 1 [38400/60000 (64%)] Loss: 0.101215
Train Epoch: 1 [39040/60000 (65%)] Loss: 0.065297
Train Epoch: 1 [39680/60000 (66%)] Loss: 0.385076
Train Epoch: 1 [40320/60000 (67%)] Loss: 0.096584
Train Epoch: 1 [40960/60000 (68%)] Loss: 0.075692
Train Epoch: 1 [41600/60000 (69%)] Loss: 0.204966
Train Epoch: 1 [42240/60000 (70%)] Loss: 0.196096
Train Epoch: 1 [42880/60000 (71%)] Loss: 0.080916
Train Epoch: 1 [43520/60000 (72%)] Loss: 0.095168
Train Epoch: 1 [44160/60000 (74%)] Loss: 0.054071
Train Epoch: 1 [44800/60000 (75%)] Loss: 0.047702
Train Epoch: 1 [45440/60000 (76%)] Loss: 0.238700
Train Epoch: 1 [46080/60000 (77%)] Loss: 0.044779
Train Epoch: 1 [46720/60000 (78%)] Loss: 0.042750
Train Epoch: 1 [47360/60000 (79%)] Loss: 0.045921
Train Epoch: 1 [48000/60000 (80%)] Loss: 0.065307
Train Epoch: 1 [48640/60000 (81%)] Loss: 0.175462
Train Epoch: 1 [49280/60000 (82%)] Loss: 0.048250
Train Epoch: 1 [49920/60000 (83%)] Loss: 0.099369
Train Epoch: 1 [50560/60000 (84%)] Loss: 0.074099
Train Epoch: 1 [51200/60000 (85%)] Loss: 0.149978
Train Epoch: 1 [51840/60000 (86%)] Loss: 0.226017
Train Epoch: 1 [52480/60000 (87%)] Loss: 0.098241
Train Epoch: 1 [53120/60000 (88%)] Loss: 0.031828
Train Epoch: 1 [53760/60000 (90%)] Loss: 0.053969
Train Epoch: 1 [54400/60000 (91%)] Loss: 0.062234
Train Epoch: 1 [55040/60000 (92%)] Loss: 0.153562
Train Epoch: 1 [55680/60000 (93%)] Loss: 0.132319
Train Epoch: 1 [56320/60000 (94%)] Loss: 0.234221
Train Epoch: 1 [56960/60000 (95%)] Loss: 0.116265
Train Epoch: 1 [57600/60000 (96%)] Loss: 0.096597
Train Epoch: 1 [58240/60000 (97%)] Loss: 0.135910
Train Epoch: 1 [58880/60000 (98%)] Loss: 0.027735
Train Epoch: 1 [59520/60000 (99%)] Loss: 0.063364
Test set: Average loss: 0.0523, Accuracy: 9822/10000 (98%)
Train Epoch: 2 [0/60000 (0%)] Loss: 0.045163
Train Epoch: 2 [640/60000 (1%)] Loss: 0.047603
Train Epoch: 2 [1280/60000 (2%)] Loss: 0.036025
Train Epoch: 2 [1920/60000 (3%)] Loss: 0.239124
Train Epoch: 2 [2560/60000 (4%)] Loss: 0.033531
Train Epoch: 2 [3200/60000 (5%)] Loss: 0.048047
Train Epoch: 2 [3840/60000 (6%)] Loss: 0.078441
Train Epoch: 2 [4480/60000 (7%)] Loss: 0.073367
Train Epoch: 2 [5120/60000 (9%)] Loss: 0.097115
Train Epoch: 2 [5760/60000 (10%)] Loss: 0.046024
Train Epoch: 2 [6400/60000 (11%)] Loss: 0.030496
Train Epoch: 2 [7040/60000 (12%)] Loss: 0.133856
Train Epoch: 2 [7680/60000 (13%)] Loss: 0.053690
Train Epoch: 2 [8320/60000 (14%)] Loss: 0.098344
Train Epoch: 2 [8960/60000 (15%)] Loss: 0.040891
Train Epoch: 2 [9600/60000 (16%)] Loss: 0.086196
Train Epoch: 2 [10240/60000 (17%)] Loss: 0.114554
Train Epoch: 2 [10880/60000 (18%)] Loss: 0.104446
Train Epoch: 2 [11520/60000 (19%)] Loss: 0.061721
Train Epoch: 2 [12160/60000 (20%)] Loss: 0.070107
Train Epoch: 2 [12800/60000 (21%)] Loss: 0.067476
Train Epoch: 2 [13440/60000 (22%)] Loss: 0.011354
Train Epoch: 2 [14080/60000 (23%)] Loss: 0.175211
Train Epoch: 2 [14720/60000 (25%)] Loss: 0.057003
Train Epoch: 2 [15360/60000 (26%)] Loss: 0.295026
Train Epoch: 2 [16000/60000 (27%)] Loss: 0.003309
Train Epoch: 2 [16640/60000 (28%)] Loss: 0.155658
Train Epoch: 2 [17280/60000 (29%)] Loss: 0.108767
Train Epoch: 2 [17920/60000 (30%)] Loss: 0.041116
Train Epoch: 2 [18560/60000 (31%)] Loss: 0.178850
Train Epoch: 2 [19200/60000 (32%)] Loss: 0.083287
Train Epoch: 2 [19840/60000 (33%)] Loss: 0.113318
Train Epoch: 2 [20480/60000 (34%)] Loss: 0.140282
Train Epoch: 2 [21120/60000 (35%)] Loss: 0.062811
Train Epoch: 2 [21760/60000 (36%)] Loss: 0.057027
Train Epoch: 2 [22400/60000 (37%)] Loss: 0.061565
Train Epoch: 2 [23040/60000 (38%)] Loss: 0.016169
Train Epoch: 2 [23680/60000 (39%)] Loss: 0.167491
Train Epoch: 2 [24320/60000 (41%)] Loss: 0.028930
Train Epoch: 2 [24960/60000 (42%)] Loss: 0.168337
Train Epoch: 2 [25600/60000 (43%)] Loss: 0.070296
Train Epoch: 2 [26240/60000 (44%)] Loss: 0.063916
Train Epoch: 2 [26880/60000 (45%)] Loss: 0.124346
Train Epoch: 2 [27520/60000 (46%)] Loss: 0.156117
Train Epoch: 2 [28160/60000 (47%)] Loss: 0.063158
Train Epoch: 2 [28800/60000 (48%)] Loss: 0.410222
Train Epoch: 2 [29440/60000 (49%)] Loss: 0.114112
Train Epoch: 2 [30080/60000 (50%)] Loss: 0.213662
Train Epoch: 2 [30720/60000 (51%)] Loss: 0.006840
Train Epoch: 2 [31360/60000 (52%)] Loss: 0.030451
Train Epoch: 2 [32000/60000 (53%)] Loss: 0.018153
Train Epoch: 2 [32640/60000 (54%)] Loss: 0.052081
Train Epoch: 2 [33280/60000 (55%)] Loss: 0.055731
Train Epoch: 2 [33920/60000 (57%)] Loss: 0.057269
Train Epoch: 2 [34560/60000 (58%)] Loss: 0.164522
Train Epoch: 2 [35200/60000 (59%)] Loss: 0.070857
Train Epoch: 2 [35840/60000 (60%)] Loss: 0.003579
Train Epoch: 2 [36480/60000 (61%)] Loss: 0.056605
Train Epoch: 2 [37120/60000 (62%)] Loss: 0.143370
Train Epoch: 2 [37760/60000 (63%)] Loss: 0.039430
Train Epoch: 2 [38400/60000 (64%)] Loss: 0.049977
Train Epoch: 2 [39040/60000 (65%)] Loss: 0.029009
Train Epoch: 2 [39680/60000 (66%)] Loss: 0.021224
Train Epoch: 2 [40320/60000 (67%)] Loss: 0.037421
Train Epoch: 2 [40960/60000 (68%)] Loss: 0.034541
Train Epoch: 2 [41600/60000 (69%)] Loss: 0.034314
Train Epoch: 2 [42240/60000 (70%)] Loss: 0.035196
Train Epoch: 2 [42880/60000 (71%)] Loss: 0.021462
Train Epoch: 2 [43520/60000 (72%)] Loss: 0.010411
Train Epoch: 2 [44160/60000 (74%)] Loss: 0.045671
Train Epoch: 2 [44800/60000 (75%)] Loss: 0.086281
Train Epoch: 2 [45440/60000 (76%)] Loss: 0.051936
Train Epoch: 2 [46080/60000 (77%)] Loss: 0.031195
Train Epoch: 2 [46720/60000 (78%)] Loss: 0.105621
Train Epoch: 2 [47360/60000 (79%)] Loss: 0.088800
Train Epoch: 2 [48000/60000 (80%)] Loss: 0.036342
Train Epoch: 2 [48640/60000 (81%)] Loss: 0.127459
Train Epoch: 2 [49280/60000 (82%)] Loss: 0.067723
Train Epoch: 2 [49920/60000 (83%)] Loss: 0.040831
Train Epoch: 2 [50560/60000 (84%)] Loss: 0.068403
Train Epoch: 2 [51200/60000 (85%)] Loss: 0.053798
Train Epoch: 2 [51840/60000 (86%)] Loss: 0.042621
Train Epoch: 2 [52480/60000 (87%)] Loss: 0.039189
Train Epoch: 2 [53120/60000 (88%)] Loss: 0.276354
Train Epoch: 2 [53760/60000 (90%)] Loss: 0.121061
Train Epoch: 2 [54400/60000 (91%)] Loss: 0.058045
Train Epoch: 2 [55040/60000 (92%)] Loss: 0.009123
Train Epoch: 2 [55680/60000 (93%)] Loss: 0.129157
Train Epoch: 2 [56320/60000 (94%)] Loss: 0.009530
Train Epoch: 2 [56960/60000 (95%)] Loss: 0.079902
Train Epoch: 2 [57600/60000 (96%)] Loss: 0.043248
Train Epoch: 2 [58240/60000 (97%)] Loss: 0.018305
Train Epoch: 2 [58880/60000 (98%)] Loss: 0.024661
Train Epoch: 2 [59520/60000 (99%)] Loss: 0.030346
Test set: Average loss: 0.0345, Accuracy: 9889/10000 (99%)
Train Epoch: 3 [0/60000 (0%)] Loss: 0.031817
Train Epoch: 3 [640/60000 (1%)] Loss: 0.070033
Train Epoch: 3 [1280/60000 (2%)] Loss: 0.068437
Train Epoch: 3 [1920/60000 (3%)] Loss: 0.037296
Train Epoch: 3 [2560/60000 (4%)] Loss: 0.025286
Train Epoch: 3 [3200/60000 (5%)] Loss: 0.040879
Train Epoch: 3 [3840/60000 (6%)] Loss: 0.010902
Train Epoch: 3 [4480/60000 (7%)] Loss: 0.013186
Train Epoch: 3 [5120/60000 (9%)] Loss: 0.056757
Train Epoch: 3 [5760/60000 (10%)] Loss: 0.009007
Train Epoch: 3 [6400/60000 (11%)] Loss: 0.012295
Train Epoch: 3 [7040/60000 (12%)] Loss: 0.065799
Train Epoch: 3 [7680/60000 (13%)] Loss: 0.055801
Train Epoch: 3 [8320/60000 (14%)] Loss: 0.042944
Train Epoch: 3 [8960/60000 (15%)] Loss: 0.016761
Train Epoch: 3 [9600/60000 (16%)] Loss: 0.056136
Train Epoch: 3 [10240/60000 (17%)] Loss: 0.009733
Train Epoch: 3 [10880/60000 (18%)] Loss: 0.149223
Train Epoch: 3 [11520/60000 (19%)] Loss: 0.024051
Train Epoch: 3 [12160/60000 (20%)] Loss: 0.007287
Train Epoch: 3 [12800/60000 (21%)] Loss: 0.025377
Train Epoch: 3 [13440/60000 (22%)] Loss: 0.046749
Train Epoch: 3 [14080/60000 (23%)] Loss: 0.082273
Train Epoch: 3 [14720/60000 (25%)] Loss: 0.002023
Train Epoch: 3 [15360/60000 (26%)] Loss: 0.024286
Train Epoch: 3 [16000/60000 (27%)] Loss: 0.248030
Train Epoch: 3 [16640/60000 (28%)] Loss: 0.036516
Train Epoch: 3 [17280/60000 (29%)] Loss: 0.060476
Train Epoch: 3 [17920/60000 (30%)] Loss: 0.168245
Train Epoch: 3 [18560/60000 (31%)] Loss: 0.048446
Train Epoch: 3 [19200/60000 (32%)] Loss: 0.048896
Train Epoch: 3 [19840/60000 (33%)] Loss: 0.248206
Train Epoch: 3 [20480/60000 (34%)] Loss: 0.008382
Train Epoch: 3 [21120/60000 (35%)] Loss: 0.025487
Train Epoch: 3 [21760/60000 (36%)] Loss: 0.062187
Train Epoch: 3 [22400/60000 (37%)] Loss: 0.229837
Train Epoch: 3 [23040/60000 (38%)] Loss: 0.290953
Train Epoch: 3 [23680/60000 (39%)] Loss: 0.068614
Train Epoch: 3 [24320/60000 (41%)] Loss: 0.008134
Train Epoch: 3 [24960/60000 (42%)] Loss: 0.054782
Train Epoch: 3 [25600/60000 (43%)] Loss: 0.004235
Train Epoch: 3 [26240/60000 (44%)] Loss: 0.033991
Train Epoch: 3 [26880/60000 (45%)] Loss: 0.009926
Train Epoch: 3 [27520/60000 (46%)] Loss: 0.033616
Train Epoch: 3 [28160/60000 (47%)] Loss: 0.010513
Train Epoch: 3 [28800/60000 (48%)] Loss: 0.194457
Train Epoch: 3 [29440/60000 (49%)] Loss: 0.004999
Train Epoch: 3 [30080/60000 (50%)] Loss: 0.206723
Train Epoch: 3 [30720/60000 (51%)] Loss: 0.131418
Train Epoch: 3 [31360/60000 (52%)] Loss: 0.091117
Train Epoch: 3 [32000/60000 (53%)] Loss: 0.190356
Train Epoch: 3 [32640/60000 (54%)] Loss: 0.028072
Train Epoch: 3 [33280/60000 (55%)] Loss: 0.016485
Train Epoch: 3 [33920/60000 (57%)] Loss: 0.004004
Train Epoch: 3 [34560/60000 (58%)] Loss: 0.003702
Train Epoch: 3 [35200/60000 (59%)] Loss: 0.053455
Train Epoch: 3 [35840/60000 (60%)] Loss: 0.264043
Train Epoch: 3 [36480/60000 (61%)] Loss: 0.018753
Train Epoch: 3 [37120/60000 (62%)] Loss: 0.166800
Train Epoch: 3 [37760/60000 (63%)] Loss: 0.320760
Train Epoch: 3 [38400/60000 (64%)] Loss: 0.032863
Train Epoch: 3 [39040/60000 (65%)] Loss: 0.090924
Train Epoch: 3 [39680/60000 (66%)] Loss: 0.040496
Train Epoch: 3 [40320/60000 (67%)] Loss: 0.013433
Train Epoch: 3 [40960/60000 (68%)] Loss: 0.084366
Train Epoch: 3 [41600/60000 (69%)] Loss: 0.044813
Train Epoch: 3 [42240/60000 (70%)] Loss: 0.092339
Train Epoch: 3 [42880/60000 (71%)] Loss: 0.096099
Train Epoch: 3 [43520/60000 (72%)] Loss: 0.003775
Train Epoch: 3 [44160/60000 (74%)] Loss: 0.017234
Train Epoch: 3 [44800/60000 (75%)] Loss: 0.014356
Train Epoch: 3 [45440/60000 (76%)] Loss: 0.059401
Train Epoch: 3 [46080/60000 (77%)] Loss: 0.022429
Train Epoch: 3 [46720/60000 (78%)] Loss: 0.031060
Train Epoch: 3 [47360/60000 (79%)] Loss: 0.002502
Train Epoch: 3 [48000/60000 (80%)] Loss: 0.046735
Train Epoch: 3 [48640/60000 (81%)] Loss: 0.061205
Train Epoch: 3 [49280/60000 (82%)] Loss: 0.046381
Train Epoch: 3 [49920/60000 (83%)] Loss: 0.049245
Train Epoch: 3 [50560/60000 (84%)] Loss: 0.013162
Train Epoch: 3 [51200/60000 (85%)] Loss: 0.105103
Train Epoch: 3 [51840/60000 (86%)] Loss: 0.002515
Train Epoch: 3 [52480/60000 (87%)] Loss: 0.039819
Train Epoch: 3 [53120/60000 (88%)] Loss: 0.317206
Train Epoch: 3 [53760/60000 (90%)] Loss: 0.035227
Train Epoch: 3 [54400/60000 (91%)] Loss: 0.057587
Train Epoch: 3 [55040/60000 (92%)] Loss: 0.038839
Train Epoch: 3 [55680/60000 (93%)] Loss: 0.036158
Train Epoch: 3 [56320/60000 (94%)] Loss: 0.196687
Train Epoch: 3 [56960/60000 (95%)] Loss: 0.005493
Train Epoch: 3 [57600/60000 (96%)] Loss: 0.008280
Train Epoch: 3 [58240/60000 (97%)] Loss: 0.050596
Train Epoch: 3 [58880/60000 (98%)] Loss: 0.035560
Train Epoch: 3 [59520/60000 (99%)] Loss: 0.085061
Test set: Average loss: 0.0351, Accuracy: 9887/10000 (99%)
Train Epoch: 4 [0/60000 (0%)] Loss: 0.022249
Train Epoch: 4 [640/60000 (1%)] Loss: 0.023039
Train Epoch: 4 [1280/60000 (2%)] Loss: 0.080782
Train Epoch: 4 [1920/60000 (3%)] Loss: 0.009227
Train Epoch: 4 [2560/60000 (4%)] Loss: 0.110548
Train Epoch: 4 [3200/60000 (5%)] Loss: 0.045937
Train Epoch: 4 [3840/60000 (6%)] Loss: 0.009661
Train Epoch: 4 [4480/60000 (7%)] Loss: 0.004341
Train Epoch: 4 [5120/60000 (9%)] Loss: 0.039671
Train Epoch: 4 [5760/60000 (10%)] Loss: 0.008958
Train Epoch: 4 [6400/60000 (11%)] Loss: 0.004605
Train Epoch: 4 [7040/60000 (12%)] Loss: 0.059503
Train Epoch: 4 [7680/60000 (13%)] Loss: 0.005586
Train Epoch: 4 [8320/60000 (14%)] Loss: 0.041664
Train Epoch: 4 [8960/60000 (15%)] Loss: 0.054319
Train Epoch: 4 [9600/60000 (16%)] Loss: 0.016082
Train Epoch: 4 [10240/60000 (17%)] Loss: 0.018330
Train Epoch: 4 [10880/60000 (18%)] Loss: 0.050755
Train Epoch: 4 [11520/60000 (19%)] Loss: 0.028240
Train Epoch: 4 [12160/60000 (20%)] Loss: 0.084743
Train Epoch: 4 [12800/60000 (21%)] Loss: 0.096246
Train Epoch: 4 [13440/60000 (22%)] Loss: 0.040874
Train Epoch: 4 [14080/60000 (23%)] Loss: 0.058291
Train Epoch: 4 [14720/60000 (25%)] Loss: 0.021153
Train Epoch: 4 [15360/60000 (26%)] Loss: 0.012191
Train Epoch: 4 [16000/60000 (27%)] Loss: 0.002721
Train Epoch: 4 [16640/60000 (28%)] Loss: 0.003195
Train Epoch: 4 [17280/60000 (29%)] Loss: 0.011672
Train Epoch: 4 [17920/60000 (30%)] Loss: 0.078698
Train Epoch: 4 [18560/60000 (31%)] Loss: 0.050549
Train Epoch: 4 [19200/60000 (32%)] Loss: 0.003490
Train Epoch: 4 [19840/60000 (33%)] Loss: 0.031895
Train Epoch: 4 [20480/60000 (34%)] Loss: 0.052208
Train Epoch: 4 [21120/60000 (35%)] Loss: 0.035184
Train Epoch: 4 [21760/60000 (36%)] Loss: 0.011007
Train Epoch: 4 [22400/60000 (37%)] Loss: 0.015482
Train Epoch: 4 [23040/60000 (38%)] Loss: 0.021405
Train Epoch: 4 [23680/60000 (39%)] Loss: 0.065724
Train Epoch: 4 [24320/60000 (41%)] Loss: 0.016041
Train Epoch: 4 [24960/60000 (42%)] Loss: 0.040696
Train Epoch: 4 [25600/60000 (43%)] Loss: 0.010206
Train Epoch: 4 [26240/60000 (44%)] Loss: 0.069689
Train Epoch: 4 [26880/60000 (45%)] Loss: 0.013439
Train Epoch: 4 [27520/60000 (46%)] Loss: 0.160770
Train Epoch: 4 [28160/60000 (47%)] Loss: 0.031039
Train Epoch: 4 [28800/60000 (48%)] Loss: 0.033234
Train Epoch: 4 [29440/60000 (49%)] Loss: 0.123461
Train Epoch: 4 [30080/60000 (50%)] Loss: 0.035576
Train Epoch: 4 [30720/60000 (51%)] Loss: 0.162622
Train Epoch: 4 [31360/60000 (52%)] Loss: 0.008411
Train Epoch: 4 [32000/60000 (53%)] Loss: 0.007099
Train Epoch: 4 [32640/60000 (54%)] Loss: 0.032651
Train Epoch: 4 [33280/60000 (55%)] Loss: 0.065300
Train Epoch: 4 [33920/60000 (57%)] Loss: 0.007988
Train Epoch: 4 [34560/60000 (58%)] Loss: 0.034462
Train Epoch: 4 [35200/60000 (59%)] Loss: 0.009092
Train Epoch: 4 [35840/60000 (60%)] Loss: 0.015822
Train Epoch: 4 [36480/60000 (61%)] Loss: 0.015659
Train Epoch: 4 [37120/60000 (62%)] Loss: 0.009652
Train Epoch: 4 [37760/60000 (63%)] Loss: 0.029330
Train Epoch: 4 [38400/60000 (64%)] Loss: 0.053672
Train Epoch: 4 [39040/60000 (65%)] Loss: 0.034806
Train Epoch: 4 [39680/60000 (66%)] Loss: 0.017358
Train Epoch: 4 [40320/60000 (67%)] Loss: 0.032102
Train Epoch: 4 [40960/60000 (68%)] Loss: 0.030450
Train Epoch: 4 [41600/60000 (69%)] Loss: 0.007481
Train Epoch: 4 [42240/60000 (70%)] Loss: 0.004294
Train Epoch: 4 [42880/60000 (71%)] Loss: 0.014999
Train Epoch: 4 [43520/60000 (72%)] Loss: 0.031101
Train Epoch: 4 [44160/60000 (74%)] Loss: 0.015222
Train Epoch: 4 [44800/60000 (75%)] Loss: 0.000280
Train Epoch: 4 [45440/60000 (76%)] Loss: 0.013675
Train Epoch: 4 [46080/60000 (77%)] Loss: 0.063409
Train Epoch: 4 [46720/60000 (78%)] Loss: 0.021591
Train Epoch: 4 [47360/60000 (79%)] Loss: 0.089902
Train Epoch: 4 [48000/60000 (80%)] Loss: 0.005891
Train Epoch: 4 [48640/60000 (81%)] Loss: 0.022563
Train Epoch: 4 [49280/60000 (82%)] Loss: 0.053116
Train Epoch: 4 [49920/60000 (83%)] Loss: 0.010057
Train Epoch: 4 [50560/60000 (84%)] Loss: 0.003650
Train Epoch: 4 [51200/60000 (85%)] Loss: 0.014271
Train Epoch: 4 [51840/60000 (86%)] Loss: 0.027841
Train Epoch: 4 [52480/60000 (87%)] Loss: 0.043027
Train Epoch: 4 [53120/60000 (88%)] Loss: 0.059816
Train Epoch: 4 [53760/60000 (90%)] Loss: 0.012417
Train Epoch: 4 [54400/60000 (91%)] Loss: 0.044598
Train Epoch: 4 [55040/60000 (92%)] Loss: 0.002129
Train Epoch: 4 [55680/60000 (93%)] Loss: 0.025880
Train Epoch: 4 [56320/60000 (94%)] Loss: 0.022374
Train Epoch: 4 [56960/60000 (95%)] Loss: 0.024055
Train Epoch: 4 [57600/60000 (96%)] Loss: 0.117871
Train Epoch: 4 [58240/60000 (97%)] Loss: 0.051740
Train Epoch: 4 [58880/60000 (98%)] Loss: 0.036338
Train Epoch: 4 [59520/60000 (99%)] Loss: 0.010673
Test set: Average loss: 0.0306, Accuracy: 9900/10000 (99%)
Train Epoch: 5 [0/60000 (0%)] Loss: 0.015028
Train Epoch: 5 [640/60000 (1%)] Loss: 0.073557
Train Epoch: 5 [1280/60000 (2%)] Loss: 0.012012
Train Epoch: 5 [1920/60000 (3%)] Loss: 0.019785
Train Epoch: 5 [2560/60000 (4%)] Loss: 0.018503
Train Epoch: 5 [3200/60000 (5%)] Loss: 0.057524
Train Epoch: 5 [3840/60000 (6%)] Loss: 0.151951
Train Epoch: 5 [4480/60000 (7%)] Loss: 0.082396
Train Epoch: 5 [5120/60000 (9%)] Loss: 0.056857
Train Epoch: 5 [5760/60000 (10%)] Loss: 0.077433
Train Epoch: 5 [6400/60000 (11%)] Loss: 0.018822
Train Epoch: 5 [7040/60000 (12%)] Loss: 0.001748
Train Epoch: 5 [7680/60000 (13%)] Loss: 0.009337
Train Epoch: 5 [8320/60000 (14%)] Loss: 0.011009
Train Epoch: 5 [8960/60000 (15%)] Loss: 0.067346
Train Epoch: 5 [9600/60000 (16%)] Loss: 0.003695
Train Epoch: 5 [10240/60000 (17%)] Loss: 0.010226
Train Epoch: 5 [10880/60000 (18%)] Loss: 0.018558
Train Epoch: 5 [11520/60000 (19%)] Loss: 0.057915
Train Epoch: 5 [12160/60000 (20%)] Loss: 0.019058
Train Epoch: 5 [12800/60000 (21%)] Loss: 0.004824
Train Epoch: 5 [13440/60000 (22%)] Loss: 0.099939
Train Epoch: 5 [14080/60000 (23%)] Loss: 0.028397
Train Epoch: 5 [14720/60000 (25%)] Loss: 0.003157
Train Epoch: 5 [15360/60000 (26%)] Loss: 0.087539
Train Epoch: 5 [16000/60000 (27%)] Loss: 0.081277
Train Epoch: 5 [16640/60000 (28%)] Loss: 0.021202
Train Epoch: 5 [17280/60000 (29%)] Loss: 0.016788
Train Epoch: 5 [17920/60000 (30%)] Loss: 0.042152
Train Epoch: 5 [18560/60000 (31%)] Loss: 0.050758
Train Epoch: 5 [19200/60000 (32%)] Loss: 0.016031
Train Epoch: 5 [19840/60000 (33%)] Loss: 0.022243
Train Epoch: 5 [20480/60000 (34%)] Loss: 0.038452
Train Epoch: 5 [21120/60000 (35%)] Loss: 0.007969
Train Epoch: 5 [21760/60000 (36%)] Loss: 0.073699
Train Epoch: 5 [22400/60000 (37%)] Loss: 0.082593
Train Epoch: 5 [23040/60000 (38%)] Loss: 0.010222
Train Epoch: 5 [23680/60000 (39%)] Loss: 0.050240
Train Epoch: 5 [24320/60000 (41%)] Loss: 0.017879
Train Epoch: 5 [24960/60000 (42%)] Loss: 0.014092
Train Epoch: 5 [25600/60000 (43%)] Loss: 0.006328
Train Epoch: 5 [26240/60000 (44%)] Loss: 0.050697
Train Epoch: 5 [26880/60000 (45%)] Loss: 0.157726
Train Epoch: 5 [27520/60000 (46%)] Loss: 0.056370
Train Epoch: 5 [28160/60000 (47%)] Loss: 0.110564
Train Epoch: 5 [28800/60000 (48%)] Loss: 0.046650
Train Epoch: 5 [29440/60000 (49%)] Loss: 0.005971
Train Epoch: 5 [30080/60000 (50%)] Loss: 0.052866
Train Epoch: 5 [30720/60000 (51%)] Loss: 0.001035
Train Epoch: 5 [31360/60000 (52%)] Loss: 0.050984
Train Epoch: 5 [32000/60000 (53%)] Loss: 0.182988
Train Epoch: 5 [32640/60000 (54%)] Loss: 0.005140
Train Epoch: 5 [33280/60000 (55%)] Loss: 0.067844
Train Epoch: 5 [33920/60000 (57%)] Loss: 0.015252
Train Epoch: 5 [34560/60000 (58%)] Loss: 0.029659
Train Epoch: 5 [35200/60000 (59%)] Loss: 0.006034
Train Epoch: 5 [35840/60000 (60%)] Loss: 0.047515
Train Epoch: 5 [36480/60000 (61%)] Loss: 0.023498
Train Epoch: 5 [37120/60000 (62%)] Loss: 0.001093
Train Epoch: 5 [37760/60000 (63%)] Loss: 0.030596
Train Epoch: 5 [38400/60000 (64%)] Loss: 0.003272
Train Epoch: 5 [39040/60000 (65%)] Loss: 0.015613
Train Epoch: 5 [39680/60000 (66%)] Loss: 0.048291
Train Epoch: 5 [40320/60000 (67%)] Loss: 0.041594
Train Epoch: 5 [40960/60000 (68%)] Loss: 0.109485
Train Epoch: 5 [41600/60000 (69%)] Loss: 0.022999
Train Epoch: 5 [42240/60000 (70%)] Loss: 0.073143
Train Epoch: 5 [42880/60000 (71%)] Loss: 0.034382
Train Epoch: 5 [43520/60000 (72%)] Loss: 0.071114
Train Epoch: 5 [44160/60000 (74%)] Loss: 0.001795
Train Epoch: 5 [44800/60000 (75%)] Loss: 0.005797
Train Epoch: 5 [45440/60000 (76%)] Loss: 0.036328
Train Epoch: 5 [46080/60000 (77%)] Loss: 0.001735
Train Epoch: 5 [46720/60000 (78%)] Loss: 0.016361
Train Epoch: 5 [47360/60000 (79%)] Loss: 0.029622
Train Epoch: 5 [48000/60000 (80%)] Loss: 0.030281
Train Epoch: 5 [48640/60000 (81%)] Loss: 0.129943
Train Epoch: 5 [49280/60000 (82%)] Loss: 0.161838
Train Epoch: 5 [49920/60000 (83%)] Loss: 0.098718
Train Epoch: 5 [50560/60000 (84%)] Loss: 0.014711
Train Epoch: 5 [51200/60000 (85%)] Loss: 0.031646
Train Epoch: 5 [51840/60000 (86%)] Loss: 0.015925
Train Epoch: 5 [52480/60000 (87%)] Loss: 0.148731
Train Epoch: 5 [53120/60000 (88%)] Loss: 0.042788
Train Epoch: 5 [53760/60000 (90%)] Loss: 0.009525
Train Epoch: 5 [54400/60000 (91%)] Loss: 0.019534
Train Epoch: 5 [55040/60000 (92%)] Loss: 0.003148
Train Epoch: 5 [55680/60000 (93%)] Loss: 0.008416
Train Epoch: 5 [56320/60000 (94%)] Loss: 0.010743
Train Epoch: 5 [56960/60000 (95%)] Loss: 0.009166
Train Epoch: 5 [57600/60000 (96%)] Loss: 0.009997
Train Epoch: 5 [58240/60000 (97%)] Loss: 0.075954
Train Epoch: 5 [58880/60000 (98%)] Loss: 0.005059
Train Epoch: 5 [59520/60000 (99%)] Loss: 0.015331
Test set: Average loss: 0.0298, Accuracy: 9903/10000 (99%)
Train Epoch: 6 [0/60000 (0%)] Loss: 0.126990
Train Epoch: 6 [640/60000 (1%)] Loss: 0.010935
Train Epoch: 6 [1280/60000 (2%)] Loss: 0.165594
Train Epoch: 6 [1920/60000 (3%)] Loss: 0.234482
Train Epoch: 6 [2560/60000 (4%)] Loss: 0.008931
Train Epoch: 6 [3200/60000 (5%)] Loss: 0.042313
Train Epoch: 6 [3840/60000 (6%)] Loss: 0.036666
Train Epoch: 6 [4480/60000 (7%)] Loss: 0.011998
Train Epoch: 6 [5120/60000 (9%)] Loss: 0.009047
Train Epoch: 6 [5760/60000 (10%)] Loss: 0.015043
Train Epoch: 6 [6400/60000 (11%)] Loss: 0.004203
Train Epoch: 6 [7040/60000 (12%)] Loss: 0.000474
Train Epoch: 6 [7680/60000 (13%)] Loss: 0.009949
Train Epoch: 6 [8320/60000 (14%)] Loss: 0.027245
Train Epoch: 6 [8960/60000 (15%)] Loss: 0.077088
Train Epoch: 6 [9600/60000 (16%)] Loss: 0.090391
Train Epoch: 6 [10240/60000 (17%)] Loss: 0.005082
Train Epoch: 6 [10880/60000 (18%)] Loss: 0.016389
Train Epoch: 6 [11520/60000 (19%)] Loss: 0.029068
Train Epoch: 6 [12160/60000 (20%)] Loss: 0.006997
Train Epoch: 6 [12800/60000 (21%)] Loss: 0.042572
Train Epoch: 6 [13440/60000 (22%)] Loss: 0.038031
Train Epoch: 6 [14080/60000 (23%)] Loss: 0.025293
Train Epoch: 6 [14720/60000 (25%)] Loss: 0.010536
Train Epoch: 6 [15360/60000 (26%)] Loss: 0.064900
Train Epoch: 6 [16000/60000 (27%)] Loss: 0.014638
Train Epoch: 6 [16640/60000 (28%)] Loss: 0.004963
Train Epoch: 6 [17280/60000 (29%)] Loss: 0.006997
Train Epoch: 6 [17920/60000 (30%)] Loss: 0.083886
Train Epoch: 6 [18560/60000 (31%)] Loss: 0.059807
Train Epoch: 6 [19200/60000 (32%)] Loss: 0.019030
Train Epoch: 6 [19840/60000 (33%)] Loss: 0.025675
Train Epoch: 6 [20480/60000 (34%)] Loss: 0.176005
Train Epoch: 6 [21120/60000 (35%)] Loss: 0.007704
Train Epoch: 6 [21760/60000 (36%)] Loss: 0.004681
Train Epoch: 6 [22400/60000 (37%)] Loss: 0.010295
Train Epoch: 6 [23040/60000 (38%)] Loss: 0.002958
Train Epoch: 6 [23680/60000 (39%)] Loss: 0.006419
Train Epoch: 6 [24320/60000 (41%)] Loss: 0.020559
Train Epoch: 6 [24960/60000 (42%)] Loss: 0.013513
Train Epoch: 6 [25600/60000 (43%)] Loss: 0.004063
Train Epoch: 6 [26240/60000 (44%)] Loss: 0.018405
Train Epoch: 6 [26880/60000 (45%)] Loss: 0.020268
Train Epoch: 6 [27520/60000 (46%)] Loss: 0.019546
Train Epoch: 6 [28160/60000 (47%)] Loss: 0.011752
Train Epoch: 6 [28800/60000 (48%)] Loss: 0.008671
Train Epoch: 6 [29440/60000 (49%)] Loss: 0.028695
Train Epoch: 6 [30080/60000 (50%)] Loss: 0.148049
Train Epoch: 6 [30720/60000 (51%)] Loss: 0.008042
Train Epoch: 6 [31360/60000 (52%)] Loss: 0.145429
Train Epoch: 6 [32000/60000 (53%)] Loss: 0.048377
Train Epoch: 6 [32640/60000 (54%)] Loss: 0.325409
Train Epoch: 6 [33280/60000 (55%)] Loss: 0.004711
Train Epoch: 6 [33920/60000 (57%)] Loss: 0.007198
Train Epoch: 6 [34560/60000 (58%)] Loss: 0.028788
Train Epoch: 6 [35200/60000 (59%)] Loss: 0.051573
Train Epoch: 6 [35840/60000 (60%)] Loss: 0.021750
Train Epoch: 6 [36480/60000 (61%)] Loss: 0.016786
Train Epoch: 6 [37120/60000 (62%)] Loss: 0.079854
Train Epoch: 6 [37760/60000 (63%)] Loss: 0.030452
Train Epoch: 6 [38400/60000 (64%)] Loss: 0.025469
Train Epoch: 6 [39040/60000 (65%)] Loss: 0.014554
Train Epoch: 6 [39680/60000 (66%)] Loss: 0.027019
Train Epoch: 6 [40320/60000 (67%)] Loss: 0.124955
Train Epoch: 6 [40960/60000 (68%)] Loss: 0.023374
Train Epoch: 6 [41600/60000 (69%)] Loss: 0.123569
Train Epoch: 6 [42240/60000 (70%)] Loss: 0.021324
Train Epoch: 6 [42880/60000 (71%)] Loss: 0.139882
Train Epoch: 6 [43520/60000 (72%)] Loss: 0.020845
Train Epoch: 6 [44160/60000 (74%)] Loss: 0.002598
Train Epoch: 6 [44800/60000 (75%)] Loss: 0.002796
Train Epoch: 6 [45440/60000 (76%)] Loss: 0.031513
Train Epoch: 6 [46080/60000 (77%)] Loss: 0.163956
Train Epoch: 6 [46720/60000 (78%)] Loss: 0.019073
Train Epoch: 6 [47360/60000 (79%)] Loss: 0.013172
Train Epoch: 6 [48000/60000 (80%)] Loss: 0.010234
Train Epoch: 6 [48640/60000 (81%)] Loss: 0.006379
Train Epoch: 6 [49280/60000 (82%)] Loss: 0.118712
Train Epoch: 6 [49920/60000 (83%)] Loss: 0.137450
Train Epoch: 6 [50560/60000 (84%)] Loss: 0.077002
Train Epoch: 6 [51200/60000 (85%)] Loss: 0.007900
Train Epoch: 6 [51840/60000 (86%)] Loss: 0.004504
Train Epoch: 6 [52480/60000 (87%)] Loss: 0.005469
Train Epoch: 6 [53120/60000 (88%)] Loss: 0.042808
Train Epoch: 6 [53760/60000 (90%)] Loss: 0.065220
Train Epoch: 6 [54400/60000 (91%)] Loss: 0.007759
Train Epoch: 6 [55040/60000 (92%)] Loss: 0.045051
Train Epoch: 6 [55680/60000 (93%)] Loss: 0.011808
Train Epoch: 6 [56320/60000 (94%)] Loss: 0.039251
Train Epoch: 6 [56960/60000 (95%)] Loss: 0.006707
Train Epoch: 6 [57600/60000 (96%)] Loss: 0.030685
Train Epoch: 6 [58240/60000 (97%)] Loss: 0.040645
Train Epoch: 6 [58880/60000 (98%)] Loss: 0.005706
Train Epoch: 6 [59520/60000 (99%)] Loss: 0.003886
Test set: Average loss: 0.0299, Accuracy: 9912/10000 (99%)
Train Epoch: 7 [0/60000 (0%)] Loss: 0.020136
Train Epoch: 7 [640/60000 (1%)] Loss: 0.490683
Train Epoch: 7 [1280/60000 (2%)] Loss: 0.004335
Train Epoch: 7 [1920/60000 (3%)] Loss: 0.004355
Train Epoch: 7 [2560/60000 (4%)] Loss: 0.038279
Train Epoch: 7 [3200/60000 (5%)] Loss: 0.053269
Train Epoch: 7 [3840/60000 (6%)] Loss: 0.011103
Train Epoch: 7 [4480/60000 (7%)] Loss: 0.009625
Train Epoch: 7 [5120/60000 (9%)] Loss: 0.030090
Train Epoch: 7 [5760/60000 (10%)] Loss: 0.028419
Train Epoch: 7 [6400/60000 (11%)] Loss: 0.003621
Train Epoch: 7 [7040/60000 (12%)] Loss: 0.001282
Train Epoch: 7 [7680/60000 (13%)] Loss: 0.007006
Train Epoch: 7 [8320/60000 (14%)] Loss: 0.043685
Train Epoch: 7 [8960/60000 (15%)] Loss: 0.001144
Train Epoch: 7 [9600/60000 (16%)] Loss: 0.000733
Train Epoch: 7 [10240/60000 (17%)] Loss: 0.010456
Train Epoch: 7 [10880/60000 (18%)] Loss: 0.003360
Train Epoch: 7 [11520/60000 (19%)] Loss: 0.013506
Train Epoch: 7 [12160/60000 (20%)] Loss: 0.047778
Train Epoch: 7 [12800/60000 (21%)] Loss: 0.127137
Train Epoch: 7 [13440/60000 (22%)] Loss: 0.004023
Train Epoch: 7 [14080/60000 (23%)] Loss: 0.024721
Train Epoch: 7 [14720/60000 (25%)] Loss: 0.022984
Train Epoch: 7 [15360/60000 (26%)] Loss: 0.003474
Train Epoch: 7 [16000/60000 (27%)] Loss: 0.012666
Train Epoch: 7 [16640/60000 (28%)] Loss: 0.002413
Train Epoch: 7 [17280/60000 (29%)] Loss: 0.088378
Train Epoch: 7 [17920/60000 (30%)] Loss: 0.099423
Train Epoch: 7 [18560/60000 (31%)] Loss: 0.001433
Train Epoch: 7 [19200/60000 (32%)] Loss: 0.016714
Train Epoch: 7 [19840/60000 (33%)] Loss: 0.016250
Train Epoch: 7 [20480/60000 (34%)] Loss: 0.013340
Train Epoch: 7 [21120/60000 (35%)] Loss: 0.005496
Train Epoch: 7 [21760/60000 (36%)] Loss: 0.037571
Train Epoch: 7 [22400/60000 (37%)] Loss: 0.004443
Train Epoch: 7 [23040/60000 (38%)] Loss: 0.081719
Train Epoch: 7 [23680/60000 (39%)] Loss: 0.008241
Train Epoch: 7 [24320/60000 (41%)] Loss: 0.002855
Train Epoch: 7 [24960/60000 (42%)] Loss: 0.096220
Train Epoch: 7 [25600/60000 (43%)] Loss: 0.073388
Train Epoch: 7 [26240/60000 (44%)] Loss: 0.044258
Train Epoch: 7 [26880/60000 (45%)] Loss: 0.004959
Train Epoch: 7 [27520/60000 (46%)] Loss: 0.000207
Train Epoch: 7 [28160/60000 (47%)] Loss: 0.029529
Train Epoch: 7 [28800/60000 (48%)] Loss: 0.026542
Train Epoch: 7 [29440/60000 (49%)] Loss: 0.007436
Train Epoch: 7 [30080/60000 (50%)] Loss: 0.104154
Train Epoch: 7 [30720/60000 (51%)] Loss: 0.063764
Train Epoch: 7 [31360/60000 (52%)] Loss: 0.034765
Train Epoch: 7 [32000/60000 (53%)] Loss: 0.010292
Train Epoch: 7 [32640/60000 (54%)] Loss: 0.123570
Train Epoch: 7 [33280/60000 (55%)] Loss: 0.076065
Train Epoch: 7 [33920/60000 (57%)] Loss: 0.003248
Train Epoch: 7 [34560/60000 (58%)] Loss: 0.003740
Train Epoch: 7 [35200/60000 (59%)] Loss: 0.002591
Train Epoch: 7 [35840/60000 (60%)] Loss: 0.023624
Train Epoch: 7 [36480/60000 (61%)] Loss: 0.001512
Train Epoch: 7 [37120/60000 (62%)] Loss: 0.002149
Train Epoch: 7 [37760/60000 (63%)] Loss: 0.029558
Train Epoch: 7 [38400/60000 (64%)] Loss: 0.036850
Train Epoch: 7 [39040/60000 (65%)] Loss: 0.044573
Train Epoch: 7 [39680/60000 (66%)] Loss: 0.032145
Train Epoch: 7 [40320/60000 (67%)] Loss: 0.020010
Train Epoch: 7 [40960/60000 (68%)] Loss: 0.003251
Train Epoch: 7 [41600/60000 (69%)] Loss: 0.005330
Train Epoch: 7 [42240/60000 (70%)] Loss: 0.020333
Train Epoch: 7 [42880/60000 (71%)] Loss: 0.023146
Train Epoch: 7 [43520/60000 (72%)] Loss: 0.176142
Train Epoch: 7 [44160/60000 (74%)] Loss: 0.031682
Train Epoch: 7 [44800/60000 (75%)] Loss: 0.090017
Train Epoch: 7 [45440/60000 (76%)] Loss: 0.087157
Train Epoch: 7 [46080/60000 (77%)] Loss: 0.002258
Train Epoch: 7 [46720/60000 (78%)] Loss: 0.046986
Train Epoch: 7 [47360/60000 (79%)] Loss: 0.020715
Train Epoch: 7 [48000/60000 (80%)] Loss: 0.006792
Train Epoch: 7 [48640/60000 (81%)] Loss: 0.102707
Train Epoch: 7 [49280/60000 (82%)] Loss: 0.056532
Train Epoch: 7 [49920/60000 (83%)] Loss: 0.011454
Train Epoch: 7 [50560/60000 (84%)] Loss: 0.005888
Train Epoch: 7 [51200/60000 (85%)] Loss: 0.028364
Train Epoch: 7 [51840/60000 (86%)] Loss: 0.107270
Train Epoch: 7 [52480/60000 (87%)] Loss: 0.078085
Train Epoch: 7 [53120/60000 (88%)] Loss: 0.002885
Train Epoch: 7 [53760/60000 (90%)] Loss: 0.006929
Train Epoch: 7 [54400/60000 (91%)] Loss: 0.021960
Train Epoch: 7 [55040/60000 (92%)] Loss: 0.019752
Train Epoch: 7 [55680/60000 (93%)] Loss: 0.086779
Train Epoch: 7 [56320/60000 (94%)] Loss: 0.030308
Train Epoch: 7 [56960/60000 (95%)] Loss: 0.056322
Train Epoch: 7 [57600/60000 (96%)] Loss: 0.004134
Train Epoch: 7 [58240/60000 (97%)] Loss: 0.026428
Train Epoch: 7 [58880/60000 (98%)] Loss: 0.055225
Train Epoch: 7 [59520/60000 (99%)] Loss: 0.063854
Test set: Average loss: 0.0277, Accuracy: 9922/10000 (99%)
Train Epoch: 8 [0/60000 (0%)] Loss: 0.075338
Train Epoch: 8 [640/60000 (1%)] Loss: 0.008074
Train Epoch: 8 [1280/60000 (2%)] Loss: 0.024476
Train Epoch: 8 [1920/60000 (3%)] Loss: 0.003908
Train Epoch: 8 [2560/60000 (4%)] Loss: 0.021384
Train Epoch: 8 [3200/60000 (5%)] Loss: 0.000932
Train Epoch: 8 [3840/60000 (6%)] Loss: 0.009920
Train Epoch: 8 [4480/60000 (7%)] Loss: 0.002661
Train Epoch: 8 [5120/60000 (9%)] Loss: 0.015734
Train Epoch: 8 [5760/60000 (10%)] Loss: 0.052622
Train Epoch: 8 [6400/60000 (11%)] Loss: 0.019402
Train Epoch: 8 [7040/60000 (12%)] Loss: 0.071534
Train Epoch: 8 [7680/60000 (13%)] Loss: 0.006797
Train Epoch: 8 [8320/60000 (14%)] Loss: 0.006788
Train Epoch: 8 [8960/60000 (15%)] Loss: 0.007393
Train Epoch: 8 [9600/60000 (16%)] Loss: 0.001774
Train Epoch: 8 [10240/60000 (17%)] Loss: 0.025654
Train Epoch: 8 [10880/60000 (18%)] Loss: 0.026471
Train Epoch: 8 [11520/60000 (19%)] Loss: 0.003025
Train Epoch: 8 [12160/60000 (20%)] Loss: 0.000636
Train Epoch: 8 [12800/60000 (21%)] Loss: 0.061195
Train Epoch: 8 [13440/60000 (22%)] Loss: 0.033223
Train Epoch: 8 [14080/60000 (23%)] Loss: 0.004138
Train Epoch: 8 [14720/60000 (25%)] Loss: 0.020122
Train Epoch: 8 [15360/60000 (26%)] Loss: 0.061547
Train Epoch: 8 [16000/60000 (27%)] Loss: 0.007905
Train Epoch: 8 [16640/60000 (28%)] Loss: 0.020564
Train Epoch: 8 [17280/60000 (29%)] Loss: 0.095406
Train Epoch: 8 [17920/60000 (30%)] Loss: 0.006842
Train Epoch: 8 [18560/60000 (31%)] Loss: 0.036393
Train Epoch: 8 [19200/60000 (32%)] Loss: 0.005329
Train Epoch: 8 [19840/60000 (33%)] Loss: 0.001270
Train Epoch: 8 [20480/60000 (34%)] Loss: 0.017200
Train Epoch: 8 [21120/60000 (35%)] Loss: 0.001552
Train Epoch: 8 [21760/60000 (36%)] Loss: 0.005893
Train Epoch: 8 [22400/60000 (37%)] Loss: 0.011306
Train Epoch: 8 [23040/60000 (38%)] Loss: 0.026739
Train Epoch: 8 [23680/60000 (39%)] Loss: 0.000592
Train Epoch: 8 [24320/60000 (41%)] Loss: 0.122688
Train Epoch: 8 [24960/60000 (42%)] Loss: 0.098567
Train Epoch: 8 [25600/60000 (43%)] Loss: 0.017651
Train Epoch: 8 [26240/60000 (44%)] Loss: 0.037192
Train Epoch: 8 [26880/60000 (45%)] Loss: 0.172926
Train Epoch: 8 [27520/60000 (46%)] Loss: 0.143965
Train Epoch: 8 [28160/60000 (47%)] Loss: 0.095228
Train Epoch: 8 [28800/60000 (48%)] Loss: 0.030516
Train Epoch: 8 [29440/60000 (49%)] Loss: 0.063266
Train Epoch: 8 [30080/60000 (50%)] Loss: 0.032149
Train Epoch: 8 [30720/60000 (51%)] Loss: 0.014092
Train Epoch: 8 [31360/60000 (52%)] Loss: 0.062975
Train Epoch: 8 [32000/60000 (53%)] Loss: 0.006685
Train Epoch: 8 [32640/60000 (54%)] Loss: 0.093086
Train Epoch: 8 [33280/60000 (55%)] Loss: 0.047476
Train Epoch: 8 [33920/60000 (57%)] Loss: 0.025191
Train Epoch: 8 [34560/60000 (58%)] Loss: 0.005494
Train Epoch: 8 [35200/60000 (59%)] Loss: 0.007443
Train Epoch: 8 [35840/60000 (60%)] Loss: 0.001841
Train Epoch: 8 [36480/60000 (61%)] Loss: 0.006735
Train Epoch: 8 [37120/60000 (62%)] Loss: 0.022190
Train Epoch: 8 [37760/60000 (63%)] Loss: 0.001142
Train Epoch: 8 [38400/60000 (64%)] Loss: 0.002211
Train Epoch: 8 [39040/60000 (65%)] Loss: 0.001839
Train Epoch: 8 [39680/60000 (66%)] Loss: 0.097344
Train Epoch: 8 [40320/60000 (67%)] Loss: 0.021180
Train Epoch: 8 [40960/60000 (68%)] Loss: 0.138566
Train Epoch: 8 [41600/60000 (69%)] Loss: 0.012287
Train Epoch: 8 [42240/60000 (70%)] Loss: 0.420171
Train Epoch: 8 [42880/60000 (71%)] Loss: 0.003825
Train Epoch: 8 [43520/60000 (72%)] Loss: 0.007335
Train Epoch: 8 [44160/60000 (74%)] Loss: 0.055388
Train Epoch: 8 [44800/60000 (75%)] Loss: 0.000814
Train Epoch: 8 [45440/60000 (76%)] Loss: 0.075581
Train Epoch: 8 [46080/60000 (77%)] Loss: 0.209742
Train Epoch: 8 [46720/60000 (78%)] Loss: 0.000466
Train Epoch: 8 [47360/60000 (79%)] Loss: 0.002551
Train Epoch: 8 [48000/60000 (80%)] Loss: 0.003525
Train Epoch: 8 [48640/60000 (81%)] Loss: 0.013830
Train Epoch: 8 [49280/60000 (82%)] Loss: 0.023169
Train Epoch: 8 [49920/60000 (83%)] Loss: 0.012012
Train Epoch: 8 [50560/60000 (84%)] Loss: 0.018435
Train Epoch: 8 [51200/60000 (85%)] Loss: 0.009352
Train Epoch: 8 [51840/60000 (86%)] Loss: 0.009852
Train Epoch: 8 [52480/60000 (87%)] Loss: 0.012402
Train Epoch: 8 [53120/60000 (88%)] Loss: 0.027528
Train Epoch: 8 [53760/60000 (90%)] Loss: 0.311435
Train Epoch: 8 [54400/60000 (91%)] Loss: 0.005958
Train Epoch: 8 [55040/60000 (92%)] Loss: 0.045856
Train Epoch: 8 [55680/60000 (93%)] Loss: 0.007033
Train Epoch: 8 [56320/60000 (94%)] Loss: 0.009017
Train Epoch: 8 [56960/60000 (95%)] Loss: 0.027962
Train Epoch: 8 [57600/60000 (96%)] Loss: 0.002828
Train Epoch: 8 [58240/60000 (97%)] Loss: 0.064553
Train Epoch: 8 [58880/60000 (98%)] Loss: 0.003628
Train Epoch: 8 [59520/60000 (99%)] Loss: 0.032909
Test set: Average loss: 0.0283, Accuracy: 9913/10000 (99%)
Train Epoch: 9 [0/60000 (0%)] Loss: 0.003443
Train Epoch: 9 [640/60000 (1%)] Loss: 0.014438
Train Epoch: 9 [1280/60000 (2%)] Loss: 0.004731
Train Epoch: 9 [1920/60000 (3%)] Loss: 0.010515
Train Epoch: 9 [2560/60000 (4%)] Loss: 0.001451
Train Epoch: 9 [3200/60000 (5%)] Loss: 0.001031
Train Epoch: 9 [3840/60000 (6%)] Loss: 0.018930
Train Epoch: 9 [4480/60000 (7%)] Loss: 0.000154
Train Epoch: 9 [5120/60000 (9%)] Loss: 0.000733
Train Epoch: 9 [5760/60000 (10%)] Loss: 0.032051
Train Epoch: 9 [6400/60000 (11%)] Loss: 0.016632
Train Epoch: 9 [7040/60000 (12%)] Loss: 0.002562
Train Epoch: 9 [7680/60000 (13%)] Loss: 0.011222
Train Epoch: 9 [8320/60000 (14%)] Loss: 0.002181
Train Epoch: 9 [8960/60000 (15%)] Loss: 0.006865
Train Epoch: 9 [9600/60000 (16%)] Loss: 0.032391
Train Epoch: 9 [10240/60000 (17%)] Loss: 0.043184
Train Epoch: 9 [10880/60000 (18%)] Loss: 0.010149
Train Epoch: 9 [11520/60000 (19%)] Loss: 0.003689
Train Epoch: 9 [12160/60000 (20%)] Loss: 0.020855
Train Epoch: 9 [12800/60000 (21%)] Loss: 0.055720
Train Epoch: 9 [13440/60000 (22%)] Loss: 0.082490
Train Epoch: 9 [14080/60000 (23%)] Loss: 0.028715
Train Epoch: 9 [14720/60000 (25%)] Loss: 0.005401
Train Epoch: 9 [15360/60000 (26%)] Loss: 0.034124
Train Epoch: 9 [16000/60000 (27%)] Loss: 0.005842
Train Epoch: 9 [16640/60000 (28%)] Loss: 0.034637
Train Epoch: 9 [17280/60000 (29%)] Loss: 0.009069
Train Epoch: 9 [17920/60000 (30%)] Loss: 0.027860
Train Epoch: 9 [18560/60000 (31%)] Loss: 0.066139
Train Epoch: 9 [19200/60000 (32%)] Loss: 0.008850
Train Epoch: 9 [19840/60000 (33%)] Loss: 0.147988
Train Epoch: 9 [20480/60000 (34%)] Loss: 0.001180
Train Epoch: 9 [21120/60000 (35%)] Loss: 0.043872
Train Epoch: 9 [21760/60000 (36%)] Loss: 0.048523
Train Epoch: 9 [22400/60000 (37%)] Loss: 0.006986
Train Epoch: 9 [23040/60000 (38%)] Loss: 0.016137
Train Epoch: 9 [23680/60000 (39%)] Loss: 0.000713
Train Epoch: 9 [24320/60000 (41%)] Loss: 0.094277
Train Epoch: 9 [24960/60000 (42%)] Loss: 0.140367
Train Epoch: 9 [25600/60000 (43%)] Loss: 0.006542
Train Epoch: 9 [26240/60000 (44%)] Loss: 0.064483
Train Epoch: 9 [26880/60000 (45%)] Loss: 0.005256
Train Epoch: 9 [27520/60000 (46%)] Loss: 0.120251
Train Epoch: 9 [28160/60000 (47%)] Loss: 0.006152
Train Epoch: 9 [28800/60000 (48%)] Loss: 0.004135
Train Epoch: 9 [29440/60000 (49%)] Loss: 0.005806
Train Epoch: 9 [30080/60000 (50%)] Loss: 0.050835
Train Epoch: 9 [30720/60000 (51%)] Loss: 0.002529
Train Epoch: 9 [31360/60000 (52%)] Loss: 0.001982
Train Epoch: 9 [32000/60000 (53%)] Loss: 0.000945
Train Epoch: 9 [32640/60000 (54%)] Loss: 0.019677
Train Epoch: 9 [33280/60000 (55%)] Loss: 0.005030
Train Epoch: 9 [33920/60000 (57%)] Loss: 0.015237
Train Epoch: 9 [34560/60000 (58%)] Loss: 0.002280
Train Epoch: 9 [35200/60000 (59%)] Loss: 0.005953
Train Epoch: 9 [35840/60000 (60%)] Loss: 0.053388
Train Epoch: 9 [36480/60000 (61%)] Loss: 0.042914
Train Epoch: 9 [37120/60000 (62%)] Loss: 0.011494
Train Epoch: 9 [37760/60000 (63%)] Loss: 0.024443
Train Epoch: 9 [38400/60000 (64%)] Loss: 0.037382
Train Epoch: 9 [39040/60000 (65%)] Loss: 0.005391
Train Epoch: 9 [39680/60000 (66%)] Loss: 0.014658
Train Epoch: 9 [40320/60000 (67%)] Loss: 0.023616
Train Epoch: 9 [40960/60000 (68%)] Loss: 0.038959
Train Epoch: 9 [41600/60000 (69%)] Loss: 0.004563
Train Epoch: 9 [42240/60000 (70%)] Loss: 0.046213
Train Epoch: 9 [42880/60000 (71%)] Loss: 0.007633
Train Epoch: 9 [43520/60000 (72%)] Loss: 0.001581
Train Epoch: 9 [44160/60000 (74%)] Loss: 0.017676
Train Epoch: 9 [44800/60000 (75%)] Loss: 0.003912
Train Epoch: 9 [45440/60000 (76%)] Loss: 0.139427
Train Epoch: 9 [46080/60000 (77%)] Loss: 0.006161
Train Epoch: 9 [46720/60000 (78%)] Loss: 0.006618
Train Epoch: 9 [47360/60000 (79%)] Loss: 0.001641
Train Epoch: 9 [48000/60000 (80%)] Loss: 0.000744
Train Epoch: 9 [48640/60000 (81%)] Loss: 0.065756
Train Epoch: 9 [49280/60000 (82%)] Loss: 0.016521
Train Epoch: 9 [49920/60000 (83%)] Loss: 0.215904
Train Epoch: 9 [50560/60000 (84%)] Loss: 0.093698
Train Epoch: 9 [51200/60000 (85%)] Loss: 0.061032
Train Epoch: 9 [51840/60000 (86%)] Loss: 0.033895
Train Epoch: 9 [52480/60000 (87%)] Loss: 0.030293
Train Epoch: 9 [53120/60000 (88%)] Loss: 0.100245
Train Epoch: 9 [53760/60000 (90%)] Loss: 0.010049
Train Epoch: 9 [54400/60000 (91%)] Loss: 0.074552
Train Epoch: 9 [55040/60000 (92%)] Loss: 0.001400
Train Epoch: 9 [55680/60000 (93%)] Loss: 0.016799
Train Epoch: 9 [56320/60000 (94%)] Loss: 0.074889
Train Epoch: 9 [56960/60000 (95%)] Loss: 0.000835
Train Epoch: 9 [57600/60000 (96%)] Loss: 0.051169
Train Epoch: 9 [58240/60000 (97%)] Loss: 0.010802
Train Epoch: 9 [58880/60000 (98%)] Loss: 0.005692
Train Epoch: 9 [59520/60000 (99%)] Loss: 0.006080
Test set: Average loss: 0.0285, Accuracy: 9912/10000 (99%)
Train Epoch: 10 [0/60000 (0%)] Loss: 0.066472
Train Epoch: 10 [640/60000 (1%)] Loss: 0.000861
Train Epoch: 10 [1280/60000 (2%)] Loss: 0.035700
Train Epoch: 10 [1920/60000 (3%)] Loss: 0.046769
Train Epoch: 10 [2560/60000 (4%)] Loss: 0.002510
Train Epoch: 10 [3200/60000 (5%)] Loss: 0.014785
Train Epoch: 10 [3840/60000 (6%)] Loss: 0.018235
Train Epoch: 10 [4480/60000 (7%)] Loss: 0.002776
Train Epoch: 10 [5120/60000 (9%)] Loss: 0.001437
Train Epoch: 10 [5760/60000 (10%)] Loss: 0.031271
Train Epoch: 10 [6400/60000 (11%)] Loss: 0.013996
Train Epoch: 10 [7040/60000 (12%)] Loss: 0.002698
Train Epoch: 10 [7680/60000 (13%)] Loss: 0.025018
Train Epoch: 10 [8320/60000 (14%)] Loss: 0.033485
Train Epoch: 10 [8960/60000 (15%)] Loss: 0.007595
Train Epoch: 10 [9600/60000 (16%)] Loss: 0.023715
Train Epoch: 10 [10240/60000 (17%)] Loss: 0.051663
Train Epoch: 10 [10880/60000 (18%)] Loss: 0.014862
Train Epoch: 10 [11520/60000 (19%)] Loss: 0.008545
Train Epoch: 10 [12160/60000 (20%)] Loss: 0.005122
Train Epoch: 10 [12800/60000 (21%)] Loss: 0.044644
Train Epoch: 10 [13440/60000 (22%)] Loss: 0.023675
Train Epoch: 10 [14080/60000 (23%)] Loss: 0.023370
Train Epoch: 10 [14720/60000 (25%)] Loss: 0.004087
Train Epoch: 10 [15360/60000 (26%)] Loss: 0.000924
Train Epoch: 10 [16000/60000 (27%)] Loss: 0.135698
Train Epoch: 10 [16640/60000 (28%)] Loss: 0.016960
Train Epoch: 10 [17280/60000 (29%)] Loss: 0.001627
Train Epoch: 10 [17920/60000 (30%)] Loss: 0.101285
Train Epoch: 10 [18560/60000 (31%)] Loss: 0.026828
Train Epoch: 10 [19200/60000 (32%)] Loss: 0.009353
Train Epoch: 10 [19840/60000 (33%)] Loss: 0.046688
Train Epoch: 10 [20480/60000 (34%)] Loss: 0.002074
Train Epoch: 10 [21120/60000 (35%)] Loss: 0.007316
Train Epoch: 10 [21760/60000 (36%)] Loss: 0.015666
Train Epoch: 10 [22400/60000 (37%)] Loss: 0.022117
Train Epoch: 10 [23040/60000 (38%)] Loss: 0.013400
Train Epoch: 10 [23680/60000 (39%)] Loss: 0.036549
Train Epoch: 10 [24320/60000 (41%)] Loss: 0.015107
Train Epoch: 10 [24960/60000 (42%)] Loss: 0.135749
Train Epoch: 10 [25600/60000 (43%)] Loss: 0.022456
Train Epoch: 10 [26240/60000 (44%)] Loss: 0.002534
Train Epoch: 10 [26880/60000 (45%)] Loss: 0.022857
Train Epoch: 10 [27520/60000 (46%)] Loss: 0.005443
Train Epoch: 10 [28160/60000 (47%)] Loss: 0.002118
Train Epoch: 10 [28800/60000 (48%)] Loss: 0.015055
Train Epoch: 10 [29440/60000 (49%)] Loss: 0.129272
Train Epoch: 10 [30080/60000 (50%)] Loss: 0.130920
Train Epoch: 10 [30720/60000 (51%)] Loss: 0.102639
Train Epoch: 10 [31360/60000 (52%)] Loss: 0.038153
Train Epoch: 10 [32000/60000 (53%)] Loss: 0.001986
Train Epoch: 10 [32640/60000 (54%)] Loss: 0.003369
Train Epoch: 10 [33280/60000 (55%)] Loss: 0.008723
Train Epoch: 10 [33920/60000 (57%)] Loss: 0.009437
Train Epoch: 10 [34560/60000 (58%)] Loss: 0.003083
Train Epoch: 10 [35200/60000 (59%)] Loss: 0.008142
Train Epoch: 10 [35840/60000 (60%)] Loss: 0.171981
Train Epoch: 10 [36480/60000 (61%)] Loss: 0.132058
Train Epoch: 10 [37120/60000 (62%)] Loss: 0.004168
Train Epoch: 10 [37760/60000 (63%)] Loss: 0.066958
Train Epoch: 10 [38400/60000 (64%)] Loss: 0.002050
Train Epoch: 10 [39040/60000 (65%)] Loss: 0.002156
Train Epoch: 10 [39680/60000 (66%)] Loss: 0.033177
Train Epoch: 10 [40320/60000 (67%)] Loss: 0.009092
Train Epoch: 10 [40960/60000 (68%)] Loss: 0.022885
Train Epoch: 10 [41600/60000 (69%)] Loss: 0.003838
Train Epoch: 10 [42240/60000 (70%)] Loss: 0.032701
Train Epoch: 10 [42880/60000 (71%)] Loss: 0.007397
Train Epoch: 10 [43520/60000 (72%)] Loss: 0.089226
Train Epoch: 10 [44160/60000 (74%)] Loss: 0.007560
Train Epoch: 10 [44800/60000 (75%)] Loss: 0.103667
Train Epoch: 10 [45440/60000 (76%)] Loss: 0.034520
Train Epoch: 10 [46080/60000 (77%)] Loss: 0.005575
Train Epoch: 10 [46720/60000 (78%)] Loss: 0.116959
Train Epoch: 10 [47360/60000 (79%)] Loss: 0.025126
Train Epoch: 10 [48000/60000 (80%)] Loss: 0.000598
Train Epoch: 10 [48640/60000 (81%)] Loss: 0.003662
Train Epoch: 10 [49280/60000 (82%)] Loss: 0.001009
Train Epoch: 10 [49920/60000 (83%)] Loss: 0.008557
Train Epoch: 10 [50560/60000 (84%)] Loss: 0.008524
Train Epoch: 10 [51200/60000 (85%)] Loss: 0.001014
Train Epoch: 10 [51840/60000 (86%)] Loss: 0.037323
Train Epoch: 10 [52480/60000 (87%)] Loss: 0.006754
Train Epoch: 10 [53120/60000 (88%)] Loss: 0.055297
Train Epoch: 10 [53760/60000 (90%)] Loss: 0.004579
Train Epoch: 10 [54400/60000 (91%)] Loss: 0.002429
Train Epoch: 10 [55040/60000 (92%)] Loss: 0.011540
Train Epoch: 10 [55680/60000 (93%)] Loss: 0.015078
Train Epoch: 10 [56320/60000 (94%)] Loss: 0.101013
Train Epoch: 10 [56960/60000 (95%)] Loss: 0.005557
Train Epoch: 10 [57600/60000 (96%)] Loss: 0.049002
Train Epoch: 10 [58240/60000 (97%)] Loss: 0.018683
Train Epoch: 10 [58880/60000 (98%)] Loss: 0.003927
Train Epoch: 10 [59520/60000 (99%)] Loss: 0.019337
Test set: Average loss: 0.0270, Accuracy: 9916/10000 (99%)
Train Epoch: 11 [0/60000 (0%)] Loss: 0.000583
Train Epoch: 11 [640/60000 (1%)] Loss: 0.019397
Train Epoch: 11 [1280/60000 (2%)] Loss: 0.022540
Train Epoch: 11 [1920/60000 (3%)] Loss: 0.010484
Train Epoch: 11 [2560/60000 (4%)] Loss: 0.008713
Train Epoch: 11 [3200/60000 (5%)] Loss: 0.026814
Train Epoch: 11 [3840/60000 (6%)] Loss: 0.019683
Train Epoch: 11 [4480/60000 (7%)] Loss: 0.020514
Train Epoch: 11 [5120/60000 (9%)] Loss: 0.002041
Train Epoch: 11 [5760/60000 (10%)] Loss: 0.013970
Train Epoch: 11 [6400/60000 (11%)] Loss: 0.023215
Train Epoch: 11 [7040/60000 (12%)] Loss: 0.003069
Train Epoch: 11 [7680/60000 (13%)] Loss: 0.021880
Train Epoch: 11 [8320/60000 (14%)] Loss: 0.000747
Train Epoch: 11 [8960/60000 (15%)] Loss: 0.003059
Train Epoch: 11 [9600/60000 (16%)] Loss: 0.016219
Train Epoch: 11 [10240/60000 (17%)] Loss: 0.033322
Train Epoch: 11 [10880/60000 (18%)] Loss: 0.005459
Train Epoch: 11 [11520/60000 (19%)] Loss: 0.092742
Train Epoch: 11 [12160/60000 (20%)] Loss: 0.012621
Train Epoch: 11 [12800/60000 (21%)] Loss: 0.001704
Train Epoch: 11 [13440/60000 (22%)] Loss: 0.001074
Train Epoch: 11 [14080/60000 (23%)] Loss: 0.000304
Train Epoch: 11 [14720/60000 (25%)] Loss: 0.024537
Train Epoch: 11 [15360/60000 (26%)] Loss: 0.006519
Train Epoch: 11 [16000/60000 (27%)] Loss: 0.002694
Train Epoch: 11 [16640/60000 (28%)] Loss: 0.063115
Train Epoch: 11 [17280/60000 (29%)] Loss: 0.007173
Train Epoch: 11 [17920/60000 (30%)] Loss: 0.039986
Train Epoch: 11 [18560/60000 (31%)] Loss: 0.034666
Train Epoch: 11 [19200/60000 (32%)] Loss: 0.047636
Train Epoch: 11 [19840/60000 (33%)] Loss: 0.023297
Train Epoch: 11 [20480/60000 (34%)] Loss: 0.005024
Train Epoch: 11 [21120/60000 (35%)] Loss: 0.025957
Train Epoch: 11 [21760/60000 (36%)] Loss: 0.007337
Train Epoch: 11 [22400/60000 (37%)] Loss: 0.037568
Train Epoch: 11 [23040/60000 (38%)] Loss: 0.003618
Train Epoch: 11 [23680/60000 (39%)] Loss: 0.005309
Train Epoch: 11 [24320/60000 (41%)] Loss: 0.001622
Train Epoch: 11 [24960/60000 (42%)] Loss: 0.013712
Train Epoch: 11 [25600/60000 (43%)] Loss: 0.084744
Train Epoch: 11 [26240/60000 (44%)] Loss: 0.111220
Train Epoch: 11 [26880/60000 (45%)] Loss: 0.012916
Train Epoch: 11 [27520/60000 (46%)] Loss: 0.024589
Train Epoch: 11 [28160/60000 (47%)] Loss: 0.031567
Train Epoch: 11 [28800/60000 (48%)] Loss: 0.004300
Train Epoch: 11 [29440/60000 (49%)] Loss: 0.029767
Train Epoch: 11 [30080/60000 (50%)] Loss: 0.023700
Train Epoch: 11 [30720/60000 (51%)] Loss: 0.001290
Train Epoch: 11 [31360/60000 (52%)] Loss: 0.000412
Train Epoch: 11 [32000/60000 (53%)] Loss: 0.034096
Train Epoch: 11 [32640/60000 (54%)] Loss: 0.006834
Train Epoch: 11 [33280/60000 (55%)] Loss: 0.001322
Train Epoch: 11 [33920/60000 (57%)] Loss: 0.010359
Train Epoch: 11 [34560/60000 (58%)] Loss: 0.016077
Train Epoch: 11 [35200/60000 (59%)] Loss: 0.016256
Train Epoch: 11 [35840/60000 (60%)] Loss: 0.002107
Train Epoch: 11 [36480/60000 (61%)] Loss: 0.041328
Train Epoch: 11 [37120/60000 (62%)] Loss: 0.007419
Train Epoch: 11 [37760/60000 (63%)] Loss: 0.005188
Train Epoch: 11 [38400/60000 (64%)] Loss: 0.009733
Train Epoch: 11 [39040/60000 (65%)] Loss: 0.007586
Train Epoch: 11 [39680/60000 (66%)] Loss: 0.010136
Train Epoch: 11 [40320/60000 (67%)] Loss: 0.019733
Train Epoch: 11 [40960/60000 (68%)] Loss: 0.022872
Train Epoch: 11 [41600/60000 (69%)] Loss: 0.000745
Train Epoch: 11 [42240/60000 (70%)] Loss: 0.015106
Train Epoch: 11 [42880/60000 (71%)] Loss: 0.010130
Train Epoch: 11 [43520/60000 (72%)] Loss: 0.013877
Train Epoch: 11 [44160/60000 (74%)] Loss: 0.002777
Train Epoch: 11 [44800/60000 (75%)] Loss: 0.002366
Train Epoch: 11 [45440/60000 (76%)] Loss: 0.013362
Train Epoch: 11 [46080/60000 (77%)] Loss: 0.056401
Train Epoch: 11 [46720/60000 (78%)] Loss: 0.005211
Train Epoch: 11 [47360/60000 (79%)] Loss: 0.047950
Train Epoch: 11 [48000/60000 (80%)] Loss: 0.002357
Train Epoch: 11 [48640/60000 (81%)] Loss: 0.001289
Train Epoch: 11 [49280/60000 (82%)] Loss: 0.021824
Train Epoch: 11 [49920/60000 (83%)] Loss: 0.105863
Train Epoch: 11 [50560/60000 (84%)] Loss: 0.007684
Train Epoch: 11 [51200/60000 (85%)] Loss: 0.035193
Train Epoch: 11 [51840/60000 (86%)] Loss: 0.022034
Train Epoch: 11 [52480/60000 (87%)] Loss: 0.023651
Train Epoch: 11 [53120/60000 (88%)] Loss: 0.034551
Train Epoch: 11 [53760/60000 (90%)] Loss: 0.003504
Train Epoch: 11 [54400/60000 (91%)] Loss: 0.054915
Train Epoch: 11 [55040/60000 (92%)] Loss: 0.009922
Train Epoch: 11 [55680/60000 (93%)] Loss: 0.036793
Train Epoch: 11 [56320/60000 (94%)] Loss: 0.001638
Train Epoch: 11 [56960/60000 (95%)] Loss: 0.015796
Train Epoch: 11 [57600/60000 (96%)] Loss: 0.072871
Train Epoch: 11 [58240/60000 (97%)] Loss: 0.023970
Train Epoch: 11 [58880/60000 (98%)] Loss: 0.006769
Train Epoch: 11 [59520/60000 (99%)] Loss: 0.001245
Test set: Average loss: 0.0275, Accuracy: 9918/10000 (99%)
Train Epoch: 12 [0/60000 (0%)] Loss: 0.018245
Train Epoch: 12 [640/60000 (1%)] Loss: 0.002880
Train Epoch: 12 [1280/60000 (2%)] Loss: 0.030436
Train Epoch: 12 [1920/60000 (3%)] Loss: 0.054957
Train Epoch: 12 [2560/60000 (4%)] Loss: 0.006171
Train Epoch: 12 [3200/60000 (5%)] Loss: 0.006099
Train Epoch: 12 [3840/60000 (6%)] Loss: 0.008752
Train Epoch: 12 [4480/60000 (7%)] Loss: 0.028347
Train Epoch: 12 [5120/60000 (9%)] Loss: 0.015304
Train Epoch: 12 [5760/60000 (10%)] Loss: 0.099505
Train Epoch: 12 [6400/60000 (11%)] Loss: 0.009682
Train Epoch: 12 [7040/60000 (12%)] Loss: 0.024153
Train Epoch: 12 [7680/60000 (13%)] Loss: 0.016994
Train Epoch: 12 [8320/60000 (14%)] Loss: 0.015284
Train Epoch: 12 [8960/60000 (15%)] Loss: 0.005004
Train Epoch: 12 [9600/60000 (16%)] Loss: 0.015800
Train Epoch: 12 [10240/60000 (17%)] Loss: 0.008052
Train Epoch: 12 [10880/60000 (18%)] Loss: 0.003762
Train Epoch: 12 [11520/60000 (19%)] Loss: 0.096711
Train Epoch: 12 [12160/60000 (20%)] Loss: 0.101324
Train Epoch: 12 [12800/60000 (21%)] Loss: 0.023606
Train Epoch: 12 [13440/60000 (22%)] Loss: 0.004191
Train Epoch: 12 [14080/60000 (23%)] Loss: 0.014392
Train Epoch: 12 [14720/60000 (25%)] Loss: 0.069814
Train Epoch: 12 [15360/60000 (26%)] Loss: 0.005322
Train Epoch: 12 [16000/60000 (27%)] Loss: 0.029572
Train Epoch: 12 [16640/60000 (28%)] Loss: 0.036131
Train Epoch: 12 [17280/60000 (29%)] Loss: 0.009320
Train Epoch: 12 [17920/60000 (30%)] Loss: 0.000546
Train Epoch: 12 [18560/60000 (31%)] Loss: 0.018851
Train Epoch: 12 [19200/60000 (32%)] Loss: 0.002398
Train Epoch: 12 [19840/60000 (33%)] Loss: 0.026500
Train Epoch: 12 [20480/60000 (34%)] Loss: 0.006554
Train Epoch: 12 [21120/60000 (35%)] Loss: 0.016205
Train Epoch: 12 [21760/60000 (36%)] Loss: 0.002378
Train Epoch: 12 [22400/60000 (37%)] Loss: 0.008967
Train Epoch: 12 [23040/60000 (38%)] Loss: 0.004931
Train Epoch: 12 [23680/60000 (39%)] Loss: 0.042849
Train Epoch: 12 [24320/60000 (41%)] Loss: 0.014766
Train Epoch: 12 [24960/60000 (42%)] Loss: 0.036539
Train Epoch: 12 [25600/60000 (43%)] Loss: 0.040763
Train Epoch: 12 [26240/60000 (44%)] Loss: 0.003107
Train Epoch: 12 [26880/60000 (45%)] Loss: 0.060773
Train Epoch: 12 [27520/60000 (46%)] Loss: 0.008122
Train Epoch: 12 [28160/60000 (47%)] Loss: 0.038294
Train Epoch: 12 [28800/60000 (48%)] Loss: 0.000962
Train Epoch: 12 [29440/60000 (49%)] Loss: 0.001718
Train Epoch: 12 [30080/60000 (50%)] Loss: 0.025688
Train Epoch: 12 [30720/60000 (51%)] Loss: 0.003557
Train Epoch: 12 [31360/60000 (52%)] Loss: 0.009379
Train Epoch: 12 [32000/60000 (53%)] Loss: 0.043338
Train Epoch: 12 [32640/60000 (54%)] Loss: 0.006043
Train Epoch: 12 [33280/60000 (55%)] Loss: 0.073951
Train Epoch: 12 [33920/60000 (57%)] Loss: 0.003510
Train Epoch: 12 [34560/60000 (58%)] Loss: 0.023834
Train Epoch: 12 [35200/60000 (59%)] Loss: 0.037913
Train Epoch: 12 [35840/60000 (60%)] Loss: 0.007221
Train Epoch: 12 [36480/60000 (61%)] Loss: 0.000641
Train Epoch: 12 [37120/60000 (62%)] Loss: 0.001838
Train Epoch: 12 [37760/60000 (63%)] Loss: 0.004391
Train Epoch: 12 [38400/60000 (64%)] Loss: 0.013336
Train Epoch: 12 [39040/60000 (65%)] Loss: 0.005487
Train Epoch: 12 [39680/60000 (66%)] Loss: 0.019220
Train Epoch: 12 [40320/60000 (67%)] Loss: 0.040792
Train Epoch: 12 [40960/60000 (68%)] Loss: 0.010977
Train Epoch: 12 [41600/60000 (69%)] Loss: 0.004653
Train Epoch: 12 [42240/60000 (70%)] Loss: 0.019384
Train Epoch: 12 [42880/60000 (71%)] Loss: 0.003416
Train Epoch: 12 [43520/60000 (72%)] Loss: 0.004081
Train Epoch: 12 [44160/60000 (74%)] Loss: 0.002371
Train Epoch: 12 [44800/60000 (75%)] Loss: 0.007803
Train Epoch: 12 [45440/60000 (76%)] Loss: 0.006351
Train Epoch: 12 [46080/60000 (77%)] Loss: 0.000488
Train Epoch: 12 [46720/60000 (78%)] Loss: 0.010394
Train Epoch: 12 [47360/60000 (79%)] Loss: 0.039691
Train Epoch: 12 [48000/60000 (80%)] Loss: 0.013394
Train Epoch: 12 [48640/60000 (81%)] Loss: 0.161356
Train Epoch: 12 [49280/60000 (82%)] Loss: 0.001381
Train Epoch: 12 [49920/60000 (83%)] Loss: 0.005011
Train Epoch: 12 [50560/60000 (84%)] Loss: 0.003702
Train Epoch: 12 [51200/60000 (85%)] Loss: 0.149166
Train Epoch: 12 [51840/60000 (86%)] Loss: 0.010490
Train Epoch: 12 [52480/60000 (87%)] Loss: 0.010572
Train Epoch: 12 [53120/60000 (88%)] Loss: 0.032527
Train Epoch: 12 [53760/60000 (90%)] Loss: 0.005528
Train Epoch: 12 [54400/60000 (91%)] Loss: 0.003912
Train Epoch: 12 [55040/60000 (92%)] Loss: 0.021487
Train Epoch: 12 [55680/60000 (93%)] Loss: 0.001777
Train Epoch: 12 [56320/60000 (94%)] Loss: 0.055389
Train Epoch: 12 [56960/60000 (95%)] Loss: 0.001266
Train Epoch: 12 [57600/60000 (96%)] Loss: 0.106136
Train Epoch: 12 [58240/60000 (97%)] Loss: 0.012853
Train Epoch: 12 [58880/60000 (98%)] Loss: 0.003468
Train Epoch: 12 [59520/60000 (99%)] Loss: 0.026972
Test set: Average loss: 0.0267, Accuracy: 9917/10000 (99%)
Train Epoch: 13 [0/60000 (0%)] Loss: 0.006483
Train Epoch: 13 [640/60000 (1%)] Loss: 0.009633
Train Epoch: 13 [1280/60000 (2%)] Loss: 0.059453
Train Epoch: 13 [1920/60000 (3%)] Loss: 0.041049
Train Epoch: 13 [2560/60000 (4%)] Loss: 0.002505
Train Epoch: 13 [3200/60000 (5%)] Loss: 0.002137
Train Epoch: 13 [3840/60000 (6%)] Loss: 0.001923
Train Epoch: 13 [4480/60000 (7%)] Loss: 0.023811
Train Epoch: 13 [5120/60000 (9%)] Loss: 0.066973
Train Epoch: 13 [5760/60000 (10%)] Loss: 0.029046
Train Epoch: 13 [6400/60000 (11%)] Loss: 0.001369
Train Epoch: 13 [7040/60000 (12%)] Loss: 0.114614
Train Epoch: 13 [7680/60000 (13%)] Loss: 0.094165
Train Epoch: 13 [8320/60000 (14%)] Loss: 0.014921
Train Epoch: 13 [8960/60000 (15%)] Loss: 0.013669
Train Epoch: 13 [9600/60000 (16%)] Loss: 0.030830
Train Epoch: 13 [10240/60000 (17%)] Loss: 0.052833
Train Epoch: 13 [10880/60000 (18%)] Loss: 0.072083
Train Epoch: 13 [11520/60000 (19%)] Loss: 0.001023
Train Epoch: 13 [12160/60000 (20%)] Loss: 0.006959
Train Epoch: 13 [12800/60000 (21%)] Loss: 0.000872
Train Epoch: 13 [13440/60000 (22%)] Loss: 0.008088
Train Epoch: 13 [14080/60000 (23%)] Loss: 0.051367
Train Epoch: 13 [14720/60000 (25%)] Loss: 0.000967
Train Epoch: 13 [15360/60000 (26%)] Loss: 0.096453
Train Epoch: 13 [16000/60000 (27%)] Loss: 0.009356
Train Epoch: 13 [16640/60000 (28%)] Loss: 0.003991
Train Epoch: 13 [17280/60000 (29%)] Loss: 0.002532
Train Epoch: 13 [17920/60000 (30%)] Loss: 0.083037
Train Epoch: 13 [18560/60000 (31%)] Loss: 0.012334
Train Epoch: 13 [19200/60000 (32%)] Loss: 0.007489
Train Epoch: 13 [19840/60000 (33%)] Loss: 0.003132
Train Epoch: 13 [20480/60000 (34%)] Loss: 0.008023
Train Epoch: 13 [21120/60000 (35%)] Loss: 0.002955
Train Epoch: 13 [21760/60000 (36%)] Loss: 0.000298
Train Epoch: 13 [22400/60000 (37%)] Loss: 0.004474
Train Epoch: 13 [23040/60000 (38%)] Loss: 0.114630
Train Epoch: 13 [23680/60000 (39%)] Loss: 0.005797
Train Epoch: 13 [24320/60000 (41%)] Loss: 0.003194
Train Epoch: 13 [24960/60000 (42%)] Loss: 0.001465
Train Epoch: 13 [25600/60000 (43%)] Loss: 0.002327
Train Epoch: 13 [26240/60000 (44%)] Loss: 0.000595
Train Epoch: 13 [26880/60000 (45%)] Loss: 0.001808
Train Epoch: 13 [27520/60000 (46%)] Loss: 0.171466
Train Epoch: 13 [28160/60000 (47%)] Loss: 0.011974
Train Epoch: 13 [28800/60000 (48%)] Loss: 0.078435
Train Epoch: 13 [29440/60000 (49%)] Loss: 0.017996
Train Epoch: 13 [30080/60000 (50%)] Loss: 0.049924
Train Epoch: 13 [30720/60000 (51%)] Loss: 0.035697
Train Epoch: 13 [31360/60000 (52%)] Loss: 0.003384
Train Epoch: 13 [32000/60000 (53%)] Loss: 0.021575
Train Epoch: 13 [32640/60000 (54%)] Loss: 0.006902
Train Epoch: 13 [33280/60000 (55%)] Loss: 0.005197
Train Epoch: 13 [33920/60000 (57%)] Loss: 0.025994
Train Epoch: 13 [34560/60000 (58%)] Loss: 0.002432
Train Epoch: 13 [35200/60000 (59%)] Loss: 0.010943
Train Epoch: 13 [35840/60000 (60%)] Loss: 0.098180
Train Epoch: 13 [36480/60000 (61%)] Loss: 0.008865
Train Epoch: 13 [37120/60000 (62%)] Loss: 0.002464
Train Epoch: 13 [37760/60000 (63%)] Loss: 0.036317
Train Epoch: 13 [38400/60000 (64%)] Loss: 0.007238
Train Epoch: 13 [39040/60000 (65%)] Loss: 0.037112
Train Epoch: 13 [39680/60000 (66%)] Loss: 0.016673
Train Epoch: 13 [40320/60000 (67%)] Loss: 0.104173
Train Epoch: 13 [40960/60000 (68%)] Loss: 0.029568
Train Epoch: 13 [41600/60000 (69%)] Loss: 0.008611
Train Epoch: 13 [42240/60000 (70%)] Loss: 0.034948
Train Epoch: 13 [42880/60000 (71%)] Loss: 0.066927
Train Epoch: 13 [43520/60000 (72%)] Loss: 0.027559
Train Epoch: 13 [44160/60000 (74%)] Loss: 0.060235
Train Epoch: 13 [44800/60000 (75%)] Loss: 0.000917
Train Epoch: 13 [45440/60000 (76%)] Loss: 0.035733
Train Epoch: 13 [46080/60000 (77%)] Loss: 0.005244
Train Epoch: 13 [46720/60000 (78%)] Loss: 0.001296
Train Epoch: 13 [47360/60000 (79%)] Loss: 0.164095
Train Epoch: 13 [48000/60000 (80%)] Loss: 0.019843
Train Epoch: 13 [48640/60000 (81%)] Loss: 0.041003
Train Epoch: 13 [49280/60000 (82%)] Loss: 0.003117
Train Epoch: 13 [49920/60000 (83%)] Loss: 0.034763
Train Epoch: 13 [50560/60000 (84%)] Loss: 0.015371
Train Epoch: 13 [51200/60000 (85%)] Loss: 0.015402
Train Epoch: 13 [51840/60000 (86%)] Loss: 0.052273
Train Epoch: 13 [52480/60000 (87%)] Loss: 0.001214
Train Epoch: 13 [53120/60000 (88%)] Loss: 0.079685
Train Epoch: 13 [53760/60000 (90%)] Loss: 0.010649
Train Epoch: 13 [54400/60000 (91%)] Loss: 0.004147
Train Epoch: 13 [55040/60000 (92%)] Loss: 0.028947
Train Epoch: 13 [55680/60000 (93%)] Loss: 0.001420
Train Epoch: 13 [56320/60000 (94%)] Loss: 0.002791
Train Epoch: 13 [56960/60000 (95%)] Loss: 0.001883
Train Epoch: 13 [57600/60000 (96%)] Loss: 0.001851
Train Epoch: 13 [58240/60000 (97%)] Loss: 0.001997
Train Epoch: 13 [58880/60000 (98%)] Loss: 0.021178
Train Epoch: 13 [59520/60000 (99%)] Loss: 0.014728
Test set: Average loss: 0.0272, Accuracy: 9918/10000 (99%)
Train Epoch: 14 [0/60000 (0%)] Loss: 0.008265
Train Epoch: 14 [640/60000 (1%)] Loss: 0.012390
Train Epoch: 14 [1280/60000 (2%)] Loss: 0.008145
Train Epoch: 14 [1920/60000 (3%)] Loss: 0.022766
Train Epoch: 14 [2560/60000 (4%)] Loss: 0.013641
Train Epoch: 14 [3200/60000 (5%)] Loss: 0.057561
Train Epoch: 14 [3840/60000 (6%)] Loss: 0.007648
Train Epoch: 14 [4480/60000 (7%)] Loss: 0.008538
Train Epoch: 14 [5120/60000 (9%)] Loss: 0.012323
Train Epoch: 14 [5760/60000 (10%)] Loss: 0.098143
Train Epoch: 14 [6400/60000 (11%)] Loss: 0.003731
Train Epoch: 14 [7040/60000 (12%)] Loss: 0.029406
Train Epoch: 14 [7680/60000 (13%)] Loss: 0.005672
Train Epoch: 14 [8320/60000 (14%)] Loss: 0.024968
Train Epoch: 14 [8960/60000 (15%)] Loss: 0.001314
Train Epoch: 14 [9600/60000 (16%)] Loss: 0.010014
Train Epoch: 14 [10240/60000 (17%)] Loss: 0.016016
Train Epoch: 14 [10880/60000 (18%)] Loss: 0.002855
Train Epoch: 14 [11520/60000 (19%)] Loss: 0.011906
Train Epoch: 14 [12160/60000 (20%)] Loss: 0.008759
Train Epoch: 14 [12800/60000 (21%)] Loss: 0.012760
Train Epoch: 14 [13440/60000 (22%)] Loss: 0.105929
Train Epoch: 14 [14080/60000 (23%)] Loss: 0.067378
Train Epoch: 14 [14720/60000 (25%)] Loss: 0.009053
Train Epoch: 14 [15360/60000 (26%)] Loss: 0.001238
Train Epoch: 14 [16000/60000 (27%)] Loss: 0.005134
Train Epoch: 14 [16640/60000 (28%)] Loss: 0.006088
Train Epoch: 14 [17280/60000 (29%)] Loss: 0.001667
Train Epoch: 14 [17920/60000 (30%)] Loss: 0.000448
Train Epoch: 14 [18560/60000 (31%)] Loss: 0.005646
Train Epoch: 14 [19200/60000 (32%)] Loss: 0.003152
Train Epoch: 14 [19840/60000 (33%)] Loss: 0.190961
Train Epoch: 14 [20480/60000 (34%)] Loss: 0.002776
Train Epoch: 14 [21120/60000 (35%)] Loss: 0.001240
Train Epoch: 14 [21760/60000 (36%)] Loss: 0.036356
Train Epoch: 14 [22400/60000 (37%)] Loss: 0.006674
Train Epoch: 14 [23040/60000 (38%)] Loss: 0.196805
Train Epoch: 14 [23680/60000 (39%)] Loss: 0.006047
Train Epoch: 14 [24320/60000 (41%)] Loss: 0.071522
Train Epoch: 14 [24960/60000 (42%)] Loss: 0.043944
Train Epoch: 14 [25600/60000 (43%)] Loss: 0.012566
Train Epoch: 14 [26240/60000 (44%)] Loss: 0.031596
Train Epoch: 14 [26880/60000 (45%)] Loss: 0.010121
Train Epoch: 14 [27520/60000 (46%)] Loss: 0.018184
Train Epoch: 14 [28160/60000 (47%)] Loss: 0.001499
Train Epoch: 14 [28800/60000 (48%)] Loss: 0.010726
Train Epoch: 14 [29440/60000 (49%)] Loss: 0.001483
Train Epoch: 14 [30080/60000 (50%)] Loss: 0.010917
Train Epoch: 14 [30720/60000 (51%)] Loss: 0.073289
Train Epoch: 14 [31360/60000 (52%)] Loss: 0.000825
Train Epoch: 14 [32000/60000 (53%)] Loss: 0.100856
Train Epoch: 14 [32640/60000 (54%)] Loss: 0.018616
Train Epoch: 14 [33280/60000 (55%)] Loss: 0.044124
Train Epoch: 14 [33920/60000 (57%)] Loss: 0.004864
Train Epoch: 14 [34560/60000 (58%)] Loss: 0.061987
Train Epoch: 14 [35200/60000 (59%)] Loss: 0.012720
Train Epoch: 14 [35840/60000 (60%)] Loss: 0.016332
Train Epoch: 14 [36480/60000 (61%)] Loss: 0.031489
Train Epoch: 14 [37120/60000 (62%)] Loss: 0.013975
Train Epoch: 14 [37760/60000 (63%)] Loss: 0.004155
Train Epoch: 14 [38400/60000 (64%)] Loss: 0.025105
Train Epoch: 14 [39040/60000 (65%)] Loss: 0.007277
Train Epoch: 14 [39680/60000 (66%)] Loss: 0.014628
Train Epoch: 14 [40320/60000 (67%)] Loss: 0.016834
Train Epoch: 14 [40960/60000 (68%)] Loss: 0.017062
Train Epoch: 14 [41600/60000 (69%)] Loss: 0.001647
Train Epoch: 14 [42240/60000 (70%)] Loss: 0.033741
Train Epoch: 14 [42880/60000 (71%)] Loss: 0.030466
Train Epoch: 14 [43520/60000 (72%)] Loss: 0.008528
Train Epoch: 14 [44160/60000 (74%)] Loss: 0.003342
Train Epoch: 14 [44800/60000 (75%)] Loss: 0.000242
Train Epoch: 14 [45440/60000 (76%)] Loss: 0.024705
Train Epoch: 14 [46080/60000 (77%)] Loss: 0.010056
Train Epoch: 14 [46720/60000 (78%)] Loss: 0.022075
Train Epoch: 14 [47360/60000 (79%)] Loss: 0.006427
Train Epoch: 14 [48000/60000 (80%)] Loss: 0.079013
Train Epoch: 14 [48640/60000 (81%)] Loss: 0.003491
Train Epoch: 14 [49280/60000 (82%)] Loss: 0.011261
Train Epoch: 14 [49920/60000 (83%)] Loss: 0.003170
Train Epoch: 14 [50560/60000 (84%)] Loss: 0.009894
Train Epoch: 14 [51200/60000 (85%)] Loss: 0.099669
Train Epoch: 14 [51840/60000 (86%)] Loss: 0.008581
Train Epoch: 14 [52480/60000 (87%)] Loss: 0.004044
Train Epoch: 14 [53120/60000 (88%)] Loss: 0.016344
Train Epoch: 14 [53760/60000 (90%)] Loss: 0.006215
Train Epoch: 14 [54400/60000 (91%)] Loss: 0.013414
Train Epoch: 14 [55040/60000 (92%)] Loss: 0.039810
Train Epoch: 14 [55680/60000 (93%)] Loss: 0.003840
Train Epoch: 14 [56320/60000 (94%)] Loss: 0.008363
Train Epoch: 14 [56960/60000 (95%)] Loss: 0.046253
Train Epoch: 14 [57600/60000 (96%)] Loss: 0.011893
Train Epoch: 14 [58240/60000 (97%)] Loss: 0.003144
Train Epoch: 14 [58880/60000 (98%)] Loss: 0.007198
Train Epoch: 14 [59520/60000 (99%)] Loss: 0.002651
Test set: Average loss: 0.0271, Accuracy: 9917/10000 (99%)
root@albertabeef-HP-Z4-G4-Workstation:/dockerx/examples/mnist#While this was running, I also ran "rocm-smi" in another console, to verify that the GPU was being used. Below is the highest GPU usage I captured:
albertabeef@albertabeef-HP-Z4-G4-Workstation~$ rocm-smi
========================= ROCm System Management Interface =========================
=================================== Concise Info ===================================
GPU Temp (DieEdge) AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU%
0 34.0c 71.0W 1883Mhz 772Mhz 20.0% auto 241.0W 3% 23%
====================================================================================
=============================== End of ROCm SMI Log ================================I consider this to be a successful validation of the PyTorch framework with the AMD Radeon Pro W7900 GPU !
https://www.amd.com/en/developer/resources/ml-radeon.html
Today I received the Front Fan for my workstation.
HP Z4 G4 Fan and Front Card Guide Kit
Installation was simple, and I now have an legit HP workstation, accelerated with an AMD GPU.
1 / 2 • HP Z4 G4 workstation + AMD Radeon Pro W7900 + Front Fan
Now that my system is legit, I can make a more serious test.
The PyTorch examples come with a Vision Transformer.
root@albertabeef-HP-Z4-G4-Workstation:/dockerx/examples# cd vision_transformer/
root@albertabeef-HP-Z4-G4-Workstation:/dockerx/examples/vision_transformer# ls
README.md best-weights.pt dataset main.py requirements.txt
root@albertabeef-HP-Z4-G4-Workstation:/dockerx/examples/vision_transformer# ls dataset
cifar-10-batches-py cifar-10-python.tar.gz
root@albertabeef-HP-Z4-G4-Workstation:/dockerx/examples/vision_transformer# python3 main.py
Files already downloaded and verified
Files already downloaded and verified
EPOCH[TRAIN]1/10: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 12500/12500 [11:06<00:00, 18.75it/s, Loss=2.812245]
EPOCH[VALID]1/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:37<00:00, 65.87it/s, Loss=2.307569]
Saved Best Weights
EPOCH[TRAIN]2/10: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 12500/12500 [11:11<00:00, 18.62it/s, Loss=2.780969]
EPOCH[VALID]2/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:37<00:00, 67.23it/s, Loss=2.305201]
Saved Best Weights
EPOCH[TRAIN]3/10: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 12500/12500 [11:10<00:00, 18.64it/s, Loss=3.020522]
EPOCH[VALID]3/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:37<00:00, 67.23it/s, Loss=2.306001]
EPOCH[TRAIN]4/10: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 12500/12500 [11:11<00:00, 18.61it/s, Loss=2.606667]
EPOCH[VALID]4/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:37<00:00, 65.93it/s, Loss=2.307188]
EPOCH[TRAIN]5/10: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 12500/12500 [11:13<00:00, 18.57it/s, Loss=3.050503]
EPOCH[VALID]5/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:37<00:00, 66.19it/s, Loss=2.309119]
EPOCH[TRAIN]6/10: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 12500/12500 [11:12<00:00, 18.58it/s, Loss=2.862878]
EPOCH[VALID]6/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:37<00:00, 66.71it/s, Loss=2.305250]
EPOCH[TRAIN]7/10: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 12500/12500 [11:08<00:00, 18.70it/s, Loss=2.799589]
EPOCH[VALID]7/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:37<00:00, 67.07it/s, Loss=2.304259]
Saved Best Weights
EPOCH[TRAIN]8/10: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 12500/12500 [11:07<00:00, 18.72it/s, Loss=2.635211]
EPOCH[VALID]8/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:37<00:00, 66.04it/s, Loss=2.305211]
EPOCH[TRAIN]9/10: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 12500/12500 [11:08<00:00, 18.70it/s, Loss=2.949558]
EPOCH[VALID]9/10: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:38<00:00, 65.71it/s, Loss=2.304510]
EPOCH[TRAIN]10/10: 100%|████████████████████████████████████████████████████████████████████████████████████████| 12500/12500 [11:14<00:00, 18.52it/s, Loss=2.810732]
EPOCH[VALID]10/10: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 2500/2500 [00:38<00:00, 64.23it/s, Loss=2.458832]
Training Loss : 2.7995891003608704
Valid Loss : 2.304258857059479
root@albertabeef-HP-Z4-G4-Workstation:/dockerx/examples/vision_transformer#While this was running, I also ran "rocm-smi" in another console, to verify how much the GPU was being used. Below are the highest GPU usage and Average Power I captured:
...
albertabeef@albertabeef-HP-Z4-G4-Workstation:~/dockerx$ rocm-smi
========================= ROCm System Management Interface =========================
=================================== Concise Info ===================================
GPU Temp (DieEdge) AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU%
0 65.0c 236.0W 2393Mhz 1124Mhz 47.84% auto 241.0W 8% 100%
====================================================================================
=============================== End of ROCm SMI Log ================================
...
albertabeef@albertabeef-HP-Z4-G4-Workstation:~/dockerx$ rocm-smi
========================= ROCm System Management Interface =========================
=================================== Concise Info ===================================
GPU Temp (DieEdge) AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU%
0 72.0c 240.0W 2385Mhz 1124Mhz 47.84% auto 241.0W 8% 89%
====================================================================================
=============================== End of ROCm SMI Log ================================
...Hard to believe I trained a Vision Transformer during only a few hours. It would be interesting to measure how much power it took for this task, and how much cost it represents.
In my region, we have the following tariffs:
- $0.06509 / kWh (up to 40kWh)
- $0.10041 / kWh (above 40kWh)
Assuming, I am already consuming the first 40 kWh, training this Vision Transformer cost:
- 0.240 kW * 1 h * $0.10041 / kWh = $0.024 for one hour
This is much lower than I anticipated ... only a fraction of what electricity costs in London (which I heard was $0.67 / kWh).
Still, something tells me my electrical bill is going to increase in 2024 !
ConclusionIn conclusion, the Vitis-AI 3.5 docker containers do not support the AMD Radeon Pro W7900 GPU.
However, the AMD ROCm PyTorch docker container does support the GPU. This opens the road for training PyTorch models.
Stay tuned for more content on PyTorch training with the AMD Radeon Pro W7900 GPU.
AcknowledgementsI wanted to thank AMD for this loaner, what a great Christmas surprise !
I you want a chance at getting this same GPU card, enroll in AMD's Pervasive AI developer contest:
https://www.hackster.io/contests/amd2023
For the purpose of this contest, a Study Guide for the Radeon Pro W7900 has been provided:
I will definitely be checking this out !
Version History- 2023/12/04 : Reception of AMD Radeon Pro W7900 GPU
- 2023/12/06 : Installing the Software
- 2023/12/18 : Hardware Upgrade 1 - 1000W Power Supply
- 2023/12/18 : Detecting the GPU card
- 2023/12/19 : Testing the Vitis-AI docker containers
- 2023/12/20 : Testing the ROCm docker containers
- 2023/12/23 : Hardware Upgrade 2 - Front Fan
- 2023/12/23 : Estimating the Power Cost
65 projects • 317 followers
Mario Bergeron specializes in embedded vision, machine learning, and robotics.
Comments