Helpneededtrainingmnistlenet5inspiredtopology

Hi, I've started working on an mnist training app, based on tiny-dnn's mnist example code, an trying to reproduce this network topology : keras-cnn-inspired-by-lenet-5.
My app source code is here, but I'm facing some issues trying to make training converge. Using the mnist dataset included in tiny-dnn, after 6 epochs the scores drops from 9832/10000 to 980/10000.
As you can see I use a custom adamax optimizer.

Could you help me understand this drop? Did I make something wrong in my code?

Thx for the help

该提问来源于开源项目：tiny-dnn/tiny-dnn