Childclassesofactivation_layerforallcurrentlysupportedactivations.

This PR is a succession to #529 which takes forward the task of decoupling layers and activations. Prior to this PR, a new base class

was created, and here several child classes for other activations have been made and used in tests.

Scope of this PR:

1. Following new layers inheriting

   1	tiny_dnn::activation_layer

   have been created in

   1

   tiny_dnn

   namespace, supporting all current activation functions of the library:

   • 
     

     1	sigmoid_layer

     
     
   
   
   • 
     

     1	tanh_layer

     
     
   
   
   • 
     

     1	tanh_p1m2_layer

     
     
   
   
   • 
     

     1	relu_layer

     (Created before in #529 )

   
   
   • 
     

     1	softmax_layer

     
     
   
   
   • 
     

     1	leaky_relu_layer

     
     
   
   
   • 
     

     1

     elu_layer

     
     
   
   



2. Refactor

   1

   make_mlp

   method to construct an MLP with

   1	fully_connected

   layers templated with identity activation, alongwith desired activation as a layer. This will make dropping of template later easy.




3. Use these newly created layers in atleast one of the tests written in test_network.h to see if they work properly.

After this PR:

1. Refactor complete codebase where occurrences of layers with templated activations are replaced by layers with identity activation as template and an attached activation layer after the original layer. I could have done it here, but it would make the PR diff huge and hard to review.




2. After this point, since all the existing layer instances will have identity activation as template parameter, the

   1	fully_connected

   layer can be experimented with and template can be removed. Some changes in fully connected OpKernels will have to be made for all backends to properly undertake this, as we will remove

   1	vector_type::aux

   from output data in absence of activation template.

该提问来源于开源项目：tiny-dnn/tiny-dnn