Neural Networks

Neural Networks

Neural networks refer to machine learning algorithms focusing on improving the various scenarios in the real world. Training datasets to automatically detect specific patterns is not always an easy task. So, implementing neural networks in real-world use cases plays an essential role in creating a more advanced human-oriented feature support. Neural networks in deep learning focus on improving performance for different models. The paper focuses on various aspects provided on the references which further the earlier learned concepts in neural networks.

1. a) What thoughts are introduced in these papers that extend those covered in the

CO3311 subject guide and relevant readings?

The paper introduces various aspects which supplement the earlier learned concepts learned earlier. Some of the ideas which have been explained further in these topics include backpropagation to train multi-layer architectures. From the source “Deep learning,” the author argues that the earlier approach to replace hand-engineering with multi-layer networks has does not fully satisfy the needs for neural networks ((LeCun, Bengio, & Hinton, 2015).

As an alternative to the earlier approach, the author suggests training multi-layer architectures using stochastic gradient descent. The algorithm assumes that, for any smooth function modules in terms of internal weights and inputs, it is possible to compute gradient using the backpropagation method. The backpropagation technique uses the chain rule method to return results in a multilayered architecture ((Li, 2019). By computing backward, given the output of the modules, one can achieve the critical derivative of the module (gradient).  All respective module’s angle can be obtained by propagating throughout the architectural layers forming the stack starting with the top tier were prediction is to be made to the bottom layer were respective external inputs are to be supplied. After computing the gradient (derivatives), each architecture module’s slope can be calculated concerning their weights.

In “DEEP LEARNING – REVIEW,” the author gives a clear concept which improves the general structure of the earlier learned concept of convolutional neural networks. The author provides an overview of how convolutional neural networks can be used to give accurate estimates of a multi-dimensional array data, especially when dealing with two-dimensional image data. The convolutional neural network has capabilities of taking into account natural signals properties. The ConvNets also take their inputs in the form of multiple layers, allow for shared weights and local connection as opposed to earlier approaches. ConvNets utilize the filter bank concept, which will enable interconnections between individual local patches within an organized mapped feature. The results obtained from the locally weighted sum are then computed by a non-linearity model such as ReLU. The same filter banks are shared within related features hence maintaining the model integrity.

1. b) Summarise each source.

A Brief Introduction to Deep Learning by Yangyan Li summary

The author focuses on five main elementary areas: Artificial neural network, Overfitting, Convolutional Layer, Backpropagation, and fully connected layer. In most cases, humans and animals learn through both supervised and unsupervised learning. Therefore, the main focus of learning neural networks is to allow systems to detect patterns and learn from them automatically. Each use case has distinctive features that uniquely identify themselves from other models (Ranzato, Hinton, & LeCun, 2015). Therefore, neural networks should readily identify patterns and clearly distinguish them from others. To enhance neural network performance, it is essential to test for over-fitting issues. The neural network should be designed to allow neural networks to learn from datasets as opposed to memorizing. This aspect ensures flexibility, especially when a new dataset is introduced in the system.

Deep Learning – Review Yann Lecun, Yoshua Bengio & Geoffrey Hinton Summary

The authors focus on various stages involved in neural network development. The earliest approach in the neural network field was based on computational models. However, the idea has been enhanced over the year by different researchers until the concept of convolutional neural networks emerged in 1990. The field has increased due to its demand on various areas such as drug molecule prediction, reconstruction of brain circuits, predicting effects of non-coding regions of DNA and language translation, and Image and speech recognition. The author tries to explain various approaches that can be adopted when dealing with multilayered neural networks. The author suggests that the backpropagation technique can be used to solve the module gradient through the various layers. The analogy presented in this case gives a clear overview of how the proposed algorithm works to counter multiple problems during the learning process. However, the primary method used in this case is supervised learning, giving room for future study to focus on unsupervised learning approach.

1. c) Are these new concepts easy to understand, given the material that you have already covered in CO3311?

The new concepts given in the resources are easy to understand, especially when someone has a piece of prior knowledge on CO3311. The aspects highlighted in these resources provide an in-depth understanding of concepts such as convolutional neural networks, backpropagation algorithms in neural networks, and dataset training models. The idea of convolutional neural networks has been emphasized in each resource, giving an insight into what is expected in the current trend. The authors provide unique representation models of backpropagation techniques in minimizing error rates while working with multiple array data. The convolutional neural network has been an ambiguous topic with the introduction of aspects such as machine learning and computer vision. However, by learning about the various elements such as ReLU and pooling layers, it has become easier to distinguish. Issues such as non-linearity need ReLu functions for every pixel bringing compelling results. In the earlier study guide, aspects such as vanishing gradient were not adequately presented. Therefore, the study resources given provide room for advancement in the fundamental study areas.

1. d) What improvements would you have liked to see in these articles?

Various vital areas need to be improved for future research. In the study cases provided, issues such as learning rate and Loss and optimizer functions have not been presented well — backpropagation technique to solve multiple layered architecture problems. However, Choosing the best optimum learning rate is crucial since it answers the hypothesis of whether global minima has been converged in the neural network or not. Therefore, the right decision is essential on whether to adopt a slower learning rate, which finally converges to the global minima or oscillating around the global minima by choosing a faster learning rate. Taking a small learning rate can cause the neural network to experience vanishing gradient issues; hence a better approach to ethical decision making is needed for future research.

All sources provided focus on supervised learning for datasets, and this gives a limited capacity system that can hardly predict emerging datasets. The over-fitting dataset tries to memorize data instead of learning from them. This aspect should be improved to cater for unsupervised learning. The current trend in the technological field is to allow machines to have features related to human beings and be able to learn from new patterns.

References

LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444. doi:10.1038/nature14539

Li, Y. (2019). A Brief History of Deep Learning. Deep Learning. doi:10.7551/mitpress/11171.003.0007

Ranzato, M., Hinton, G., & LeCun, Y. (2015). Guest Editorial: Deep Learning. International Journal of Computer Vision, 113(1), 1-2. doi:10.1007/s11263-015-0813-1