<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD Journal Publishing DTD v2.3 20070202//EN" "journalpublishing.dtd">
<article article-type="research-article" dtd-version="2.3" xml:lang="EN" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">Front. Comput. Sci.</journal-id>
<journal-title>Frontiers in Computer Science</journal-title>
<abbrev-journal-title abbrev-type="pubmed">Front. Comput. Sci.</abbrev-journal-title>
<issn pub-type="epub">2624-9898</issn>
<publisher>
<publisher-name>Frontiers Media S.A.</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="publisher-id">786964</article-id>
<article-id pub-id-type="doi">10.3389/fcomp.2021.786964</article-id>
<article-categories>
<subj-group subj-group-type="heading">
<subject>Computer Science</subject>
<subj-group>
<subject>Original Research</subject>
</subj-group>
</subj-group>
</article-categories>
<title-group>
<article-title>Singular Learning of Deep Multilayer Perceptrons for EEG-Based Emotion Recognition</article-title>
<alt-title alt-title-type="left-running-head">Guo et&#x20;al.</alt-title>
<alt-title alt-title-type="right-running-head">Singular Learning for Emotion Recognition</alt-title>
</title-group>
<contrib-group>
<contrib contrib-type="author">
<name>
<surname>Guo</surname>
<given-names>Weili</given-names>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
<xref ref-type="aff" rid="aff2">
<sup>2</sup>
</xref>
<uri xlink:href="https://loop.frontiersin.org/people/1460076/overview"/>
</contrib>
<contrib contrib-type="author" corresp="yes">
<name>
<surname>Li</surname>
<given-names>Guangyu</given-names>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
<xref ref-type="aff" rid="aff2">
<sup>2</sup>
</xref>
<xref ref-type="corresp" rid="c001">&#x2a;</xref>
</contrib>
<contrib contrib-type="author">
<name>
<surname>Lu</surname>
<given-names>Jianfeng</given-names>
</name>
<xref ref-type="aff" rid="aff2">
<sup>2</sup>
</xref>
</contrib>
<contrib contrib-type="author" corresp="yes">
<name>
<surname>Yang</surname>
<given-names>Jian</given-names>
</name>
<xref ref-type="aff" rid="aff1">
<sup>1</sup>
</xref>
<xref ref-type="aff" rid="aff2">
<sup>2</sup>
</xref>
<xref ref-type="corresp" rid="c001">&#x2a;</xref>
</contrib>
</contrib-group>
<aff id="aff1">
<sup>1</sup>
<institution>PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, Jiangsu Key Lab of Image and Video Understanding for Social Security, Nanjing University of Science and Technology</institution>, <addr-line>Nanjing</addr-line>, <country>China</country>
</aff>
<aff id="aff2">
<sup>2</sup>
<institution>School of Computer Science and Engineering, Nanjing University of Science and Technology</institution>, <addr-line>Nanjing</addr-line>, <country>China</country>
</aff>
<author-notes>
<fn fn-type="edited-by">
<p>
<bold>Edited by:</bold> <ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/1207483/overview">Xiaopeng Hong</ext-link>, Xi&#x2019;an Jiaotong University, China</p>
</fn>
<fn fn-type="edited-by">
<p>
<bold>Reviewed by:</bold> <ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/874895/overview">Lianyong Qi</ext-link>, Qufu Normal University, China</p>
<p>
<ext-link ext-link-type="uri" xlink:href="https://loop.frontiersin.org/people/1517955/overview">Liangfei Zhang</ext-link>, University of St Andrews, United&#x20;Kingdom</p>
</fn>
<corresp id="c001">&#x2a;Correspondence: Guangyu Li, <email>guangyu.li2017@njust.edu.cn</email>; Jian Yang, <email>csjyang@njust.edu.cn</email>
</corresp>
<fn fn-type="other">
<p>This article was submitted to Human-Media Interaction, a section of the journal Frontiers in Computer Science</p>
</fn>
</author-notes>
<pub-date pub-type="epub">
<day>21</day>
<month>12</month>
<year>2021</year>
</pub-date>
<pub-date pub-type="collection">
<year>2021</year>
</pub-date>
<volume>3</volume>
<elocation-id>786964</elocation-id>
<history>
<date date-type="received">
<day>30</day>
<month>09</month>
<year>2021</year>
</date>
<date date-type="accepted">
<day>05</day>
<month>11</month>
<year>2021</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright &#xa9; 2021 Guo, Li, Lu and Yang.</copyright-statement>
<copyright-year>2021</copyright-year>
<copyright-holder>Guo, Li, Lu and Yang</copyright-holder>
<license xlink:href="http://creativecommons.org/licenses/by/4.0/">
<p>This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these&#x20;terms.</p>
</license>
</permissions>
<abstract>
<p>Human emotion recognition is an important issue in human&#x2013;computer interactions, and electroencephalograph (EEG) has been widely applied to emotion recognition due to its high reliability. In recent years, methods based on deep learning technology have reached the state-of-the-art performance in EEG-based emotion recognition. However, there exist singularities in the parameter space of deep neural networks, which may dramatically slow down the training process. It is very worthy to investigate the specific influence of singularities when applying deep neural networks to EEG-based emotion recognition. In this paper, we mainly focus on this problem, and analyze the singular learning dynamics of deep multilayer perceptrons theoretically and numerically. The results can help us to design better algorithms to overcome the serious influence of singularities in deep neural networks for EEG-based emotion recognition.</p>
</abstract>
<kwd-group>
<kwd>emotion recognition</kwd>
<kwd>EEG</kwd>
<kwd>deep multilayer perceptrons</kwd>
<kwd>singular learning</kwd>
<kwd>theoretical and numerical analysis</kwd>
</kwd-group>
<contract-sponsor id="cn001">National Natural Science Foundation of China<named-content content-type="fundref-id">10.13039/501100001809</named-content>
</contract-sponsor>
</article-meta>
</front>
<body>
<sec id="s1">
<title>1 Introduction</title>
<p>Emotion recognition is a fundamental task in affective computing and has attracted many researchers&#x2019; attention in recent years (<xref ref-type="bibr" rid="B17">Mauss and Robinson, 2009</xref>). Human emotion can be expressed through external signals and internal signals, where external signals usually include facial expressions, body actions, and speeches, and electroencephalograph (EEG) and galvanic skin response (GSR) are typical internal signals. EEG is the method to measure electrical activities of the brain by using electrodes along the scalp skin and it is rather reliable; therefore, EEG has played a more significant role in investigating human emotion recognition problem in recent years (<xref ref-type="bibr" rid="B32">Yin et&#x20;al., 2021</xref>).</p>
<p>For the emotion recognition problem based on EEG signals, researchers mainly investigate this issue from two aspects: how to extract better features from EEG signals and how to construct a model with better performance. For aspect 1, researchers have investigated the feature extraction methods of EEG signals from a time domain, frequency domain, and time&#x2013;frequency domain, respectively, and a series of results have been given previously (<xref ref-type="bibr" rid="B8">Fang et&#x20;al., 2020</xref>; <xref ref-type="bibr" rid="B19">Nawa et&#x20;al., 2020</xref>). In this paper, we mainly focus on aspect 2, i.e.,&#x20;the computational model problem, and researchers have proposed many models to recognize emotions through EEG signals (<xref ref-type="bibr" rid="B39">Zong et&#x20;al., 2016</xref>; <xref ref-type="bibr" rid="B28">Yang et&#x20;al., 2018a</xref>; <xref ref-type="bibr" rid="B33">Zhang et&#x20;al., 2019</xref>; <xref ref-type="bibr" rid="B7">Cui et&#x20;al., 2020</xref>). In recent years, deep learning technology has achieved great success in many fields (<xref ref-type="bibr" rid="B29">Yang et&#x20;al., 2018b</xref>; <xref ref-type="bibr" rid="B31">Yang et&#x20;al., 2019</xref>; <xref ref-type="bibr" rid="B5">Basodi et&#x20;al., 2020</xref>; <xref ref-type="bibr" rid="B38">Zhu and Zhang, 2021</xref>), and many works are devoted to addressing the EEG emotion recognition issue by applying deep neural networks (DNNs) (<xref ref-type="bibr" rid="B6">Cao et&#x20;al., 2020</xref>; <xref ref-type="bibr" rid="B18">Natarajan et&#x20;al., 2021</xref>), where the performances based on deep learning also show significant superiority of conventional methods (<xref ref-type="bibr" rid="B20">Ng et&#x20;al., 2015</xref>; <xref ref-type="bibr" rid="B25">Tzirakis et&#x20;al., 2017</xref>; <xref ref-type="bibr" rid="B11">Hassan et&#x20;al., 2019</xref>). However, the learning dynamics of DNNs, including deep multilayer perceptrons (MLPs), deep belief networks and deep convolution neural networks, are often affected by singularities, which exist in the parameter space of DNNs (<xref ref-type="bibr" rid="B21">Nitta, 2016</xref>).</p>
<p>Due to the influence of singularities, the training of DNNs often becomes very slow and the plateau phenomenon can often be observed. When the DNNs are applied to EEG-based emotion recognition, the severe negative effect of singularities on the learning process of DNNs is also inevitable, where the efficiency and performance of networks can also not be guaranteed. However, up to now, there are rarely literatures investigating this problem. In this paper, we mainly concern this problem. The main contribution of this paper is to take the theoretical and numerical analysis of singular learning in DNNs for EEG-based emotion recognition. We choose deep MLPs as the learning machine, where deep MLPs are of typical DNNs and the results are also representative for other DNNs. The types of singularities in parameter space are analyzed and the specific influence of the singularities is clearly shown. Based on the obtained results in this paper, we can further design the related algorithms to overcome this&#x20;issue.</p>
<p>The rest of this paper is organized as follows. A brief review of related work is presented in <xref ref-type="sec" rid="s2">Section 2</xref>. In <xref ref-type="sec" rid="s3">Section 3</xref>, theoretical analysis of singularities in deep MLPs for EEG-based emotion recognition is taken and then the learning dynamics near singularities are numerically analyzed in <xref ref-type="sec" rid="s4">Section 4</xref>. <xref ref-type="sec" rid="s5">Section 5</xref> states conclusion and discussion.</p>
</sec>
<sec id="s2">
<title>2 Related Work</title>
<p>In this section, we provide a brief overview of previous work on EEG-based emotion recognition and singular learning of&#x20;DNNs.</p>
<p>In recent years, due to the high accuracy and stabilization of EEG signals, EEG-based algorithms have attracted ever-increasing attention in emotion recognition field. To extract better features of EEG signals, researchers have proposed various feature extraction models (<xref ref-type="bibr" rid="B36">Zheng et&#x20;al., 2014</xref>; <xref ref-type="bibr" rid="B35">Zheng, 2017</xref>; <xref ref-type="bibr" rid="B24">Tao et&#x20;al., 2020</xref>; <xref ref-type="bibr" rid="B34">Zhao et&#x20;al., 2021</xref>), such as power spectral density (PSD), differential entropy (DE), and differential asymmetry (DASM). By using PSD and DE to extract dimension reduced features of EEG signals, <xref ref-type="bibr" rid="B8">Fang et&#x20;al. (2020)</xref> chose the original features and dimension reduced features as the multi-feature input and verified the validity of the proposed method in the experiment part. <xref ref-type="bibr" rid="B12">Li et&#x20;al. (2020)</xref> integrated psychoacoustic knowledge and raw waveform embedding within an augmented feature space. <xref ref-type="bibr" rid="B23">Song et&#x20;al. (2020)</xref> employed an additional branch to characterize the intrinsic dynamic relationships between different EEG channels and a type of sparse graphic representation was presented to extract more discriminative features. Besides the feature extraction methods, more attention is paid to study the emotion classification. Given that the deep learning technology has excellent capabilities, various types of DNNs have been widely used in emotion classification (<xref ref-type="bibr" rid="B13">Li et&#x20;al., 2018</xref>; <xref ref-type="bibr" rid="B14">Li et&#x20;al., 2019</xref>; <xref ref-type="bibr" rid="B16">Ma et&#x20;al., 2019</xref>; <xref ref-type="bibr" rid="B4">Atmaja and Akagi, 2020</xref>; <xref ref-type="bibr" rid="B7">Cui et&#x20;al., 2020</xref>; <xref ref-type="bibr" rid="B37">Zhong et&#x20;al., 2020</xref>), including deep convolution neural networks, deep MLPs, long short term memory (LSTM)-based recurrent neural networks, and graph neural networks. The obtained results show that these DNN models can provide superior performance compared to previous models (<xref ref-type="bibr" rid="B27">Yang et&#x20;al., 2021a</xref>; <xref ref-type="bibr" rid="B30">Yang et&#x20;al., 2021b</xref>).</p>
<p>As mentioned above, various DNNs have been widely used in EEG-based emotion recognition; however, the training processes of DNNs often encounter many difficulties. Even if numerous research studies have been developed to conduct explanatory research, it is still very far to revealing the mechanism. As there are singularities in the parameter space of DNNs where the Fisher information matrix is singular, the singular learning dynamics of DNNs have been studied and have attracted more and more attention. As the basis of DNNs, traditional neural networks often suffer from the serious influence of various singularities (<xref ref-type="bibr" rid="B3">Amari et&#x20;al., 2006</xref>; <xref ref-type="bibr" rid="B10">Guo et&#x20;al., 2018</xref>; <xref ref-type="bibr" rid="B9">Guo et&#x20;al., 2019</xref>), and the learning dynamics of DNNs are also easy to be influenced by the singularities. <xref ref-type="bibr" rid="B21">Nitta (2016</xref>, <xref ref-type="bibr" rid="B22">2018)</xref> analyzed the types of singularity in DNNs and deep complex-value neural networks. <xref ref-type="bibr" rid="B2">Ainsworth and Shin (2020)</xref> investigated the plateau phenomenon in Relu-based neural networks. By using the spectral information of Fisher information matrix, <xref ref-type="bibr" rid="B15">Liao et&#x20;al. (2020)</xref> proposed an algorithm to accelerate the training process of&#x20;DNNs.</p>
<p>In view of the serious influence of singularities to DNNs, the training processes of DNNs will also encounter difficulties when applying DNNs to EEG-based emotion recognition. Thus, it is necessary to take the theoretical and numerical analysis to reveal the mechanism and propose related algorithms to overcome the influence of singularities.</p>
</sec>
<sec id="s3">
<title>3 Theoretical Analysis of Singular Learning Dynamics of Deep Multilayer Perceptrons</title>
<p>In this section, we theoretically analyze the learning dynamics near singularities of deep MLPs for the EEG-based emotion recognition.</p>
<sec id="s3-1">
<title>3.1 Learning Paradigm of Deep Multilayer Perceptrons</title>
<p>Firstly, we introduce a typical learning paradigm of deep MLPs. For a typical deep multilayer perceptrons with <italic>L</italic> hidden layers (the architecture of the networks is shown in <xref ref-type="fig" rid="F1">Figure&#x20;1</xref>), assuming <italic>M</italic>
<sub>
<italic>i</italic>
</sub> is the neuron number of hidden layer <italic>i</italic>, <italic>M</italic>
<sub>0</sub> is the dimension of the input layer and <italic>M</italic>
<sub>
<italic>L</italic>&#x2b;1</sub> is the dimension of the output layer, we denote that: <inline-formula id="inf1">
<mml:math id="m1">
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> represents the weight connecting from the <italic>j</italic>th node of the previous layer to the <italic>k</italic>th node of hidden layer <italic>i</italic>, and <inline-formula id="inf2">
<mml:math id="m2">
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>p</mml:mi>
<mml:mi>q</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>L</mml:mi>
<mml:mo>&#x2b;</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> represents the weight connecting from the <italic>p</italic>th node of hidden layer <italic>L</italic> to the <italic>q</italic>th node of output layer for 1 &#x2264; <italic>i</italic>&#x20;&#x2264; <italic>L</italic>, 1 &#x2264; <italic>j</italic>&#x20;&#x2264; <italic>M</italic>
<sub>
<italic>i</italic>&#x2212;1</sub>, 1 &#x2264; <italic>k</italic>&#x20;&#x2264; <italic>M</italic>
<sub>
<italic>i</italic>
</sub>, 1 &#x2264; <italic>p</italic>&#x20;&#x2264; <italic>M</italic>
<sub>
<italic>L</italic>
</sub>, and 1 &#x2264; <italic>q</italic>&#x20;&#x2264; <italic>M</italic>
<sub>
<italic>L</italic>&#x2b;1</sub>. Then <bold>
<italic>&#x3b8;</italic>
</bold> &#x3d; {<bold>
<italic>W</italic>
</bold>
<sup>(1)</sup>, <bold>
<italic>W</italic>
</bold>
<sup>(2)</sup>, &#x2026;, <bold>
<italic>W</italic>
</bold>
<sup>(<italic>L</italic>&#x2b;1)</sup>} represents all the parameters of the networks, where <inline-formula id="inf3">
<mml:math id="m3">
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
<mml:mo>&#x3d;</mml:mo>
<mml:mrow>
<mml:mo stretchy="false">[</mml:mo>
<mml:mrow>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>,</mml:mo>
<mml:mo>&#x2026;</mml:mo>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>M</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mo stretchy="false">]</mml:mo>
</mml:mrow>
</mml:math>
</inline-formula> and <inline-formula id="inf4">
<mml:math id="m4">
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x3d;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">[</mml:mo>
<mml:mrow>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>,</mml:mo>
<mml:mo>&#x2026;</mml:mo>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>M</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>i</mml:mi>
<mml:mo>&#x2212;</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msub>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mo stretchy="false">]</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mrow>
<mml:mi>T</mml:mi>
</mml:mrow>
</mml:msup>
</mml:math>
</inline-formula> for 1 &#x2264; <italic>i</italic>&#x20;&#x2264; <italic>L</italic>&#x20;&#x2b; 1 and 1 &#x2264; <italic>j</italic>&#x20;&#x2264; <italic>M</italic>
<sub>
<italic>i</italic>
</sub>. In this paper, the widely used log-sigmoid function <inline-formula id="inf5">
<mml:math id="m5">
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>x</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mo>&#x3d;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
<mml:mo>&#x2b;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi>e</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mo>&#x2212;</mml:mo>
<mml:mi>x</mml:mi>
</mml:mrow>
</mml:msup>
</mml:mrow>
</mml:mfrac>
</mml:math>
</inline-formula> is adopted as the activation of hidden layers and the purelin function <italic>&#x3c8;</italic>(<italic>x</italic>) &#x3d; <italic>x</italic> is adopted as the activation function of output layer, then for the input <inline-formula id="inf6">
<mml:math id="m6">
<mml:mi mathvariant="bold-italic">x</mml:mi>
<mml:mo>&#x2208;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="double-struck">R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold-italic">M</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi mathvariant="bold">0</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:msup>
</mml:math>
</inline-formula>, by denoting the input to hidden layer <italic>k</italic> as <bold>
<italic>X</italic>
</bold>
<sup>(<italic>k</italic>&#x2212;1)</sup> for 1 &#x2264; <italic>k</italic>&#x20;&#x2264; <italic>L</italic> and the input to output layer as <bold>
<italic>X</italic>
</bold>
<sup>(<italic>L</italic>)</sup>, the mathematical model of the networks can be described as follows:<disp-formula id="e1">
<mml:math id="m7">
<mml:mi>f</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">x</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mo>&#x3d;</mml:mo>
<mml:mi>&#x3c8;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:msup>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>L</mml:mi>
<mml:mo>&#x2b;</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mrow>
<mml:mi>T</mml:mi>
</mml:mrow>
</mml:msup>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">X</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>L</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mo>&#x3d;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>L</mml:mi>
<mml:mo>&#x2b;</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mrow>
<mml:mi>T</mml:mi>
</mml:mrow>
</mml:msup>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">X</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>L</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
<mml:mo>.</mml:mo>
</mml:math>
<label>(1)</label>
</disp-formula>
</p>
<fig id="F1" position="float">
<label>FIGURE 1</label>
<caption>
<p>Architecture of deep MLPs.</p>
</caption>
<graphic xlink:href="fcomp-03-786964-g001.tif"/>
</fig>
<p>For 1 &#x2264; <italic>k</italic>&#x20;&#x2264; <italic>L</italic>, <bold>
<italic>X</italic>
</bold>
<sup>(<italic>k</italic>)</sup> can be computed as <inline-formula id="inf7">
<mml:math id="m8">
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">X</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
<mml:mo>&#x3d;</mml:mo>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">X</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>k</mml:mi>
<mml:mo>&#x2212;</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
<mml:mo>,</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mo>&#x3d;</mml:mo>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:msup>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>k</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mrow>
<mml:mi>T</mml:mi>
</mml:mrow>
</mml:msup>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">X</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>k</mml:mi>
<mml:mo>&#x2212;</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:math>
</inline-formula> and <bold>
<italic>X</italic>
</bold>
<sup>(0)</sup> is the input&#x20;<bold>
<italic>x</italic>
</bold>.</p>
<p>We choose the square loss function to measure the error:<disp-formula id="e2">
<mml:math id="m9">
<mml:mi>l</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>y</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi mathvariant="bold-italic">x</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mo>&#x3d;</mml:mo>
<mml:mfrac>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:mfrac>
<mml:msup>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>y</mml:mi>
<mml:mo>&#x2212;</mml:mo>
<mml:mi>f</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">x</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msup>
<mml:mo>,</mml:mo>
</mml:math>
<label>(2)</label>
</disp-formula>and use the gradient descent method to minimize the loss:<disp-formula id="e3">
<mml:math id="m10">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>t</mml:mi>
<mml:mo>&#x2b;</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
<mml:mo>&#x2212;</mml:mo>
<mml:mi>&#x3b7;</mml:mi>
<mml:mfrac>
<mml:mrow>
<mml:mi>&#x2202;</mml:mi>
<mml:mi>l</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>y</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi mathvariant="bold-italic">x</mml:mi>
<mml:mo>,</mml:mo>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mrow>
<mml:mi>&#x2202;</mml:mi>
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>t</mml:mi>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:mfrac>
<mml:mo>,</mml:mo>
</mml:math>
<label>(3)</label>
</disp-formula>where <italic>&#x3b7;</italic> is the learning&#x20;rate.</p>
</sec>
<sec id="s3-2">
<title>3.2 Singularities of Deep Multilayer Perceptrons in Electroencephalograph-Based Emotion Recognition</title>
<p>In this paper, we mainly focus on the mechanism of singular learning dynamics of deep MLPs applied to EEG-based emotion recognition domain, not seeking the best performance; therefore, the size of the networks need not to be very large, and an appropriate size that can capture the essence of singular learning dynamics can satisfy the requirement. Without loss of generality, we choose the deep MLPs with two hidden layers and a single output neuron, i.e.,&#x20;<italic>L</italic>&#x20;&#x3d; 2 and <italic>M</italic>
<sub>3</sub> &#x3d; 1, i.e.,&#x20;<inline-formula id="inf8">
<mml:math id="m11">
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
<mml:mo>&#x3d;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x3d;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">[</mml:mo>
<mml:mrow>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>11</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>21</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>,</mml:mo>
<mml:mo>&#x2026;</mml:mo>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>M</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mo stretchy="false">]</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mrow>
<mml:mi>T</mml:mi>
</mml:mrow>
</mml:msup>
</mml:math>
</inline-formula>, we simply denoted as <inline-formula id="inf9">
<mml:math id="m12">
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
<mml:mo>&#x3d;</mml:mo>
<mml:mrow>
<mml:mo stretchy="false">[</mml:mo>
<mml:mrow>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>,</mml:mo>
<mml:mo>&#x2026;</mml:mo>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>M</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mo stretchy="false">]</mml:mo>
</mml:mrow>
</mml:math>
</inline-formula>. Then, the deep MLPs can be rewritten as:<disp-formula id="e4">
<mml:math id="m13">
<mml:mtable class="align" columnalign="left">
<mml:mtr>
<mml:mtd columnalign="right">
<mml:mi>f</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">x</mml:mi>
<mml:mo>,</mml:mo>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mtd>
<mml:mtd columnalign="left">
<mml:mo>&#x3d;</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
<mml:mrow>
<mml:mi>T</mml:mi>
</mml:mrow>
</mml:msup>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">x</mml:mi>
<mml:mo>,</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mo>,</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mtd>
</mml:mtr>
<mml:mtr>
<mml:mtd columnalign="right"/>
<mml:mtd columnalign="left">
<mml:mo>&#x3d;</mml:mo>
<mml:munderover accentunder="false" accent="false">
<mml:mrow>
<mml:mo>&#x2211;</mml:mo>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
<mml:mo>&#x3d;</mml:mo>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:msub>
<mml:mrow>
<mml:mi>M</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
</mml:mrow>
</mml:munderover>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">x</mml:mi>
<mml:mo>,</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mo>.</mml:mo>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:math>
<label>(4)</label>
</disp-formula>
</p>
<p>Next, we analyze the types of singularities. From <xref ref-type="disp-formula" rid="e4">Eq. 4</xref>, we can see that if one output weight equals zero, e.g., <inline-formula id="inf10">
<mml:math id="m14">
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x3d;</mml:mo>
<mml:mn>0</mml:mn>
</mml:math>
</inline-formula>, whatever the values of <bold>
<italic>W</italic>
</bold>
<sup>(1)</sup> and <inline-formula id="inf11">
<mml:math id="m15">
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> be, the output of unit <italic>j</italic> will be always 0 and the unit seems to be vanished. As the values of <bold>
<italic>W</italic>
</bold>
<sup>(1)</sup> and <inline-formula id="inf12">
<mml:math id="m16">
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> have no effect on the output of the deep MLPs, the training process will encounter difficulties on the subspace <inline-formula id="inf13">
<mml:math id="m17">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="normal">R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mrow>
<mml:mo stretchy="false">{</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x3d;</mml:mo>
<mml:mn>0</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">}</mml:mo>
</mml:mrow>
</mml:math>
</inline-formula>. Besides the above singularity, if there are two elements of weight <bold>
<italic>W</italic>
</bold>
<sup>(2)</sup> overlap, e.g., <inline-formula id="inf14">
<mml:math id="m18">
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x3d;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula>, then<disp-formula id="equ1">
<mml:math id="m19">
<mml:mtable class="align-star" columnalign="left">
<mml:mtr>
<mml:mtd columnalign="right"/>
<mml:mtd columnalign="left">
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">x</mml:mi>
<mml:mo>,</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mtd>
</mml:mtr>
<mml:mtr>
<mml:mtd columnalign="right"/>
<mml:mtd columnalign="left">
<mml:mspace width="1em"/>
<mml:mo>&#x2b;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">x</mml:mi>
<mml:mo>,</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mtd>
</mml:mtr>
<mml:mtr>
<mml:mtd columnalign="right"/>
<mml:mtd columnalign="left">
<mml:mo>&#x3d;</mml:mo>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x2b;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi>&#x3d5;</mml:mi>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">x</mml:mi>
<mml:mo>,</mml:mo>
<mml:msup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
<mml:mo>,</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mtd>
</mml:mtr>
</mml:mtable>
</mml:math>
</disp-formula>remains the same value when <inline-formula id="inf15">
<mml:math id="m20">
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x2b;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> takes a fixed value, regardless of particular values of <inline-formula id="inf16">
<mml:math id="m21">
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> and <inline-formula id="inf17">
<mml:math id="m22">
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula>. Therefore, we can identify their sum <inline-formula id="inf18">
<mml:math id="m23">
<mml:mi>W</mml:mi>
<mml:mo>&#x3d;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x2b;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula>; nevertheless, each of <inline-formula id="inf19">
<mml:math id="m24">
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> and <inline-formula id="inf20">
<mml:math id="m25">
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> remains unidentifiable. Thus, the training will also suffer difficulties on the subspace <inline-formula id="inf21">
<mml:math id="m26">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="normal">R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mrow>
<mml:mo stretchy="false">{</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x3d;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mo stretchy="false">}</mml:mo>
</mml:mrow>
</mml:math>
</inline-formula>.</p>
<p>To sum up the above analysis, it can be seen that there are at least two types of singularities:<list list-type="simple">
<list-item>
<p>(1) Zero weight singularity: <inline-formula id="inf22">
<mml:math id="m27">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="normal">R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>1</mml:mn>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mrow>
<mml:mo stretchy="false">{</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x3d;</mml:mo>
<mml:mn>0</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">}</mml:mo>
</mml:mrow>
</mml:math>
</inline-formula>,</p>
</list-item>
<list-item>
<p>(2) Overlap singularity: <inline-formula id="inf23">
<mml:math id="m28">
<mml:msub>
<mml:mrow>
<mml:mi mathvariant="normal">R</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
</mml:msub>
<mml:mo>&#x3d;</mml:mo>
<mml:mrow>
<mml:mo stretchy="false">{</mml:mo>
<mml:mrow>
<mml:mi mathvariant="bold-italic">&#x3b8;</mml:mi>
<mml:mo stretchy="false">&#x7c;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>i</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
<mml:mo>&#x3d;</mml:mo>
<mml:msubsup>
<mml:mrow>
<mml:mi mathvariant="bold-italic">W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mi>j</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>2</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:mrow>
<mml:mo stretchy="false">}</mml:mo>
</mml:mrow>
</mml:math>
</inline-formula>.</p>
</list-item>
</list>
</p>
<p>Till now, we have theoretically analyzed the types of singularity that existed in the parameter space of deep MLPs; in the next section, we will numerically analyze the influence of singularities to solve EEG-based emotion recognition problem.</p>
</sec>
</sec>
<sec id="s4">
<title>4 Numerical Analysis of Learning Dynamics Near Singularities</title>
<p>In this section, we take the numerical analysis of singularities by taking experiments on the dataset of EEG signals. For the EEG datasets, the SEED dataset is a typical benchmark dataset that is developed by SJTU and has been widely used to evaluate the proposed methods on EEG-based emotion recognition. In this paper, the training process will be carried out using the SEED dataset.</p>
<sec id="s4-1">
<title>4.1 Data Preprocessing</title>
<p>The SEED dataset (<xref ref-type="bibr" rid="B26">Zheng and Lu, 2015</xref>) is collected from 62-channel EEG device and contains EEG signals of three emotions (positive, neutral, and negative) from 15 subjects. Due to the low signal-to-noise ratio of raw EEG signals, it is rather necessary to take the preprocessing step to extract meaningful features. As is known, there are five frequency bands for each EEG channel: delta (1&#x2013;3&#xa0;Hz), theta (4&#x2013;7&#xa0;Hz), alpha (8&#x2013;13&#xa0;Hz), beta (14&#x2013;30&#xa0;Hz), and gamma (31&#x2013;50&#xa0;Hz). That means, for one subject, the data are the form 5 &#xd7; 62, the dimension of raw EEG signal is very large, and then we use principal component analysis (PCA) (<xref ref-type="bibr" rid="B1">Abdi and Williams, 2010</xref>) to extract the features of the EEG signal. After the PCA step, the form of EEG signals becomes 5 &#xd7; 5, and then by putting every element of the data to a vector, the dimension of the input can be finally reduced to be&#x20;25.</p>
</sec>
<sec id="s4-2">
<title>4.2 Learning Trajectories Near Singularities</title>
<p>Now, we take experiments on the SEED dataset, and the learning dynamics near singularities will be numerically analyzed. We choose the neuron numbers of two hidden layers as <italic>L</italic>
<sub>1</sub> &#x3d; 8 and <italic>L</italic>
<sub>2</sub> &#x3d; 8; thus, the architecture of the deep MLPs is 25&#x2212;8&#x2212;8&#x2212;1. As there are three emotions in the SEED dataset, we set values 1, 2, and 3 corresponding to labels positive, neutral, and negative, respectively. We choose the training sample number and testing sample number to be 1,000 and 500, respectively. Then, by setting the learning rate to <italic>&#x3b7;</italic> &#x3d; 0.002, the target error to 0.05, and the maximum epochs to 8,000, we use <xref ref-type="disp-formula" rid="e3">Eq. 3</xref> to accomplish the experiment. By analyzing the experiment results, two cases of learning dynamics will be shown. Besides training error, classification accuracy is also used to measure the performance. In the following figures of experiment results, &#x201c;&#x25e6;&#x201d; and &#x201c;&#xd7;&#x201d; represent the initial state and final state, respectively. The experiments were run by using Matlab 2013a on a PC with an Intel Core i7-9700K CPU @3.60&#xa0;GHz, 32&#xa0;GB RAM and NVIDIA GeForce RTX 2070&#x20;GPU.</p>
<p>
<statement content-type="case" id="Case_1">
<label>Case 1</label>
<p>Fast convergence: The learning process fast converges to the global minimum.</p>
<p>For this case, the learning dynamics does not suffer from any influence of singularity and the parameters fast converge to the optimal value. The initial value of <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup> is <bold>
<italic>W</italic>
</bold>
<sup>(3)(0)</sup> &#x3d; [0.8874, 0.6993, 0.5367, &#x2212;0.9415, &#x2212;0.8464, &#x2212;0.9280, 0.3335, &#x2212;0.7339]<sup>
<italic>T</italic>
</sup> and the final value of <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup> is <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup> &#x3d; [3.1443, 2.5868, 2.3291, &#x2212;1.1544, &#x2212;1.2281, &#x2212;2.9704, 2.9650, &#x2212;1.8221]<sup>
<italic>T</italic>
</sup>. The experiment results are shown in <xref ref-type="fig" rid="F2">Figure&#x20;2</xref>, which represent the trajectories of training error, output weights <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup>, and classification accuracy, respectively.</p>
<p>As can be seen from <xref ref-type="fig" rid="F2">Figure&#x20;2A</xref>, the learning dynamics quickly converge to the global minimum and have not been affected by any singularity.</p>
</statement>
</p>
<fig id="F2" position="float">
<label>FIGURE 2</label>
<caption>
<p>
<xref ref-type="statement" rid="Case_1">
<bold>Case 1</bold>
</xref> (Fast Convergence). <bold>(A)</bold> Trajectory of training error. <bold>(B)</bold> Trajectory of <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup>. <bold>(C)</bold> Trajectory of classification accuracy.</p>
</caption>
<graphic xlink:href="fcomp-03-786964-g002.tif"/>
</fig>
<p>
<statement content-type="case" id="Case_2">
<label>Case 2</label>
<p>Zero weight singularity: the learning process is affected by the elimination singularity.</p>
<p>For this case, one output weight crosses 0 during the learning process and a plateau phenomenon can be obviously observed. The initial value of <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup> is <bold>
<italic>W</italic>
</bold>
<sup>(3)(0)</sup> &#x3d; [0.4825, 0.9885, &#x2212;0.9522, &#x2212;0.3505, &#x2212;0.5004, 0.9749, &#x2212;0.9111, &#x2212;0.5056]<sup>
<italic>T</italic>
</sup>, and the final student parameters are <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup> &#x3d; [3.0297, 3.1006, &#x2212;1.7413, 0.1717, &#x2212;1.9567, 3.5131, &#x2212;1.9037, &#x2212;0.9143]<sup>
<italic>T</italic>
</sup>. The experiment results are shown in <xref ref-type="fig" rid="F3">Figure&#x20;3</xref>, which represent the trajectories of training error, output weights <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup> and classification accuracy, respectively.</p>
<p>From <xref ref-type="fig" rid="F3">Figure&#x20;3B</xref>, we can see that <inline-formula id="inf24">
<mml:math id="m29">
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>4</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> crosses 0 in the learning process and the learning process is affected by elimination singularity. During the stage <inline-formula id="inf25">
<mml:math id="m30">
<mml:msubsup>
<mml:mrow>
<mml:mi>W</mml:mi>
</mml:mrow>
<mml:mrow>
<mml:mn>4</mml:mn>
</mml:mrow>
<mml:mrow>
<mml:mrow>
<mml:mo stretchy="false">(</mml:mo>
<mml:mrow>
<mml:mn>3</mml:mn>
</mml:mrow>
<mml:mo stretchy="false">)</mml:mo>
</mml:mrow>
</mml:mrow>
</mml:msubsup>
</mml:math>
</inline-formula> crosses 0, the plateau phenomenon can be obviously observed (<xref ref-type="fig" rid="F3">Figure&#x20;3A</xref>). Then, the student parameters escape the influence of elimination singularity. After the training process, we can see that the training error is bigger than that in <xref ref-type="statement" rid="Case_1">Case 1</xref> and the classification accuracy is also lower than that in <xref ref-type="statement" rid="Case_1">Case 1</xref>, which means that the parameters do not reach the optimum.</p>
</statement>
</p>
<fig id="F3" position="float">
<label>FIGURE 3</label>
<caption>
<p>
<bold>
<xref ref-type="statement" rid="Case_2">Case 2</xref>
</bold> (Zero weight singularity). <bold>(A)</bold> Trajectory of training error. <bold>(B)</bold> Trajectory of <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup>. <bold>(C)</bold> Trajectory of classification accuracy.</p>
</caption>
<graphic xlink:href="fcomp-03-786964-g003.tif"/>
</fig>
<p>
<statement content-type="case" id="Case_3">
<label>Case 3</label>
<p>Extending training time of Case&#x20;2.</p>
<p>In this experiment, we only increase the training epochs to 15,000, and the rest of the experiment setup remains the same with that in <xref ref-type="statement" rid="Case_2">Case 2</xref>. The experiment results are shown in <xref ref-type="fig" rid="F4">Figure&#x20;4</xref>. Compared to <xref ref-type="fig" rid="F3">Figure&#x20;3</xref>, it can be seen that the learning process that is affected by the zero weight singularity can arrive at the optimum, but it costs much more time. This means that the zero weight singularities will greatly reduce the efficiency of deep&#x20;MLPs.</p>
</statement>
</p>
<fig id="F4" position="float">
<label>FIGURE 4</label>
<caption>
<p>
<bold>
<xref ref-type="statement" rid="Case_3">Case 3</xref>
</bold> (Extending training time of <xref ref-type="statement" rid="Case_2">
<bold>Case 2</bold>
</xref>). <bold>(A)</bold> Trajectory of training error. <bold>(B)</bold> Trajectory of <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup>. <bold>(C)</bold> Trajectory of classification accuracy.</p>
</caption>
<graphic xlink:href="fcomp-03-786964-g004.tif"/>
</fig>
<p>
<statement content-type="case" id="Case_4">
<label>Case 4</label>
<p>Changing initial value of Case&#x20;2.</p>
<p>In order to confirm that the plateau phenomenon corresponds to the zero weight singularity, a supplementary experiment is carried out here where only the initial value of <bold>
<italic>W</italic>
</bold>
<sup>(3)(0)</sup> has been changed and the rest of the experiment setup remains the same. The initial value of <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup> is <bold>
<italic>W</italic>
</bold>
<sup>(3)(0)</sup> &#x3d; [&#x2212;0.5056, &#x2212;0.9111, 1.7749, &#x2212;0.5004, 1.6495, &#x2212;0.9522, 0.9885, 1.2825]<sup>
<italic>T</italic>
</sup>, and the final student parameters are <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup> &#x3d; [&#x2212;1.3660, &#x2212;1.8232, 3.2529, &#x2212;1.9425, 3.2450, &#x2212;1.6325, 1.9452, 3.4158]<sup>
<italic>T</italic>
</sup>. The experiment results are shown in <xref ref-type="fig" rid="F5">Figure&#x20;5</xref>, which represent the trajectories of training error, output weights <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup>, and classification accuracy, respectively. As can be seen in <xref ref-type="fig" rid="F5">Figure&#x20;5</xref>, there is not any weight of <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup> that becomes zero. Also, no plateau phenomenon can be observed, and the classification accuracy has reached a comparatively high value. By comparing the experiment results shown in <xref ref-type="fig" rid="F3">Figures 3</xref>, <xref ref-type="fig" rid="F5">5</xref>, we can conclude that the plateau phenomenon is indeed caused by zero weight singularity.</p>
</statement>
</p>
<fig id="F5" position="float">
<label>FIGURE 5</label>
<caption>
<p>
<bold>
<xref ref-type="statement" rid="Case_4">Case 4</xref>
</bold> (Changing initial value of <bold>
<xref ref-type="statement" rid="Case_2">Case 2</xref>
</bold>). <bold>(A)</bold> Trajectory of training error. <bold>(B)</bold> Trajectory of <bold>
<italic>W</italic>
</bold>
<sup>(3)</sup>. <bold>(C)</bold> Trajectory of classification accuracy.</p>
</caption>
<graphic xlink:href="fcomp-03-786964-g005.tif"/>
</fig>
<p>
<statement content-type="remark" id="Remark_1">
<label>Remark 1</label>
<p>From the results shown in <xref ref-type="fig" rid="F2">Figures 2</xref>&#x2013;<xref ref-type="fig" rid="F5">5</xref> and <xref ref-type="table" rid="T1">Table&#x20;1</xref>, we can see that the training and testing accuracy in <xref ref-type="statement" rid="Case_2">Case 2</xref> is the lowest. This means that when the training process is affected by the zero weight singularity, the parameters cannot achieve the optimum after the same training time with that in fast convergence case. When we extend the training time in <xref ref-type="statement" rid="Case_2">Case 2</xref>, the parameters can escape the influence of zero weight singularity and finally arrive at the optimum, which is shown in <xref ref-type="statement" rid="Case_3">Case 3</xref>. Thus, the points in zero weight singularity are saddle points, not local minimum. To sum up, the zero weight singularity will seriously delay the training process, and it is worthy to investigate algorithms to overcome the influence of zero weight singularities.</p>
</statement>
</p>
<table-wrap id="T1" position="float">
<label>TABLE 1</label>
<caption>
<p>Training and testing classification accuracy.</p>
</caption>
<table>
<thead valign="top">
<tr>
<th align="left"/>
<th align="left">Iteration number</th>
<th align="center">Training classification accuracy</th>
<th align="center">Testing classification accuracy</th>
</tr>
</thead>
<tbody valign="top">
<tr>
<td align="left">Case 1</td>
<td align="char" char=".">8,000</td>
<td align="char" char=".">0.948</td>
<td align="char" char=".">0.941</td>
</tr>
<tr>
<td align="left">Case 2</td>
<td align="char" char=".">8,000</td>
<td align="char" char=".">0.901</td>
<td align="char" char=".">0.894</td>
</tr>
<tr>
<td align="left">Case 3</td>
<td align="char" char=".">15,000</td>
<td align="char" char=".">0.944</td>
<td align="char" char=".">0.938</td>
</tr>
<tr>
<td align="left">Case 4</td>
<td align="char" char=".">8,000</td>
<td align="char" char=".">0.924</td>
<td align="char" char=".">0.920</td>
</tr>
</tbody>
</table>
</table-wrap>
<p>
<statement content-type="remark" id="Remark_2">
<label>Remark 2</label>
<p>When taking the experiments, we do not observe the learning dynamics of deep MLPs that are affected by overlap singularities. The results are in accordance with the conclusion where we analyze the learning dynamics of shallow neural networks (<xref ref-type="bibr" rid="B10">Guo et&#x20;al., 2018</xref>); i.e.,&#x20;the overlap singularities mainly influence the neural networks with low dimension and the large-scale networks predominantly suffer from zero weight singularities. Thus, we should pay more attention to how to overcome the influence of zero weight singularities.</p>
<p>In this section, we have numerically analyzed the learning dynamics near singularities of deep MLPs for EEG-based emotion recognition and showed the singular case. We can obtain that the learning dynamics of deep MLPs are mainly influenced by zero weight singularities and rarely affected by overlap singularities.</p>
</statement>
</p>
</sec>
</sec>
<sec id="s5">
<title>5 Conclusion and Discussion</title>
<p>Deep learning technology has been widely used in EEG-based emotion recognition and has shown superior performance compared to traditional methods. However, for various DNNs, there exist singularities in the parameter space, which cause singular behaviors in the training process. In this paper, we investigate the singular learning dynamics of DNNs when applied to EEG-based emotion recognition. By choosing deep MLPs as the learning machine, we firstly take the theoretical analysis of singularities of deep MLPs, and obtained that there are at least two types of singularities: overlap singularity and zero weight singularity. Then, by doing several experiments, the numerical analysis is taken. The experiment results show that the learning dynamics of deep MLPs are seriously influenced by zero weight singularities and rarely affected by overlap singularities. Furthermore, the plateau phenomenon is caused by zero weight singularity. Thus, we should pay more attention to how to overcome the serious influence of zero weight singularity to improve the efficiency of DNNs in EEG-based emotion recognition in the future.</p>
</sec>
</body>
<back>
<sec id="s6">
<title>Data Availability Statement</title>
<p>The original contributions presented in the study are included in the article/Supplementary Material, Further inquiries can be directed to the corresponding authors.</p>
</sec>
<sec id="s7">
<title>Author Contributions</title>
<p>WG and GL: Methodology. WG and JY: Validation and investigation. WG: Writing&#x2014;original draft preparation. GL, JL, and JY: Formal analysis, data curation. JL and JY: Writing&#x2014;reviewing and editing, and supervision. All authors have read and agreed to the published version of the manuscript.</p>
</sec>
<sec id="s8">
<title>Funding</title>
<p>This work was supported by the National Natural Science Foundation of China under Grant Nos. 61906092, 61802059, 62006119, and 61876085, the Natural Science Foundation of Jiangsu Province of China under Grant Nos. BK20190441, BK20180365, and BK20190444, and the 973 Program No. 2014CB349303.</p>
</sec>
<sec sec-type="COI-statement" id="s9">
<title>Conflict of Interest</title>
<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.</p>
</sec>
<sec sec-type="disclaimer" id="s10">
<title>Publisher&#x2019;s Note</title>
<p>All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.</p>
</sec>
<ref-list>
<title>References</title>
<ref id="B1">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Abdi</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Williams</surname>
<given-names>L. J.</given-names>
</name>
</person-group> (<year>2010</year>). <article-title>Principal Component Analysis</article-title>. <source>Wires Comp. Stat.</source> <volume>2</volume>, <fpage>433</fpage>&#x2013;<lpage>459</lpage>. <pub-id pub-id-type="doi">10.1002/wics.101</pub-id> </citation>
</ref>
<ref id="B2">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Ainsworth</surname>
<given-names>M.</given-names>
</name>
<name>
<surname>Shin</surname>
<given-names>Y.</given-names>
</name>
</person-group> (<year>2020</year>). <source>Plateau Phenomenon in Gradient Descent Training of Relu Networks: Explanation, Quantification and Avoidance</source>. </citation>
</ref>
<ref id="B3">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Amari</surname>
<given-names>S.-i.</given-names>
</name>
<name>
<surname>Park</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Ozeki</surname>
<given-names>T.</given-names>
</name>
</person-group> (<year>2006</year>). <article-title>Singularities Affect Dynamics of Learning in Neuromanifolds</article-title>. <source>Neural Comput.</source> <volume>18</volume>, <fpage>1007</fpage>&#x2013;<lpage>1065</lpage>. <pub-id pub-id-type="doi">10.1162/neco.2006.18.5.1007</pub-id> </citation>
</ref>
<ref id="B4">
<citation citation-type="book">
<person-group person-group-type="author">
<name>
<surname>Atmaja</surname>
<given-names>B. T.</given-names>
</name>
<name>
<surname>Akagi</surname>
<given-names>M.</given-names>
</name>
</person-group> (<year>2020</year>). &#x201c;<article-title>Deep Multilayer Perceptrons for Dimensional Speech Emotion Recognition</article-title>,&#x201d; in <conf-name>Proceedings of Asia-Pacific Signal and Information Processing Association Annual Summit and Conference</conf-name>, <conf-loc>Auckland, New Zealand</conf-loc>, <conf-date>December 7&#x2013;10, 2020</conf-date> (<publisher-loc>Auckland</publisher-loc>: <publisher-name>IEEE</publisher-name>), <fpage>325</fpage>&#x2013;<lpage>331</lpage>. </citation>
</ref>
<ref id="B5">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Basodi</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Ji</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Zhang</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Pan</surname>
<given-names>Y.</given-names>
</name>
</person-group> (<year>2020</year>). <article-title>Gradient Amplification: An Efficient Way to Train Deep Neural Networks</article-title>. <source>Big Data Min. Anal.</source> <volume>3</volume>, <fpage>196</fpage>&#x2013;<lpage>207</lpage>. <pub-id pub-id-type="doi">10.26599/bdma.2020.9020004</pub-id> </citation>
</ref>
<ref id="B6">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Cao</surname>
<given-names>Q.</given-names>
</name>
<name>
<surname>Zhang</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Zhu</surname>
<given-names>Y.</given-names>
</name>
</person-group> (<year>2020</year>). <article-title>Deep Learning-Based Classification of the Polar Emotions of &#x201d;moe&#x201d;-Style Cartoon Pictures</article-title>. <source>Tsinghua Sci. Technology</source> <volume>26</volume>, <fpage>275</fpage>&#x2013;<lpage>286</lpage>. </citation>
</ref>
<ref id="B7">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Cui</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Liu</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Zhang</surname>
<given-names>X.</given-names>
</name>
<name>
<surname>Chen</surname>
<given-names>X.</given-names>
</name>
<name>
<surname>Wang</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Chen</surname>
<given-names>X.</given-names>
</name>
</person-group> (<year>2020</year>). <article-title>Eeg-based Emotion Recognition Using an End-To-End Regional-Asymmetric Convolutional Neural Network</article-title>. <source>Knowledge-Based Syst.</source> <volume>205</volume>, <fpage>106243</fpage>. <pub-id pub-id-type="doi">10.1016/j.knosys.2020.106243</pub-id> </citation>
</ref>
<ref id="B8">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Fang</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Yang</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Zhang</surname>
<given-names>X.</given-names>
</name>
<name>
<surname>Liu</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Tao</surname>
<given-names>B.</given-names>
</name>
</person-group> (<year>2020</year>). <article-title>Multi-feature Input Deep forest for Eeg-Based Emotion Recognition</article-title>. <source>Front. Neurorobotics</source> <volume>14</volume>, <fpage>617531</fpage>. </citation>
</ref>
<ref id="B9">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Guo</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Ong</surname>
<given-names>Y.-S.</given-names>
</name>
<name>
<surname>Zhou</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Hervas</surname>
<given-names>J.&#x20;R.</given-names>
</name>
<name>
<surname>Song</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Wei</surname>
<given-names>H.</given-names>
</name>
</person-group> (<year>2019</year>). <article-title>Fisher Information Matrix of Unipolar Activation Function-Based Multilayer Perceptrons</article-title>. <source>IEEE Trans. Cybern.</source> <volume>49</volume>, <fpage>3088</fpage>&#x2013;<lpage>3098</lpage>. <pub-id pub-id-type="doi">10.1109/tcyb.2018.2838680</pub-id> </citation>
</ref>
<ref id="B10">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Guo</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Wei</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Ong</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Hervas</surname>
<given-names>J.&#x20;R.</given-names>
</name>
<name>
<surname>Zhao</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Wang</surname>
<given-names>H.</given-names>
</name>
<etal/>
</person-group> (<year>2018</year>). <article-title>Numerical Analysis Near Singularities in RBF Networks</article-title>. <source>J.&#x20;Mach. Learn. Res.</source> <volume>19</volume>, <fpage>1</fpage>&#x2013;<lpage>39</lpage>. </citation>
</ref>
<ref id="B11">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Hassan</surname>
<given-names>M. M.</given-names>
</name>
<name>
<surname>Alam</surname>
<given-names>M. G. R.</given-names>
</name>
<name>
<surname>Uddin</surname>
<given-names>M. Z.</given-names>
</name>
<name>
<surname>Huda</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Almogren</surname>
<given-names>A.</given-names>
</name>
<name>
<surname>Fortino</surname>
<given-names>G.</given-names>
</name>
</person-group> (<year>2019</year>). <article-title>Human Emotion Recognition Using Deep Belief Network Architecture</article-title>. <source>Inf. Fusion</source> <volume>51</volume>, <fpage>10</fpage>&#x2013;<lpage>18</lpage>. <pub-id pub-id-type="doi">10.1016/j.inffus.2018.10.009</pub-id> </citation>
</ref>
<ref id="B12">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Li</surname>
<given-names>J.-L.</given-names>
</name>
<name>
<surname>Huang</surname>
<given-names>T.-Y.</given-names>
</name>
<name>
<surname>Chang</surname>
<given-names>C.-M.</given-names>
</name>
<name>
<surname>Lee</surname>
<given-names>C.-C.</given-names>
</name>
</person-group> (<year>2020</year>). <article-title>A Waveform-Feature Dual branch Acoustic Embedding Network for Emotion Recognition</article-title>. <source>Front. Comput. Sci.</source> <volume>2</volume>, <fpage>13</fpage>. <pub-id pub-id-type="doi">10.3389/fcomp.2020.00013</pub-id> </citation>
</ref>
<ref id="B13">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Li</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Zhang</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>He</surname>
<given-names>H.</given-names>
</name>
</person-group> (<year>2018</year>). <article-title>Hierarchical Convolutional Neural Networks for Eeg-Based Emotion Recognition</article-title>. <source>Cogn. Comput.</source> <volume>10</volume>, <fpage>368</fpage>&#x2013;<lpage>380</lpage>. <pub-id pub-id-type="doi">10.1007/s12559-017-9533-x</pub-id> </citation>
</ref>
<ref id="B14">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Li</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Liu</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Si</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Li</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Li</surname>
<given-names>F.</given-names>
</name>
<name>
<surname>Zhu</surname>
<given-names>X.</given-names>
</name>
<etal/>
</person-group> (<year>2019</year>). <article-title>EEG Based Emotion Recognition by Combining Functional Connectivity Network and Local Activations</article-title>. <source>IEEE Trans. Biomed. Eng.</source> <volume>66</volume>, <fpage>2869</fpage>&#x2013;<lpage>2881</lpage>. <pub-id pub-id-type="doi">10.1109/tbme.2019.2897651</pub-id> </citation>
</ref>
<ref id="B15">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Liao</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>Drummond</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Reid</surname>
<given-names>I.</given-names>
</name>
<name>
<surname>Carneiro</surname>
<given-names>G.</given-names>
</name>
</person-group> (<year>2020</year>). <article-title>Approximate Fisher Information Matrix to Characterize the Training of Deep Neural Networks</article-title>. <source>IEEE Trans. Pattern Anal. Mach. Intell.</source> <volume>42</volume>, <fpage>15</fpage>&#x2013;<lpage>26</lpage>. <pub-id pub-id-type="doi">10.1109/tpami.2018.2876413</pub-id> </citation>
</ref>
<ref id="B16">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Ma</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Tang</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Zheng</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Lu</surname>
<given-names>B.</given-names>
</name>
</person-group> (<year>2019</year>). &#x201c;<article-title>Emotion Recognition Using Multimodal Residual LSTM Network</article-title>,&#x201d; in <conf-name>Proceedings of the 27th ACM International Conference on Multimedia</conf-name>, <conf-loc>Nice, France</conf-loc>, <conf-date>October 21&#x2013;25, 2019</conf-date>, <fpage>176</fpage>&#x2013;<lpage>183</lpage>. <pub-id pub-id-type="doi">10.1145/3343031.3350871</pub-id> </citation>
</ref>
<ref id="B17">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Mauss</surname>
<given-names>I. B.</given-names>
</name>
<name>
<surname>Robinson</surname>
<given-names>M. D.</given-names>
</name>
</person-group> (<year>2009</year>). <article-title>Measures of Emotion: A Review</article-title>. <source>Cogn. Emot.</source> <volume>23</volume>, <fpage>209</fpage>&#x2013;<lpage>237</lpage>. <pub-id pub-id-type="doi">10.1080/02699930802204677</pub-id> </citation>
</ref>
<ref id="B18">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Natarajan</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Srihari</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Chandragandhi</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Raja</surname>
<given-names>R. A.</given-names>
</name>
<name>
<surname>Dhiman</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Kaur</surname>
<given-names>A.</given-names>
</name>
</person-group> (<year>2021</year>). <article-title>Analysis of Protein-Ligand Interactions of Sars-Cov-2 against Selective Drug Using Deep Neural Networks</article-title>. <source>Big Data Min. Anal.</source> <volume>4</volume>, <fpage>76</fpage>&#x2013;<lpage>83</lpage>. </citation>
</ref>
<ref id="B19">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Nawaz</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Cheah</surname>
<given-names>K. H.</given-names>
</name>
<name>
<surname>Nisar</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Yap</surname>
<given-names>V. V.</given-names>
</name>
</person-group> (<year>2020</year>). <article-title>Comparison of Different Feature Extraction Methods for Eeg-Based Emotion Recognition</article-title>. <source>Biocybernetics Biomed. Eng.</source> <volume>40</volume>, <fpage>910</fpage>&#x2013;<lpage>926</lpage>. <pub-id pub-id-type="doi">10.1016/j.bbe.2020.04.005</pub-id> </citation>
</ref>
<ref id="B20">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Ng</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Nguyen</surname>
<given-names>V. D.</given-names>
</name>
<name>
<surname>Vonikakis</surname>
<given-names>V.</given-names>
</name>
<name>
<surname>Winkler</surname>
<given-names>S.</given-names>
</name>
</person-group> (<year>2015</year>). &#x201c;<article-title>Deep Learning for Emotion Recognition on Small Datasets Using Transfer Learning</article-title>,&#x201d; in <conf-name>Proceedings of the 2015 ACM on International Conference on Multimodal Interaction</conf-name>, <conf-loc>Seattle</conf-loc>, <conf-date>November 09&#x2013;13, 2015</conf-date>. Editors <person-group person-group-type="editor">
<name>
<surname>Zhang</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>Cohen</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Bohus</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Horaud</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Meng</surname>
<given-names>H.</given-names>
</name>
</person-group>, <fpage>443</fpage>&#x2013;<lpage>449</lpage>. <pub-id pub-id-type="doi">10.1145/2818346.2830593</pub-id> </citation>
</ref>
<ref id="B21">
<citation citation-type="confproc">
<person-group person-group-type="author">
<name>
<surname>Nitta</surname>
<given-names>T.</given-names>
</name>
</person-group> (<year>2016</year>). &#x201c;<article-title>On the Singularity in Deep Neural Networks</article-title>,&#x201d; in <conf-name>Proceedings of 23rd International Conference on Neural Information Processing</conf-name>, <conf-loc>Kyoto, Japan</conf-loc>, <conf-date>October 16&#x2013;21, 2016</conf-date> (<publisher-loc>Kyoto</publisher-loc>: <publisher-name>Lecture Notes in Computer Science</publisher-name>), <fpage>389</fpage>&#x2013;<lpage>396</lpage>. <pub-id pub-id-type="doi">10.1007/978-3-319-46681-1_47</pub-id> </citation>
</ref>
<ref id="B22">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Nitta</surname>
<given-names>T.</given-names>
</name>
</person-group> (<year>2018</year>). <article-title>Resolution of Singularities via Deep Complex-Valued Neural Networks</article-title>. <source>Math. Meth Appl. Sci.</source> <volume>41</volume>, <fpage>4170</fpage>&#x2013;<lpage>4178</lpage>. <pub-id pub-id-type="doi">10.1002/mma.4434</pub-id> </citation>
</ref>
<ref id="B23">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Song</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Liu</surname>
<given-names>S.</given-names>
</name>
<name>
<surname>Zheng</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Zong</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Cui</surname>
<given-names>Z.</given-names>
</name>
</person-group> (<year>2020</year>). &#x201c;<article-title>Instance-adaptive Graph for EEG Emotion Recognition</article-title>,&#x201d; in <conf-name>Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence</conf-name>, <conf-loc>New York</conf-loc>, <conf-date>February 7&#x2013;12, 2020</conf-date>, <fpage>2701</fpage>&#x2013;<lpage>2708</lpage>. <pub-id pub-id-type="doi">10.1609/aaai.v34i03.5656</pub-id>
<source>Aaai</source>
<volume>34</volume> </citation>
</ref>
<ref id="B24">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Tao</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Li</surname>
<given-names>C.</given-names>
</name>
<name>
<surname>Song</surname>
<given-names>R.</given-names>
</name>
<name>
<surname>Cheng</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Liu</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Wan</surname>
<given-names>F.</given-names>
</name>
<etal/>
</person-group> (<year>2020</year>). <article-title>Eeg-based Emotion Recognition via Channel-wise Attention and Self Attention</article-title>. <source>IEEE Trans. Affective Comput.</source>, <fpage>1</fpage>. <pub-id pub-id-type="doi">10.1109/TAFFC.2020.3025777</pub-id> </citation>
</ref>
<ref id="B25">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Tzirakis</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Trigeorgis</surname>
<given-names>G.</given-names>
</name>
<name>
<surname>Nicolaou</surname>
<given-names>M. A.</given-names>
</name>
<name>
<surname>Schuller</surname>
<given-names>B. W.</given-names>
</name>
<name>
<surname>Zafeiriou</surname>
<given-names>S.</given-names>
</name>
</person-group> (<year>2017</year>). <article-title>End-to-end Multimodal Emotion Recognition Using Deep Neural Networks</article-title>. <source>IEEE J.&#x20;Sel. Top. Signal. Process.</source> <volume>11</volume>, <fpage>1301</fpage>&#x2013;<lpage>1309</lpage>. <pub-id pub-id-type="doi">10.1109/jstsp.2017.2764438</pub-id> </citation>
</ref>
<ref id="B26">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Wei-Long Zheng</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Bao-Liang Lu</surname>
<given-names>B.</given-names>
</name>
</person-group> (<year>2015</year>). <article-title>Investigating Critical Frequency Bands and Channels for Eeg-Based Emotion Recognition with Deep Neural Networks</article-title>. <source>IEEE Trans. Auton. Ment. Dev.</source> <volume>7</volume>, <fpage>162</fpage>&#x2013;<lpage>175</lpage>. <pub-id pub-id-type="doi">10.1109/tamd.2015.2431497</pub-id> </citation>
</ref>
<ref id="B27">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Yang</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Fu</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>Zhan</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Liu</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>Jiang</surname>
<given-names>Y.</given-names>
</name>
</person-group> (<year>2021a</year>). <article-title>Semi-supervised Multi-Modal Multi-Instance Multi-Label Deep Network with Optimal Transport</article-title>. <source>IEEE Trans. Knowl. Data Eng.</source> <volume>33</volume>, <fpage>696</fpage>&#x2013;<lpage>709</lpage>. </citation>
</ref>
<ref id="B28">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Yang</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Wu</surname>
<given-names>Q. M. J.</given-names>
</name>
<name>
<surname>Zheng</surname>
<given-names>W.-L.</given-names>
</name>
<name>
<surname>Lu</surname>
<given-names>B.-L.</given-names>
</name>
</person-group> (<year>2018a</year>). <article-title>Eeg-based Emotion Recognition Using Hierarchical Network with Subnetwork Nodes</article-title>. <source>IEEE Trans. Cogn. Dev. Syst.</source> <volume>10</volume>, <fpage>408</fpage>&#x2013;<lpage>419</lpage>. <pub-id pub-id-type="doi">10.1109/tcds.2017.2685338</pub-id> </citation>
</ref>
<ref id="B29">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Yang</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Wu</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Zhan</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Liu</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>Jiang</surname>
<given-names>Y.</given-names>
</name>
</person-group> (<year>2018b</year>). &#x201c;<article-title>Complex Object Classification: A Multi-Modal Multi-Instance Multi-Label Deep Network with Optimal Transport</article-title>,&#x201d; in <conf-name>Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &#x26; Data Mining</conf-name>, <conf-loc>London, UK</conf-loc>, <conf-date>August 19&#x2013;23, 2018</conf-date>, <fpage>2594</fpage>&#x2013;<lpage>2603</lpage>. </citation>
</ref>
<ref id="B30">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Yang</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Zhan</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Wu</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Liu</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>Xiong</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Jiang</surname>
<given-names>Y.</given-names>
</name>
</person-group> (<year>2021b</year>). <article-title>Semi-supervised Multi-Modal Clustering and Classification with Incomplete Modalities</article-title>. <source>IEEE Trans. Knowl. Data Eng.</source> <volume>33</volume>, <fpage>682</fpage>&#x2013;<lpage>695</lpage>. </citation>
</ref>
<ref id="B31">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Yang</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Zhou</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Zhan</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Xiong</surname>
<given-names>H.</given-names>
</name>
<name>
<surname>Jiang</surname>
<given-names>Y.</given-names>
</name>
</person-group> (<year>2019</year>). &#x201c;<article-title>Adaptive Deep Models for Incremental Learning: Considering Capacity Scalability and Sustainability</article-title>,&#x201d; in <conf-name>Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &#x26; Data Mining</conf-name>, <conf-loc>Anchorage, AK</conf-loc>, <conf-date>August 4&#x2013;8, 2019</conf-date>, <fpage>74</fpage>&#x2013;<lpage>82</lpage>. </citation>
</ref>
<ref id="B32">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Yin</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Zheng</surname>
<given-names>X.</given-names>
</name>
<name>
<surname>Hu</surname>
<given-names>B.</given-names>
</name>
<name>
<surname>Zhang</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Cui</surname>
<given-names>X.</given-names>
</name>
</person-group> (<year>2021</year>). <article-title>EEG Emotion Recognition Using Fusion Model of Graph Convolutional Neural Networks and LSTM</article-title>. <source>Appl. Soft Comput.</source> <volume>100</volume>, <fpage>106954</fpage>. <pub-id pub-id-type="doi">10.1016/j.asoc.2020.106954</pub-id> </citation>
</ref>
<ref id="B33">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Zhang</surname>
<given-names>T.</given-names>
</name>
<name>
<surname>Zheng</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Cui</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>Zong</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Li</surname>
<given-names>Y.</given-names>
</name>
</person-group> (<year>2019</year>). <article-title>Spatial-temporal Recurrent Neural Network for Emotion Recognition</article-title>. <source>IEEE Trans. Cybern.</source> <volume>49</volume>, <fpage>839</fpage>&#x2013;<lpage>847</lpage>. <pub-id pub-id-type="doi">10.1109/tcyb.2017.2788081</pub-id> </citation>
</ref>
<ref id="B34">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Zhao</surname>
<given-names>L.</given-names>
</name>
<name>
<surname>Yan</surname>
<given-names>X.</given-names>
</name>
<name>
<surname>Lu</surname>
<given-names>B.</given-names>
</name>
</person-group> (<year>2021</year>). &#x201c;<article-title>Plug-and-play Domain Adaptation for Cross-Subject Eeg-Based Emotion Recognition</article-title>,&#x201d; in <conf-name>Proceedings of thirty-Fifth AAAI Conference on Artificial Intelligence</conf-name>, <conf-loc>Virtual Event</conf-loc>, <conf-date>February 2&#x2013;9, 2021</conf-date> <fpage>863</fpage>&#x2013;<lpage>870</lpage>. </citation>
</ref>
<ref id="B35">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Zheng</surname>
<given-names>W.</given-names>
</name>
</person-group> (<year>2017</year>). <article-title>Multichannel Eeg-Based Emotion Recognition via Group Sparse Canonical Correlation Analysis</article-title>. <source>IEEE Trans. Cogn. Dev. Syst.</source> <volume>9</volume>, <fpage>281</fpage>&#x2013;<lpage>290</lpage>. <pub-id pub-id-type="doi">10.1109/tcds.2016.2587290</pub-id> </citation>
</ref>
<ref id="B36">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Zheng</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Zhu</surname>
<given-names>J.</given-names>
</name>
<name>
<surname>Peng</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Lu</surname>
<given-names>B.</given-names>
</name>
</person-group> (<year>2014</year>). <article-title>Eeg-based Emotion Classification Using Deep Belief Networks</article-title>. <source>IEEE Int. Conf. Multimedia Expo</source>, <fpage>1</fpage>&#x2013;<lpage>6</lpage>. <pub-id pub-id-type="doi">10.1109/icme.2014.6890166</pub-id> </citation>
</ref>
<ref id="B37">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Zhong</surname>
<given-names>P.</given-names>
</name>
<name>
<surname>Wang</surname>
<given-names>D.</given-names>
</name>
<name>
<surname>Miao</surname>
<given-names>C.</given-names>
</name>
</person-group> (<year>2020</year>). <article-title>Eeg-based Emotion Recognition Using Regularized Graph Neural Networks</article-title>. <source>IEEE Trans. Affective Comput.</source>, <fpage>1</fpage>. <pub-id pub-id-type="doi">10.1109/TAFFC.2020.2994159</pub-id> </citation>
</ref>
<ref id="B38">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Zhu</surname>
<given-names>K.</given-names>
</name>
<name>
<surname>Zhang</surname>
<given-names>T.</given-names>
</name>
</person-group> (<year>2021</year>). <article-title>Deep Reinforcement Learning Based mobile Robot Navigation: A Review</article-title>. <source>Tsinghua Sci. Technol.</source> <volume>26</volume>, <fpage>674</fpage>&#x2013;<lpage>691</lpage>. <pub-id pub-id-type="doi">10.26599/tst.2021.9010012</pub-id> </citation>
</ref>
<ref id="B39">
<citation citation-type="journal">
<person-group person-group-type="author">
<name>
<surname>Zong</surname>
<given-names>Y.</given-names>
</name>
<name>
<surname>Zheng</surname>
<given-names>W.</given-names>
</name>
<name>
<surname>Cui</surname>
<given-names>Z.</given-names>
</name>
<name>
<surname>Li</surname>
<given-names>Q.</given-names>
</name>
</person-group> (<year>2016</year>). <article-title>Double Sparse Learning Model for Speech Emotion Recognition</article-title>. <source>Electron. Lett.</source> <volume>52</volume>, <fpage>1410</fpage>&#x2013;<lpage>1412</lpage>. <pub-id pub-id-type="doi">10.1049/el.2016.1211</pub-id> </citation>
</ref>
</ref-list>
</back>
</article>
