Reducing semiconductor optical Amplifier’s non-linearity through probabilistic amplitude shaping of optical QAM signal

Optical communication is an efficient technology for high-speed, long-distance data transmission. Semiconductor optical amplifiers (SOAs) are particularly promising for optical signal amplification in O-band transmission due to their compact size, low power consumption, and ease of integration into photonic networks. However, SOA-based systems suffer from nonlinear impairments, which degrade signal quality, especially at higher amplification levels. These nonlinear effects arise from changes in the intrinsic properties of the SOA’s waveguide material, such as refractive index variations, when subjected to high optical field intensities. In this paper, we investigate the potential of probabilistic amplitude shaping (PAS) as a signal-shaping technique to mitigate SOA-induced nonlinearities. PAS leverages a non-uniform probability distribution of constellation points (e.g., QAM symbols) to reduce the average transmit power while maintaining the same information rate. Our simulation results demonstrate that applying PAS to standard QAM signals significantly improves received signal quality, as measured by bit error rate (BER), error vector magnitude (EVM), and mutual information (MI), compared to conventional uniform QAM signaling. Furthermore, forward error correction (FEC) is employed to further enhance the system performance.

1 Introduction

Optical communication is a highly efficient alternative for high-rate, long-distance information transmission. With internet traffic expected to rise significantly over the next decade, the importance of optical communication will continue to grow. To meet the increasing demand for high data throughput, more effective and efficient optical transmission techniques are required. However, high-speed data transmission introduces several technical challenges in communication systems [1]. Despite its advantages, such as high bandwidth capacity and low attenuation, optical communication systems experience impairments similar to conventional electrical-domain communication. These impairments are primarily nonlinear effects, collectively known as the Kerr effect, which degrade system performance. Nonlinear effects occur when propagating light interacts with various components of an optical communication system, such as optical fibers, amplifiers, and modulators. In such cases, the system’s response becomes nonlinear, meaning it is no longer proportional to the input signal power [2]. Among these nonlinearities, the most significant contributions come from the optical fiber channel and optical amplifiers, which boost the signal to higher power levels. While nonlinear effects are negligible at low optical power levels, they become significant once the optical signal surpasses a certain threshold, typically when amplified by optical amplifiers.

Optical signals in the O-band (1260–1360 nm) are well-suited for long-distance transmission, particularly in metro and regional networks operating at high baud rates. However, fiber attenuation in the O-band is relatively higher than in the C-band (1530–1565 nm) [3]. Among the most commonly used optical amplifiers are Erbium-doped fiber amplifiers (EDFAs) and semiconductor optical amplifiers (SOAs). EDFAs provide high gain for C-band signals, making them ideal for long-haul telecommunications. However, they are unsuitable for O-band transmission, as Erbium ions do not offer gain in this wavelength range. Consequently, O-band transmission distances reported in previous research have been relatively short [4–6]. In contrast, SOAs can be designed for a broad range of wavelengths supported by semiconductor lasers while offering wide gain bandwidth [7]. Compared to EDFAs, SOAs have several advantages, including a simple structure, compact size, low power consumption, ease of integration into photonic networks, and cost-effectiveness for metro optical network applications [8, 9]. As a result, SOAs are considered a promising candidate for O-band signal amplification in optical communication systems.

Despite their advantages, SOAs introduce self-induced nonlinear impairments, degrading signal quality. These effects, commonly referred to as Kerr effects, stem from variations in the SOA’s refractive index due to high electric fields generated by ultra-short optical pulses. As a result, the refractive index fluctuates with signal intensity [10, 11]. SOA-induced nonlinearities distort the signal constellation of advanced modulation formats, making compensation particularly challenging for M-ary quadrature amplitude modulation (m-QAM). Key nonlinear impairments include self-gain modulation, self-phase modulation, cross-gain modulation, cross-phase modulation, and four-wave mixing [2].

For advanced modulation formats and coherent receivers, digital back-propagation has been proposed to compensate for fiber-induced nonlinearities by numerically solving the nonlinear Schrödinger equation via the split-step Fourier method [12]. Similarly [13, 14], extend digital back-propagation to mitigate SOA-induced impairments by applying it as a numerical inverse SOA. In intensity modulation/direct detection (IM/DD) systems, digital post-compensation at the receiver minimizes SOA gain saturation effects, which cause pattern-dependent distortions, particularly in PAM signals, degrading BER performance. While nonlinearity compensation via digital signal processing can alleviate these effects, it increases receiver complexity [3, 15, 16]. SOA-based amplification also introduces amplitude distortions due to bit-pattern effects, arising from the SOA’s slow gain recovery time relative to symbol duration. This can be mitigated by ensuring symbol rates remain below the inverse of the gain recovery time [17]. Additionally, chromatic dispersion, coupled with SOA-induced nonlinearities, further complicates field reconstruction [13]. In high-speed data transmission, the symbol period is much shorter than the SOA carrier lifetime, leading to insufficient gain recovery. To mitigate these nonlinear effects, techniques such as optical filtering and gain clamping can be applied [18, 19]. Recently, digital signal processing methods, including decision feedback equalizers [20], Volterra nonlinear equalizers [21], and machine learning approaches such as artificial neural networks (ANNs) [22, 23], convolutional neural networks (CNNs) [24], and recurrent neural networks (RNNs) [25], have proven effective in compensating SOA-induced signal distortions. Optical phase conjugation further reduces nonlinear phase distortion and suppresses bit-pattern effects [26]. A training-based technique in [27] demonstrated a 2.52 dB Q-factor improvement through adaptive SOA nonlinearity compensation. Additionally, multi-wavelength gain clamping and polarization-division-multiplexed (PDM) self-homodyne coherent detection have shown effectiveness in suppressing SOA-induced amplitude and phase distortions [28].

This paper focuses on probabilistic shaping (PS), a scheme that transmits symbols of a QAM signal with a non-uniform probability distribution at the channel input [29]. It investigates how PS can help mitigate SOA-induced nonlinear impairments that distort amplified signals. While various techniques exist for compensating these effects, PS is preferred for its low complexity. A key advancement in PS is probabilistic amplitude shaping (PAS) [30], which integrates a shaping outer code, known as a distribution matcher (DM), with an inner forward error correction (FEC) code [31]. PAS has proven highly effective in coherent optical transmission systems, with its benefits demonstrated in multiple optical transmission experiments [32]. To the best of our knowledge, PAS has not yet been explored for mitigating SOA-induced nonlinear effects.

Recent advances in O-band optical amplification have seen notable progress, for example, praseodymium-doped fiber amplifiers (PDFAs) for the 1270 nm–1350 nm range achieved small-signal gains above 50 dB under optimized bidirectional pumping [33]. Bismuth-doped fiber amplifiers (BDFAs) have also extended amplification into the O-band to U-band regions, with reviews summarizing their evolving gain and noise figure performance [34]. Specific implementations of cladding-pumped BDFAs in the 1.4 to 1.5 $\mu m$ region further illustrate the maturation of these technologies [35]. Moreover, investigations into doping profiles for Bi-doped $Ge{O}_{2}Si{O}_2$ glass fibers demonstrate the material-engineering efforts behind O-band amplification platforms [36]. Despite these promising developments, many fibre-based O-band amplifiers remain challenged by integration complexity, pump architecture bulkiness, and limited scalability in compact photonic circuits. In contrast, SOAs offer a more favourable combination of compactness, low power consumption, and ease of integration into photonic networks, making them particularly attractive for O-band deployment. In this work, we therefore focus on the application of PAS in SOA based O-band transmission systems to mitigate nonlinearities and improve performance.

The remainder of this paper is organized as follows. Section II introduces probabilistic constellation shaping, with a focus on PAS. It examines both conventional uniform m-QAM and probabilistically shaped QAM (PS-mQAM) in a wavelength division multiplexing (WDM) based optical communication system, where m represents the QAM modulation order. Their performance is evaluated based on bit error rate (BER), error vector magnitude (EVM), and mutual information (MI). Section III integrates SOAs into both systems, amplifying signals to the SOA saturation level. Results demonstrate that PS-mQAM is less affected by SOA-induced nonlinearities, outperforming standard m-QAM. Section IV applies FEC alongside probabilistic shaping to further enhance signal quality. The performance of all models is evaluated based on BER, EVM, and MI against optical signal-to-noise ratio (OSNR). Section V concludes the paper.

2 Probabilistic constellation shaping

Increasing spectral efficiency is crucial in optical fiber systems. Due to the nonlinear and noise characteristics of the fiber, signals with a uniform input distribution are not optimal in terms of information rate. Therefore, signal shaping and especially probabilistic shaping, is used to adapt the input distribution to better match the capacity-achieving distribution of the channel. There are two main types: geometric shaping, which uses non-uniformly spaced constellations with equiprobable symbols, and probabilistic shaping, where symbols remain on a uniform grid but have varying probabilities. Both techniques offer an signal to noise ratio (SNR) gain of up to 1.53 dB in an additive white Gaussian noise (AWGN) channel [37]. PAS integrates a DM with FEC [31], as illustrated in Figure 1. PAS is unique due to three key properties [1]: seamless integration with existing FEC [2], operation near the Shannon limit, and [3] rate adaptation by adjusting probability distributions without modifying FEC [29]. These advantages make PAS highly effective in coherent optical transmission systems.

Figure 1

Figure 1. The layered PS architecture. Adapted from [53], licensed under CC BY.

Recently, probabilistic constellation shaping (PCS) has gained significant research interest, as it optimizes the probability of constellation points rather than their positions to approximate Gaussian signaling [38], as shown in Figure 2. PCS transmits non-uniform data symbols using modulation schemes like QAM, where constellation points are selected based on symbol probability [39]. This approach minimizes average transmit power, reducing sensitivity to non-linear impairments from optical amplifiers or WDM channels, thereby enhancing performance in terms of BER and energy efficiency. The key objective is to maximize the achievable channel capacity gain with shaped QAM signals.

Figure 2

Figure 2. Probabilistic Constellation Shaping.

2.1 Probabilistic amplitude shaping for QAM-based optical signal

Probabilistic shaping imposes a non-uniform probability distribution on transmitted QAM symbols, resulting in constellation points with varying probabilities. This approach increases the average Euclidean distance between constellation points compared to a standard uniformly distributed m-QAM signal of the same average power. In conventional m-QAM, each symbol represents a fixed number of bits, maintaining a binary interface between source and channel coding [40]. However, in our scheme, some data originates from an amplitude source $P_A$, eliminating this binary interface. Therefore, a device is required to take uniformly distributed independent bits $U_{k_c}$ as input and generate a non-uniform amplitude-shaped output Ã_nc as shown in Figure 3. The recommended properties for this device are outlined in [30]. According to [30], the input should produce a binary interface to the source coding part of the digital communication system. Also, the input can be recovered from the output, which means the mapping should be invertible. Finally, the output should be close to the output of amplitude source $P_A$.

Figure 3

Figure 3. The matcher transforms uniform data blocks of length $k_c$ into amplitudes $n_c$ which are distributed according to the desired distribution $P_A$.

The device with such properties is known as DM and it is the function block that performs rate-adaptive shaping in the PAS architecture [41], as shown in Figure 2. The DM transforms uniformly distributed input information bits to Maxwell-Boltzmann (MB) distributed PAM output symbols. The MB distribution helps to estimate the probability of constellation points. The DM generates only the positive amplitudes of m-PAM symbols [38]. The PAS architecture shown in Figure 1 attains PCS by independently shaping each signal dimension on a M-ary PAM template in order to build a PS $M^2$-QAM constellation. This probability distribution can be seen in Figure 2 where the bars indicate probability of each modulated QAM symbol with orthogonal in-phase and quadrature dimensions. Further details are discussed in [30, 42, 43].

2.2 Constant composition distribution matching

The distribution matcher reconstructs independently distributed input bits into output symbols based on the desired probability distribution. A key feature of the proposed constant composition distribution matcher (CCDM) is its asymptotic optimality, achieved through constant composition codes indexed via arithmetic coding [31]. The CCDM encoder generates M-ary symbol codewords from a binary input sequence, ensuring each M-ary symbol appears with a fixed frequency within the codeword. Primarily designed for PAS, the CCDM encoder uniquely maps each input to a distinct codeword.

2.2.1 CCDM encoder

Several algorithms have been employed to generate the non-uniform probability distribution of M-ary symbols required for CCDM encoding. Among them, the MB distribution is widely used due to its lower complexity and is also adopted in our approach. By default, the CCDM encoder constructs codewords with symbol occurrences following the MB distribution to maximize entropy. The entropy of this distribution is approximately equal to the ratio of k and n, which corresponds to the code rate. The key parameters defining the CCDM encoder are [44, 45]:

1. The length of the input bits sequence, represented by $k$.

2. The length of the output memory sequence is represented by $n$.

3. And the cardinality is represented by $L$.

The cardinality $L$ defines the number of unique symbols used within the output sequence referred as the codewords, e.g., $L$ = 4 for PAM-4. The CCDM distribution matcher for symbols distribution is shown in Figure 4.

Figure 4

Figure 4. CCDM Encoder for symbols distribution.

Similar to encoding, a CCDM decoder operates at the receiver side. The transmitted M-ary sequence, represented as a codeword, is received and processed by the CCDM decoder. The decoder then reconstructs the original binary signal from the received codeword.

2.3 Maxwell-Boltzmann distribution

The method used by the CCDM encoder in our PAS model for generating a non-uniform probability distribution of M-ary symbols is known as the MB distribution. The probability distribution of M-ary symbols in the constellation points is determined by the MB equation [38]:

$P_X(x)=\frac{e^{-\nu \mid r_x{\mid }^2}}{{\mathrm{\Sigma }}_{x^{\prime }\in X}{e}^{-\nu \mid r_{x^{\prime }}{\mid }^2}}(1)$

The representation of different symbols of above equation are:

1. $P$ represents the probability of symbol $x$.

2. $x$ represents symbol in the symbol set $X$.

3. $r$ represents the distance between $x$ and the origin that directly correspond with M-ary level.

The probability of constellation points x $\in X$ is commonly generated according to the MB distribution. The distribution can be seen in Equation 1 where the probability $P$ of each symbol $x$ is related to the symbol’s location r and a shaping parameter $\nu$. For a fixed symbol location, the distribution can be changed using the single parameter $\nu$, which is the shaping parameter for PAS. When $\nu$ = 0, the MB distribution degrades to a uniform distribution, indicating the maximum-entropy distribution for set of symbols $X$ under an average-power constraint. This means every symbol would have equal probability of occurrence. Increasing the value of $\nu$ (i.e., $\nu >0$), shapes the distribution of symbols while decreasing the entropy. This also transforms the distribution of symbols from uniform to probabilistic shaped in a M-ary constellation plane, perceiving the rate adaptation as a reduced average symbol energy. The rate parameter $\nu$ controls the entropy rate 2H(X) of the PS QAM signal in bits/symbol [38].

In the PAS architecture of Figure 1, having code rate $R_c$ and entropy rate 2$\mathbb{H}(X)$, the information rate in bits/symbol per two dimensions can be estimated as [30, 42]:

$IR=2(\mathbb{H}(X)-\mathit{m}(1-R_c))(2)$

On right-hand side of above Equation 2, the term 2$\mathbb{H}(P_x)$ refer to the largest number of information bits that can be accommodated within a complex symbol (per two dimensions), with the probability distribution $P_x$, that is controlled by shaping or rate parameter $\nu$ in a MB distribution. While, the term 2m(1-$R_c$) determines the FEC overhead in bits/symbol per two dimensions.

3 Semiconductor optical amplifier (SOA)

Unlike other optical amplifiers, SOAs are pumped electronically through an applied injection current, eliminating the need for a separate pump laser. However, one of the key challenges with SOAs is their polarization sensitivity, meaning that their operation depends on the polarization state of the incident light. Since the transmitted optical signal in our study is dual-polarized, special effort is required to ensure polarization-independent operation, which is crucial for practical field applications.

There are two main types of SOAs: i) Fabry-Perot SOAs ii) Traveling-Wave SOAs.

In our study, we have used the Traveling-Wave SOA due to its advantages, including a large optical bandwidth, high saturation power, and low polarization sensitivity. These characteristics make it well-suited for high-speed optical communication systems. Further details on the structure and working principles of SOAs can be found in [46, 47].

There are certain operational limitations in the process of signal amplification by SOA, one of the most critical being gain saturation. The gain of an SOA is influenced by both the input signal power and the internal noise generated during the amplification process. Beyond the SOA’s saturation point, any further increase in signal power leads to a decrease in the amplifier’s gain. To elaborate this, Figure 5 illustrates the gain (dB) response as a function of output power (dBm) for a typical SOA. Under a given injection current, the SOA exhibits a nearly constant or flat gain within a small signal region, also known as the stable gain region, over a certain range of output power. However, once the output power surpasses a threshold value, the gain starts to decline sharply, indicating the saturation of the SOA device. Therefore, to ensure linear amplification of the input signal, the SOA must operate within a signal power range constrained by its saturation power. The reduction in optical gain at higher output power levels is attributed to the depletion of charge carriers in the SOA’s active region, which limits the available gain. Further insights into gain saturation and SOA performance characteristics can be found in [46, 48]. The parameters set for Travelling-Wave SOA used in our work are listed in the following Table 1.

Figure 5

Figure 5. Gain response against the output power of typical SOA.

Table 1

Table 1. SOA’s parameters used in the simulation.

Figure 6 presents two SOA-Gain response curves: one for the uniform m-QAM signal, where the symbols follow a uniform probability distribution, and another for the PS m-QAM signal, where the symbols have a non-uniform probability distribution. At lower values of applied injection current (mA), the SOA-Gain response for both m-QAM and PS m-QAM signals remains nearly identical, causing their curves to overlap. However, as the applied injection current increases beyond 150–160 mA, the two curves start to diverge. When the applied injection current exceeds 150–160 mA, the output power of the SOA increases significantly, leading to the generation of non-linear impairments in the communication system. These impairments degrade the transmitted signal quality. However, PS m-QAM signals are more resilient to nonlinearity because of containing fewer high-energy symbols. Consequently, the SOA-Gain curve for PS m-QAM remains higher compared to conventional uniform m-QAM for injection currents beyond 150–160 mA. For instance, at 400 mA, the SOA-Gain for PS m-QAM is 40.33 dB, while for uniform m-QAM, it is only 38.43 dB, as shown in Figure 6. This suggests that for both uniform m-QAM and PS m-QAM optical signals, the applied injection current should not exceed 150–160 mA. Beyond this range, SOA’s internal noise, particularly due to amplified spontaneous emission (ASE). The overall output power surpasses 6–7 dBm, leading to nonlinear distortion and signal quality degradation, indicating that the SOA has reached saturation. Thus, PS m-QAM signals, owing to their lower power nature, maintain higher SOA-Gain while suffering less from nonlinear distortions compared to uniform m-QAM signals. This SOA-Gain response behavior remains consistent across different QAM modulation orders (m).

Figure 6

Figure 6. SOA-Gain [dB] response against applied values of injection current[mA].

4 Simulation setup

The uniform m-QAM and PS m-QAM dual-polarized optical signal transmission link incorporating a traveling-wave SOA is shown in Figure 7. The SOA is used for amplifying both uniform and PS m-QAM signals. Previously, the operation of SOA under varying conditions was discussed, particularly focusing on the effects of applied injection current and output signal power. It was concluded that to ensure stable operation in the linear gain region, the applied injection current should not exceed 150–160 mA, and the output power should remain below 6–7 dBm. To maintain stable SOA performance, the uniform m-QAM and PS m-QAM signal transmission models are analyzed with SOA operating within this linear range. The performance evaluation is conducted not only for lower-order modulation formats like 16-QAM but also for higher-order formats such as 32-QAM, 64-QAM, and 128-QAM. Different values of applied injection current are selected, ensuring operation within the stable gain region of the SOA. Table 2 presents the selected applied injection current values for different modulation orders of both m-QAM and PS m-QAM signals. In Table 2, ${\mathrm{I}}_{\mathrm{1}}$ and ${\mathrm{I}}_{\mathrm{2}}$ represent the low and high values of SOA’s applied injection current, respectively. The value ${\mathrm{I}}_{\mathrm{2}}$ corresponds to the maximum injection current at which the SOA maintains a stable gain response without reaching saturation, whereas ${\mathrm{I}}_{\mathrm{1}}$ is approximately half of ${\mathrm{I}}_{\mathrm{2}}$. The performance of the simulation model shown in Figure 7 is analyzed at both these selected injection current values.

Figure 7

Figure 7. The uniform m-QAM and PS-mQAM dual polarized optical signal transmission link in the presence of SOA.

Table 2

Table 2. Applied injection currents for different order of QAM signals.

Observing Table 2, it is evident that the maximum range of applied injection current to the SOA decreases as the modulation order of the QAM signals increases. This is because higher-order QAM signals, such as 64-QAM and 128-QAM, exhibit greater sensitivity to the applied injection current due to nonlinear distortions, noise enhancement and phase noise sensitivity [49].

5 Performance analysis of uniform and PS QAM signals

5.1 Bit error rate

The plots in Figure 8a and (b) show the estimated BER against OSNR (14–22 dB) for various QAM signals amplified under two different values of applied injection current of SOA. It can be observed that the BER of PS m-QAM signals is lower compared to uniform m-QAM signals for a certain OSNR value. This is because the PS signal is less prone to errors due to the reduced probability of higher-amplitude QAM symbols, making it less affected by SOA-induced nonlinear impairments during amplification. For instance, considering the 16-QAM case in Figure 8a, which demonstrates the BER response under high SOA injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$), achieving a BER level of $10^{-2}$ requires approximately 1 dB higher OSNR (18 dB) for the uniform 16-QAM signal compared to the probabilistic shaped PS-16QAM (17 dB). This indicates that the uniform 16-QAM signal requires more power than the PS-16QAM signal to maintain the same BER level.

Figure 8

Figure 8. BER results of mQAM and PS-mQAM: (a) On SOA’s high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) and (b) On SOA’s low injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$).

When analyzing the results in terms of SOA’s applied injection current, the values of BER for different OSNRs is slightly lower for low injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) compared to high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$). A higher applied injection current results in higher output signal power, making the signal more susceptible to nonlinear impairments introduced by the SOA, leading to an increased error rate. Comparing Figures 8a,b, the BER of the PS-16QAM signal under high SOA injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) is approximately $10^{-2}$ for an OSNR of 17 dB, whereas the BER response of the uniform 16-QAM signal under low SOA injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) is slightly lower than $10^{-2}$ for the same OSNR value. A similar BER response is observed for higher-order QAM signals (e.g., 32-QAM, 64-QAM, 128-QAM) through both uniform (m-QAM) and probabilistic shaped (PS-mQAM) transmission models in the presence of SOA operating under high ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) and low ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) applied injection currents. The PS-mQAM signal consistently experiences lower BER than its corresponding uniform m-QAM signal. In terms of SOA injection current, the BER response is better at low injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) compared to high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) for any QAM modulation order, as shown in Figures 8a,b.

5.2 Error vector magnitude

The EVM plots of both uniform QAM and PS-QAM signals amplified under two different values of applied injection currents are shown in Figures 9a,b, respectively. The plots indicate that the EVM result for PS-QAM signals is lower than that of traditional uniform m-QAM signals for any given OSNR value. As OSNR increases, the corresponding EVM decreases, and the gap between the uniform m-QAM and PS-QAM curves gradually widens. This is because higher OSNR implies a lower noise component, making the probabilistic shaped signal less susceptible to nonlinear impairments introduced by the SOA. Due to its lower probability of high power symbols, the PS-QAM signal achieves a further reduction in EVM compared to uniform m-QAM.

Figure 9

Figure 9. EVM results of mQAM and PS-mQAM: (a) On SOA’s high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) and (b) On SOA’s low injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$).

Comparing the EVM results in Figures 9a,b, it is observed that the EVM for the case of low injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) is slightly lower than that for high SOA injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) at any fixed OSNR value. Additionally, the difference in EVM between uniform m-QAM and PS-QAM is smaller when SOA operates at a lower injection current. For example, at OSNR = 20 dB, in Figure 9a, the EVM for uniform 16-QAM is approximately 18.5%, while for PS-16QAM is around 17%, resulting in a difference of about 1.5%. In contrast, Figure 9b shows that for the same OSNR, the EVM for 16-QAM is about 17.5%, and for PS-16QAM is approximately 16.5%, reducing the difference to around 1%. This behavior is due to the higher power of PS-16QAM signals under SOA’s high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$), which makes the signals more susceptible to nonlinear effects induced by the SOA, leading to increased EVM values. Similar trends are observed for higher-order QAM signals such as 32-QAM, 64-QAM and 128-QAM. In all cases, the PS-mQAM signal exhibits lower EVM than its corresponding uniform m-QAM signal. Furthermore, for any given OSNR, the EVM is lower when SOA operates at a lower injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) compared to a higher injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$).

5.3 Mutual information

The MI of the link shown in Figure 7 is simulated for both uniform m-QAM and PS-mQAM dual-polarized signals amplified by SOA under two different values of injection currents, and the results are shown in Figures 10a,b, respectively. The results indicate that the MI response of the PS signal is significantly better and closer to the Shannon Limit than that of the uniform signal for any OSNR value within the given range. Comparing MI in terms of SOA’s applied injection current, the MI response of both 16-QAM and PS-16QAM signals amplified under SOA’s low injection current is slightly better than high injection current. For instance, at OSNR = 17 dB, the MI is slightly above 3.7 bits/symbol for 16-QAM and exceeds 3.8 bits/symbol for PS-16QAM at low injection current compared to at high injection current. This is because signals amplified under lower injection currents experience fewer nonlinear impairments. Additionally, the PS-mQAM signal curve is consistently closer to the maximum channel capacity, approaching the Shannon Limit more efficiently compared to the uniform m-QAM signal. As OSNR increases, both m-QAM and PS-QAM signals gradually approach the theoretical channel capacity. For instance, while the ideal Shannon capacity for a 16-QAM signal is 4 bits/symbol, the simulated MI achieved for 16-QAM and PS-16QAM in this study is approximately 3.95 bits/symbol.

Figure 10

Figure 10. MI response of mQAM and PS-mQAM: (a) On SOA’s high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) and (b) On SOA’s low injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$).

Figure 11 presents the constellation diagrams of both uniform m-QAM and PS-mQAM signals after amplification through the SOA under high ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) and low ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) injection currents. The impact of PAS is evident in the form of a lower density of higher-power points in the PS-mQAM constellation compared to uniform m-QAM. Additionally, it can be observed that for signals amplified under high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$), the constellation points exhibit increased noise and phase distortion compared to those amplified under low injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$). This degradation is due to the stronger nonlinear impairments induced by the SOA at higher injection currents, leading to greater distortion in the received signal.

Figure 11

Figure 11. The constellation diagrams of mQAM and PS-mQAM signals amplified on SOA’s low ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) and high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$). (a) 16QAM (SOA-${\mathrm{I}}_{\mathrm{1}}$), (b) 32QAM (SOA-${\mathrm{I}}_{\mathrm{1}}$), (c) 64QAM (SOA-${\mathrm{I}}_{\mathrm{1}}$), (d) 128QAM (SOA-${\mathrm{I}}_{\mathrm{1}}$), (e) PS-16QAM (SOA-${\mathrm{I}}_{\mathrm{1}}$), (f) PS-32QAM (SOA-${\mathrm{I}}_{\mathrm{1}}$), (g) PS-64QAM (SOA-${\mathrm{I}}_{\mathrm{1}}$), (h) PS-128QAM (SOA-${\mathrm{I}}_{\mathrm{1}}$), (i) 16QAM (SOA-${\mathrm{I}}_{\mathrm{2}}$), (j) 32QAM (SOA-${\mathrm{I}}_{\mathrm{2}}$), (k) 64QAM (SOA-${\mathrm{I}}_{\mathrm{2}}$), (l) 128QAM (SOA-${\mathrm{I}}_{\mathrm{2}}$), (m) PS-16QAM (SOA-${\mathrm{I}}_{\mathrm{2}}$), (n) PS-32QAM (SOA-${\mathrm{I}}_{\mathrm{2}}$), (o) PS-64QAM (SOA-${\mathrm{I}}_{\mathrm{2}}$) and (p) PS-128QAM (SOA-${\mathrm{I}}_{\mathrm{1}}$).

In multi-carrier formats such as OFDM and WDM, PAS can be independently applied to each subcarrier or wavelength. Consistent shaping distributions are required to maintain uniform performance, although minor deviations may arise from channel-dependent nonlinearities. Inter-carrier interference can slightly affect PAS efficiency, but this can be mitigated through adaptive equalization and moderate shaping depth, ensuring stable performance across all carriers.

6 FEC based on Low-Density Parity-Check coding

To approach the capacity of an AWGN channel, constellation shaping must be combined with FEC codes. In this section, we simulate a PS-mQAM signal integrated with systematic binary FEC, demonstrating an improvement in system performance compared to the conventional PS-mQAM transmission model. The PS-mQAM signal is generated using MB distribution approach detailed in Section II. For error correction, we employ Low-Density Parity-Check (LDPC) coding. LDPC codes are chosen due to their performance, which closely approaches Shannon capacity across various channels, as well as their relatively lower decoding complexity compared to other FEC schemes.

6.1 LDPC coding of IEEE 802.16e standard

The LDPC codes defined with IEEE 802.16e belong to the structured category. These codes are also known as quasi-cyclic structured LDPC codes. The base parity-check matrix for this structured code is composed of [50]:

$H_{bm}=[H1\mid H2](3)$

Where ${[H1]}_{\mathit{mxk}}$ in Equation 3 indicates the information bits, which is sparse matrix generated using pseudo random design of the matrix (IEEE P802.16e) with special periodicity constraints. ${[H2]}_{\mathit{mxm}}$ represents parity bits, which is also sparse and lower triangular matrix. The design of the base matrix is given in [50], which is discussed in detail in [51, 52].

The LDPC codes selected by the WiMAX standard (IEEE 802.16e) support 19 different codeword sizes with four different code rates (k/n) and six unique class codes, shown in (46, Table 1). The codeword length of N = 2208 with expansion factor $(zxz)=92\times 92$ is used in the formation of parity-check matrix for applying FEC codes based on LDPC WiMAX standard (IEEE 802.16e) in our study of PS m-QAM signals.

6.2 Implementation of LDPC coded system

The basic model of LDPC coded system is shown in Figure 12. It consists of five major building blocks which include encoder, modulator, channel, demodulator and decoder. The LDPC encoder divides the information sequence, i.e., PS-mQAM signal into message blocks of u = ($u_0,u_1$,……,$u_{k-1}$) and each block consists of $k$ information bits. The total number of different possible messages would be $2^k$. The encoder then converts each message block u independently into n-tuple of discrete symbols called as codeword.

Figure 12

Figure 12. Basic model of LDPC coded system.

The encoded codeword is passed to the modulator, which prepares it for transmission over the channel. In this case, the modulator employs PS-mQAM to map the codewords onto one or more information-carrying signals. The channel is assumed to follow an AWGN distribution. At the receiver, the signal is first demodulated, producing a received vector that may contain errors due to channel impairments. This received vector is then processed by the LDPC decoder, which detects and corrects errors, improving the reliability of the transmitted data.

7 Performance analysis of uniform and PS QAM signals in presence of FEC

To achieve the maximum capacity of an AWGN channel, constellation shaping of the PS mQAM signal is combined with FEC. The application of FEC further enhances the performance of the received signal. The simulation is conducted on a dual-polarized PS mQAM signal with systematic binary FEC in a back-to-back configuration using LDPC coding. The parameters used for the simulation of the PS mQAM signal with the FEC model are listed in Table 3.

Table 3

Table 3. PS-mQAM with FEC simulation module parameters.

According to parameters specified in 3 for FEC, the simulation has been performed on dual polarized PS mQAM signal along with FEC. The parameters of modules shown in Figure 7 for simulation of PS mQAM signal will remain same as mentioned in Table 1, 2. Different signal metrics such as BER, EVM and MI are estimated.

7.1 Bit error rate

The BER results for higher-order PS m-QAM signals combined with FEC are evaluated against an OSNR range of 14–22 dB, as shown in Figure 13. The results clearly indicate a significant reduction in BER by applying PS together with FEC to a uniform m-QAM signal. For instance, as observed in Figure 13, the uniform 16-QAM signal requires an OSNR of approximately 20.5 dB to achieve a BER of $10^{-3}$. In contrast, the same 16-QAM signal with PS and FEC (PS-16QAM FEC) requires an OSNR of approximately 18.8 dB to maintain the same BER level of $10^{-3}$. This implies that the uniform 16-QAM signal requires around 1.7 dB higher OSNR to achieve the same signal quality compared to the PS 16QAM FEC signal. Similar improvements are observed for higher-order modulation formats shown in the plots.

Figure 13

Figure 13. BER response obtained for different QAM order signals by mQAM, PS-mQAM and PS-mQAM with FEC.

7.2 Error vector magnitude

The estimated EVM values for higher-order PS-QAM signals combined with FEC, evaluated against an OSNR range of 14–22 dB, are plotted in Figure 14. For comparison, the EVM curves of uniform m-QAM and PS-mQAM signals are also shown. It is evident that the EVM of m-QAM signals is significantly reduced by applying both PS and FEC (PS-mQAM FEC) compared to their corresponding uniform m-QAM signals. For higher-order QAM signals such as 64-QAM, the FEC method is more effective in reducing EVM than for lower-order QAM signals like 16-QAM, as illustrated in Figure 14. For instance, the uniform 16-QAM signal exhibits an EVM of approximately 17% at an OSNR of 20 dB. However, when PS is applied together with FEC, the estimated EVM is reduced to approximately 15% for the same OSNR level. This indicates that the combined effect of PS and FEC reduces EVM by nearly 2% compared to the standard uniform 16-QAM signal.

Figure 14

Figure 14. EVM results of different order QAM signals for m-QAM, PS-mQAM and PS-mQAM with FEC.

7.3 Mutual information

The MI results against OSNR for PS-mQAM signals combined with FEC are plotted in Figure 15. The curves clearly indicate that the MI response of dual-polarized optical signals is significantly improved by implementing PS together with FEC (PS-mQAM FEC) compared to the corresponding uniform m-QAM. For instance, for a 16-QAM PS signal with FEC (PS-16QAM FEC), it is evident that applying FEC to a PS-16QAM signal brings the MI response closer to the Shannon limit (4 bits/symbol) compared to a uniform 16-QAM signal for any given OSNR. Since it is a 16-QAM signal, the MI limit of all signals including uniform 16-QAM, PS-16QAM, and PS-16QAM FEC, approaches 4 bits/symbol.

Figure 15

Figure 15. MI response for different order of m-QAM, PS-mQAM and PS-mQAM signals with FEC.

8 Performance analysis of SOA amplified PS signals in the presence of FEC

In Section V discusses the integration of a SOA within the PS-mQAM signal transmission link shown in Figure 7. The primary function of the SOA is to amplify weak signals that have been attenuated during long-distance transmission. However, the SOA not only amplifies the signal power but also introduces ASE noise. Furthermore, the signal amplification by the SOA is limited, as high-power signals can induce nonlinear effects in both the SOA and the WDM channel. To mitigate these nonlinearities, PS-mQAM was first applied to a uniform m-QAM signal, thereby reducing the peak-to-average power ratio by employing a non-uniform probability distribution of symbols. As a result, the signal becomes less susceptible to SOA-induced nonlinear impairments.

Building upon this, we now apply FEC based on LDPC coding to the PS-mQAM signal in the presence of the SOA. It is important to note that the parameters used for the simulation of PS-mQAM FEC signals remain the same as those listed in Table 3. Similarly, the standard SOA parameters, particularly the injection currents $\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$ and $\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$ for different m-QAM modulation orders, are maintained as specified in 2. The system parameters listed in 1 also remain unchanged.

Integrating PAS with advanced FEC enhances error resilience but increases digital processing complexity and power consumption due to additional distribution matching and decoding stages. However, since PAS lowers the required SNR for a given BER, the system can operate at reduced optical launch power or with lower FEC overhead. As a result, the overall energy efficiency may remain comparable to or better than that of conventional uniform signaling with high-overhead FEC.

8.1 Bit error rate

The BER versus OSNR plots of PS m-QAM signals such as 32-QAM, 64-QAM, 128-QAM, combined with FEC and amplified using a SOA under two different injection current levels, high ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) and low ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) are shown in Figures 16a,b. The BER trends are generally consistent across different modulation format. The PS m-QAM signals with FEC (PS-mQAM FEC) exhibit slightly higher BER values when amplified under the higher injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) compared to those amplified at the lower injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$). For instance, the BER plot for the PS-16QAM signal with FEC under $\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$, shown in Figure 16a, reveals a BER slightly above $10^{-3}$ at an OSNR of 19 dB. In contrast, the same signal amplified under $\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$, shown in Figure 16b, achieves a BER approximately equal to $10^{-3}$ at the same OSNR level. This degradation in performance under high injection current is attributed to increased nonlinear effects induced by the SOA due to the higher signal power.

Figure 16

Figure 16. BER results of mQAM, PS-mQAM and PS-mQAM FEC signals: (a) SOA’s high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) and (b) SOA’s low injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$).

8.2 Error vector magnitude

The EVM plots against OSNR for higher-order QAM signals such as 32-QAM, 64-QAM, 128-QAM with PS and FEC, amplified under two different SOA injection current levels—high ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) and low ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$), are shown in Figures 17a,b, respectively. The general trend remains consistent. The m-QAM signals employing PS combined with FEC (PS-mQAM FEC) exhibit lower EVM compared to their corresponding uniform m-QAM counterparts. Furthermore, the EVM performance is consistently better when the SOA operates at a lower injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) compared to a higher injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$). For instance, the EVM of the 16-QAM signal is significantly improved by applying PS and FEC compared to the uniform 16-QAM signal. Additionally, when comparing the EVM results based on SOA injection current, it can be observed that the PS-16QAM FEC signal exhibits slightly lower EVM under $\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$ than under $\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$, for any given OSNR value. Specifically, as shown in Figure 17a, the PS-16QAM FEC signal amplified at high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) exhibits an EVM of approximately 20% at an OSNR of 18 dB. In contrast, Figure 17b shows that the same signal amplified under low injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) achieves a slightly reduced EVM of approximately 19% at the same OSNR.

Figure 17

Figure 17. EVM results of mQAM, PS-mQAM and PS-mQAM FEC signals: (a) On SOA’s high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) and (b) On SOA’s low injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$).

8.3 Mutual information

The MI plots versus OSNR for all QAM signals with PS and FEC and amplified using SOAs under high and low injection currents—$\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$ and $\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$, are presented in Figures 18a,b, respectively. As before, lower SOA injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$) yields improved MI performance compared to higher injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$), for a given OSNR. For example, the PS-16QAM FEC signal demonstrates superior MI values, approaching the Shannon limit (4 bits/symbol for 16-QAM), compared to the corresponding uniform 16-QAM signals in both Figures 18a,b. When comparing the MI performance across different SOA injection current levels, it is evident that amplification with $\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$ yields better MI results. For instance, at an OSNR of 15 dB, the estimated MI for the PS-16QAM FEC signal in Figure 18a, amplified with high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$), is approximately 3.6 bits/symbol. In contrast, Figure 18b shows a slightly improved MI, exceeding 3.6 bits/symbol, when the same signal is amplified using the lower injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$).

Figure 18

Figure 18. MI results of mQAM, PS-mQAM and PS-mQAM FEC signals: (a) On SOA’s high injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{2}}$) and (b) On SOA’s low injection current ($\mathrm{S}\mathrm{O}\mathrm{A}-{\mathrm{I}}_{\mathrm{1}}$).

9 Discussion and conclusion

Nonlinear impairments are a formidable challenge in optical communication systems, arising from the interaction of propagating light with various optical components, such as fiber channels and optical amplifiers. This paper focuses on the mitigation of nonlinear effects induced by SOAs, which occur due to changes in the refractive index of the active region when a high-power optical signal passes through the amplifier during the amplification process. SOAs are often preferred in optical systems due to their low power consumption, compact size, and ease of integration into photonic networks. Additionally, they are considered a promising candidate for signal amplification in the O-band (1260–1360 nm), where other optical amplifiers are less effective.

To address the challenge of nonlinear impairments, we employ a technique known as PAS. PAS imposes a non-uniform probability distribution on the transmitted QAM symbols in the I/Q constellation plane, assigning higher probabilities to symbols closer to the origin, which possess greater entropy, while symbols farther from the origin have a lower probability of occurence. This approach effectively reduces the average power required for QAM signal transmission, thereby mitigating the nonlinear effects induced by the SOA that degrade signal quality.

Performance analysis demonstrate that the application of PS to QAM signals significantly improves the quality of the received signal compared to uniform m-QAM signals, as evidenced by improvements in BER, EVM, and MI. Furthermore, the application of FEC based on LDPC coding to PS QAM signals further enhances the overall performance of the communication system.

While PAS offers significant performance gains in mitigating nonlinearities, its implementation in real optical transceivers introduces certain practical challenges. The most notable among these are the processing latency and hardware complexity associated with the DM and de-matcher modules, which are responsible for generating and recovering the shaped symbol sequences. At very high symbol rates, the sequential nature of some DM algorithms, such as CCDM, can lead to latency that may limit throughput. Furthermore, additional digital signal processing (DSP) blocks are required to integrate PAS with FEC and modulation mapping, which increases hardware resource utilization. Nevertheless, several recent studies and prototype implementations have shown that low-latency PAS architectures can be realized using optimized algorithms (e.g., shell mapping, arithmetic coding) and parallelized FPGA or ASIC implementations. These approaches enable near–real-time operation even for high-order QAM formats, suggesting that PAS is increasingly becoming feasible for deployment in practical high-speed optical systems.

In practical optical links, the SOA’s operating characteristics—such as gain, carrier lifetime, and refractive index—can vary with temperature, bias current, and input optical power. These variations affect the system’s nonlinear behavior and, consequently, the optimum shaping configuration. Adaptive PAS schemes have been proposed to address this issue by dynamically tuning the shaping distribution according to real-time estimates of channel or amplifier conditions. Such adaptability can be realized through feedback-based control, where the receiver periodically evaluates link metrics (e.g., SNR, BER, or mutual information) and communicates the optimal shaping parameters back to the transmitter. Alternatively, machine learning–assisted PAS architectures have shown promise in predicting optimal shaping distributions without explicit channel modeling. Although adaptive PAS introduces additional computational and control overhead, it has the potential to sustain optimal performance even under rapidly changing conditions, such as those caused by SOA gain saturation or temperature drift. Future work will focus on implementing and experimentally validating such adaptive PAS schemes in dynamically varying optical link environments. The present study is based on numerical simulations. Experimental validation using a lab-scale O-band optical link with SOAs and DSP-based transceivers is planned for future work to confirm the simulation insights regarding BER, EVM, and MI performance.

While PAS primarily targets SOA-induced nonlinearities, it can also reduce the impact of fiber nonlinearities such as SPM and XPM by lowering the average transmit power. Furthermore, PAS can be effectively combined with conventional nonlinear mitigation techniques—such as digital back-propagation, pre-distortion, or multi-channel equalization—to enhance overall system performance.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

SA: Writing – original draft. SG: Methodology, Writing – review and editing. BA: Validation, Writing – original draft. AA: Funding acquisition, Writing – review and editing. MIm: Project administration, Writing – review and editing. MIj: Validation, Writing – review and editing. LP: Conceptualization, Writing – review and editing.

Funding

The authors declare that financial support was received for the research and/or publication of this article. The authors would like to thank the support of Erasmus + Student/Staff Mobility Exchange Program (G.A. n. 2023-1-IT02-KA171-HED-000141442).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The authors declare that no Generative AI was used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Irukulapati NV, Wymeersch H, Johannisson P, Agrell E. Stochastic digital backpropagation. IEEE Trans Commun (2014) 62(11):3956–3968. doi:10.1109/tcomm.2014.2362534

CrossRef Full Text | Google Scholar

2. P, Urquhart editor. Advances in optical amplifiers. London, United Kingdom: IntechOpen Limited (2011). doi:10.5772/571

CrossRef Full Text | Google Scholar

3. Wang K, Zhang J, Zhao L, Li X, Yu J. Mitigation of pattern-dependent effect in SOA at O-band by using DSP. J Lightwave Technology (2019) 38(3):590–597. doi:10.1109/jlt.2019.2946188

CrossRef Full Text | Google Scholar

4. Yamazaki H, Kanazawa S, Nakanishi Y, Ueda Y, Kobayashi W, Muramoto Y, et al. Ultra-broadband EA-DFB laser module for 200-Gbit/s PAM4 transmitter. In: 2017 Optical Fiber Communications Conference and Exhibition (OFC); Los Angeles, CA: IEEE. (2017). p. 1–3.

Google Scholar

5. Chang F, Bhoja S, Riani J, Hosagrahar I, Wu J, Herlekar S, et al. Link performance investigation of industry first 100G PAM4 IC chipset with real-time DSP for data center connectivity. In: 2016 Optical Fiber Communications Conference and Exhibition (OFC); Anaheim, CA: IEEE. (2016). p. 1–3.

Google Scholar

6. El-Fiky E, Osman M, Sowailem M, Samani A, Patel D, Li R, et al. 200 Gb/s transmission using a dual-polarization O-Band silicon photonic intensity modulator for stokes vector direct detection applications. Opt Express (2017) 25(24):30336–48. doi:10.1364/oe.25.030336

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Vedala G, Hameed MA, Hui R. Digital compensation of SSBI in direct detection multicarrier system with SOA nonlinearities. IEEE Photon Technol Lett (2017) 29(4):369–72. doi:10.1109/LPT.2016.2647561

CrossRef Full Text | Google Scholar

8. Hameed MA, O’Sullivan M, Hui R. Impact of SOA-induced nonlinear impairments in CO-OFDM and nyquist sinc-pulse transmission. Asia Commun Photon Conf (2013) AF3E.4. doi:10.1364/acpc.2013.af3e.4

CrossRef Full Text | Google Scholar

9. Khaleghi H, Sharaiha A, Rampone T, Morel P, Guegan M. Semiconductor optical amplifiers in coherent Optical-OFDM systems. Photon Technology Lett IEEE (2012) 24(7):560–562. doi:10.1109/lpt.2012.2183346

CrossRef Full Text | Google Scholar

10. Hosseini SR, Razaghi M, Das NK. Analysis of non-linear refractive index influences on four-wave mixing conversion efficiency in semiconductor optical amplifiers. Opt and Laser Technology (2012) 44(3):528–533. doi:10.1016/j.optlastec.2011.08.016

CrossRef Full Text | Google Scholar

11. Paschotta R. Encyclopedia of laser physics and technology, I. Frauenfeld, Switzerland: RP Photonics AG (2008).

Google Scholar

12. Ip E. Nonlinear compensation using backpropagation for polarization-multiplexed transmission. J Lightw Technol (2010) 28(6):939–951. doi:10.1109/jlt.2010.2040135

CrossRef Full Text | Google Scholar

13. Li X, Li G. Electrical postcompensation of SOA impairments for fiber-optic transmission. IEEE Photon Technol Lett (2009) 21(9):581–3. doi:10.1109/LPT.2009.2015149

CrossRef Full Text | Google Scholar

14. Li X, Li G. Joint fiber and SOA Impairment compensation using digital backward propagation. IEEE Photon J (2010) 2(5):753–8. doi:10.1109/JPHOT.2010.2068042

CrossRef Full Text | Google Scholar

15. Yu J, Jeppesen P. Increasing input power dynamic range of SOA by shifting the transparent wavelength of tunable optical filter. J Lightw Technol (2001) 19(9):1316–25. doi:10.1109/50.948279

CrossRef Full Text | Google Scholar

16. Yu J, Jeppesen P. Improvement of cascaded semiconductor optical amplifier gates by using holding light injection. J Lightw Technol (2001) 19(5):614–23. doi:10.1109/50.923474

CrossRef Full Text | Google Scholar

17. Xu J, Zhang X, Moerk J. Investigation of patterning effects in ultrafast SOA-based optical switches. IEEE J Quan Electron (2010) 46(1):87–94. doi:10.1109/jqe.2009.2027341

CrossRef Full Text | Google Scholar

18. Rizou ZV, Zoiros KE, Hatziefremidis A. Comparison of basic notch filters for semiconductor optical amplifier pattern effect mitigation. Appl Sci (2017) 7(8):783. doi:10.3390/app7080783

CrossRef Full Text | Google Scholar

19. Yadav GS, Yan J-H, Feng K-M. Radial basis function network equalizer for 112 Gb/s PAM4 IM-DD inter-data centers transmission with gain-clamped semiconductor optical amplifier. In: Proc. Opto-Electron. Commun. Conf. (2020). p. 1–3. doi:10.1109/oecc48412.2020.9273600

CrossRef Full Text | Google Scholar

20. Sun C, Bae SH, Kim H. Transmission of 28-Gb/s duobinary and PAM-4 signals using DML for optical access network. IEEE Photon Technol Lett (2017) 29(1):130–133. doi:10.1109/lpt.2016.2629623

CrossRef Full Text | Google Scholar

21. Xue L, Lin R, Kerrebrouck JV, Yi L, Chen J, Yin X. 100G PAM-4 PON with 34 dB power budget using joint nonlinear tomlinson-harashima precoding and volterra equalization. In: Proc. Eur. Conf. Opt. Commun. (2021). p. 1–4. doi:10.1109/ecoc52684.2021.9606041

CrossRef Full Text | Google Scholar

22. Xue L, Yi L, Lin R, Huang L, Chen J. SOApattern effect mitigation by neural network based pre-equalizer for 50G PON. Opt Exp (2021) 29(16):24714–22. doi:10.1364/oe.426781

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Reza AG, Rhee JK. Nonlinear equalizer based on neural networks for PAM-4 signal transmission using DML. IEEE Photon Technol Lett (2018) 30(15):1416–1419. doi:10.1109/lpt.2018.2852327

CrossRef Full Text | Google Scholar

24. Chuang C, Liu LC, Wei CC, Liu JJ, Henrickson L, Huang WJ, et al. Convolutional neural network based nonlinear classifier for 112-Gbps high speed optical link. In: Proc. Opt. Fiber Commun. Conf. Exhib. (2018). p. W2A.43. Paper W2A.43. doi:10.1364/ofc.2018.w2a.43

CrossRef Full Text | Google Scholar

25. Huang X, Zhang D, Hu X, Ye C, Zhang K. Recurrent neural network-based equalizer with embedded parallelization for 100Gbps/λ PON. In: Proc. Opt. Fiber Commun. Conf. (2021). p. M3G.2. Paper M3G.2. doi:10.1364/ofc.2021.m3g.2

CrossRef Full Text | Google Scholar

26. Sobhanan A, Pelusi M, Inoue T, Venkitesh D, Namiki S. Compensation of SOA-induced nonlinear phase distortions by optical phase conjugation. Opt Express (2021) 29(8):12252–65. doi:10.1364/oe.416955

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Hamaoka F, Okamoto S, Nakamura M, Matsushita A, Kisaka Y. Adaptive compensation for SOA-induced nonlinear distortion with training-based estimation of SOA device parameters. In: 2018 European Conference on Optical Communication (ECOC). IEEE (2018). p. 1–3.

CrossRef Full Text | Google Scholar

28. Li W, Chen Y, Zeng Y, Zhou Z, Zhang M, Chen J, et al. Unleashing 100-km multi-channel PDM self-homodyne coherent transmission by SOAs and all-optical nonlinear distortion mitigations. J Lightwave Technology (2023) 42:1805–18. doi:10.1109/jlt.2023.3328306

CrossRef Full Text | Google Scholar

29. Böcherer G. Probabilistic shaping and its applications for optical communications. In: 2019 Conference on Lasers and Electro-Optics Europe and European Quantum Electronics Conference (CLEO/Europe-EQEC); Munich, Germany (2019). p. 1. doi:10.1109/cleoe-eqec.2019.8873256

CrossRef Full Text | Google Scholar

30. Böcherer G, Steiner F, Schulte P. Bandwidth efficient and rate-matched low-density parity-check coded modulation. IEEE Trans Commun (2015) 63(12):4651–65. doi:10.1109/TCOMM.2015.2494016

CrossRef Full Text | Google Scholar

31. Schulte P, Bocherer G. Constant composition distribution matching. IEEE Trans Inf Theor (2016) 62(1):430–434. doi:10.1109/tit.2015.2499181

CrossRef Full Text | Google Scholar

32. Buchali F, Steiner F, ocherer GB, Schmalen L, Schulte P, Idler W. Rate adaptation and reach increase by probabilistically shaped 64-QAM: an experimental demonstration. J Lightw Technol (2016) 34(8):1599–609. doi:10.1109/jlt.2015.2510034

CrossRef Full Text | Google Scholar

33. Alharbi AG, Mirza J, Raza M, Ghafoor S. Performance enhancement of praseodymium doped fiber amplifiers. Comput Mater Continua (2022) 73(3):5411–22. doi:10.32604/cmc.2022.029317

CrossRef Full Text | Google Scholar

34. Alyshev S, Khegai A, Umnikov A, Firstov S. Bismuth-doped fiber lasers and amplifiers operating from O- to U-Band: current state of the art and outlook. Photonics (2024) 11(7):663. doi:10.3390/photonics11070663

CrossRef Full Text | Google Scholar

35. Vakhrushev AS, Kharakhordin AV, Alyshev SV, Khegai AM, Firstova EG, Melkumov MA, et al. Cladding-pumped bismuth-doped fiber lasers operating at a wavelength region of 1.4–1.5 µm. Dokl Ross Akad Nauk Fiz., Tekh Nauk (2024) 514(1):5–13. doi:10.31857/S2686740024010014

CrossRef Full Text | Google Scholar

36. Alyshev S, Vakhrushev A, Khegai A, Firstova E, Riumkin K, Melkumov M, et al. Impact of doping profiles on the formation of laser-active centers in bismuth-doped GeO₂ − −SiO₂ glass fibers. Photon Res (2024) 12:260–70. doi:10.1364/prj.498782

CrossRef Full Text | Google Scholar

37. Fehenberger T, Alvarado A, Böcherer G, Hanik N. On probabilistic shaping of quadrature amplitude modulation for the nonlinear fiber channel. J Lightwave Technology (2016) 34(21):5063–73. doi:10.1109/JLT.2016.2594271

CrossRef Full Text | Google Scholar

38. Cho J, Winzer PJ. Probabilistic constellation shaping for optical fiber communications. J Lightwave Technology (2019) 37(6):1590–607. doi:10.1109/JLT.2019.2898855

CrossRef Full Text | Google Scholar

39. Sergeev A, Shaniiazov R. Analysis of the capacity gain of probability shaping QAM. In: Y Koucheryavy, and A Aziz, editors. Internet of things, smart spaces, and next generation networks and systems. NEW2AN 2022. Lecture notes in computer science, 13772. Cham: Springer (2023). p. 595–605. doi:10.1007/978-3-031-30258-9_53

CrossRef Full Text | Google Scholar

40. Gallager RG. Principles of digital communication. Cambridge, U.K.: Cambridge Univ. Press (2008).

Google Scholar

41. Böcherer G, Mathar R. Matching dyadic distributions to channels. In: Proc. Data Compression Conf. (DCC) (2011). p. 23–32. doi:10.1109/dcc.2011.10

CrossRef Full Text | Google Scholar

42. Cho J, Chen X, Chandrasekhar S, Winzer P. On line rates, information rates, and spectral efficiencies in probabilistically shaped QAM systems. Opt Express (2018) 26(8):9784–91. doi:10.1364/oe.26.009784

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Qu Z, Djordjevic IB, Jon A. Two-dimensional constellation shaping in fiber-optic communications. Appl Sciences (2019) 9(9):1889. doi:10.3390/app9091889

CrossRef Full Text | Google Scholar

44. Kaller KLC, Raytchev M, Galib Reza A, Troncoso Costas M, Barry L, Atieh A. Investigating probabilistic constellation shaping for dual-polarization PAM8 signals at different data rates. Next-Generation Optical Communication: Components, Sub-systems, and Systems XII (2023) 12429 p. 264–70. doi:10.1117/12.2648047

CrossRef Full Text | Google Scholar

45. Reza AG, Venkatasubramani LN, Raj Gautam A, Troncoso-Costas M, Raytcheva M, Atieh A, et al. 4×130 Gbit/s PS-PAM-16 transmissions using an integrated SOA-PIN design for intra-DCIs enabled by machine learning. 49th Eur Conf Opt Commun (ECOC 2023) Glasgow, Scotland (2023) 2023:373–376. doi:10.1049/icp.2023.2083

CrossRef Full Text | Google Scholar

46. Malik Y. Gain control in semiconductor optical amplifier. University of Glasgow (2014). PhD thesis.

Google Scholar

47. Connelly MJ. Semiconductor optical amplifiers. Boston: Kluwer Academic (2002).

Google Scholar

48. Connelly M. Semiconductor optical amplifiers and their applications. In: Presented at 3rd Spanish meeting of optoelectronics, OPTOEL, 3 (2003).

Google Scholar

49. de Valicourt G, Adrover MAM, Moroz ND, Pointurier Y. Semiconductor optical amplifier for next generation of high data rate optical packet-switched networks. In: Some advanced functionalities of optical amplifiers, 16 (2015). doi:10.5772/61990

CrossRef Full Text | Google Scholar

50. Nidagundi JC, Siddarama RP. High throughput structured LDPC layered decoder. In: 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET). IEEE (2017). p. 1559–63.

Google Scholar

51. Falcao G, Silva V, Marinho J, Sousa L. LDPC decoders for the WiMAX (IEEE 802.16e) based on multicore architectures, WIMAX new developments. In: UD Dalal, and YP Kosta, editors (2009).

Google Scholar

52. Nurellari E. LDPC coded OFDM and its application to DVB-T2, DVB-S2 and IEEE 802.16e. Eastern Mediterranean University (EMU) PhD Thesis (2012). doi:10.13140/2.1.3603.5361

CrossRef Full Text | Google Scholar

53. Gültekin YC, Fehenberger T, Alvarado A, Willems FMJ. Probabilistic shaping for finite blocklengths: distribution matching and sphere shaping. Entropy (2020) 22(5):581. doi:10.3390/e22050581

CrossRef Full Text | Google Scholar