Optimized Pulse Shaping for Intra-channel Nonlinearities Mitigation in a 10 Gbaud Dual-Polarization 16-QAM System Benot Chtelain1, Charles Laperle2, Kim Roberts2, Xian Xu1, Mathieu Chagnon1, Andrzej Borowiec2, Franois Gagnon3, John C. Cartledge4, and David V. Plant1 1McGill University, Montreal, Quebec, Canada, H3A 2A7 2Ciena Corporation, Ottawa, Ontario, Canada, K2H 8E9 3cole de technologie suprieure, Montreal, Quebec, Canada, H3C 1K3 4Queens University, Kingston, Ontario, Canada, K7L 3N6 Email: benoit.chatelainmail.mcgill.ca Abstract: An optimized pulse shape is shown to reduce intra-channel nonlinear effects in a 10 Gbaud dual-polarization 16-QAM EDFA-amplified system without optical dispersion compensation. OCIS codes: (060.2330) Fiber optics communications; (060.4080) Modulation 1. Introduction Intra-channel nonlinear distortion is an important source of signal degradation in optical communication systems using advanced modulation formats such as quadrature amplitude modulation (QAM). Its main impact is to reduce the maximal power that can be launched into the fiber, thereby limiting optical signal-to-noise ratio (OSNR) levels at the receiver, reducing system margins and reducing the maximum propagation distance that can be achieved. In this paper, the nonlinear tolerance of an optimized pulse and a root-raised cosine (RRC) pulse is compared in terms of bit error rate (BER) performance, for varying propagation distances and launch powers. Experimental results show that the optimized pulse considerably reduces intra-channel nonlinear effects in a long haul, 10 Gbaud dual-polarization (DP) 16-QAM system relying on erbium-doped fiber amplifiers (EDFAs) and G.652 fiber (no optical dispersion compensation). The improved nonlinear tolerance of the return-to-zero (RZ) pulse shape was recently highlighted in 1, for a 112 Gb/s DP-quadrature phase shift keying (QPSK) system without optical dispersion compensation. Comparing the RZ pulse to the non return-to-zero (NRZ) pulse, the authors reported an increase in single channel maximum propagation distance of 18%. However, the improved nonlinear tolerance achieved using the RZ pulse comes at the expense of increased spectral content. For instance, the bandwidth of a 50% RZ pulse is twice the bandwidth of the NRZ pulse or the RRC pulse with a roll-off factor () of 1. The use of a RZ pulse thus reduces spectral efficiency by a factor of two, and for a system using digital signal processing at the transmitter and receiver, at the Nyquist rate, it doubles the required bandwidth and sampling frequency of the digital-to-analog converters (DACs) and analog-to-digital converters (ADCs).（a）(b)Figure 1. Impulse (a) and frequency (b) responses of the RRC and optimized pulses.In comparison to the RRC pulse, the optimized pulse shape presented in 2 was shown to improve the nonlinear performance of DP-QPSK systems without optical dispersion compensation and to increase the maximum transmission distance by as much as 22%. The specialized pulse was obtained by numerical optimization, with the primary objective formulated to reduce its width, but constrained to have a bandwidth equal to the bandwidth of a NRZ or RRC pulse ( = 1). Therefore, the optimized pulse can be used without penalizing spectral efficiency and without using higher speed DACs and ADCs. Fig. 1 exhibits the time and frequency characteristics of the optimized and RRC pulses. It can be seen that the optimized pulse is narrower than the RRC pulse with = 1, and that its first-null bandwidth is equal to the baud rate. It is also apparent that the reduced width of the optimized pulse translates in higher energy content for frequencies between 6 GHz and 10 GHz. Through dispersion, the higher energy frequencycomponents produce a propagating pulse and a propagating waveform that is more broadened, with lower peak power excursions. Since intra-channel nonlinear distortion is proportional to peak power, the optimized pulse makes the system more tolerant to nonlinear effects. In this work and as in 25, the RRC pulse is used as the reference pulse for its ability to minimize pulse-induced intersymbol interference (ISI) and to minimize out-of-band power. 2. Experimental Setup Fig. 2 illustrates the experimental setup. The symbol sources consist of random symbol sequences of length 214 and were followed by pulse shaping finite impulse response (FIR) filters. DACs with 6-bit resolution drive the in-phase-quadrature (I-Q) modulators associated with the two polarizations. The emitting wavelength was set to 1547.715 nm. A polarization scrambler (PS) was inserted after the polarization beam combiner (PBC) in order to randomly rotate the polarization state. A fiber link consisting of 15 spans of 80 km G.652 fiber was used, together with EDFAs at each span. At the receiver, a