This dissertation makes contribution to the sparse approximation methods in relation to reproducing kernel and learning theory. The approximation methods are based on some learning algorithms with several applications including system identification, solving partial differential equations (PDEs), etc. Two rational approximation methods are proposed. One is complex support vector machine (SVM) based on Szego kernels, the other is a novel learning algorithm ¨ based on general rational orthogonal basis (Takenaka-Malmquist (TM) system). We apply the two methods to system identification respectively. SVMs and the Tikhonov regularization method are combined to numerically solve the Dirichlet problem and inversion of heat conduction, in which the kernels are newly developed. The thesis is structured as follows. Chapter 1 is an introduction. We briefly introduce the background of approximation methods, system identification and numerical PDEs. In Chapter 2, a brief introduction of Core adaptive Fourier decomposition (AFD) is given. In Chapter 3, we discuss generalization bounds for complex data learning which serves as a theoretical foundation for complex SVM. Draw on the generalization bounds, a complex SVM approach based on the Szego kernel of ¨ H2 (D) is formulated. It is applied to the frequency domain identification problem. In Chapter 4, a novel learning algorithm based on discrete TM system is presented. Unlike traditional identification method, the algorithm randomly select the parameters of TM system. Theoretical foundation is established in this chapter. After that, in Chapter 5, after viewing a maximal-energy AFD in computing the Hilbert transform, I propose a complex SVM algorithm to compute the Hilbert transform. In Chapter 6 and 7, we combine SVMs with the Tikhonov regularization method to numerically solve the Dirichlet problem and inversion of heat convolution, respectively. In Chapter 8, conclusions are given