Abstract: For geotechnical stability problems, the limit equilibrium method (LEM) and shear strength reduction finite element method (SSRFEM) have been commonly used. In the traditional elasto-plastic finite element method, a large maximum allowable number of nonlinear iterations (such as 200 or 500) are often set in the SSRFEM, so that the calculation is generally time-consuming; besides, the equilibrium iteration and stress integration algorithm may probably lead to inaccurate calculation of plastic zone and stability. Based on the Hellinger-Reissner mixed variational principle and finite element method, a new shear strength reduction finite element method is proposed based on the finite element method of second-order cone programming (FEM-SOCP). In the mathematical programming finite element framework, the elasto-plastic finite element problem can be cast into a form of second-order cone programming (SOCP), and when being utilized in conjunction with the strength reduction technique, the resultant approach named SSRFEM-SOCP can be applied to geotechnical stability analysis. When being applied to plane strain problems, it is observed that SSRFEM-SOCP is reliable and efficient, and particularly the plastic zone attained by the SSRFEM-SOCP is generally smoother than that by the conventional SSRFEM method.