Abstract: The integral safety of high arch dams mainly depends on the stability of supporting rock system. There is no standard for the instability criteria for high arch dams due to its complexity. The instability criteria for the FEM stability analysis of high arch dam and its cusp catastrophe model are discussed firstly. Then the advantages of plastic zone connection criteria, catastrophe criteria of plastic strain energy and displacement are analyzed. The case shows that the plastic zone connection criteria are affected obviously by dam local factor and artificial factor, and that the displacement catastrophe criteria may produce different stability analysis results due to different monitoring points. Whereas the plastic strain energy is unique and reflected correctly for high arch dams under typical load condition, the catastrophe of the plastic strain energy of high arch dams is a better criterion.