Abstract: This paper first divides the instability of stopes or roadways into two types:(1)instability controlled by construction(faults and joints) and (2) instability controlled by energy.Then mechanics of instability and acting factors are studied.Based on the researches,forecasting theory of instability is put forward for both types respectively.For the instability controlled by construction a key block theory is applied, which will search for and determine possible movable block bodies in the surrounding rock of a stope or a roadway. Then a reliability theory and analyzing method are established. Combining the key block theory and reliability analyses,the instability probability of movable blocks and effect on the other blocks are given. With the help of instability probability we are capable of evaluating the stability of the surrounding rock of the stope or the roadway.Forecasting method for the instability controlled by energy is based on a catastrophe theory. By means of Finite Element Method(FEM) and Respond Surface Method(RSM),a limit state equation is obtained and a reliability analysis is performed. Then stable or unstable probability of the surrounding rock can be obtained,which evaluates the stability of surrounding rock and filled body.This forecasting theory and forecasting method were applied in Xishimen iron mine. The results indicate the forecasting theory and forecasting method in this paper can more accurately forecast the instability of surrounding rock of a stope and a roadway.