Abstract: Based upon strain gradient plasticity theory, in the first time,a theoretical analysis of stability and unstable sliding of the system composed of shear band and elastic rock mass is conducted. The relation between the shear stress and the relative sliding displacement of the shear band is got. In the strain softening stage, the slope of shear stress-relative sliding displacement can be used to assess the stability of the system.The results obtained show that if the width of elastic zone outside the shear band is large and the width of the shearing bands (or the internal length of rock material) is small,and if the descending modulus is high and the shear elastic modulus is low, then the stability of the system is bad.Lower shear modulus of test machine can lead to instability of the system.