The shearing behavior at the interface between granular materials and saw-tooth surfaces is investigated using a three-dimensional (3D) discrete element method (DEM). Three key factors influencing the interface shearing behavior have been studied, i.e., the interface roughness, particle sphericity and initial fabric of granular material. A 3D DEM model is firstly established and calibrated by using a series of laboratory interface shear tests (ISTs). Then a series of DEM ISTs are carried out with different interface roughness and loading parameters. The macroscopic observation shows that the shear strength of interface increases as the interface roughness increases. Stress softening and bulk dilatancy of the granular material are observed in the tests featuring rough surfaces. The particle-scale analysis illustrates the formation process of localized band and its thickness is affected by interface roughness and normal stress which ranges between 4 and 5 times that of the median grain diameter. A thicker localized band is formed when the material shearing on a rough interface. After the localized band appears, the granular material structuralizes into two regions: the interface zone and the upper zone. The mechanical behavior in the interface zone is representative of the interface according to the local average stress analysis. Shear at the interface creates an anisotropic material fabric and leads to the rotation of the major principal stress. For the effects of particle sphericity and initial fabric, three types of clustered particles are designed to represent irregular particles featuring various sphericities. The results show that the shear strength of the interface increases as particle sphericity decreases. The particle-scale analysis indicates that the thickness of the localized band is affected by the initial fabric while independent of the particle sphericity. A thicker localized band is formed in the material composed of inclined and randomly placed particles. The analysis of local coordination numbers suggests that the dilation mostly occurs inside the interface zone. Anisotropy is induced by the interface shearing of the initial isotropic specimens, The direction of shear-induced anisotropy relates to the shearing direction. The initial fabric of the granular material affects significantly the evolution of the anisotropy during the interface shearing.