In some mechanical nonlinear systems, the transient motion will be undergoing a very long process and the attractor-basin boundaries are so complicated that some difficulties occur in analyzing the system global behavior. To solve this problem a mixed cell mapping method based on the point mapping and the principle of simple cell mapping is developed. The algorithm of the mixed cell mapping is studied. A dynamic model of a gear pair is established with the backlash, damping, transmission error and the time-varying stiffness taken into consideration. The global behaviors of this system are analyzed. The coexistence of the system attractors and the respective attractor-basin of each attractor with different parameters are obtained, thus laying a theoretical basis for improvement of the dynamic behaviors of gear system.