The dynamic model experiment of the rock filling embankment was carried out to investigate the vibration compaction mechanism. The rock filling materials were compacted by the plate-vibrated compactor, and the characteristics of the rock filling materials, such as settlement, pressure change and response waveform, were measured by the dynamic earth pressure gauge and accelerometer. Moreover, a new method for detecting the compactness of the rock filling embankment was proposed based on the maximum dry density and modulus of deformation. The results show that the process of vibration compaction includes compact, elastic deformation and loose stages, and the vibratory pressure transfers to the surroundings from the vibration center in non-linear rule. Furthermore, the test results obtained by the present method are basically in agreement with those obtained by the traditional method, and the maximum relative error between them is about 0.5%.