This paper first presents a stochastic structural model to describe the random geometrical features of rock and soil aggregates. The stochastic structural model uses mixture ratio, rock size and rock shape to construct the microstructures of aggregates,and introduces two types of structural elements (block element and jointed element) and three types of material elements (rock element, soil element, and weaker jointed element)for this microstructure. Then, continuum-based discrete element method is used to study the deformation and failure mechanism of rock and soil aggregate through a series of loading tests. It is found that the stress-strain curve of rock and soil aggregates is nonlinear, and the failure is usually initialized from weaker jointed elements. Finally, some factors such as mixture ratio, rock size and rock shape are studied in detail. The numerical results are in good agreement with in situ test. Therefore, current model is effective for simulating the mechanical behaviors of rock and soil aggregates.