Polarized magnetic system has a series of features, such as small volume, light weight, low power consumption, high sensitivity, quick movement and so on, widely used in the products of the military aerospace electromagnetic relay. The typical polarized magnetic system has mainly four structures and its simplified equivalent magnetic circuits model is the base of the design of the electromagnetic relay with permanent magnet. In the past, the analysis method that people used was difficult to build the unified mathematical models, which divided the work gap magnetic flux into "permanent magnet flux" and "electromagnetic flux". Based on the analysis method of the work gap magnetic voltage, this paper founds the unified mathematical model of the polarized magnetic system and divides the attractive torque into permanent magnet torque, polarized torque and electromagnetic torque through the energy balance formula. It analyses the influence of permanent magnet sizes on permanent magnet torque, polarized torque and electromagnetic torque through the energy balance formula and the conclusions can direct the design of aerospace electromagnetic relay with permanent magnet.