In order to effectively solve the problems of low accuracy, large amount of computation and complex logic of deep learning algorithms in behavior recognition, a kind of behavior recognition based on the fusion of 3 dimensional batch normalization visual geometry group(3D-BN-VGG) and long short-term memory ( LSTM) network is designed. In this network, 3D convolutional layer is used to extract the spatial domain features and time domain features of video sequence at the same time, multiple small convolution kernels are stacked to replace large convolution kernels, thus the depth of neural network is deepened and the number of network parameters is reduced. In addition, the latest batch normalization algorithm is added to the 3-dimensional convolutional network to im-prove the training speed. Then the output of the full connection layer is sent to LSTM network as the feature vectors to extract the sequence information. This method, which directly uses the output of the whole base level without passing through the full connection layer, reduces the parameters of the whole fusion network to 15324485, nearly twice as much as those of 3D-BN-VGG. Finally, it re-veals that the proposed network achieves 96 . 5％ and 74 . 9％ accuracy in the UCF-101 and HMDB-51 respectively, and the algorithm has a calculation speed of 1066 fps and an acceleration ratio of 1, which has a significant predominance in velocity.