Anomaly detection refers to identify the true anomalies from a given data set. We present an ensemble anomaly detection method called Relative mass and half-space tree based forest (RMHSForest), which detect anomalies, including global and local anomalies, based on relative mass estimation and half-space tree. Different from density or distance based measure, RMHSForest utilizes a novel relative mass estimation to improve the detection of local anomaly. Meanwhile, half-space tree based on augmented mass can estimate a mass distribution efficiently without density or distance calculations or clustering. Our empirical results show that RMHSForest outperforms the current popular anomaly detection algorithms in terms of AUC and processing time in the test data sets.