We describe a new tracking and predicting scheme applied to a lab-made ping pong robot. The robot has a monocular vision system comprised of a camera and a light. We propose an optimized strategy to calibrate the light center using the least square method. An ellipse fitting method is used to precisely locate the center of ball and shadow on the captured image. After the triangulation of the ball position in the world coordinates, a tracking algorithm based on a Kalman filter outputs an accurate estimation of the flight states including the ball position and velocity. Furthermore, a neural network model is constructed and trained to predict the following flight path. Experimental results show that this scheme can achieve a good predicting precision and success rate of striking an incoming ball. The robot can achieve a success rate of about 80% to return a flight ball of 5 m/s to the opposite court. We describe a new tracking and predicting scheme applied to a lab-made ping pong robot. The robot has a monocular vision system comprised of a camera and a light. We propose an optimized strategy to calibrate the light center using the least square method. An ellipse fitting method is used to precisely locate the center of ball and shadow on the captured image. After the triangulation of the ball position in the world coordinates, a tracking algorithm based on a Kalman filter outputs an accurate estimation of the flight states including the ball position and velocity. Furthermore, a neural network model is constructed and trained to predict the following flight path. Experimental results show that this scheme can achieve a good predicting precision and success rate of striking an incoming ball. The robot can achieve a success rate of about 80% to return a flight ball of 5 m / s to the opposite court.