The TMT tertiary mirror system is facing a challenge of a rigid requirement for image jitter performance. The design of the mechanical structure and the devices selection will affect the image motion jitter for the telescope observation. In this paper,a jitter test experiment has been done on a similar configuration telescope to support the design of the TMT tertiary mirror system. A method of position tracking is explored for jitter test using a 32 bit absolute angle encoder. Several kinds of disturbance sources which can affect the system jitter have been discussed. According to the experimental results,during the low speed tracking,the jitter is less than 10 milliarcsec RMS,and the design of the system can meet the jitter requirement the TMT tertiary mirror system. The NFIRAOS error rejection transfer function is given for the further correction of jitter. The TMT tertiary mirror system is facing a challenge of a rigid requirement for image jitter performance. The design of the mechanical structure and the devices selection will affect the image motion jitter for the telescope observation. In this paper, a jitter test experiment has been done A method of position tracking is explored for jitter test using a 32 bit absolute angle encoder. Several kinds of jitter test which a can cause the system jitter have been discussed. to the experimental results, during the low speed tracking, the jitter is less than 10 milliarcsec RMS, and the design of the system can meet the jitter requirement the TMT tertiary mirror system. The NFIRAOS error rejection transfer function is given for further step of jitter.