As a complex adaptive system,artificial financial market can be built in computer.By doing so,principles of financial market and investors behaviors are researched,which is one of the most important methods to research into financial market.In this paper,ant colony algorithm on the evolution of agent investment behavior in artificial stock market is proposed to research the effects of the investors behaviors on the stability of stock market.Results show that there is dynamic balance existing in artificial stock market during the stock exchange.Stock price fluctuates within a certain range around its intrinsic value.Non-linear relationship exists among investment behavior,stock price and stability of the market.Investors with different investment behaviors trade stocks in the artificial stock market,which forms the complexity of the artificial stock market.