Deep learning algorithm can autonomously mine the representative information which is hidden in the data and provides a new idea for damage identification of building structures. Keeping in view that a structural damage identification based on time series data has low accuracy, a new method for structural damage identification based on IASC-ASCE SHM benchmark is proposed which combines the advantages of Hilbert-Huang transform (HHT) and deep neural network. The damage signal of the Benchmark model is first analyzed by HHT. After that the obtained time-frequency graph and the marginal spectrum of the signal are used as the input of the convolutional neural network. In addition, the structural parameters of CNN model are adaptively optimized by particle swarm optimization (PSO) algorithm to ensure better performance of CNN. Compared with other traditional methods (ANN, SVM), the experimental results show that the newly proposed damage identification method has significant performance advantages. Accuracy of the proposed CNN model is improved more than 10% after getting optimized by PSO in comparison with non-adaptive CNN model. CNN model also showed better robustness against noise interference.