Background: An artificial intelligence system of Faster Region-based Convolutional Neural Network (Faster R-CNN) is newly developed for the diagnosis of metastatic lymph node (LN) in rectal cancer patients. The primary objective of this study was to comprehensively verify its accuracy in clinical use. Methods: Four hundred fourteen patients with rectal cancer discharged between January 2013 and March 2015 were collected from 6 clinical centers, and the magnetic resonance imaging data for pelvic metastatic LNs of each patient was identified by Faster RCNN. Faster R-CNN based diagnoses were compared with radiologist based diagnoses and pathologist based diagnoses for methodological verification, using correlation analyses and consistency check. For clinical verification, the patients were retrospectively followed up by telephone for 36 months, with post-operative recurrence of rectal cancer as a clinical outcome; recurrence-free survivals of the patients were compared among different diagnostic groups, by methods of Kaplan-Meier and Cox hazards regression model. Results: Significant correlations were observed between any 2 factors among the numbers of metastatic LNs separately diagnosed by radiologists, Faster R-CNN and pathologists, as evidenced by r(radiologist-Faster) R-CNN of 0.912, r(Pathologist-radiologist) of 0.134, and r(Pathologist-Faster) (R-CNN )of 0.448 respectively. The value of kappa coefficient in N staging between Faster R-CNN and pathologists was 0.573, and this value between radiologists and pathologists was 0.473. The 3 groups of Faster R-CNN, radiologists and pathologists showed no significant differences in the recurrence-free survival time for stage N0 and N1 patients, but significant differences were found for stage N2 patients. Conclusion: Faster R-CNN surpasses radiologists in the evaluation of pelvic metastatic LNs of rectal cancer, but is not on par with pathologists.