Aim: To evaluate how computer-aided detection (CAD) affects observer performance in detecting lung nodules on computed tomography (CT) scans. Methods: Two hundred chest CT scans of healthy people and 80 patients' CT scans containing 96 lung nodules were retrospectively included. The CAD technique is based on sparse non-negative matrix factorization (NMF) model learning. Six observers, including two senior chest radiologists, two secondary chest radiologists and two junior radiology residents, were asked to find out the potential lung nodules on the CT scans, first without and subsequently with the assist of CAD scheme. McNemar's test was used to compare observer sensitivity without and with CAD. Results: Of the 96 nodules contained within these scans, 89 (92.7%) nodules were correctly detected by the computer, with an average 0.09 FP (false positive) annotations per CT scan. With use of the CAD scheme, the average sensitivity improved from 87.3% to 96.9% for the 6 radiologists, from 77.6% to 94.8% for junior radiology residents, from 89.1% to 97.9% for secondary chest radiologists, and from 95.3% to 97.9% for senior chest radiologists. The sensitivities of all the observers increased after reviewing the CAD annotations, however only the difference of observer D, E and F were statistically significant (p = 0.022, 0.008, < 0.001, respectively). Conclusion: Our study suggests that the CAD system can improve observer sensitivity for the detection of lung nodules in CT images.