(1) Background: The clinical characteristics and the prognostic factors of HIV-negative Pneumocystis jirovecii pneumonia (PJP) patients (non-HIV-PJP) with interstitial lung disease (ILD) remain unclear. Our objectives were to describe the clinical characteristics and to explore the prognostic factors of non-HIV-ILD-PJP patients. (2) Methods: The enrolled patients in this retrospective study were stratified based on the presence or absence of ILD and fibrotic ILD (FILD). The log-rank test and Cox regression models were used to analyze the prognostic factors. (3) Results: Among 378 non-HIV-PJP patients, there were 133 patients with ILD-PJP, and 70 patients were classified as having FILD-PJP. The all-cause mortality rate for the ILD-PJP group is higher than that of the ILD-PJP group (57.9% vs. 38.4%, p < 0.001). However, the all-cause mortality is similar between the FILD-PJP group and non-FILD-PJP group. Preexisting ILD (HR: 2.156, p = 0.003) and honeycomb appearance on the chest HRCT (HR = 16.3, p < 0.001) are independent survival risk factors for ILD-PJP. Non-invasive ventilation is an independent survival risk factor for ILD-PJP patients (HR = 928.56, p < 0.01) and FILD-PJP patients (HR = 33.86, p < 0.001). (4) Conclusions: Pre-existing ILD and honeycomb appearance on the chest HRCT are independent survival risk factors for PJP patients. Non-invasive ventilation is associated with poor survival for both ILD-PJP and FILD-PJP patients.