Tumor-infiltrating immune cells have been implicated in the tumorigenesis and progression of esophageal squamous cell carcinoma (ESCC). However, the functionalities and clinical significance of immune cells remain largely unveiled. In this study, the gene expression data from the Cancer Genome Atlas (TCGA) and Gene Expression Omnibus (GEO) were extracted. The relative infiltrating levels were estimated by single-sample gene set enrichment analysis. Some cytotoxic immune cells were attenuated, and resting cytotoxic immune cells were accumulated in ESCC. Remarkably, we also observed that infiltrating levels of macrophage M2 and resting natural killer (NK) cells were increased in nonresponders of CRT, and T cells that had anticancer activities such as activated memory CD4 and T helper 2 (Th2) cells were significantly reduced in ESCC tissues of the nonresponders. Moreover, the high infiltrations of the resting natural killer (NK) and dendritic cell (DC) were observed to result in a shorter overall survival in ESCC. Consistently, high expression of immune checkpoint genes, CTLA4 and HAVCR2, was associated with poor prognosis. Furthermore, STAT5B, a key transcription factor, as well as its target genes, involved in the regulation of T cells, was significantly downregulated in ESCC, especially subgroup I, indicating that downregulation of STAT5B might be associated with reduced T cell-mediated anticancer activity. In conclusion, the present study significantly improved our understanding of the regulatory roles of immune cells in ESCC.