Microtia-atresia is a congenital malformation of the external ear, often affecting one side and being associated with severe-to-profound unilateral conductive hearing loss (UCHL). Although the impact of unilateral hearing loss (UHL) on speech recognition, sound localization and brain plasticity has been intensively investigated, less is known about the subjects with unilateral microtia-atresia (UMA). Considering these UMA subjects have hearing loss from birth, we hypothesize it has a great effect on brain organization. A questionnaire on speech recognition and spatial listening ability was administered to 40 subjects with UMA and 40 age- and sex-matched controls. UMA subjects showed poorer speech recognition in laboratory and poorer spatial listening ability. However, cognitive scores determined by the Montreal Cognitive Assessment (MoCA) and Wechsler Intelligence Scale for Children (WISC-IV) did not differ significantly in these two groups. The impact of hearing loss in UMA on brain functional organization was examined by comparing resting-state fMRIs (rs-fMRI) in 27 subjects with right-sided UMA and 27 matched controls. UMA subjects had increased nodal betweenness in visual networks and DMN but decreases in auditory and attention networks. These results indicate that UCHL in UMA causes significant abnormalities in brain organization. The impact of UCHL on cognition should be further examined with a battery of tests that are more challenging and better focused on the cognitive networks identified.