Objective Adipose tissue distribution and glucose metabolism differ between men and women. Few studies have investigated sex differences in adipose tissue insulin resistance (adipose-IR). Herein, we investigated sex differences in adipose-IR in adults ranging from overweight to obese and the potential factors associated with sex differences in adipose-IR. Methods A total of 424 adults had their body mass index (BMI), adipose-IR, and sex hormones evaluated. Based on BMI, males and females were assigned to 4 groups. Results In total, males (n = 156) had higher adipose-IR than females with similar BMI levels (n = 268) (P < 0.05). Adipose-IR progressively increased from overweight to class III obesity in both males and females (all P < 0.0001); however, only in the class III obesity group was the adipose-IR significantly higher in males than in females (P = 0.025). There were significant differences in testosterone between males and females (all P < 0.01); testosterone levels were negatively correlated with adipose-IR (r = -0.333, P < 0.001) in males but positively correlated with adipose-IR (r = 0.216, P < 0.001) in females. For the logistic regression analysis, testosterone was an independent protective factor against adipose-IR in males, with an odds ratio of 0.858 (B = -0.153 [95% CI, 0.743-0.991], P = 0.037). Conclusions Adipose-IR reflects the progressive deterioration in adipose tissue insulin sensitivity from overweight to obesity in both males and females. Males with class III obesity have more severe adipose-IR than similarly obese females. The sex difference is associated with testosterone, and low testosterone levels may contribute to more severe adipose-IR in obese males.