This paper addresses algorithmic fairness in skin cancer diagnosis, focusing on performance disparities across different skin tone groups. Due to the entanglement of lesion characteristics and skin tone in visual data, existing bias-suppression methods are inherently constrained. To address this, the authors propose a framework that influences image-text pairs to separate lesion conditions from skin tone. Their model comprises a shared text encoder and two specialized image encoders, which align separate image features with lesion and skin tone descriptions. By using techniques such as measuring the semantic distances between lesion and skin color in both image and text modalities, the framework achieved optimal representation alignment. Experiments on two benchmark datasets (PAD-UFES-20 and Fitzpatrick17k), which include a wide spectrum of skin tones, show that their method improves classification accuracy and fairness across various evaluation parameters. Specifically, the framework improved classification accuracy by 3.8% and reduced fairness disparity by 7.2% compared to baseline methods.