arXiv:2607.28236v2 Announce Type: replace
Abstract: Pre-trained language models have significantly improved sentence representation learning, yet their embeddings remain sensitive to textual perturbations such as synonym substitution, masking, and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embeddings by jointly optimizing contrastive and reconstruction objectives to learn perturbation-invariant representations. We evaluate the proposed framework under multiple perturbation strategies and strengths, and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbation, with the gap widening as the perturbation strength increases, highlighting perturbation-invariant learning as a promising direction for improving the stability of sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE
Abstract: Pre-trained language models have significantly improved sentence representation learning, yet their embeddings remain sensitive to textual perturbations such as synonym substitution, masking, and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embeddings by jointly optimizing contrastive and reconstruction objectives to learn perturbation-invariant representations. We evaluate the proposed framework under multiple perturbation strategies and strengths, and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbation, with the gap widening as the perturbation strength increases, highlighting perturbation-invariant learning as a promising direction for improving the stability of sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE
This article has been indexed from cs.AI updates on arXiv.org
Read the original article: