Cognitive diagnosis (CD), inferring student knowledge mastery based on historical response records, is crucial for personalized educational services such as adaptive practice and learning path planning. Existing CD models were built based on the assumption that student's response data is integral, overlooking the nonrandom missingness of data caused by student answering exercises selectively. This missingness generally leads to biased and incomplete observations, where confounders, such as selection bias and exposure bias, significantly undermine the accuracy of student knowledge modeling. To address missingness, we propose a Debiased Cognitive Diagnosis (DBCD) framework through the perspective of counterfactual modeling to remove exogenous confounders from the response data. Specifically, the proposed DBCD achieves debiasing for CD by applying the idea of contrastive learning to constrain the model's prediction distributions on both factual and counterfactual data. For a student, the factual data is his/her original response records, while the counterfactual data is generated by sampling the same number of exercises from all exercises of each concept through a similarity-based counterfactual sampling strategy. Considering the difficulty of directly removing the exogenous confounders for student, we devise a β-Variational Autoencoder to model their exogenous confounders within the latent representations of knowledge proficiency by leveraging exercise priors and student response patterns. Then, the learned representations are further combined with the vanilla student's ability embedding via a gating mechanism-based fusion for final diagnosis prediction of the model. Extensive experiments on three real-world educational datasets demonstrate that the proposed DBCD effectively mitigates confounders and even outperforms existing methods, thereby validating the feasibility and effectiveness of the DBCD framework.