Affective Brain-Computer Interfaces has achieved remarkable advancements, enabling researchers to interpret labeled EEG data accurately. However, the annotation of EEG data is time-consuming and requires substantial effort, which limits the application in practical scenarios. In this paper, we propose a self-supervised EEG representation learning framework based on temporal prediction and spatial reconstruction (EEG-TPSR) to learn EEG representations from a large amount of unlabeled data. Our model consists of two stages: 1) In the pre-training stage, we use contrastive temporal prediction and spatial reconstruction as proxy tasks, which utilize the spatio-temporal information to learn the generic representations from EEG data; 2) In the fine-tuning stage, few data is used to calibrate the pre-trained model. We conduct extensive experiments on three emotion EEG datasets. The results demonstrate that our proposed model achieves excellent performance, with over 20% relative accuracy improvement and more than 15% improvement using only 1% labeled data.