While Vision Language Models (VLMs) excel at understanding videos, their application to hour-long videos is hampered by two intertwined challenges: prohibitive computational costs and a qualitative failure in sustained temporal reasoning. Consequently, models often produce responses based on speculation rather than concrete visual information, leading to both factual inaccuracies and plausible hallucinations. This issue is exacerbated by existing benchmarks that, by focusing only on final answers, lack a rigorous mechanism to verify if reasoning is grounded in specific visual evidence. This makes it difficult to distinguish genuine comprehension from plausible fabrication, hindering targeted model improvement. To address these intertwined challenges of model fallibility and evaluation inadequacy, we propose a two-pronged approach. First, we introduce EV²-Bench, a large-scale benchmark that pioneers an evaluation paradigm centered on spatio-temporal visual evidence, compelling models to justify their answers with verifiable clues. Second, we propose DynamicSelect, an adaptive token compression framework that efficiently distills salient information using a dynamic semantic selector and a hierarchical compression strategy. Extensive experiments show that DynamicSelect substantially outperforms baselines on EV²-Bench and other public benchmarks. Our work provides not only a more effective method for long-video understanding but also a more rigorous evaluation paradigm, highlighting the path toward developing more robust and faithful models.