Recent Large Vision-Language Models (LVLMs) have demonstrated promising reasoning capabilities on text-rich images derived from charts, tables, and documents. However, the abundant text within such images potentially increases the model's sensitivity to language. Given that, it is crucial to explore how well LVLMs perform when faced with cross-lingual text-rich visual input, where the language of the text within the input image differs from the subsequent instructions. To explore this scenario, we present \textbf{XT-VQA} (\textbf{Cross}-Lingual \textbf{Text}-Rich \textbf{V}isual \textbf{Q}uestion \textbf{A}nswering), a benchmark designed to investigate how LVLMs handle language inconsistency between text in the image and the question. Derived from five existing text-rich VQA datasets and a newly collected dataset XPaperQA, our XT-VQA covers a wide range of text-rich scenarios and requires faithful recognition and comprehension of visual information despite language inconsistency. Through a comprehensive evaluation of a series of prominent LVLMs on XT-VQA, we observe a severe decline in their visual-text cross-lingual performance, even for those with multilingual capabilities. We further analyze the performance gap from a mutual information perspective and reveal that such gap is mainly attributed to cross-lingual questions failing to fully activate relevant visual information. To address this limitation, we introduce \textbf{MVCL-MI} (\textbf{M}aximization of \textbf{V}ision-Language \textbf{C}ross-\textbf{L}ingual \textbf{M}utual \textbf{I}nformation), where a visual-text cross-lingual alignment is built by maximizing mutual information between the model's outputs and visual information. Specifically, We distill the knowledge from the monolingual setting to the cross-lingual setting by minimizing the KL divergence between output distributions for cross-lingual and monolingual scenarios, where the monolingual output logits serve as a teacher. Experimental results on the XT-VQA benchmark demonstrate that MVCL-MI effectively reduces the visual-text cross-lingual performance disparity while preserving the inherent capabilities of LVLMs, shedding new light on the potential practice for improving LVLMs.