Human cognition excels at transcending sensory input and forming latent representations that structure our understanding of the world. Although Large Language Models (LLMs) can produce chain‐of‐thought reasoning, they lack a principled framework to capture latent structures and model uncertainty, especially in compositional reasoning tasks. In this work, we explore for the first time how to bridge LLMs with probabilistic graphical models (PGMs) to address LLM reasoning under uncertainty. To this end, we introduce Verbalized Probabilistic Graphical Modeling (vPGM), a LLM-based Bayesian framework that (i) guides LLMs in following key principles of PGMs through natural language and (ii) refines the resulting posterior distributions via numerical Bayesian inference. Unlike many traditional probabilistic methods requiring substantial domain expertise, vPGM bypasses expert‐driven model design, making it well‐suited for scenarios with limited assumptions. We evaluated our model on several compositional reasoning tasks, both close-ended and open-ended. Our results indicate that the model effectively enhances confidence calibration and text generation quality.