Endoscopic images often suffer from diverse and co-occurring degradations such as low lighting, smoke, and bleeding, which obscure critical clinical details. Existing methods are typically task-specific, and some require prior knowledge of the degradation type. To tackle this, we propose EndoIR, an all-in-one, degradation-agnostic diffusion-based framework that restores multiple degradation types using a single model. To provide informative task hints for guiding restoration across varying degradations, we introduce a Dual-Domain Prompter that extracts joint spatial-frequency features, coupled with an adaptive embedding that encodes shared and task-specific cues as conditioning for denoising. To avoid confusion of corrupted data distributions in the diffusion learning, EndoIR employs a Dual-Stream Diffusion design that separately processes separately encodes degraded and noise-scheduled inputs, followed by a Rectified Fusion Block that integrates them in a structured, degradation-aware manner. Furthermore, we propose a Noise-Aware Routing Decoder that improves efficiency by dynamically selecting relevant features during denoising. Experiments on SegSTRONG-C and CEC datasets demonstrate superior restoration quality of our EndoIR framework, with downstream segmentation confirming its clinical utility.