The increasing frequency and intensity of natural disasters demand more sophisticated approaches for prompt and reliable damage assessment. To tackle this issue, researchers have developed various methods to detect damaged areas after a disaster using benchmark datasets constructed from satellite imagery. However, challenges remain when applying the existing methods especially to unseen regions during training because the benchmark datasets have the limited coverage of diverse geographical landscapes and disaster types. We present DAVI (Disaster Assessment with VIsion foundation model), which overcomes domain disparities and detects structural damage (e.g., building) without requiring ground-truth labels of target regions. DAVI integrates task-specific knowledge from a model trained on source regions with an image segmentation foundation model to generate pseudo labels of possible damage in target regions. It then employs a two-stage refinement process, targeting both the pixel and overall image, to more accurately pinpoint changes in disaster-struck areas based on before-and-after images. Comprehensive evaluations, including a case study on a recent disaster, demonstrate that DAVI achieves exceptional performance across diverse terrains (e.g., USA, Indonesia, and Türkiye) and disaster types (e.g., wildfires, hurricanes, tsunamis, and earthquakes). This confirms its robustness in assessing disaster impact without relying on ground-truth labels and emphasizes its practical applicability.