Large language models (LLMs) have shown promising performance across language-related tasks. However, their potential for adversarial attacks against neural ranking models (NRMs) has not been well explored, introducing new technical challenges such as accurately capturing the ranking preferences of NRMs and controlling the imperceptibility of perturbations. We introduce a novel ranking attack framework named Attack-in-the-Chain, which tracks interactions between LLMs and NRMs based on chain-of-thought (CoT) prompting to generate adversarial examples under black-box settings. Our approach starts by identifying anchor documents with higher ranking positions than the target document as nodes in the reasoning chain. We then dynamically assign the number of perturbation words to each node and prompt LLMs to execute attacks. Finally, we verify the attack performance of all nodes at each reasoning step and proceed to generate the next reasoning step. Empirical results on two web search benchmarks show the effectiveness of our method.