Recent advancements in audio generation have been significantly propelled by the capabilities of Large Language Models (LLMs). Prior research on audio LLM has predominantly focused on enhancing the architecture and scale of audio language models and leveraging larger datasets. However, in the field of audio tokenization, acoustic codecs such as EnCodec have often been utilized. These codecs were initially designed to compress audio rather than for audio language models. This misalignment means the design may not be optimal for the audio LLM. Our research highlights the shortcomings of codecs in current audio LLM, particularly their challenges in maintaining semantic integrity in generated audio. For instance, methods like VALL-E, which condition acoustic token generation on text transcriptions, frequently result in content inaccuracies and elevated word error rates (WER), stemming from the semantic misinterpretations of acoustic tokens, leading to word skipping and errors. To address these issues, we propose a straightforward but effective approach termed X-Codec, which incorporates semantic features from a pre-trained semantic encoder prior to the Residual Vector Quantization (RVQ) stage and introduces a semantic reconstruction loss after RVQ. In this way, we can preserve the structure of the prior audio codec and seamlessly integrate semantic information through a concatenation strategy. We find that enhanced semantic ability leads to more accurate content in speech synthesis tasks. And we extend the benefits to non-speech applications, including music and sound generation. Our experiments in text-to-speech, music continuation, and text-to-sound tasks demonstrate that the integration of semantic information substantially enhances the overall performance of language models in audio generation. The demo page can be found at https://x-codec-audio.github.io