I assume you have chunksize set to high (2048-ish):
int Mix_OpenAudio(int frequency, Uint16 format, int channels, int chunksize)
Lower chunksize to 512; good trade-off between sufficiently long buffer and quick sound response. To short and you'll hear fragments, too long and you get the sound...