There is always going to be "flutter" due to one side of the belt being in tension and one not during operation. Any time you add an erratic load (your compressor), it only makes it worse. As stated above, there are various methods of solving it. Personally I would NOT crank up the tension, but rather do as darryl suggested with something flat almost, but not quite touching the belt.
"I am, and ever will be, a white-socks, pocket-protector, nerdy engineer -- born under the second law of thermodynamics, steeped in the steam tables, in love with free-body diagrams, transformed by Laplace, and propelled by compressible flow."