Instead of the ER collet, try a TG collet instead. Say TG100, which are pretty inexpensive used. The shallower taper angle than that of the ER collets...