They made things to do this co-ordinate based drilling...... Jig borers. Drill the hole, then bore it out to size, all on location to a tenth or maybe less.
If you DID use bushings, you could do the job with a drill press, outfitted with a bushing holder, and an x-y table. X-Y table to be as accurate as your needs require, possibly with a DRO or the like.
Of course your mill could do that too, and already has an X-Y table attached to it, all you need is the drill bushings and holder. I betcha you could make the holder, and choose a suitable size drill to use as the pilot for your holes. Then you could drill pilots for all holes, and maybe that is good enough.
If not, make more bushings for the sizes you need, and drill all the holes with them.
1601
Keep eye on ball.
Hashim Khan