As others have stated, drilling is the least accurate method of hole production. When I set up for drilling in the lathe using the QCTP, I proceed as follows. 1. Center drill the work. 2. Square tool block to the work. 3.Use the conical point of an edge finder in the drill chuck to the center drilled hole. Using the cross slide on the lathe, and the height adjustment screw on the tool block, adjust X axis and tool block height until you feel no transition between the conical point and finder body. 4. Secure tool block. That should be centered accurately enough for drilling. If the conical point shifts slightly, in relation to the body, when you secure the tool block, repeat steps 3 and 4 until you are satisfied with the result. Securing the tool block, depending upon how loosely you have it to be able to adjust, may shift when tool block is locked in place. Takes longer to type how to than actually do. Again, if you need very accurate holes, bore them. Are you using a wedge or piston type QCTP? I use the above method with a wedge type.
Paying Attention Is Not That Expensive.