If there is no end play or side play I would live with it. It probably means that the inside of your spindle is slightly deformed. Could be wear, stress or a ding you can't detect. I would live with it.