The collets you have appear to cover all the end mill shank diameters you're likely to need; what else do you want to hold?
But "just for the heck of it" is a good enough reason too. I'd lean toward the idea of making what you need from the 4130 and using them as-is. Don't worry about hardening. As long as you're not a total ham-fist when you use them, they ought to stand up okay. If, after 20 years, they wear out, make new ones.
Or use pre-hardened 4140. It's not so hard that you can't machine it.
Try to make a living, not a killing. -- Utah Phillips
Don't believe everything you know. -- Bumper sticker
Everybody is ignorant, only on different subjects. -- Will Rogers
Law of Logical Argument - Anything is possible if you don't know what you are talking about.