Interesting project! I took your idea and attempted to make a holder for a 2.5mm allen key to fit my new Hychika mini electric screwdriver, because I use M3 hex head bolts and annoyingly they are H2.5 and the screwdriver comes with H0.9, H1, H2 and H3. Aaaargh!!! I've ordered an H2.5 (Hex 4mm screwdriver end) from Aliexpress but that's months away, so I had a stab at printing one. Total fail printing a 4mm H2.5 complete bit because while 4mm hex comes out fine on a 3D printer, my Elegoo Neptune3Pro can't quite manage the H2.5 tip, it's pretty much a cylinder rather than hex. Again with the AAARGH! And designing an allen holder like yours for a 4mm chuck leaves the joint where the key exits the holder so weak that it falls apart at the slightest twist. I've bulked up the design and it's up for testing today. If it succeeds, then it would be fun to try to make it attract into the magnetic chuck.
So, I'm wondering just how much ferrous metal needs to be on the back end of the allen holder to give sufficient magnetic attraction to hold it in the chuck. My first thought was either a little screw or a small nail/tack. Glueing a patch of metal to the back end of the holder carries the risk that if it detaches then you've got a bit of metal stuck in your magnetic chuck which will be very hard to remove. The advantage of a screw (screwed into the end) or a nail (glued into the end) is that it's much less likely to detach. So I guess the next test is to see how big a screw/tack/nail needs to be to get a positive attraction to the magnetic chuck :o)