This is a great design. I did put a diode in series with the one of the power lines to reduce the voltage because the max voltage for the chip is 3.6V and I'm using RSR123 Li cells. If you can find lifep04 cr123 cells, then you wouldn't need a diode.
If I were thinking properly, I would use the cells that have a built in USB port for charging. That way i could just plug in instead of removing the cell. In fact i could have just added a hole in the back for the plug. (edited)
