This is quite a large range.
If you can provide details of the DHT10 & ESP8266, ie pics would help.
If the devices you have are similar to what is linked below then you need something that generates 5V.
4xAAA’s would do that 2xCR2032s would not.
Knowing the voltage requirements of the devices would help greatly in providing advice.
Something like this would provide enough power and is 5V output, but it is quite large.
L=95.0mm W=26.5mm H=22.0
Generally power banks will not output at low current levels. They are designed to recharge a phone not act as a power supply.
Another solution would be to use a LiPO (link below), but it would need a step up circuit to provide 5V.
They provide a good output for their size.
This would provide the 3.7V to 5V boost and allows recharging when the LiPo is depleted.
Regards
Jim
