I can’t remember where I heard this, but I recall that the wizards were basically constantly holding back their power as part of their existence in middle earth. They could do wild stuff, but the idea is to not interfere with the progression of the world too much, much like a star trek crew