All wooden and mineral blocks can and must be removed, while the black, stone blocks are there to be the last support between your totem and the ground.
Watch out for winds though, nothing is stable when the storm is coming! The totem as well as the blocks that you need to remove are easily affected by the wind, so you'll have to move fast!
This is just the start, so there are only 40 levels, but more are coming!