I think there's far too much hysteria surrounding asbestos. It all depends upon what type it is and what you do with it.

Asbestos fibres which become airborne are what causes the problem and many forms of what I'd call "soft" material contain asbestos (such as insulation stucco or boards) and you don't want any truck with that.

Hard panels (made with cement) are OK to move if you don't break, saw, grind or drill them or do anything else to release the fibres. To be on the safe side, keeping it wet prevents any stray fibres becoming airborne. What you do with it afterwards is another matter!