If you go caving you should always obtain permission from the land owner or manager.
Most caves require a permit. In National Parks, such as the Leeuwin-Naturaliste, this would be from the Department of Parks and Wildlife (DPaW).
If you are really interested in caving you should contact the caving club or go on a trip with a commercial operator.