If you're given the script and need to determine how to represent the payment address, you need to know how to interpret it. But given the address, it's not possible to tell what it was the hash of (other than basic differences like p2sh prefix, etc)