If you have access to the private key you can simply build a keypair object then, so there is no need to look into txes to find the delegated public as you can directly extract it from the object.
See Keypair in lightwallet
That is not to activate or deactivate, only the public key of the delegated account is needed for that and you can already use a custom one in Nano to do so.
Private key is needed to lock and unlock accounts harvesting (start/stop), which is not the same than activating and deactivating a remote (making an account your remote / de-attributing the remote).
I could ask users to provide the delegated private key if they import an account manually with remote harvesting active, will see what devs thinks about it, but I think the best solution would be to completely switch for a new remote. In the end it’ll be better.