base

Only remove client after the hardware is teared down, so a new client
is rejected before old client is done.

Also check mUsers in all cases to make sure there are no existing client.