Britain's decision to leave the Europe Union may lead to a net fiscal loss for the public finances, says think tank.