First, Dumb Charlie and Smart Charlie must have left.
They leave when you tell them to go to your office for the diamonds.