For a temporary marriage to end before it's time, the man must 'forgive' the remaining period. Just saying I'm leaving you is not enough.
If he has forgiven the remaining time on the marriage, then to remarry him there is no need to keep a waiting time