This phrase moves the story ahead to when Zechariah's service was ended.
Zechariah did not live in Jerusalem, where the temple was located. He traveled to his home town.