"go down from the roof of the house"
It would be natural for Peter not to want to go with them, because they were strangers and they were Gentiles.