So now, if it is displeasing to you

"So if you do not want me to continue going"

with the leaders of Balak

"with the leaders whom Balak had sent." See how you translated "Balak" in Numbers 22:2.