please let my brother Amnon go with us

The oldest son could often represent his father in Israelite culture. Amnon was David's oldest son.

Why should Amnon go with you?

David knew that Amnon was not Absalom's friend.