Allowing only incoming connections from dynamic ip

I have a situation where a business has 4 office locations. The primary office has a phone server located on their network. Current router is port forwarding traffic to that phone server. Because the ports are forwarded we are receiving a lot of bogus traffic hitting the phone server. I would like to filter this traffic to only the other offices. The three remote offices don't have fixed ip addresses. I am wondering if I could set up a dyndns host name for each office then block all but those host names with the firewall?

I understand that the NAT is going to block all but what is forwarded so I then want the firewall to further filter the wheat from the chafe so to speak.

If you have any other suggestions as to how this could be accomplished please let me know.