If you have System Setup > Mail Relay > SMTP Controls > Unknown Sender Domain enabled a mail will be rejected if the sending email domain does not have an 'A' or 'MX' record i.e. the domain has to be missing both an A and MX record for mail to be rejected.
You can't get around this with:
- Whitelisting IP Address
- Putting the HELO in the the Allowed HELO Hostname Restrictions
One way to get around it is to create a DNS zone and an A record for the sending hostname on the "local" DNS servers. The IP address of the A record does not have to match where the message is coming from. You just need the "A" to make it so that the sending domain is no longer "Unknown".
You could create an MX record for the domain, but this requires more steps and may affect mail delivery, so creating an A record is the recommended approach.