Thursday, April 14, 2011

PIX and SMTP troubles

Twice in our current environment, we've had to face issues with PIX interfering with very specific SMTP communications. The first time, one of our users could not receive a particular type of pdf attachments from one source. After a hell of troubleshooting, it finally turned out to be that PIX ESMTP filtering was the source of the problem.

This time, there was another similar issue. Some domains with a big list of entries in the "ADDITIONAL SECTION" of the DNS response could not me sent emails. The error being

Delivery Failure
Attempted Delivery to: Unknown 
Error: 554 5.4.4 [internal] domain lookup failed 



This error though was not correct, since the DNS lookups were successful in the nslookup utility on the messaging gateway.

Digging further in pointed to the issue being related to large DNS query responses which were being rejected by PIX DNS restrictions. DNS UDP Packets of size greater than 512 bytes were not allowed by PIX. So Once that was removed, things started working well.