It could be a problem with sendmail configuration that is causing the delay. I would recommend using a code profiling tool like blackfire to identify where time is being spent. On production you can use New Relic too which can be very useful. 

These kinds of delays are typically throgh misconfiguration or inaccessible third party services.