Site crashing intermittently - DB issue “Can't connect to MySQL server on 'db01'”

[defiancesports]

We have serious issue we cannot seem to figure out. Need help please. Randomly early last month we started to notice in our var/report logs around 80 times a day we are generating this same error message that is created by various pages when loaded on our site. When this error happens the Magento crash page will occur do to lack of DB connection. Here is the error:

 a:4:{i:0;s:76:"SQLSTATE[HY000] [2003] Can't connect to MySQL server on '192.168.100.72' (4)";i:1;s:3368:"#0 /mtn/nfs/www.ourwebsite.com.com/files/html/lib/Zend/Db/Adapter/Pdo/Mysql.php(111): Zend_Db_Adapter_Pdo_Abstract->_connect()
#1 /mtn/nfs/www.ourwebsite.com.com/files/html/lib/Varien/Db/Adapter/Pdo/Mysql.php(320): Zend_Db_Adapter_Pdo_Mysql->_connect()
#2 /mtn/nfs/www.ourwebsite.com.com/files/html/lib/Zend/Db/Adapter/Abstract.php(460): Varien_Db_Adapter_Pdo_Mysql->_connect()
#3 /mtn/nfs/www.ourwebsite.com.com/files/html/lib/Zend/Db/Adapter/Pdo/Abstract.php(238): Zend_Db_Adapter_Abstract->query('SET NAMES utf8', Array)
#4 /mtn/nfs/www.ourwebsite.com.com/files/html/lib/Varien/Db/Adapter/Pdo/Mysql.php(428): Zend_Db_Adapter_Pdo_Abstract->query('SET NAMES utf8', Array)
#5 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/Resource.php(179): Varien_Db_Adapter_Pdo_Mysql->query('SET NAMES utf8')
#6 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/Resource.php(110): Mage_Core_Model_Resource->_newConnection('pdo_mysql', Object(Mage_Core_Model_Config_Element))
#7 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/Resource/Db/Abstract.php(320): Mage_Core_Model_Resource->getConnection('core_write')
#8 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/Resource/Db/Abstract.php(350): Mage_Core_Model_Resource_Db_Abstract->_getConnection('write')
#9 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/Resource/Db/Abstract.php(335): Mage_Core_Model_Resource_Db_Abstract->_getWriteAdapter()
#10 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/Resource/Db/Abstract.php(360): Mage_Core_Model_Resource_Db_Abstract->_getReadAdapter()
#11 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/Resource/Db/Collection/Abstract.php(... Mage_Core_Model_Resource_Db_Abstract->getReadConnection()
#12 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/Config.php(1348): Mage_Core_Model_Resource_Db_Collection_Abstract->__construct(Object(Mage_Core_Model_Resource_Website))
#13 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/Config.php(1380): Mage_Core_Model_Config->getModelInstance('core_resource/w...', Object(Mage_Core_Model_Resource_Website))
#14 /mtn/nfs/www.ourwebsite.com.com/files/html/app/Mage.php(492): Mage_Core_Model_Config->getResourceModelInstance('core/website_co...', Object(Mage_Core_Model_Resource_Website))
#15 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/Abstract.php(208): Mage::getResourceModel('core/website_co...', Object(Mage_Core_Model_Resource_Website))
#16 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/Abstract.php(213): Mage_Core_Model_Abstract->getResourceCollection()
#17 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/App.php(608): Mage_Core_Model_Abstract->getCollection()
#18 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/App.php(466): Mage_Core_Model_App->_initStores()
#19 /mtn/nfs/www.ourwebsite.com.com/files/html/app/code/core/Mage/Core/Model/App.php(349): Mage_Core_Model_App->_initCurrentStore('', 'store')
#20 /mtn/nfs/www.ourwebsite.com.com/files/html/app/Mage.php(685): Mage_Core_Model_App->run(Array)
#21 /mtn/nfs/www.ourwebsite.com.com/files/html/index.php(89): Mage::run('', 'store')
#22 {main}";s:3:"url";s:10:"/markdowns";s:11:"script_name";s:10:"/index.php";}

In working with the hosting company we have already tried a few things. We have three dedicated servers. Front-end, back-end & DB. When we get the error shown above, the same time we get an error on the DB side.

2016-05-27 12:21:55 5949 [Warning] Aborted connection 13527446 to db: 'dbnameexample' user: 'usernamexample' host: '192.168.00.00' (Got timeout reading communication packets).

Our hosting company recommended pushing our connection timeouts from 60s, to 360s. And our max packet size from 16MB to 256MB. These changes helped with our timeout communication packets on the DB. But we are still getting "Can't connect to MySQL server" errors.

We think the next avenue to pursue is to look at why Magento is giving up so soon when the database is clearly online and available. Perhaps this is a php retry or maybe a socket timeout of some kind. We have already pushed the php socket timeouts up to higher values. Any ideas would be greatly appreciated.

[Pronto]

You probably have limited number on connections available to MySQL and this number has been exhausted. Either those connection have been left hanging (in which case restarting MySQL helps) or you need to reconfigure it to allow more.

[nagaraja]

Hi,

We have a similar problem, mysql cant connect with error code 2003

have you found any fix so far or any env variable set to adjust same?

please let me know

[defiancesports]

Hi there Nagaraja, 

This issue actually turned out to be a faulty network cable between the front-end servers and the database. (This was hard to track down). Try to see if you can run a test from your different servers if you are in a similar configuration for packet loss and see if that is your cause. 

-Nick