No, there is no specific restriction when it comes to using Magento Open Source (formerly known as Community Edition or CE) for stores with a high volume of transactions. I've worked with hundreds of well-established merchants and enterprises using Magento Open Source, and I've never seen one outgrow this free version of the platform.
There are cases where you'd want to engage in some customization, but there is no licensing or other restriction regarding the transaction volume.
When users compare Magento Open Source and Magento Commerce (formerly known as Enterprise Edition or EE), the tipping point is usually features and functionality. Big Companies may want to get a lot of extra modules as part of Magento's core, as opposed to adding extensions and other customizations to Magento Open Source. In that sense, it usually comes down to the scope of work, and comparing which version of Magento will offer a better long-term total cost of ownership (TCO).
