With caching, our hosting environments clock in with a TTFB of under 200ms for Magento 2.3.x sites. Without caching, these sites are usually closer to 2 seconds. A well-configured Varnish instance will go a long way. To quote Magento's Devdocs:
"We strongly recommend you use Varnish in production. The built-in full-page caching (to either the file system or database ) is much slower than Varnish, and Varnish is designed to accelerate HTTP traffic." - https://devdocs.magento.com/guides/v2.4/config-guide/varnish/config-varnish.html
It's also important to think about where you're testing from. If you're testing TTFB from another continent, you may see poorer scores. Tests like https://gf.dev/ttfb-test will give you a good idea of how you fair in different regions of the world.
I'll add that while the default Luma theme is perfectly fine in many respects, it's aged. It launched with Magento 2 back in 2015. It's very hard to get really good speed scores, like Core Web Vitals scores, with it. For new projects, we're seeing more users look at paid themes like Hyva.io, which is engineered to be lighter/faster, and easier to work with. (Quick slide deck on Hyva: https://www.slideshare.net/vinaikopp/hyva-from-a-developer-perspective). Rather than try to use plugins/extensions or otherwise reverse engineer and address speed issues, it's a way of starting from something inherently faster and more modern.
