The problem was because the cron was starting the queue consumers and they are configured to run with 10000 --max-messages by default (Magento documentation says 1000).
When the cron runs it adds a bunch of background processes for the queue consumers:
$ kubectl exec -n my-store magento-75788af261-adwdc -c magento ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 18048 1932 ? Ss 14:13 0:00 /bin/bash /setup.sh
root 16 0.0 0.0 4284 588 ? S 14:13 0:00 /bin/sh /usr/sbin/apache2ctl -D FOREGROUND
root 18 0.0 0.0 274652 19168 ? S 14:13 0:00 /usr/sbin/apache2 -D FOREGROUND
www-data 20 0.9 0.2 344228 90216 ? S 14:13 1:50 /usr/sbin/apache2 -D FOREGROUND
www-data 24 0.9 0.2 344232 86344 ? S 14:13 1:41 /usr/sbin/apache2 -D FOREGROUND
www-data 59 0.0 0.2 291524 94128 ? S 14:15 0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start commonErrorManagement --pid-file-path=commonErrorManagement-magento76799dc865rglqw.pid --max-messages=10000
www-data 61 0.0 0.2 291524 94024 ? S 14:15 0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start ping --pid-file-path=ping-magento76799dc865rglqw.pid --max-messages=10000
www-data 63 0.0 0.3 317288 118348 ? S 14:15 0:05 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start productConsumer --pid-file-path=productConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data 65 0.0 0.2 293740 95408 ? S 14:15 0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start stockConsumer --pid-file-path=stockConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data 67 0.0 0.2 293752 95652 ? S 14:15 0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start productReturnUpdatedConsumer --pid-file-path=productReturnUpdatedConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data 69 0.0 0.3 293704 97224 ? S 14:15 0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start refundUpdatedConsumer --pid-file-path=refundUpdatedConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data 71 0.0 0.3 317288 118240 ? S 14:15 0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderCancellation --pid-file-path=orderCancellation-magento76799dc865rglqw.pid --max-messages=10000
www-data 73 0.0 0.3 317288 118116 ? S 14:15 0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderGetNotifications --pid-file-path=orderGetNotifications-magento76799dc865rglqw.pid --max-messages=10000
www-data 75 0.0 0.3 317288 118440 ? S 14:15 0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderResendNotification --pid-file-path=orderResendNotification-magento76799dc865rglqw.pid --max-messages=10000
www-data 77 0.0 0.3 317292 118508 ? S 14:15 0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderConsumer --pid-file-path=orderConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data 79 0.0 0.2 291524 93944 ? S 14:15 0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderUpdatedConsumer --pid-file-path=orderUpdatedConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data 82 0.0 0.3 295756 97992 ? S 14:15 0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start customerShipmentDoneConsumer --pid-file-path=customerShipmentDoneConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data 84 0.0 0.2 291656 93784 ? S 14:15 0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderLineCancelledConsumer --pid-file-path=orderLineCancelledConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data 86 0.0 0.2 291656 94968 ? S 14:15 0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start shippingAddressUpdated --pid-file-path=shippingAddressUpdated-magento76799dc865rglqw.pid --max-messages=10000
www-data 92 0.0 0.3 317292 119656 ? S 14:15 0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start serviceBusDiscover --pid-file-path=serviceBusDiscover-magento76799dc865rglqw.pid --max-messages=10000
www-data 94 0.0 0.3 317292 118524 ? S 14:15 0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start async.operations.all --pid-file-path=async.operations.all-magento76799dc865rglqw.pid --max-messages=10000
www-data 178 1.6 0.2 336144 81796 ? S 16:24 0:53 /usr/sbin/apache2 -D FOREGROUND
www-data 179 1.2 0.2 336220 83776 ? S 16:24 0:42 /usr/sbin/apache2 -D FOREGROUND
www-data 195 1.3 0.2 336160 81964 ? S 16:27 0:42 /usr/sbin/apache2 -D FOREGROUND
www-data 198 1.0 0.2 333196 74692 ? S 16:30 0:31 /usr/sbin/apache2 -D FOREGROUND
www-data 199 1.0 0.2 333196 74696 ? S 16:30 0:31 /usr/sbin/apache2 -D FOREGROUND
www-data 200 1.0 0.2 333212 74708 ? S 16:30 0:29 /usr/sbin/apache2 -D FOREGROUND
www-data 201 1.3 0.2 333140 74632 ? S 16:30 0:40 /usr/sbin/apache2 -D FOREGROUND
www-data 210 1.1 0.2 333140 74636 ? S 16:30 0:34 /usr/sbin/apache2 -D FOREGROUND
root 440 0.0 0.0 36640 2712 ? Rs 17:19 0:00 ps aux
And Kubernetes will just kill the cron pod when the timeout is hit (1 hour by default). An easy fix is running the consumers on Supervisord and run Supervisord on its own pod. Also make sure to disable the queue consumers from running on cron by adding to the env.php:
...
'cron_consumers_runner' => array(
'cron_run' => false
),
...
