Now the site seems to be working fine, I can add products in my cart. If I stop the nginx of the server I'm on, I do get sent to one of the other servers (using haproxy for this) but the cart becomes empty.
I checked and the usual session directories (/tmp, var/session in magento ..) are empty, so it doesn't look like it's using the files handler, but it doesn't look like it's using memcache either.
I apologize, looks like I'm in the wrong section, it should be in magento 1. I had no idea both were still used, I'm just the sysadmin, I don't usually deal with the applications. Can someone move that thread ? I don't see how to do it myself.