I've tried using system cache, varnish cache, I've re-indexed manually, I've updated all my extensions, I've cleared the cache, I've deleted the var/cache folder. Nothing helps. I've entered in new values, doesn't help.
I have it configured right in Stores -> Configuration -> Catalog -> Catalog (Product Listing Sort by: is set to Position). Even if I set it to something else, it doesn't work, it just sorts by how they are entered in (third screenshot).