When you sort by a custom attribute in Magento 2 that is a dropdown or swatch, ElasticSearch sorts the attributes by attribute option ID and not by name.
Ideally, you would want it to reorder by the position you have assigned in you attribute and using sort_order.
Previously I was able to tap into this in the setCollection with
$attributeId = $this->_eavAttribute->getIdByCode('catalog_product', $this->getCurrentOrder());
->joinLeft(array('cpei' => 'catalog_product_entity_int'), 'e.entity_id = cpei.entity_id AND cpei.attribute_id = ' . $attributeId . ' AND cpei.store_id = 0', 'cpei.value as product_on_top')
->joinLeft(array('eao' => 'eav_attribute_option'), 'eao.option_id = cpei.value', 'eao.sort_order as sort_order')
->order('sort_order' . ' ' . 'asc');
But now this doesn't influence the order at all.
Any pointers, tips or someone else who has achieve this and willing to share?