Add and remove product images per store view

[avahle]

At first some background information to understand my problem:

I'm programming a product import for two different languages in Magento 2. Therefore I'm using two different store views which present the languages. In my code I add all data for each language/store view within a foreach loop. Everything works fine apart from adding and removing product images via media gallery.

First I try to remove the current product images with the following code snippet

$mediaGalleryEntries = $mageProduct->getMediaGalleryEntries();
if ($mediaGalleryEntries) {
    foreach ($mediaGalleryEntries as $key => $imageObject) {
        unset($mediaGalleryEntries[$key]);
    }
    $mageProduct->setMediaGalleryEntries($mediaGalleryEntries);
    $this->productRepository->save($mageProduct);
}

After that I add the images

if ($imagePath) {
    $mageProduct->addImageToMediaGallery($imagePath, ['image', 'small_image', 'thumbnail'], false, false);
}

My code causes two problems:

1. the images seems to be saved "globally" and not per store view. I've set the store view before with 

   $mageProduct->setStoreId($storeId);

   and it works fine for all the other attributes I set within the import process.
2. The images don't get deleted reliably. I tried to debug the $key and there I could see, that my program just find images in the first store view. I always got the keys 0 and 1, but the code only deletes one image.
   With the next import i got three images. And if I start the import again it seems that two images are deleted, because the number of images then remains.