Are you talking about product detail page images? If yes
Check app/design/frontend/{package}/{theme}/template/catalog/product/view/media.phtml.
If your theme is somehow similar with the default one you should find something like this:
<a href="#" onclick="popWin('<?php echo $this->getGalleryUrl($_image) ?>', 'gallery', 'width=300,height=300,left=0,top=0,location=no,status=yes,scrollbars=yes,resizable=yes'); return false;" title="<?php echo $this->htmlEscape($_image->getLabel()) ?>"><img src="<?php echo $this->helper('catalog/image')->init($this->getProduct(), 'thumbnail', $_image->getFile())->resize(56); ?>" width="56" height="56" alt="<?php echo $this->htmlEscape($_image->getLabel()) ?>" /></a>
Focus on the popWin. That one opens the image in a new page.
Remove this link and insert your logic for zoom in the same file.
If you want to use default Magento pop-up you can use windows.js
This link will help to implement pop-up
If my answer is useful, please Accept as Solution & give Kudos