I know there are endpoints for both fetching an address by id and deleting one by id.
Listed below: (customerAddressRepositoryV1) as listed here http://devdocs.magento.com/swagger/index_20.html#/
1) To get a address
2) To delete a address
The issue with either of them is that they are not accessible by customer token. They seem to only work with Admin Token Authentication.
If a customer adds multiple addresses and selects a specific one to fetch entire details or delete it, it should be allowed from his customer access.