I'm fairly certain you don't have to put the item back into stock.
However, the process isn't simple.
Here's the steps:
1. Find the Order.
2. Click on the Invoice.
3. View the Invoice.
4. Create a credit memo.
5. Make sure the return to stock item is deselected.
Hope this solution works out.