Issue Description
You've designed a custom-shaped window and inserted it into your wall. While the 2D representation appears accurate, the 3D is not, you notice that the cutout does not conform to the custom shape; instead, it retains a square cut.
Solution
1: Go back to the elements that make up your Custom Window.
2: Using the Slab Tool create a Slab that follows the shape of the element perfectly.
3: Select the slab and open up the settings dialogue box. Under Classifications and Properties, type in the ID field the word 'wallhole'. This will allow Archicad to use the slab to cut a hole in the wall to the exact form.
4: Select all the elements including this slab and save them as a window object.
File > Libraries and Object > Save selection as > window.
Now when it is placed in the wall the cut will follow the shape perfectly.