As a preface this is not a question asking for code, I am simply seeking a process, or math formula.
I'm working on a 2D game that has an isometric perspective to simulate a 3D world. The objects in the game are rectangular prisms as seen below (and they are not always uniform cubes). Please note the game or engine are not 3D.
I need to determine which objects should be drawn in front of others to maintain correct depth ordering.
Because objects are not uniform in shape, a rectangular (painter's algorithm) doesn't work. This is because there are some edge cases where the forumla does not correctly determine the object in front. This problem seems to be referred to as "z-fighting" where irregular shaped, or multi-tiled objects fight for the depth.
Due to the isometric projection, their bounding shapes appear as concave hexagons when viewed without their 2D artwork. mage of the bounds forming a hexagon:
Since this is a purely 2D game,I do not have access to actual 3D coordinates - only the 2D screen coordinates (however that can be converted into world coordinates).
The coordinates I have access to is the origin of the object (its anchor), and the bounding vertices of the object shape (which forms the hexagon shape). The origin of the object is the bottom-left point of the hexagon.
Is it possible to determine which is in front from the 6 vertices or do I need more data?
One approach seems to be to determine the min and max values of x, y , and h? as seen here. But I've not had luck with this yet.
I also believe that if I form a cube by calculate the other two points (using linear algebra), then from the cube i can find the faces which intersect thus determining which should be in front. mage of determining additional 2 points to form a cube:
The end result would be like this (yellow is showcasing which is "in front"):
I tried the painters algorithm, however it did not work for irregular shapes. I also tried a similar approach to this exmaple..