I think is is 12 choose 3 minus 12, because that is the number of triangles that share a side with the polygon. Is that correct?
$\endgroup$2 Answers
$\begingroup$You're not quite right. For each side of the 12-gon, there are 12-4=8 non-adjacent vertices which you can use to form a triangle, so there are 12*8=96 such triangles. There are also those triangles which share two sides with the 12-gon, of which there are 12 (one for each vertex of the 12-gon--choose the two sides adjacent to the vertex). So there are 108 total excluded triangles, and thus the answer is $\binom{12}{3}-108 = 220-108 = 112$.
This figure shows two of the 8 possible triangles sharing the right edge of the 12-gon:
Here's another way to get the answer: For a triangle to not share an edge with the 12-gon, between each pair of triangle vertices there must be another vertex of the 12-gon. Number the vertices of the 12-gon 1 through 12. Consider two cases for a valid triangle:
Vertex 1 is not on the triangle. In this case, consider getting the 12-gon by starting with a 9-gon and putting 3 triangle vertices between the existing 9 vertices. There are $\binom{9}{3} = 84$ ways to do this.
Vertex 1 is on the triangle. In this case, consider getting the 12-gon by starting with a 10-gon consisting of vertex 1 and the 9 other vertices which are not triangle vertices. Then the remaining 2 triangle vertices can go between vertices 2 & 3, 3 & 4, ..., 9 & 10, so there are 8 places to put them. There are $\binom{8}{2} = 28$ ways to do this.
Any valid triangle falls into case 1 or 2, so there are $84+28 = 112$ possible triangles.
$\endgroup$ 3 $\begingroup$Another method, choosing the points of the triangle one by one then dividing by $6$ because each triangle has been chosen $6$ times (by choosing the vertices in different orders):
Choose the first vertex. $12$ options.
Choose the second vertex.
- case (a) a next-but-one vertex: $2$ options
- case (b) any other vertex: $7$ options
Choose the last vertex.
- in case (a): $7$ options
- in case (b): $6$ options
This gives $12 \times (2\times 7 + 7\times 6) = 672$ options, then the number of triangles is $ 672/6 = \fbox{112}$.
The reason case (a) gives more selection options for the third vertex is that the next-but-one point shares a neighbour with the first point.
$\endgroup$
