I didn't, but it should match too; it's the same as my example but with the added constraint that the object should be a descendant a Set. Note that in Set[1, 2, 3] does not call Set.[](1, 2, 3)...
Sets are supposed to be unordered (any ordering is an implementation detail). If Set[1, 2, 3] matches in your example, so should Set[3, 2, 1], since Set[1, 2, 3] == Set[3, 2, 1]. We could attempt to sort the elements of the set before pattern matching, but some sets contain unsortable elements (e.g. elements of different types). If pattern matching can work correctly when using in Set[...], then maybe this would be desirable, but I'm not sure if that is possible.
To answer your question, in my opinion, yes, we shouldn't implement deconstruct using to_a.