Affects Version/s: 6.1.1 CE GA2, 6.1.20 EE GA2
Since day one, the test-integration is not really usable as the tests break in a total random manner. Exact same code ran twice, two different error codes, and the same code can pass successfully on some people's machine.
The root cause of this is that we have some junit test methods with dependence, means you have to run certain test method before others, as the some of them are preparing data for others to pick up. This is technically wrong, as each test*() method should be independent.
But to make people less confused, we should at least run all the test methods in a fixed order rather than random order, this way we could at least see the error in a consistent manner.
Junit fetches all test* methods by reflection, according to javadoc the Class.getMethods() is returning methods without any particular order, so different JDK, different build or even different run may give you different order.
This fix is sorting the test* methods in alphabetizing order by name. So if the ant format-source is not complaining your test code, the order you have in your test class is the order that is going to run with.
But again, the real proper fix is to make your test methods not depending on each other in the first place. This fix is just ruling out the random part of the test results.
An easy way to see which tests have dependence is to reverse the comparing order in AlphabetizingDescriptionComparator, whatever test methods that used to pass but break now, must have a dependence on others.
Developers who own junit tests, please try this out on your own, and fix your tests.
A quick run my side shows at least WebDAVOSXTest and SocialRelationLocalServiceTest have dependence within their test methods.