I just did a scan of minicpan to see how often test.pl is used and how often both test.pl and t/*.t are used. Results attached. Here's the summary.
12156 distributions scanned (sometimes multiple versions of a single dist)
2206 with only test.pl
130 with both
I've poked around in some of them that distribute both and have seen a few patterns...
1) Using it as a human-parsable benchmarking/utility script (DBI, Class::Tree)
2) Using it to run the t/*.t tests against the installed version (CGI::AppBuilder)
3) Leftover test stub from h2xs generation (Crypt::Passwd, Data::Match, WWW::Form).
4) A few tests are in test.pl and most are in t/*.t (Inline::Ruby)
#1 is the only legit use and its pretty rare.
#2 means the tests are normally run twice. Its also redundant now that we have prove.
#3 and #4 are likely simply oversights on the author's part.
So making "no_test_pl" a kwalitee test is a good idea.