On 7/26/06, Bill Shipley wrote:

Hello. Douglas Bates has explained in a previous posting to R why he does

not output residual degrees of freedom, F values and probabilities in the

mixed model (lmer) function: because the usual degrees of freedom (obs -

fixed df -1) are not exact and are really only upper bounds. I am

interpreting what he said but I am not a professional statistician, so I

might be getting this wrong...

Does anyone know of any more recent results, perhaps from simulations, that

quantify the degree of bias that using such upper bounds for the demoninator

degrees of freedom produces? Is it possible to calculate a lower bounds for

such degrees of freedom?

I have not seen any responses to your request yet Bill. I was hoping

that others might offer their opinions and provide some new

perspectives on this issue. However, it looks as if you will be stuck

with my responses for the time being.

You have phrased your question in terms of the denominator degrees of

freedom associated with terms in the fixed-effects specification and,

indeed, this is the way the problem is usually addressed. However,

that is jumping ahead two or three steps from the iniital problem

which is how to perform an hypothesis test comparing two nested models

- a null model without the term in question and the alternative model

including this term.

If we assume that the F statistic is a reasonable way of evaluating

this hypothesis test and that the test statistic will have an F

distribution with a known numerator degrees of freedom and an unknown

denominator degrees of freedom then we can reduce the problem of

testing the hypothesis to one of approximating the denominator degrees

of freedom. However, there is a lot of assumption going on in that

argument. These assumptions may be warranted or they may not.

As far as I can see, the usual argument made for those assumptions is

by analogy. If we had a balanced design and if we used error strata

to get expected and observed mean squares and if we equated expected

and observed mean squares to obtain estimates of variance components

then the test for a given term in the fixed effects specification

would have a certain form. Even though we are not doing any of these

things when estimating variance components by maximum likelihood or by

REML, the argument is that the test for a fixed effects term should

end up with the same form. I find that argument to be a bit of a

stretch.

Because the results from software such as SAS PROC MIXED are based on

this type of argument many people assume that it is a well-established

result that the test should be conducted in this way. Current

versions of PROC MIXED allow for several different ways of calculating

denominator degrees of freedom, including at least one, the

Kenward-Roger method, that uses two tuning parameters - denominator

degrees of freedom and a scale factor.

Some simulation studies have been performed comparing the methods in

SAS PROC MIXED and other simulation studies are planned but for me

this is all putting the cart before the horse. There is no answer to

the question "what is the _correct_ denominator degrees of freedom for

this test statistic" if the test statistic doesn't have a F

distribution with a known numerator degrees of freedom and an unknown

denominator degrees of freedom.

I don't think there is a perfect answer to this question. I like the

approach using Markov chain Monte Carlo samples from the posterior

distribution of the parameters because it allows me to assess the

distribution of the parameters and it takes into account the full

range of the variation in the parameters (the F-test approach is

conditional on estimates of the variance components). However, it

does not produce a nice cryptic p-value for publication.

I understand the desire for a definitive answer that can be used in a

publication. However, I am not satisfied with any of the "definitive

answers" that are out there and I would rather not produce an answer

than produce an answer that I don't believe in.