Random Assignment, Random Ashimement

Woohoo! I now have better job security than I had yesterday. This evening's sample comes from Quasi-Confused in MA. who received the following lecture on experimental designs from their submission:

"We think that the definition of an experimental design is problematic and probably not strict enough. . .Aren't interventions that use randomisation on classroom or school-levels quasi-experimental designs?"

Well Reviewer X, let's have a quick review, courtesy of Yahoo! Answers
So there you have it. Randomization=Experiment, Randomization of classes=Experiment. For more information you can check out Wikipedia here

http://en.wikipedia.org/wiki/Random_assignment

Yes, even Wikipedia got one right.....

You're Doing it Wrong

While this phrase tends to be more prevalent for fans of Fark.com, it seems to be aptly applied here. Now I know that Hierarchical Linear Modeling has only been around since I was watching Danger Mouse and Heathcliff and playing my brand new Atari; however, one would think there has been enough time to get the terminology down.

Now I will say that multilevel models do tend to go by a host of names including: hierarchical linear model, generalized linear mixed models, nested models, random coefficients model, random effects model, random parameter model, HLM, variance components modeling, and probably a few others. One name it does NOT go by, however, is hierarchical regression. Hierarchical regression refers to a method of multiple regression where one can look at the unique variance an indepenent variable contributes to the prediction of some outcome, after other independent variables have been entered into a regression equation. The order of entry for the predictors are based on theory (one hopes!)

Anywho, today's "failure" comes to us from Regressing in Florida who wrote the following in his/her paper:

"As children were nested within centers and treatment occurred in small groups within each center, Hierarchical Linear Modeling (Raudenbush & Bryk, 200) was used to nest children within center"

Reviewer's comment: P6. "Hierarchical modeling should only be used for verification of a model, not for finding a model (multilevel modeling is more appropriate for this)."

I'm going to stop typing so I can cry....this one really hurts.....

2 groups + 1 DV = MADNESS

Today's interesting submission comes from SW who recently received a comment from a reviewer that is confused by math.

"Under participants, why did you use an ANOVA to compare differences between two groups? A t-test is more appropriate."

(sigh)...okay, kids, time for a review of mathematical equivalence: When the number of groups = 2, ANOVA and ttest give the same results.

Group
1	10	25	19	23	17	29	21	14	19	19.67	28.50
2	19	12	21	17	22	15	16	18	17	15.89	9.28

So, Dr./Mr./Miss/Mrs./? reviewer, there you have it. When there are 2 groups, the t-test and ANOVA are mathematically equivalent, but i guess a ttest is more appropriate because it..ummm...who knows why. Anywho, keep sending your snippets and keep noting them.

ROC Curve = Too Complex!!

I recently submitted to a nice little manuscript that several colleagues and i put together that looked at the diagnostic validity of oral reading fluency in predicting reading comprehension. I thought that a Receiver Operating Characteristic Curve analysis would be appropriate, but little did i know that a little convex curve would nearly cost publication...

"ROC is not commonly used in education and although the procedure used for generating the ROC information appears to be correct, it is too complex an analysis and is better suited and understood by the medical world"

I guess if I had inserted some unicorns, it would have distracted the evil curve :)