")) cat("\n") ``` ```{r, results = "asis", echo = FALSE} ci <- confint(mcrg, level = 0.90, adjust = "scheffe") xport <- print(ci, export = TRUE) cat("\n") knitr::kable(xport$summary, align = "r") for (a in xport$annotations) cat(paste(a, "

")) cat("\n") ``` [Back to Contents](#contents) ### Using weights {#weights} As we have mentioned, `emmeans()` uses equal weighting by default, based on its foundations in experimental situations. When you have observational data, you are more likely to use unequal weights that more accurately characterize the population. Accordingly, a `weights` argument is provided in `emmeans()`. For example, using `weights = "cells"` in the call will weight the predictions according to their cell frequencies (recall this information is retained in the reference grid). This produces results comparable to ordinary marginal means: ```{r} emmeans(mod4, "percent", weights = "cells") ``` Note that, as in the ordinary marginal means we obtained long ago, the highest estimate is for `percent = 15` rather than `percent = 18`. It is interesting to compare this with the results for a model that includes only `percent` as a predictor. ```{r} mod6 <- lm(inverse(conc) ~ factor(percent), data = pigs) emmeans(mod6, "percent") ``` The EMMs in these two tables are identical, so in some sense, `weights = "cells"` amounts to throwing-out the uninvolved factors. However, note that these outputs show markedly different standard errors. That is because the model `mod4` accounts for variations due to `source` while `mod6` does not. The lesson here is that it is possible to obtain statistics comparable to ordinary marginal means, while still accounting for variations due to the factors that are being averaged over. [Back to Contents](#contents) ### Multivariate responses {#multiv} The **emmeans** package supports various multivariate models. When there is a multivariate response, the dimensions of that response are treated as if they were levels of a factor. For example, the `MOats` dataset provided in the package has predictors `Block` and `Variety`, and a four-dimensional response `yield` giving yields observed with varying amounts of nitrogen added to the soil. Here is a model and reference grid: ```{r} MOats.lm <- lm (yield ~ Block + Variety, data = MOats) ref_grid (MOats.lm, mult.name = "nitro") ``` So, `nitro` is regarded as a factor having 4 levels corresponding to the 4 dimensions of `yield`. We can subsequently obtain EMMs for any of the factors `Block`, `Variety`, `nitro`, or combinations thereof. The argument `mult.name = "nitro"` is optional; if it had been excluded, the multivariate levels would have been named `rep.meas`. [Back to Contents](#contents) ## Objects, structures, and methods {#emmobj} The `ref_grid()` and `emmeans()` functions are introduced previously. These functions, and a few related ones, return an object of class `emmGrid`. From previously defined objects: ```{r} class(RG4) class(EMM.source) ``` If you simply show these objects, you get different-looking results: ```{r} RG4 EMM.source ``` This is based on guessing what users most need to see when displaying the object. You can override these defaults; for example to just see a quick summary of what is there, do ```{r} str(EMM.source) ``` The most important method for `emmGrid` objects is `summary()`. It is used as the print method for displaying an `emmeans()` result. For this reason, arguments for `summary()` may also be specified within most functions that produce `these kinds of results.`emmGrid` objects. For example: ```{r} # equivalent to summary(emmeans(mod4, "percent"), level = 0.90, infer = TRUE)) emmeans(mod4, "percent", level = 0.90, infer = TRUE) ``` This `summary()` method for `emmGrid` objects) actually produces a `data.frame`, but with extra bells and whistles: ```{r} class(summary(EMM.source)) ``` This can be useful to know because if you want to actually *use* `emmeans()` results in other computations, you should save its summary, and then you can access those results just like you would access data in a data frame. The `emmGrid` object itself is not so accessible. There is a `print.summary_emm()` function that is what actually produces the output you see above -- a data frame with extra annotations. [Back to Contents](#contents) ## P values, "significance", and recommendations {#pvalues} There is some debate among statisticians and researchers about the appropriateness of *P* values, and that the term "statistical significance" can be misleading. If you have a small *P* value, it *only* means that the effect being tested is unlikely to be explained by chance variation alone, in the context of the current study and the current statistical model underlying the test. If you have a large *P* value, it *only* means that the observed effect could plausibly be due to chance alone: it is *wrong* to conclude that there is no effect. The American Statistical Association has for some time been advocating very cautious use of *P* values (Wasserstein *et al.* 2014) because it is too often misinterpreted, and too often used carelessly. Wasserstein *et al.* (2019) even goes so far as to advise against *ever* using the term "statistically significant". The 43 articles it accompanies in the same issue of *TAS*, recommend a number of alternatives. I do not agree with all that is said in the main article, and there are portions that are too cutesy or wander off-topic. Further, it is quite dizzying to try to digest all the accompanying articles, and to reconcile their disagreeing viewpoints. I do agree with one frequent point: that there is really no substantive difference between $P=.051$ and $P=.049$, and that one should avoid making sweeping statements based on a hard cutoff at $P=.05$ or some other value. For some time I included a summary of Wasserstein *et al.*'s recommendations and their *ATOM* paradigm (Acceptance of uncertainty, Thoughtfulness, Openness, Modesty). But in the meantime, I have handled a large number of user questions, and many of those have made it clear to me that there are more important fish to fry in a vignette section like this. It is just a fact that *P* values are used, and are useful. So I have my own set of recommendations regarding them. #### A set of comparisons or well-chosen contrasts is more useful and interpretable than an omnibus *F* test {#recs1} *F* tests are useful for model selection, but don't tell you anything specific about the nature of an effect. If *F* has a small *P* value, it suggests that there is some effect, somewhere. It doesn't even necessarily imply that any two means differ statistically. #### Use *adjusted* *P* values When you run a bunch of tests, there is a risk of making too many type-I errors, and adjusted *P* values (e.g., the Tukey adjustment for pairwise comparisons) keep you from making too many mistakes. That said, it is possible to go overboard; and it's usually reasonable to regard each "by" group as a separate family of tests for purposes of adjustment. #### It is *not* necessary to have a significant *F* test as a prerequisite to doing comparisons or contrasts {#recs2} ... as long as an appropriate adjustment is used. There do exist rules such as the "protected LSD" by which one is given license to do unadjusted comparisons provided the $F$ statistic is "significant." However, this is a very weak form of protection for which the justification is, basically, "if $F$ is significant, you can say absolutely anything you want." #### Get the model right first Everything the **emmeans** package does is an interpretation of the model that you fitted to the data. If the model is bad, you will get bad results from `emmeans()` and other functions. Every single limitation of your model, be it presuming constant error variance, omitting interaction terms, etc., becomes a limitation of the results `emmeans()` produces. So do a responsible job of fitting the model. And if you don't know what's meant by that... #### Consider seeking the advice of a statistical consultant {#recs3} Statistics is *hard*. It is a lot more than just running programs and copying output. We began this vignette by emphasizing we need to start with a good model; that is an artful task, and certainly what is shown here only hints at what is required; you may need help with it. It is *your* research; is it important that it be done right? Many academic statistics and biostatistics departments can refer you to someone who can help. [Back to Contents](#contents) ## Summary of main points {#summary} * EMMs are derived from a *model*. A different model for the same data may lead to different EMMs. * EMMs are based on a *reference grid* consisting of all combinations of factor levels, with each covariate set to its average (by default). * For purposes of defining the reference grid, dimensions of a multivariate response are treated as levels of a factor. * EMMs are then predictions on this reference grid, or marginal averages thereof (equally weighted by default). * Reference grids may be modified using `at` or other arguments for `ref_grid()` * Reference grids and `emmeans()` results may be plotted via `plot()` (for parallel confidence intervals) or `emmip()` (for an interaction-style plot). * Be cautious with the terms "significant" and "nonsignificant", and don't ever interpret a "nonsignificant" result as saying that there is no effect. * Follow good statistical practices such as getting the model right first, and using adjusted *P* values for appropriately chosen families of comparisons or contrasts. [Back to Contents](#contents) ### References Wasserstein RL, Lazar NA (2016) "The ASA's Statement on *p*-Values: Context, Process, and Purpose," *The American Statistician*, **70**, 129--133, https://doi.org/10.1080/00031305.2016.1154108 Wasserstein RL, Schirm AL, Lazar, NA (2019) "Moving to a World Beyond 'p < 0.05'," *The American Statistician*, **73**, 1--19, https://doi.org/10.1080/00031305.2019.1583913 ## Further reading {#more} The reader is referred to other vignettes for more details and advanced use. The strings linked below are the names of the vignettes; i.e., they can also be accessed via `vignette("`*name*`", "emmeans")` * Models that are supported in **emmeans** (there are lots of them) ["models"](models.html) * Confidence intervals and tests: ["confidence-intervals"](confidence-intervals.html) * Often, users want to compare or contrast EMMs: ["comparisons"](comparisons.html) * Working with response transformations and link functions: ["transformations"](transformations.html) * Multi-factor models with interactions: ["interactions"](interactions.html) * Working with messy data and nested effects: ["messy-data"](messy-data.html) * Making predictions from your model: ["predictions"](predictions.html) * Examples of more sophisticated models (e.g., mixed, ordinal, MCMC) ["sophisticated"](sophisticated.html) * Utilities for working with `emmGrid` objects: ["utilities"](utilities.html) * Frequently asked questions: ["FAQs"](FAQs.html) * Adding **emmeans** support to your package: ["xtending"](xtending.html) [Back to Contents](#contents) [Index of all vignette topics](vignette-topics.html)