https://en.wikipedia.org/wiki/Mertonian_norms

Introduction to the role of "training data" in machine learning: https://www.youtube.com/watch?v=x2mRoFNm22g

Needs-aware AI has been proposed: https://arxiv.org/abs/2202.04977 (short) and https://arxiv.org/abs/2203.03715 (longer)

Sometimes we may appreciate these, such as when the designers "values" align with our "values". Other times we may not appreciate the "values" embedded into the design when they contrast with our "values".

Here is a good video introduction to the tradeoffs of fairness and accuracy in machine learning: https://youtu.be/p5yY2MyTJXA?t=230

This is a good article on what does it mean to be "fair":

What Is Fairness? Philosophical Considerations and Implications For FairML https://arxiv.org/abs/2205.09622

Here is a review on the latest research about "Green AI": https://arxiv.org/pdf/2301.11047.pdf

https://www.oii.ox.ac.uk/people/profiles/rebecca-eynon/

https://camd.northeastern.edu/faculty/nabeel-gillani/

Looking forward into the future, this video provides some useful considerations on how AI might change our lives: https://www.youtube.com/watch?v=RzkD_rTEBYs

These are often "Generative AI".

With potential implications to our epistemologies: https://www.cigionline.org/articles/chatgpt-strikes-at-the-heart-of-the-scientific-world-view/?utm_source=cigi_newsletter&utm_medium=email&utm_campaign=chatgpt-whats-at-stake

This is a critical point and expanding here by Judea Pearl: https://www.youtube.com/watch?v=CsMV5o3hotY

If you are not familiar with Linear Regression, here is a quick introduction to the concept: https://www.youtube.com/watch?v=iIUq0SqBSH0

If you are not familiar with machine learning (ML) here is a very basic introduction: https://www.youtube.com/watch?v=f_uwKZIAeM0

If you are not familiar with this term, this video may help: https://youtu.be/dejy4PPCFHY?t=120

Human-Centered AI (HCAI) is a sub-field of AI, here is an introduction by a colleague at the Univ. of Maryland: https://www.youtube.com/watch?v=bgJZ97q3Q9w

Here is more on the GSU "proactive advising" program: https://theuia.org/blog/post/the-national-institute-for-student-success

And a video interview: https://www.youtube.com/watch?v=FdR8sI0D5i8

Here is an example of ITS being used to Georgia Tech: https://www.youtube.com/watch?v=C79x_tK1mg0

Though using the term "bias" here, this is not referring to "human bias" (e.g., prejudice) but rather statistical/computational bias. Here is a good video on the bias-variance tradeoff (statistical bias) in machine learning, then later in the article they get to "human bias", both are important: https://www.youtube.com/watch?v=EuBBz3bI-aA

Here now they talking about "human bias", which is different form the technical bias-variance tradeoff discussed before, though the two do relate to each other at times. Here is more information on "human bias" in AI: https://www.nist.gov/news-events/news/2022/03/theres-more-ai-bias-biased-data-nist-report-highlights

See the video above to understand why this is. Most tools for explaining AI outputs (a field known as Explainable AI or XAI) are thus post-hoc (ie., done after to determine what factors had the most influence on the output). Even then, interpreting the results can be complicated.

More recently (Dec. 1, 2022) the company OpenAI release ChatGPT (or GPT 3.5). You can watch this video of the workshop we did at GW on ChatGPT (Jan. 18, 2023):

https://vimeo.com/790955091

If you are unfamiliar with neural networks, this video explains them well:

https://www.youtube.com/watch?v=aircAruvnKk

If you are unfamiliar with this term, watch this video:

https://www.youtube.com/watch?v=HZGCoVF3YvM

This is the definition I use:

"Artificial intelligence is the science and engineering of making computers behave in ways that, until recently, we thought required human intelligence. "

Andrew Moore, former Dean of the School of Computer Science at Carnegie Mellon University (in Forbes, 2017)

For carefully written manuscripts, authors will have done this in the writing process already. Instructional designers, for instance, routinely do this and it becomes habit.

Correlation between Pre-Class Engagement and In-Class Engagement. In other words, those who were engaged Pre-Class were also typically engaged In-Class. For every 1 point increase in "engaged Pre-Class" there was a 0.616 increase in "engaged In-Class".

Here is a short review on interpreting correlations (sorry about the ads on the page): https://www.dummies.com/education/math/statistics/how-to-interpret-a-correlation-coefficient-r/

Means (M) and Standard Deviations (SD)

Reminder on interpreting the p-value:

https://www.youtube.com/watch?v=eyknGvncKLw&t=8s

Though statistically significant, a correlation of 0.27 is fairly low and may not represent any practical significance.

r = correlation

Correlations go between -1 and 1, with 0 being no correlation.

Here is a good article describing how to read a regression table: https://www.statology.org/how-to-read-and-interpret-a-regression-table/

These are just multiple ways that researchers can confirm that their data meets the underlying assumptions required in order to do the regressions. Here is a good overview of the basic assumptions: https://pareonline.net/getvn.asp?n=2&v=8

Chi Square

Here is a good video describing Pearson Chi Square, which is one of the most common "goodness-of-fit indices" and it thereby explains the basic concept:

https://www.youtube.com/watch?v=8CFIZ7YtGWI

Video on MLE:

https://www.youtube.com/watch?v=XepXtl9YKwc

And for those who want to get a little more technical:

https://www.youtube.com/watch?v=BfKanl1aSG0

Refer back to Figure 1 to visualize the paths (the arrows between the boxes). This is a fairly simple path model and it is easy to interpret, but they can get more complicated with more arrows going in different directions between variables.

Video on effective sizes (including Cohen's recommendations):

https://www.youtube.com/watch?v=rL32SjaHq2A

Here is a video overview of multiple regression. It builds on the previous video on simple linear regression (start with this video if regression is a new topic for you):

Multiple Regression: https://www.youtube.com/watch?v=dQNpSa-bq4M

Linear Regression: https://www.youtube.com/watch?v=ZkjP5RJLQF4

Video on what is ANOVA:

https://www.youtube.com/watch?v=ITf4vHhyGpc

A two-variable correlation

https://www.statisticshowto.datasciencecentral.com/bivariate-analysis/

The extent to which we believe that students can make accurate estimates about their growth from the beginning of the semester to the end of the semester should be considered as we later look at the data on this variable.

Again, this would better as two questions.

Typically researchers will not use "and" in a question since it indicates that you are asking two questions. For example, if a participant here used the pre-classes lectures but not the other materials, should they mark yes or no. It would be better asked as two questions (one for materials and one for lectures).

While it is somewhat common for researchers to select only a subset of questions from a validated instrument, it should be noted that the instrument was only valid in the form that was validated. Variation on the validated instrument may or may not be valid, and a separate study should be done to validate the subset of items to see if they are still valid (and reliable). In this study they do provide the "internal consistency" scores for the items as used, but this is more about reliability (i.e., does it consistently measure what you want to measure) and not as much about validity (i.e., is it accurately measuring the desired construct).

Note again, they modified an instrument.

Cronbach's alpha = measure of internal consistency of items

It is not clear why this is not labeled with the type of coefficient (number) that it is. It is likely the Cronbach's alpha (the most common measure for internal consistency) but later on the page they describe another coefficient as the Cronbach's alpha, thus we are left to wonder if these are or are not similar.

This looks to be an editing error since the value should between 0 and 1 (e.g., 0.7).

Video on internal consistency:

https://www.youtube.com/watch?v=vnP4KTJHYRs

Note that this is not their actual learning outcome, but what they thought that they would get.

All data was collected at the end of the semester from the students. This is not necessarily a problem, but when reading the results we should keep in mind that this captures where they are at the end of the semester and not at the beginning (for example, questions about their engagement would be attached to the engagement at the end of the semester and not at the beginning or middle of the semester).

It appears that this is an estimate based on instructor experience rather than data actually collected from the students. Given that the data on student use would be in Blackboard (the LMS) it would have been good for them to include the actual time students spent pre-class engaging with the materials.

A physics and a chemistry class were used in the study, both used a FL approach. There was no comparison group of students (ideally randomly assigned) taking the same courses but using a FL approach.

Research questions are an alternative to making hypotheses, though based on the research literature generally researchers should make testable hypotheses. Typically they will leave it at research questions if (a) there is not enough supporting past research to make a good estimate of what is expected, or (b) they haven't done enough background research and therefore can't make a good estimate of what is expected.

Figures are often quite useful for understanding what the researcher is doing or proposing, so take time to study them.

This is an important statement since the authors describe how they are going to define "engagement" within the context of this study.

Is FL a pedagogy or tool? You can decide:

https://en.wikipedia.org/wiki/Pedagogy

Here is a video introducing what is "epistemology":

https://www.youtube.com/watch?v=X3IcbRNQR4c

Again, though technically accurate this may be a little too strong of a statement given the mixed results of FL.

As mentioned in Molnar (2017), these finding have actually been quite inconsistent. So while it is true that some studies have documented improvements, others have not.

This is a good example of why it is important to look at the references linked to citations. Molnar's study was not a broad examination of the "numerous reports of improved academic performance and enhanced learning". Rather Molnar actually looked mostly at student perceptions of FL learning. Molnar did include some information on grades, but only for part of the sample and with limited control conditions. So while Molnar's article may generally support that some research has shown positive impacts, it balances that with "Performance measures between traditional and flipped classrooms have also generated inconsistent findings." In the end, after looking at Molnar, I do not think that I agree with the author that Molnar's article is a citation for this statement (not that the statement may not be true, but this conclusion was not the results of Molnar's research).

This section is a standard element for the British Journal of Educational Technology, but it is not common to most research journals. Since the statements do not include citations, you will want to cross check them with statements in the article to determine if you agree (through the citations) that the statements are warranted.

n squares (or eta squared) is a statistical tool for examining the effect size of a result. The effect size is an indicator to readers distinguish between results that are statistically significant and those that are practically significant, though in the end the practical significance (or utility) is left to the interpretation of the reader. For example, a difference with a substantial effect size may have practical significance for teachers in urban schools, but at the same time have little practical significance to those in rural schools, based on other characteristics of the study (such a participant profiles, the technologies used, capacity of internet access, etc.).

This video gives a good explanation. https://www.youtube.com/watch?v=RkbmA6WszTo

Depending on the topic and discipline, self-citation (i.e., referencing your own work) can be acceptable. But readers would want to note if the author(s) reference primarily their own work since this could be an indicator of potential bias in the article. Though in some cases, very few researchers are working on a given topic and there are not many others to cite... thus there is no clear "rule" on how much self-citation is acceptable.

Reference sections are very important to science. Not only do they allow readers the opportunity to review (and interpret for themselves) the research being cited by the authors, but they also give readers a quick way to discover what influential articles are guiding the researchers. Many seasoned readers of research will start by skimming the References before going back to read the article -- as a preview of coming attractions.

Researchers should also acknowledge who is paying for their research. It is not that research paid for by group's with special interest is "bad" research, but readers should be aware of any potential connections (similar to the conflict of interest statement above).

In this section the authors should indicate if there are conflicts that might influence how people interpret their research (for example, if research on lung cancer is being done by researchers who are owners of a vape company).

Researcher will commonly identify limitations to their research and these are important for readers since they may influence how you interpret the findings and implications.

This statement gets back to why it would be important for the researchers to monitor and measure the "use" of the mobile technologies of individual teachers in this research.

Be sure to note the legend below the table for the meaning of , , and . These are rather routine and typical, but sometimes authors will use different p-values.

More accurately "...no statistically significant difference..."

Reminder video on how to interpret the p-value

https://www.youtube.com/watch?v=eyknGvncKLw

This video does a good job of describing how ANOVA calculations are made, and how the F test is calculated (including what the 1 and 28 would represent from this example).

https://www.youtube.com/watch?v=-yQb_ZJnFXw

ANOVA are statistical tests for differences in variation between groups, or in this case pre-intervention measures and post-intervention measures. In other words, rather than just comparing the mean difference (3.03 vs 2.74), the ANOVA uses the variation (from the mean) for individual scores.

With the posttest, we see that the math and science teachers are more in agreement (i.e., less variation and a lower standard deviation) than they were at the time of the pretest (i.e., dropping from .98 to .70).

The "perceived ease of use" has a greater standard deviation than most of the other scores, and this illustrates that for math and science teachers there was greater variation in how easy they found the mobile devices to use (i.e., more spread in their individual scores).

SD = Standard Deviation

As a reminder, here is a short introduction to SD.

https://www.youtube.com/watch?v=MRqtXL2WX2M

As with "use" above, "adoption" here is not defined and no data is provided on actual teacher use. Since there is no control group (i.e., without mobile devices) we don't really know with that data if levels of adoption influenced levels of anxiety, or if some other variables were more influential on levels of anxiety.

The "use" of the mobile devices is essential to this study and yet it is not "operationally defined". In other words, you want the researcher to define how they operationalize the key variables in their study. In this case, how will know if teachers actually used the mobile devices? How will they measure this? How will discern differences? For example, one teacher may use the devices on a daily basis and another may use them once a week -- are these measured and considered equivalent (i.e., do both cases fall into the definition that the teachers "used" the mobile devices)? This would be important to our interpretation of the research findings.

It would also be useful for the researchers to include what was taught in these workshops, number of hours of training, and participation rates among teachers. Since the study is quasi-experimental (i.e., no control group) then as readers we would benefit from additional details on what interventions the teachers participated in during the study.

It is not clear if the research got consent of the principal and the teachers, or just the principal. Typically, all participants have to give consent to participate in a study.

You can watch this for more information on why informed consent is important to ethical research: https://www.youtube.com/watch?v=BXQHDCWSt-Q

It is important to note if the researchers made any changes to the instruments that they are using in the study. It is common for researchers to make minimal changes in order for the questions to make sense to the participants, but it is critical that these do not go so far as to change the validity or reliability of the instrument. Unfortunately, researchers do often go to far and in this case they used a mix of instruments together. This is not necessarily "wrong", but as the reader we do have to question if this has influenced the validity and reliability of the original instruments -- thus compromising the results of the current study. There is typically not clear answer to this question, as in the current paper. But as the reader you should be aware of the potential problems that may stem from altering an instrument from its original state (i.e., the state was tested for validity and reliability).

Here is an overview on validity and reliability of instruments: https://www.youtube.com/watch?v=O4FvB-W4Siw

Likert-type scales are those that you are commonly used to completing where 1 might indicate that you disagree and a 5 might indicate that you disagree. There is an interesting history to these scales and you can read more about them at on Wikipedia - https://en.wikipedia.org/wiki/Likert_scale

"n" refers to the sample size. In this case there were 29 language and humanities teachers among the 62 total teachers in the study.

Pre - Post designs are common in research, with the Pre being a measure taken before the intervention and Post being the application of the measure after the intervention. For control group studies, the control group would also have the Pre and Post measures but without an intervention in-between.

Sample sizes are typically important since they relate to the "power" of a quantitative research study. Here is a quick overview.

https://www.youtube.com/watch?v=TrbVti5Wxlg

Be sure to note demographics of participants in the research. Here, there location (i.e., Hong Kong) might be influential in how you interpret, and potentially apply, the results. Other demographics, such as age, gender, etc. may also be worth noting.

Studies usually present the hypothesis in the "alternative hypothesis" format, rather the "null hypothesis" format, even though the later is the one that is actually tested. Here is a review of the two types.

https://www.youtube.com/watch?v=WtdiMUwWX0k

If you don't recall why scientists use the null hypothesis, here is a review.

https://www.youtube.com/watch?v=-X8Xfl0JdTQ&t=309s

Wiki description of ICTs in Education, with lots of example use cases.

https://en.wikibooks.org/wiki/ICT_in_Education/The_Uses_of_ICTs_in_Education

It is important to look at the cited references for context. For example, you will note the Evan's research in 2008 looked at podcast use in higher education. Thus, when the researcher here is discussing "using devices in classrooms" the evidence they are applying is related to higher education, and not K12 education -- this may, or may not, influence your interpretation.

Introductory video to quasi-experimental research designs (i.e., studies without control groups).

https://www.youtube.com/watch?v=Ji_osZc7z5Q

Short video introduction to Repeated Measures ANOVA

https://www.youtube.com/watch?v=iErm-ksxguY

ANOVA = Analysis of Variance

ANOVA is a statistical technique used to compare the variations found among two or more groups.

Resources: Wikipedia

Article on what is included in a good research article abstract.

A DOI is an unique Digital Object Identified. They get assigned to articles so that they can be tracked and easily located.

ORCID is a non-profit organization that offers unique identifiers for researchers to automatically link together their contributions.

https://www.parsingscience.org/wp-content/uploads/2019/03/ParsingScience045-MacDonald.mp3#t=00:07:20

https://www.youtube.com/watch?v=CiGFRLCC-Ao&feature=youtu.be

Wikipedia description

https://youtu.be/FJPTnxDgQgc

JUST CONCEPT FOR THIS -- DO NOT USE

R Jupyter Notebook with code for this analysis: https://rnotebook.io/anon/fa877c0366dc965d/notebooks/DRAFT%20OF%20CONCEPT.ipynb

Or R Jupyter Notebook with Binder: https://github.com/matthewfeickert/R-in-Jupyter-with-Binder

Null Hypothesis Significance Testing (NHST)

Defining technology use:

https://www.parsingscience.org/wp-content/uploads/2019/04/ParsingScience047-Orben.mp3#t=00:21:47

Doing SCA:

https://www.parsingscience.org/wp-content/uploads/2019/04/ParsingScience047-Orben.mp3#t=00:14:59

Summary of SCA

https://www.parsingscience.org/wp-content/uploads/2019/04/ParsingScience047-Orben.mp3#t=00:14:59

Discussion of forking paths:

https://www.parsingscience.org/wp-content/uploads/2019/04/ParsingScience047-Orben.mp3#t=00:11:49

Andrew Gelman unpublished paper on the topic: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf

Story of their collaboration:

https://www.parsingscience.org/wp-content/uploads/2019/04/ParsingScience047-Orben.mp3#t=00:30:50

Supplemental tables:

https://static-content.springer.com/esm/art%3A10.1038%2Fs41562-018-0506-1/MediaObjects/41562_2018_506_MOESM1_ESM.pdf

CSV data for Supplemental Table 3:

https://static-content.springer.com/esm/art%3A10.1038%2Fs41562-018-0506-1/MediaObjects/41562_2018_506_MOESM3_ESM.csv

Amy discussing measures of adolescent well-being:

https://youtu.be/liw5AIcny5k?t=792

2015 article proposing SCA:

https://repository.upenn.edu/marketing_papers/378/

Millenium Cohort Study Variable Guide:

https://osf.io/hskjw/

Amy's background:

https://www.parsingscience.org/wp-content/uploads/2019/04/ParsingScience047-Orben.mp3#t=00:01:05

https://www.youtube.com/watch?v=tH7bVlGNjWU

https://www.parsingscience.org/wp-content/uploads/2019/03/ParsingScience045-MacDonald.mp3#t=00:21:40

https://www.parsingscience.org/wp-content/uploads/2019/03/ParsingScience045-MacDonald.mp3#t=00:17:00

https://www.parsingscience.org/wp-content/uploads/2019/03/ParsingScience045-MacDonald.mp3#t=00:12:56

https://www.parsingscience.org/wp-content/uploads/2019/03/ParsingScience045-MacDonald.mp3#t=00:10:10

https://www.parsingscience.org/wp-content/uploads/2019/03/ParsingScience045-MacDonald.mp3#t=00:05:48

What is an aurora:

https://www.parsingscience.org/wp-content/uploads/2019/03/ParsingScience045-MacDonald.mp3#t=00:03:19

https://www.parsingscience.org/wp-content/uploads/2019/03/ParsingScience045-MacDonald.mp3#t=00:00:41

https://www.parsingscience.org/wp-content/uploads/2019/03/ParsingScience045-MacDonald.mp3#t=00:21:41

Yes, reduce poor decisions -- but that isn't the same as guaranteeing good decisions.

We must still realize that we probably can't incorporate all of the perspectives, so some will get left out.

Do we have to view through each discipline? Or is it better for us (as interdisciplinary researchers) to question, listen, interpret, and communicate what the experts in each discipline see when they view it through their lens. Me looking through the lens of a biologist isn't of much good, but I can ask questions and listen, and then try to give interpretations.

This is key -- especially as the complexity of the problems is realized.

Though, getting a "complete" understanding is next to impossible with many of the complex challenges we want to investigate. Even interdisciplinary will remain partial since it can't possibly go as deep into the all of the specifics for each sub-discipline.

Influential professors, courses, readings? Parents?

Can we really change our position, or do we just become more aware of the positions of others?

What are current examples of research "elephants"? AI? Poverty?

"partial understanding" seems key to this. It is not that any perspective embodies all the necessary dimensions, but that each is missing something. For instance, the sociologist will be missing somethings that the lawyer might pick up.

Hopefully we can come up with some examples to discuss in class.

This is an odd statement, esp. since the author gives no context for it.

Which may lead us to discount the knowledge generated in other fields. Or even to downgrade the knowledge of our fields.

Related to the word Epistemology: https://en.wikipedia.org/wiki/Epistemology

The book below on "epistemic cultures" in science actually led to this reading being considered, so I should mention it here: https://www.amazon.com/Epistemic-Cultures-Sciences-Make-Knowledge/dp/0674258940 It is not a great book, but a great concept. We will discuss it in class in a few weeks.

If you are benchmarking others then you are always playing a game of catch-up. Remember, those that you are benchmarking are not sitting still -- they are innovating and move forward. So even when you catch up to where they are today, they are still several steps ahead of you. Not that benchmarking is always bad, you just have to balance it with innovations of your own.

Tacit knowledge is that which has not be codified (e.g., written down, archived, shared, etc.).

Here is a helpful link on different types of knowledge: http://www.knowledge-management-tools.net/different-types-of-knowledge.html

And here is the definition from Webster's Dictionary: http://www.merriam-webster.com/dictionary/knowledge

Here is a link to sample of articles by Peter Drucker: http://www.druckerinstitute.com/peter-druckers-life-and-legacy/a-drucker-sampler/

It did work. Do you now see this?