Tag: research

Posted on

Measurement Theory: The Basics

MEASUREMENT refers to the process of assigning numbers or labels to objects, events, or characteristics according to specific rules.  Measurement is fundamental in transforming abstract concepts and complex social phenomena into observable and quantifiable data, allowing for empirical investigation and statistical analysis.  Effective measurement enables public administrators to quantify and analyze variables, thereby:

  • facilitating the examination of relationships, patterns, and trends
  • enabling the comparison of variables across different groups, times, or conditions
  • facilitating the testing of hypotheses by providing empirical data that can be analyzed to support or refute theoretical predictions
  • informing evidence-based decision-making and policy formulation by providing measurable and reliable data

Measurement is not as simple as measuring a tangible object or substance.  It involves defining, operationalizing, and quantifying variables in a manner that ensures consistency, reliability, and validity.  We must be as precise and transparent as possible when discussing our measurements.

Conceptualization and Operationalization

The first step in empirical research is to identify CONCEPTS (sets of characteristics or specific information representing constructs or social phenomena) to study. Concepts help us organize and understand our data and can be thought of as the “building blocks” for constructing statistical models.  Concepts can be abstract/unobservable (ex: public trust in government) or concrete/observable (ex: government spending, public service delivery). Once you have identified the concept you want to research, you need to narrow your concept to a more precise CONCEPTUAL DEFINITION, or an explanation of a concept based on theoretical constructs (i.e., what theory says about the concept you are researching) outlining the concept’s meaning and the essential characteristics and/or attributes related to different aspects or components of the concept.  An effective conceptual definition:

  • provides a theoretical framework for understanding the concept, establishing a common language, understanding, and foundation for measurement 
  • is clear enough that others can understand what you mean, facilitating the replication of studies
  • distinguishes the concept being studied from other different (but related) concepts, reducing MEASUREMENT ERROR (i.e., the difference between the true value and the value obtained through measurement) and helping ensure we are using valid, reliable measures
  • is not circular

Once you have developed a conceptual definition, you need to operationalize the concept.  An OPERATIONAL DEFINITION is a clear, precise description of the procedures or steps used to measure a concept; in essence, it translates abstract concepts into measurable variables.  Operational definitions are essential for enhancing the reliability and validity of data collection and measurement and for facilitating replication.  In operationally defining the concept, you will identify one or more INDICATORS (i.e., specified observations used to measure the concept). 

Variables

A VARIABLE refers to an empirical (observable and measurable) characteristic, attribute, or quantity that can vary (i.e., take on different values among individuals, objects, or events being studied).  Variables are used to describe and quantify concepts in an analysis. 

Variables are related to indicators.  Multiple indicators may be used to represent a single variable (ex: a four-item scale, with each item associated with a distinct indicator, is often used to measure public trust in federal government).  Alternatively, a single indicator may be used to represent a single variable OR to represent more than one variable, if it measures different aspects or dimensions of those variables.  

Variables can be classified into three main types:

  • DEPENDENT VARIABLES, or the outcome variables we are interested in explaining or predicting. Sometimes called outputs or responses
  • INDEPENDENT VARIABLES, or the predictor variables we are using to explain or predict variation in the dependent variable(s). Sometimes called predictors, inputs, or features
  • CONTROL VARIABLES, or variables that are theoretically related to the dependent variables of interest, which researchers hold constant or account for to isolate the relationship between the independent variable(s) and the dependent variable; control variables are important because they help ensure that the results are attributable to the variables of interest, rather than being influenced by extraneous factors

CODING refers to the process of preparing data for analysis.  To allow for statistical analysis, variable values are coded numerically.  Some of these numerical values represent categories or labels (ex: gender); other numerical values correspond to real numbers (ex: per capita state-level spending on secondary education). 

Reliability and Validity

RELIABILITY is the extent to which a measure consistently captures whatever it measures.  Simply put – reliability is repeatability.  VALIDITY is the extent to which a measure captures what it is intended to measure.  Simply put – validity is accuracy.  

A measure CAN be reliable and not valid, but a measure CANNOT be valid if it is not reliable.  To illustrate this point, let’s consider an example.  Let’s say that your bathroom scale always displays your weight as ten pounds below your actual weight.  Is the weight displayed by the scale a reliable measure?  Yes: the scale consistently displays your weight as being ten pounds below your actual weight.  Is the weight displayed by the scale a valid measure?  No: the scale does not display your actual weight.  The weight displayed by the scale is reliable but not valid.

Levels of Measurement

The level of measurement of a variable impacts the statistics that can be examined and reported and the types of statistical analyses that can be used.  Levels of measurement are hierarchical: what applies to the lowest level of measurement will also apply to variables measured at higher levels.

Nominal

The lowest level of variable measurement is NOMINAL, in which observations are classified into a set of two (BINARY or DICHOTOMOUS variable) or more (MULTINOMIAL variable) categories that:

  1. have no direction (i.e., they are “yes/no”, “on/off”, or “either/or” in nature — there cannot be “more” or “less” of the variable) 
  2. are MUTUALLY EXCLUSIVE (i.e., there is no overlap between categories) 
  3. are EXHAUSTIVE (i.e., there is a category for every possible outcome)

When coding nominal variables, each category is assigned a numerical value to allow for statistical analysis; these numerical values have no meaning aside from what we assign them. 

Ordinal

The next level of variable measurement is ORDINAL, in which observations are classified into a set of categories that:

  1. have direction
  2. are mutually exclusive and exhaustive
  3. have intervals between categories that cannot be assumed to be equal (i.e., there is no objective basis for differentiating numerical values between categories/responses; ex: the difference between “strongly disagree” and “disagree” may vary from one person to another)

Ordinal variables are often ranking variables measured using a LIKERT SCALE. 

When coding ordinal variables measured using Likert scales, categories should be assigned numerical codes arranged in either:

  • ASCENDING ORDER, i.e., from lower numbers (less of [variable]) to higher numbers (more of [variable])
  • DESCENDING ORDER, i.e., from higher numbers (more of [variable]) to lower numbers (less of [variable])

Interval

The next level of variable measurement is INTERVAL, in which observations:

  1. have direction
  2. are mutually exclusive and exhaustive
  3. correspond to real numbers
  4. have equal intervals of measurement
  5. do not have an absolute zero

We rarely see true interval variables in public and non-profit administration.  An example of an interval variable is temperature measured by Fahrenheit or Celsius.  Fahrenheit and Celsius both lack an absolute zero (i.e., 0 degrees Fahrenheit or 0 degrees Celsius do not represent the absence of temperature).  By extension, you can have negative values (-20 Fahrenheit or -10 Celsius), and you cannot assume that 80 degrees is twice as hot as 40 degrees: 80 degrees is certainly warmer than 40 degrees, and the distance between each degree is the same, but without an absolute zero, we cannot say it is twice as hot.  

Ratio

The final level of variable measurement is RATIO, in which observations:

  1. have direction
  2. are mutually exclusive and exhaustive
  3. correspond to real numbers
  4. have equal intervals of measurement
  5. have an absolute zero

Because ratio variables have an absolute zero, we can make assumptions that cannot be made with interval variables.  For example, an income of zero dollars indicates the absence of income, and someone who is 80 years old has twice as much age (in years) as someone who is 40 years old.

Replication and Generalizability

One of the hallmarks of social science research is REPLICATION (i.e., another researcher or administrator should be able to repeat the study or experiment).  Replication is important for two reasons:

  1. it confirms the reliability of the research findings
  2. it helps us to determine the GENERALIZABILITY of the research findings (i.e., how well the results can be applied to other contexts, populations, or settings / the conditions under which the results hold; generalizability is also referred to as EXTERNAL VALIDITY)

Replication cannot occur without precise explanations of our research.

Posted on

Sources of Secondary Data: Open Data 

OPEN DATA refers to data made public for research, information, or transparency purposes.  Often, open data is published by governments, research institutions, or other organizations (such as interest groups or nonprofit organizations) that aim to promote transparency, accountability, and innovation.  The federal government maintains a comprehensive list of open data sets here. 

Posted on

Qualitative (Non-Statistical) vs. Quantitative (Statistical) Research

Non-statistical (qualitative) and statistical (quantitative) research are two fundamental approaches to conducting research, each with its own methods, purposes, and strengths.  

QUALITATIVE (NON-STATISTICAL) RESEARCH aims to explore complex phenomena, understand meanings, and gain insights into people’s experiences, behaviors, and interactions.  It focuses on providing a deep, contextual understanding of a specific issue or topic.  Data is often obtained via interviews, focus groups, participant observations, and content analysis.  Data analysis involves identifying patterns, themes, and narratives and is often interpretative and subjective, relying on the researcher’s ability to understand and articulate the meanings within the data.

QUANTITATIVE (STATISTICAL) RESEARCH aims to identify relationships or causal effects between concepts and/or phenomena.  It seeks to produce results that can be generalized to larger populations.  Data is often obtained via original data obtained through surveys or experiments and secondary data that has already been collected (such as information collected by the U.S. Census Bureau).  Analysis involves using statistical methods to analyze numerical data.  Techniques can range from basic descriptive statistics (ex: mean, median, mode) to complex inferential statistics (ex: linear regression analysis, ANOVA).  Data analysis is typically more objective and replicable, with clear rules and procedures for conducting statistical tests.

While qualitative and quantitative research have distinct differences, they are often used together in mixed-methods research to provide a comprehensive understanding of a research problem.  Qualitative research can provide context and depth to quantitative findings, while quantitative research can offer generalizability and precision to qualitative insights.

Privacy Policy Proudly powered by WordPress