Impact Assessment of
Health Promotion Agencies

: Methods, Challenges and Insights

Contents

Introduction

This case study offers a comprehensive overview of how these organizations assess their impact, adapt to challenges, and derive insights to shape the future of health promotion.

Impact assessment plays a crucial role in evaluating the effectiveness of health promotion agencies. It helps determine if the goals of programs, policies, and initiatives are being met, while also identifying areas for improvement. These assessments guide future strategies and ensure continuous improvement in public health outcomes.

Exploring

This case study explores the methods, challenges, and insights from impact assessments used by leading health promotion organizations, including the Victorian Health Promotion Foundation (VicHealth) in Australia, the Health and Wellbeing Queensland (HWQld) in Australia, the Thai Health Promotion Foundation (ThaiHealth), the Korea Health Promotion Institute (KHEPI), and the Singapore Health Promotion Board (HPB). Each agency adopts unique frameworks and tools tailored to their objectives and contexts. By examining their approaches, this study provides valuable lessons on evaluating health initiatives effectively and addressing the complexities inherent in public health promotion.

Challenges

Health promotion agencies face common challenges such as measuring long-term outcomes, attributing impacts to specific interventions, and addressing cultural and contextual differences. Despite these challenges, innovative methodologies, ranging from macro- and micro-level analyses to equity-focused frameworks, have emerged to better capture the nuances of impact.

Comprehensive overview

This case study offers a comprehensive overview of how these organizations assess their impact, adapt to challenges, and derive insights to shape the future of health promotion. Through these insights, it highlights the importance of continuous learning, stakeholder involvement, and the integration of quantitative and qualitative measures to inform meaningful and equitable public health practices.

Methodologies
for Impact Assessment

Health promotion agencies use a variety of methods to assess the impact of their programs. These approaches are designed to measure effectiveness, identify improvements, and ensure that resources are used efficiently.

Victorian Health Promotion Foundation

VicHealth, Victoria, Australia

Contents

  • Impact & Evaluation Framework
  • Mixed methods
  • Implementation
  • Evaluation

Impact & Evaluation Framework

VicHealth’s 2023-2033 Impact & Evaluation Framework monitors population health, tracks system shifts, and identifies barriers and enablers to change. It leverages data and insights for continuous improvement, evaluates the effectiveness, reach, and equity of funded programs, and captures unintended consequences to refine future interventions. The framework consists of four interconnected impact layers that measure influence and effectiveness.

  1. Aspirations focus on long-term health, equity, and economic well-being, guiding VicHealth’s strategic vision.
  2. Systems shifts in policies, environments, connections, power dynamics and mental models within Neighbourhood & Built Systems, Commercial & Economic Systems, and Food Systems to identify impact, trends and opportunities.
  3. Systems Ways of Working assess how VicHealth engages with partners, communities, and policymakers to drive sustainable change.
  4. Vichealth-funded initiatives evaluate the effectiveness, reach, and equity of programs, capturing lessons and unintended consequences.

Mixed methods

VicHealth uses a mix of economic, data-driven, and qualitative methods to measure impact and improve programs.

  1. Return on Investment (ROI): Economic evaluations to measure the social and financial value of interventions.
  2. Population Indicators & Policy Indexes: Tracking long-term changes in health and wellbeing.
  3. Systems Monitoring & Evaluation: Using case studies, narratives, and reflective practices to assess system shifts.
  4. Minimum Data Set (MDS): Standardized data collection from funded initiatives to simplify reporting and support strategic decision-making.

VicHealth follows a structured and developmental approach to implementing and evaluating its programs, ensuring they are effective, scalable, and impactful. Below are the key elements:

Implementation

  1. Integration into Processes:
    • Established a “Learning Rhythm” to align framework activities with planning cycles.
    • Embedding “System Impact Champions” to enhance internal capacity for data collection and strategic learning.
  2. Data Collection and Analysis:
    • Documenting system shifts across initiatives to derive strategic insights.
    • Standardizing data collection through a Minimum Data Set (MDS) for VicHealth-funded initiatives.
  3. Stakeholder Engagement:
    • Consulting experts and communities for continuous refinement.
    • Promoting collaboration through partnerships and reflective practices.

Evaluation

  1. Monitoring Systems Change:
    • Tracking changes using indicators, impact logs, and case studies.
    • Focusing on shifts in neighborhood, commercial, economic, and food systems as lead indicators of change.
  2. Annual Impact Reports:
    • Providing analyses of system outcomes, population trends, and strategic opportunities.
    • Highlighting intermediate and long-term impacts while exploring unintended consequences.
  3. Continuous Learning:
    • Adopting methods like experimentation, consultation, structured reflection, and collective sensemaking.
    • Using integrative evaluation questions to align organizational and program-level assessments.

The VicHealth 2023-2033 Impact & Evaluation Framework tracks health trends, system shifts, and barriers to change, ensuring continuous improvement. It evaluates programs’ effectiveness, equity, and reach while using methods like ROI analysis, population indicators, and standardized data (MDS). Integrated into planning cycles, it fosters collaboration, strengthens capacity, and aligns initiatives with VicHealth’s vision for a healthier, fairer future.

It fosters collaboration, strengthens capacity, and aligns initiatives with VicHealth’s vision for a healthier, fairer future.

Health and Wellbeing Queensland

HWQld, Queensland, Australia

HWQld’s approach demonstrates how flexibility, community involvement, and systemic thinking can lead to more meaningful and sustainable health outcomes.

Measurement, Evaluation, and Learning Framework (MEL-F)

HWQld uses a Measurement, Evaluation, and Learning Framework (MEL-F) based on a theory of change. This framework assesses:

  1. Systems shifts: Measuring shifts in policies, practices, networks, and mindsets rather than just health outcomes. The example indicators can be tracking improvements in food policies that increase access to healthy meals in schools, encouraging schools and workplaces to adopt wellness programs, strengthening collaborations between healthcare providers, local governments, and communities, and promoting a shift in societal attitudes, such as normalizing healthy eating or reducing weight stigma.
  2. Innovative methods: Incorporating tools like Most Significant Change and Reflection Workshops.
    • Most Significant Change: This method gathers qualitative stories from participants to highlight transformative changes. For example, after launching the “Pick of the Crop” program, HWQld collected stories from schools on how students were learning to grow and enjoy vegetables, fostering lifelong healthy eating habits.
    • Reflection Workshops: These regular sessions with stakeholders, including educators and healthcare professionals, are used to evaluate what’s working and adapt interventions accordingly. For instance, feedback from workshops on obesity prevention led to refining the “Making Healthy Happen 2032” strategy.
  3. Community insights: Collecting feedback from stakeholders during design, implementation, and evaluation phases to ensure equity and transparency. In the “Logan Healthy Living” initiative, HWQld partnered with local communities to combine specialist services, general practitioners, and community organizations. This helped tailor the program to local needs, ensuring it addressed barriers like transportation and access to resources. Community feedback also shaped the GenQ Grants Program, which funds grassroots initiatives for health promotion, targeting specific groups like First Nations communities.
  4. Adaptation and agility: Using real-time data and feedback together with iterative milestones to adjust programs as needed. Programs are evaluated regularly, allowing adjustments before final outcomes are measured. For instance, the “Pick Me” campaign in retail environments tracks customer responses to healthy food promotions and adjusts messaging based on sales data. The “Podsquad” kids’ wellbeing app relies on user feedback to refine features, making them more engaging and effective.
  5. Complex determinants: Acknowledging factors such as cultural, commercial, digital and ecological influences, and explicitly defining the realistic scope of impact through accountability models. The Wellbeing Model incorporates factors such as food security and advertising influences. The “Gather + Grow” strategy aims to improve food security in remote First Nations communities by addressing systemic barriers, including supply chain challenges and cultural dietary needs.

HWQld’s approach demonstrates how flexibility, community involvement, and systemic thinking can lead to more meaningful and sustainable health outcomes. Their focus on innovation, equity, and learning offers a model for other health promotion agencies worldwide.

Thai Health Promotion Foundation

ThaiHealth, Thailand

ThaiHealth is a global leader in health promotion, known for its innovative approaches to assessing and improving the impact of its programs.

Diverse methodologies

ThaiHealth is a global leader in health promotion, known for its innovative approaches to assessing and improving the impact of its programs. By leveraging diverse methodologies such as macro-level analysis, micro-level evaluation, and Social Return on Investment (SROI), ThaiHealth addresses health challenges at both national and community levels.

  1. Macro-Level Analysis (Top-Down Approach): ThaiHealth assesses the overall impact of its initiatives on national public health indicators. This approach focuses on changes in health trends and outcomes across the country. Tobacco Control Policies are outstanding examples. ThaiHealth implemented comprehensive measures such as public smoking bans, graphic health warnings on cigarette packs, and anti-smoking campaigns. With macro-level analysis, they can track a nationwide reduction in smoking rates over time; for instance, smoking prevalence decreased significantly, reducing tobacco-related diseases and associated healthcare costs. These results highlight how ThaiHealth’s policies contributed to improved public health and economic savings.
  2. Micro-Level Analysis (Bottom-Up Approach): At the community level, ThaiHealth evaluates specific projects to understand their localized impact. Data is collected from individual interventions and aggregated to measure broader outcomes. Community Health Programs are excellent examples. In a rural district, ThaiHealth supported a program to increase physical activity and reduce alcohol consumption, organizing community exercise programs and conducting health screenings and surveys before and after the intervention. Outcomes included improved physical fitness and reduced alcohol use among participants, demonstrating the program’s effectiveness in promoting healthier lifestyles.
  3. Social Return on Investment (SROI): This method is used to quantify the social, economic, and environmental benefits of its programs relative to the financial investment. ThaiHealth’s anti-smoking programs, which included public education campaigns and nicotine cessation support, were analyzed using SROI. The results showed that for every baht invested, the program generated 18 baht in social benefits, including reduced healthcare costs from fewer smoking-related diseases, economic gains from increased workforce productivity, and societal value from improved health and quality of life.
  4. Stakeholder Involvement: ThaiHealth ensures its assessments reflect the experiences of those directly affected by its programs by involving stakeholders in the evaluation process. For a school-based nutrition program, ThaiHealth engaged teachers, parents, students, and health workers. Methods included stakeholder meetings to discuss program effectiveness and gather feedback, and surveys and interviews to collect qualitative insights. This participatory approach ensured the program addressed community needs effectively and provided actionable recommendations for improvement.

ThaiHealth’s multi-faceted approach demonstrates its commitment to addressing health challenges with innovative, evidence-based, and culturally sensitive solutions. Their emphasis on both macro and micro-level analyses, along with stakeholder involvement, ensures that their programs are impactful, sustainable, and equitable.

Korea Health Promotion Institute

KHEPI, Korea

National Health Plan, a 10-year strategy, provides a roadmap for improving public health through clear goals, performance indicators, and collaborations across various sectors.

  • A core goal of KHEPI’s work is to ensure health equity.

Overview

The Korea Health Promotion Institute (KHEPI) is dedicated to improving public health in South Korea by focusing on disease prevention and promoting healthy lifestyles. Its primary tool for achievement is the National Health Plan, a 10-year strategy created under the National Health Promotion Act. It provides a roadmap for improving public health through clear goals, performance indicators, and collaborations across various sectors.

Performance indicators

The indicators include

  • Healthy Life Expectancy by monitoring the average number of years people live in good health.
  • Regional Disparities by measuring differences in health outcomes.

KHEPI uses performance indicators to track progress and provide feedback to policymakers and local governments. The indicators include Healthy Life Expectancy by monitoring the average number of years people live in good health, and Regional Disparities by measuring differences in health outcomes between wealthier and less-developed areas. For instance, regional governments are required to develop annual health promotion plans aligned with the National Health Plan. A rural area might focus on increasing access to healthcare facilities while urban areas may prioritize campaigns to reduce smoking or alcohol use.

A rural area might focus on increasing access to healthcare facilities while urban areas may prioritize campaigns to reduce smoking or alcohol use.

Health equity

A core goal of KHEPI’s work is to ensure health equity, which means reducing gaps in health outcomes between different groups. KHEPI’s main concept lies in focusing on vulnerable populations, such as low-income families or rural residents, and reducing the gap in healthy life expectancy between the highest and lowest income groups to less than 7.6 years by 2030. One important example is to support low income households. Programs are tailored to provide free or subsidized health check-ups for families who may not otherwise access medical care.

A core goal of KHEPI’s work is to ensure health equity.

KHEPI’s main concept lies in focusing on vulnerable populations.

Diverse health needs

KHEPI ensures accountability by monitoring progress and reporting results annually. This feedback helps refine strategies and policies. If data shows an increase in youth smoking in a particular region, KHEPI may prioritize anti-smoking campaigns and policies in that area.

By combining structured planning, local adaptations, and a focus on equity, KHEPI ensures its programs address the diverse health needs of South Korea’s population. Their data-driven and collaborative approach serves as a model for health promotion efforts globally.

Health Promotion Board

HPB, Singapore

The Singapore Health Promotion Board (HPB) focuses on improving public health through strategic planning, innovative methods, and the use of data and technology.

HPB’s Logic Map Framework

The Singapore Health Promotion Board (HPB) focuses on improving public health through strategic planning, innovative methods, and the use of data and technology. HPB’s Logic Map Framework guides its programs, ensuring that they are aligned with national health goals while addressing the specific needs of the population.

  1. Connect individual programs to broader national health objectives: HPB uses the Logic Map Framework to connect individual programs to broader national health objectives. This framework includes three levels of objectives:
    1. L1 (Impact-Level – National objectives): Reducing risk factors for non-communicable diseases (e.g., diabetes, heart disease) such as a nationwide campaign encouraging healthy eating to lower obesity rates.
    2. L2 (Outcome-Level – Program goals): Changing specific behaviors, knowledge, or practices within target groups such as promoting healthier food choices in schools to reduce childhood obesity.
    3. L3 (Output-Level – Measurable deliverables): Monitoring the direct results of specific programs such as counting the number of schools adopting HPB’s nutrition guidelines.
  2. Strategic Enablers and Behavior Change: Focuses on creating supportive environments (strategic enablers) and influencing individual behavior change. Strategic Enablers which are policies and regulations that shape healthier choices. The Nutri-Grade Labeling system for beverages grades drinks based on their sugar and saturated fat content. Drinks with high sugar levels (Grade C or D) must display clear labels, and advertising for Grade D drinks is banned. Behavior Change is a campaign encouraging healthier lifestyles. The Eat, Drink, Shop Healthy Challenge, which rewards Singaporeans with points for purchasing healthy food and drinks. Points can be redeemed for vouchers, creating a fun incentive for better choices.
  3. Data-Driven Health Promotion: HPB integrates data and technology into its health initiatives to understand public behavior and improve program effectiveness. Healthy 365 App tracks users’ health activities, offers personalized recommendations, and rewards healthy habits. This app also links to other programs such as the Eat, Drink, Shop Healthy Challenge. LumiHealth Program is developed in partnership with Apple, LumiHealth personalizes health journeys for users, encouraging them to meet fitness goals through reminders and rewards.

Challenges in measuring impact

ThaiHealth, Thailand

This section highlights key challenges and examples based on practices from VicHealth, HWQld, ThaiHealth, KHEPI, and HPB.

Measuring the impact of health promotion programs is essential but often complex. Health promotion agencies face a range of challenges, from data limitations to the difficulty of isolating the effects of specific interventions. This section highlights key challenges and examples based on practices from VicHealth, HWQld, ThaiHealth, KHEPI, and HPB.

  1. Long-Term Nature of Outcomes: Health promotion often aims to prevent diseases or change behaviors, but the results may take years to become visible. An example from HWQld, chronic disease indicators, like obesity or diabetes rates, are measured over long periods, yet immediate decisions about programs must be made. Pre- and post-comparisons can be difficult because changes are gradual and influenced by many factors.
  2. Attribution of Impact: Determining whether a specific program caused an observed change can be challenging, as many factors influence health outcomes. In programs targeting teenage pregnancy, external factors such as national education campaigns or economic changes also play a role. ThaiHealth addresses this by comparing communities with and without interventions to better isolate the program’s effects.
  3. Data Limitations: Incomplete or inconsistent data can hinder accurate measurement and evaluation. Tobacco control efforts rely on national health surveys to measure smoking rates. Variations in survey methods or missing data make it hard to track trends accurately. To overcome this, ThaiHealth invests in standardized data collection and monitoring systems.
  4. Multiple Determinants of Health: Health outcomes are shaped by a combination of social, economic, cultural, and environmental factors, making it hard to measure the direct impact of interventions. HWQld’s Wellbeing Model includes commercial, digital, and ecological determinants of health. They recognize that systemic changes, such as shifts in policies or community mindsets, are harder to quantify but critical for sustainable health improvements.
  5. Quantifying Intangible Benefits: Some benefits, like improved mental health, community cohesion, or cultural shifts, are difficult to measure using traditional metrics. Mental health awareness campaigns may result in less absenteeism and increased productivity, but these benefits are often intangible. ThaiHealth uses proxies like work attendance and qualitative methods, such as interviews, to capture the broader value.
  6. Cultural and Contextual Variability: Health promotion programs must account for local contexts, as effectiveness varies by culture, socioeconomic conditions, and infrastructure. Urban areas with better facilities may benefit more from programs promoting physical activity than rural areas with limited infrastructure. ThaiHealth tailors its assessments to reflect these differences, ensuring accurate evaluations.
  7. Dependency on Perspectives: The outcomes selected for evaluation often depend on the viewpoint used, such as the funder, target community, or broader society. This can influence what is considered valuable or successful. Economic evaluations can focus on cost savings for healthcare systems or broader societal benefits. HWQld uses its Equity Framework to include multiple perspectives, ensuring that marginalized voices are considered in assessments.
  8. Overlapping Interventions: Health promotion often involves multiple programs operating simultaneously, making it hard to distinguish which intervention contributed most to the observed changes. In Melbourne, programs promoting physical activity, healthier food environments, and urban design projects run concurrently. Evaluating the distinct impact of each initiative is complex.
  9. Balancing Equity in Reporting: Comparing health outcomes between groups can unintentionally perpetuate stigma or blame individuals instead of highlighting systemic issues. Reporting that a specific group has worse health outcomes can reinforce negative stereotypes. HWQld uses inclusive data methods and focuses on systemic issues, such as access to resources, to provide a fairer narrative.
  10. Long-Term Tracking: Many health interventions yield results only after several years, making it hard to sustain evaluations and maintain focus on long-term goals. Programs to reduce childhood obesity may not show measurable health improvements immediately. Long-term studies are needed to track changes in obesity rates and related health outcomes over time.

How agencies address these challenges

Health promotion agencies use innovative strategies to tackle these challenges

  • Addressing data inconsistencies
  • Ensuring a more holistic view of impact
  • Assessments reflect local realities and account for cultural differences
  • Improving relevance and accuracy

Health promotion agencies use innovative strategies to tackle these challenges:

  1. Standardized frameworks such as VicHealth’s Impact and Evaluation Framework or HWQld’s MEL-F provide structured approaches to address data inconsistencies.
  2. Mixed methods combine quantitative data with qualitative insights, ensuring a more holistic view of impact.
  3. Community involvement ensures that assessments reflect local realities and account for cultural differences.
  4. Tailored evaluations focus on specific contexts, such as rural or urban settings, to improve relevance and accuracy.

Innovative approaches and insights

Health promotion agencies are leveraging innovative approaches to better assess their impact, overcome challenges, and adapt to complex health environments.

Notable innovations and insights

Health promotion agencies are leveraging innovative approaches to better assess their impact, overcome challenges, and adapt to complex health environments. Here are some notable innovations and insights drawn from the agencies studied:

  1. Flexibility in Frameworks: VicHealth and HWQld emphasize frameworks that allow for continuous learning and adaptation. For instance, HWQld’s Measurement, Evaluation, and Learning Framework (MEL-F) focuses on systemic change indicators like policies, practices, and mindsets rather than only individual health outcomes​​.
  2. Mixed Methods for Holistic Insights: Combining quantitative data with qualitative stories helps agencies capture the richness of program impacts. For example, HWQld uses Most Significant Change techniques to document transformative personal and community stories​​. ThaiHealth incorporates qualitative feedback from stakeholders, ensuring that assessments reflect real-world impacts​.
  3. Data-Driven Strategies: HPB’s Healthy 365 App uses real-time data to engage citizens in healthy behaviors and rewards them for making better choices. This data integration helps refine programs dynamically​.
  4. Equity-Focused Assessments: HWQld and ThaiHealth prioritize equity by tailoring programs to diverse populations. ThaiHealth uses Social Return on Investment (SROI) to quantify benefits like reduced healthcare costs and improved productivity​​.
  5. Precision Health Interventions: HPB leverages data analytics for targeted health messaging and interventions, making public health programs more effective by addressing sub-population needs​.
  6. Sustainable Funding Models: ThaiHealth’s sin tax on tobacco and alcohol ensures continuous funding for health promotion initiatives. This model not only generates revenue but also reduces harmful behaviors​.

Conclusion

Impact assessment is vital for understanding the effectiveness of health promotion programs and guiding future strategies. Leading health promotion agencies like VicHealth, HWQld, ThaiHealth, KHEPI, and HPB demonstrate how innovative frameworks, data-driven strategies, and community involvement can overcome challenges in measuring impact.

By focusing on systemic changes, leveraging technology, and prioritizing equity, these agencies set examples for global health promotion efforts. Their work highlights that holistic, adaptive, and inclusive approaches are essential for addressing the complex determinants of health and achieving meaningful, long-term public health outcomes.

Articles, Insights

Related Posts