INTRODUCTION

Volunteer participation in science projects effectively elevates awareness, learning and citizen empowerment and is now embraced by large international organizations. The introduction of UNESCO's Open Science Recommendation (UNESCO, 2021a) welcomed the epistemology of citizen science (CS), aiming to integrate science theory and practice and society (UNESCO, 2021b). CS involves multidirectional research to democratize scientific processes to advance science (Bonney et al., 2016; Hecker et al., 2018; Irwin, 2018; Ruiz-Mallén et al., 2016; Wiggins & Crowston, 2011). From passive individual observations to developing networked digital platforms (Baudry et al., 2021), CS has resurged thanks to the evolution of accessible technologies. Moreover, Education 4.0 (Miranda et al., 2021) and the 4th Industrial Revolution (Schwab, 2015) have set new standards for managing data and its interactions to solve complex problems in CS projects (Robinson et al., 2017). Ultimately, CS project design has not only been shaped by a general interest in phenomena but also by emergencies that have boosted the evolution and application of citizen-led science solutions.

Considering these aspects, we understand that greater effectiveness in problem-solving in CS projects is closely linked to holistic designs. Therefore, this work examined the significant number of CS projects that keep emerging to provide a framework and typology that guides identifying and monitoring the status of CS projects, in order to plan their further development to achieve holistic impact. We did this by scrutinizing the state of the art of CS projects from a Theory of Change (ToC) (UNDAF, 2016; 2017) perspective, which considers technology, communication, and emergency contexts. The work highlights key frameworks that support CS and demonstrate the interactions and articulation that develop the sub-competencies of complex thinking.

FROM CS STAGNATION TO THE 21ST-CENTURY EXPRESSWAY

The evolution of technology and the degradation of ecosystems have also driven the interest in CS. While citizens may often be unaware of the origin and consequences of the events surrounding them (Ballerini & Berth, 2021; Callaghan et al., 2020), their participation in CS may motivate them to get involved in public policies. (García-Holgado et al., 2020; Strasser et al., 2019). New paradigms have emerged where citizens engage in collaborative relationships to analyze, improve, and discover information (McCurdy & Vinogradova-Shiffer, 2021; Stubbs et al., 2021). Such developments in Industry 4.0 technologies have spilt into the Education 4.0 framework, where learners experience real-life challenges and apply their knowledge to develop technology-based solutions using complex thinking skills (Miranda et al., 2021). This macro-competency comprises critical, systemic, scientific, and innovative thinking (Ramírez-Montoya et al., 2022). CS projects involve real-life scenarios that enable learners to reason in the face of complexity (Araújo et al., 2022; Belluigi & Cundill, 2017; Castell et al., 2021; Constant & Roberts, 2017; Zourou & Tseliou, 2020). Moreover, numerous examples of imminent hazards demonstrate citizens' critical role in reacting more quickly than their governing authorities or regulators.

Lessons learned during emergencies have spotlighted the response potential of CS. Cases such as the Fukushima nuclear accident (Hultquist & Cervone, 2017) and the recent COVID-19 pandemic (Goehrke, 2020) correspond to CS studies that focus on disaster risk reduction (DRR). These highlight the potential to bring diverse communities together and the need to broaden the spectrum to consider ethics and gender, among other topics (Paul et al., 2021). Moreover, these emergencies have triggered calls to generate policies to assess the impact of CS data in the legal arena (Peel, 2020). From this perspective, the democratization of CS makes it possible to strengthen the empowerment of citizens and their influence in bottom-up decision-making where data transmission is expected, and there is the committed involvement of quintuple helix organizations.

FOSTERING CS INITIATIVES

To integrate the general public into the scientific process, several frameworks and programs have been advanced to standardize protocols to achieve social impact and contribute to Sustainable Development Goals (SDG):

● UNESCO Recommendation on Open Science: Complex challenges have left their mark in the first quarter of the 21st century, affecting the achievement of the Sustainable Development Goals. As a strategy, member states agreed to propose a normative instrument based on good practices regarding the openness of science. The process resulted in the UNESCO Open Science Recommendation (UNESCO, 2021a).

● United Nations Environment Program: Currently, UNEP focuses on the triple planetary crisis (climate change, nature, and biodiversity loss) through strategies to achieve the 2030 Agenda for Sustainable Development. Their goal is to achieve global integration of CS into the formal process to report SDGs at the national level, promoting confidence in the data it collects through examples of good practice (UNEP, 2019).

● European Commission Framework Programs (FP): (European Commission, 2016) Continuously customized for global contexts, FPs have mainly addressed CS projects under the Horizon 2020 framework (European Commission, 2020). It is worth noting the case of the Marie Sklodowska-Curie Actions (MSCA) 25th-anniversary Presidency conference, which included a round table to discuss the topic "Closing the gap between research and citizens: building trust in science with MSCA" (Marie Sklodowska-Curie Actions [MSCA], 2022).

● European Citizen Science Association: A prime promoter of CS in Europe, ECSA seeks to foster the democratization of science for all. Its ten principles underpin good practices aimed for everyone to participate freely throughout the scientific process to advance sustainable research impact and public policies (ECSA, 2015).

CITIZEN SCIENCE THRESHOLD

When considering the various components interacting in CS projects, we conceive a threshold where projects either stall or transcend to meet the requirements of Open Science. To build frameworks for the participation of citizens in science projects, we deemed it necessary to hold an integral perspective that considers the contextual aspects linked to the project (e.g., technology, infrastructure, methods) and the intellectual development of the participants. The proposal must meet the necessary criteria of a CS project to effectively be a genuine full-cycle CS initiative. In searching for a solution, we considered it fundamental to rely on a theory of transcendence that could guide the CS interventions to achieve transformation, namely the ToC.

ToC is an approach that uses a causal analysis based on available evidence to explain how a given modification or group of variations is likely to lead to developing a certain change. ToC can assist in identifying solutions to challenges that impede development and form conclusions on which route to pursue, considering the benefits, odds, and uncertainties inherent in any change process. In addition, ToC aids in identifying elemental conjectures and hazards, which must be understood and revisited throughout the process to ensure that the strategy results in the intended change (United Nations Development Assistant Framework [UNDAF], 2017).

Method

We used the Evidence-Based Educational Innovation (EBEI) model as a methodology to develop our typology (Sarango-Lapo et al., 2021). The EBEI model was adopted because it promotes an approach of practical experiences to attain digital competencies to manage technological resources that lead to innovation (Sarango-Lapo et al., 2017), which is the case of technology-supported CS projects. Four stages guided the development of the proposed typology: analyzing the CS scientific evidence frameworks, constructing a framework of CS threshold dimensions, proposing the components of the CS typology, and presenting a prospective of the expected benefit of the CS typology (see Figure 1).

Note: EBEI is an empirical model suitable when technologies are involved.

CS Frameworks

The ultimate aim of CS to integrate experts and amateurs to generate knowledge keeps citizen participation in focus. Although participation in CS projects is voluntary, mechanisms exist to reward individuals, often monetary and sometimes through public recognition (Cappa et al., 2018). Moreover, despite the notion that CS focuses on engaging ordinary audiences, most participants in CS initiatives are not idealistic amateurs but students and workers in science or related spheres such as writers, journalists, and educators (Tancoigne, 2017).

Over the past decade, frameworks and typologies continue to emerge, providing interpretations necessary to better understand the evolution of CS participation and design projects with better scale, quality, and relevance. Evidence of this can be observed in the increasing number of scientific articles published in recent years, indicating a growing trend in the subject. The broad focus spectrum includes the following clusters: CS participant engagement (Fischer et al., 2021; Haklay, 2018; Lotfian et al., 2020; Wiggins & Crowston, 2011); ethical issues in CS (Groot & Abma, 2022; Resnik et al, 2015); virtual CS projects (Reed et al., 2012); incorporating communities in CS (García-Holgado et al., 2020; Katapally, 2019; Nardi et al., 2021; Pandya, 2012); CS data quality improvement (Antelio et al., 2012; Cerquides et al., 2021; Garriga et al, 2017); evaluation and design of CS programs (Bolici & Colella, 2019; Chase & Levine., 2016; Hennig et al., 2019); prevention and control (Asingizwe et al., 2018; Coulson & Woods, 2021; Li et al., 2019; Yang et al., 2019); education & research (Hiller et al., 2019; Spasiano et al., 2021); surveillance and monitoring via CS (Arazy & Malkinson, 2021; Callaghan et al, 2020; Welvaert & Caley, 2016); CS data collecting & processing (Callaghan et al., 2021; Eagles-Smith, 2020; Hyvönen et al., 2021; Teng & Albayrak, 2017), CS and interest in developing critical thinking (Araújo et al., 2022; Belluigi & Cundill., 2017; Castell et al., 2021; Constant & Roberts, 2017) (See Figure 2).

Note: The highest concentration of publications occurred in 2018 (182) and 2021 (176).

Note: There are no studies related to citizen science and its impact on developing the macro-competency of complex thinking.

To interpret the challenges faced by the world's population, we propose a threshold for CS that highlights the key points that should be addressed to successfully integrate individuals as citizen scientists in projects that advance scientific knowledge (Warnecke et al., 2019). Given the variety of possibilities for individuals to engage in CS, it is helpful to classify the entry routes for participants and their boundaries to guide the efforts supporting the processes and the participants' development.

Proposed dimensions for the CS Threshold

ToC and applied ladder patterns (Warnecke et al., 2019) allowed us to analyze, first, the various components; then, we were able to structure the main components, acting on the processes of a given phenomenon for future validation. The significant assets are the key elements to measure individual levels of development and their contribution to clusters linking various components and, in general, the overall project theme. In this framework, the approach combination contributes positively to improving the analysis for future CS proposals.

To establish the CS threshold dimension, we conceived eight fundamental components for measuring the performance of a CS project. These arise out of combining core features from CS frameworks explained above, relevant CS typologies (Van Kleek & Simperl, 2017; Wiggins & Crowston, 2011), and considering the building blocks of complex thinking (i.e., critical thinking, innovative thinking, scientific thinking, and systemic thinking). Within this proposed structure's perspective, multiple combinations of mastering the eight selected components would enable progress in the overall performance of the initiatives. For the indicative purposes of this work, we describe three dimensions as transitional channels for these combinations: a baseline dimension ("Bounded CS"), a transitional dimension ("Threshold CS"), and a fulfillment dimension ("Full-cycle CS"). The three dimensions incorporate the ToC approach, where analyzing a set of circumstances that requires change and articulating the desired outcome is the first step. It entails identifying inputs, outcomes, assumptions, and risks (UNDAF, 2016).

Bounded CS dimension

We envisaged the Bounded CS dimension as the development baseline. In analogy to the ToC, this domain focuses on setting out the resources (inputs) needed to achieve results.

Threshold CS dimension

In the transition towards an ideal of Full-cycle CS integration, the scaffolding allows multiple-component, circumstantial, contextual combinations. Following the ToC sequence, we reference the CS activities deliverables (outputs) obtained by transforming the initial resources (inputs) and measuring the effects that they generate (outcomes).

Full-cycle CS dimension

At the other end of the spectrum is the Full-cycle CS, the ultimate plateau to which CS projects aspire, where functions reach the maximum transformation possibilities. In ToC terms, this dimension represents the intended long-term effects (impacts) of CS projects linked to the results of the previous dimensions.

Definition of CS Threshold components

Each of the eight components detailed below is fundamental to promoting a CS project from the Bounded CS baseline to the fulfillment dimension of Full-cycle CS. The dimensional progress development of the component depends on the characteristics of the project; therefore, each project will have its own ToC. In principle, it is proposed that all eight components be considered from the outset of any CS project (See Table 1).

Context awareness

This component refers to the contextual awareness of the addressed problem, which is reflected in controlling the proposal's originality. A CS community that participates in an existing project or replicates a project from another region remains at baseline. Improving an existing project moves it to the Threshold dimension, and once an original project is generated, it transcends to the Full-cycle CS dimension.

Citizen engagement

Both the participants' and the project operators' perspectives are covered in this component. It entails establishing protocols and procedures such as informed consent and data privacy. Participants progress from a relatively passive position to co-creation and eventually initiative leadership.

Infrastructure leverage

This dimension considers the infrastructure accessible to the project and how the CS participants' interactive dynamics influence mobility between the dimensions. Ideally, participants come to improve existing interactions within the infrastructure.

Technological innovation

In this dimension, we address the use or creation of technology supporting the project administration, the storage and analysis of the data, and the ability to disseminate the information to the participants. At the Bounded level, general-purpose technology applied to the project is used with no particular development, but project participants can communicate with each other. At the Full-cycle level, we find specialized software to collect and analyze data or software developed to suit the project to meet contextual requirements.

Educational innovation

This component reflects the importance of incorporating new sustainable processes to strengthen education to impact lifelong learning. At the most basic level, CS initiatives may lack educational vision. Using educational resources, participants move the projects to transcend the threshold; they propose reusable resources for formal and non-formal education to continue making impacts.

Outreach and Scale

The scope of the project is measured by this component, considering the number of benefited people, the number of citizens participating in the data collection, and the geographical scope of the data retrieved and analyzed. At the Bounded level, CS projects occur within a locality with the participation of citizens who live in the community. At the Full-cycle level, several global localities participate, and interested citizens living in these communities work at different levels.

Network building

This component allows us to identify the number of actors involved in the CS projects, as the participation of representatives of the quintuple helix is completed —Political, Social, Environmental, Economic and Educational systems— it will be possible to move from the limited dimension to the full cycle.

Complex thinking

Proposing solutions to complex problems involves using complex thinking competency, consisting of systemic thinking, scientific thinking, critical thinking, and innovative thinking. Only one sub-competency is developed at the Bounded level, while at the Full-cycle level, the four types of sub-competencies are developed integratively. The use of each type of thinking brings a dimension to the problem solution.

By addressing the crucial components for crossing the CS threshold, researchers and science amateurs are encouraged to design full-experiential cycle CS projects to enhance personal learning and the initiative's impact. Actions taken in each dimension encompass the type of activity and interaction undertaken by participants, linked to the type of contribution. The intended impact results from strategically combining the various components per the project characteristics. Due to the diverse nature of CS projects, the passage through each component's dimensions will progress differently in every case (see Figure 4).

Note: All eight components progress at different rates through the dimensions.

DISCUSSION AND CONCLUSION

CS projects have focused on the "doing" of science, overlooking the study of how the participants think. The analysis of the frameworks identified a lack of attention to developing the sub-competencies of complex thinking (Figure 3). The growing interest in the macro-competency of complex thinking in educational innovation (Ramírez et al., 2022) is a warning that insufficient attention is being paid to the development of its sub-competencies in CS projects (Araújo et al., 2022; Belluigi & Cundill, 2017; Castell et al., 2021; Constant & Roberts, 2017). For the community involved in the design of CS projects, this finding is a wake-up call to consider measuring the development of complex thinking with their metrics.

An evidence-based foundation is suitable to advance the cascade of CS projects. Our analysis revealed that in the last decade, CS frameworks have emerged with remarkable growth (Figure 2) due primarily to the application of technologies and connectivity (Cappa et al., 2018; McCurdy & Vinogradova-Shiffer, 2021; Nesbit et al., 2020; Paul et al., 2021). Notably, large international organizations have integrated CS into their guidelines (UNESCO, 2021a) and fostered CS through financial programs (European Commission, 2020). These have contributed to developing partnerships with Higher Education Institutions in several countries (Zourou & Tseliou, 2020). Recognition at the level of governments and major international institutions is an indication of the prominence and expected prosperity of CS initiatives

There is room to improve the design of CS projects more comprehensively. Three main clusters of emphasis were identified within the CS frameworks: prevention and control, project evaluation and design, and education and research (Figure 2). Frameworks that consider the participation of individuals and community engagement in CS projects focus on their involvement role rather than on developing competencies (Fischer et al., 2021; Haklay, 2018; Lotfian et al., 2020; Nardi et al. 2021; Pandya, 2012; Spasiano et al., 2021; Wiggins & Crowston, 2011; Yang et al., 2019). The rationale for designing CS projects is at a mature level with thematic frameworks that support efficiency.

The notion that CS projects should have a holistic impact is highly relevant. The key deliverables of this study are the EBEI-based methodological analysis of the four-stage guide to developing the CS framework (Figure 1) and the proposal of three ToC dimensions to advance CS projects (Table 1). On the one hand, EBEI allowed the structuring of the direction of research to host technology-enhanced, innovative CS projects (Sarango-Lapo et al., 2021). On the other hand, ToC provided the intended scaffolding to build the dimensions that included numerous actors and circumstances when dealing with complexity (UNDAF, 2016). Our proposed structure should allow CS projects to design customized ToC with various application possibilities.

Improving the design and scope of CS projects involves identifying the components to be measured and monitoring their performance from multiple perspectives. A typology of eight components has been proposed to measure the impact of CS projects: Context awareness, Citizen engagement, Infrastructure leverage, Technological innovation, Educational innovation, Outreach and Scale, Network building, and Complex thinking (Table 1, Figure 4). In addition to the components and the diverse approaches, higher education institutions can play a crucial role as a link between society and science (Zourou & Tseliou, 2020). Awareness of the impact of integrated CS project components is critical to designing and measuring its scope and impact.

Among the study's limitations, we point out the need to propose the sub-levels that the projects could experience in the dimensions. It is equally important to exemplify the potential component combinations according to the type of CS project being monitored. Moreover, the nature of CS projects that increasingly train amateur participants to potentially transcend the threshold dimension calls for protocols and mechanisms to strengthen and sustain ethical CS. Future studies shall address how the CS Threshold framework and its typology may give rise to creating innovative educational tools that measure CS projects' performance within its framework. Also, from the research perspective, we foresee the need to present case studies where the dimensions and components of the framework are applied to define in more detail the Full-cycle CS and, if desirable, to plan the transition of CS projects.

Acknowledgments

The authors wish to acknowledge the financial and technical support of Writing Lab,

The authors wish to acknowledge the financial and technical support or Writing Lab, Institute for the Future of Education, Tecnologico de Monterrey, Mexico, in the production of this work, the OpenResearchLab and the SKILIKET4C project from the Reasoning for Complexity Interdisciplinary Research Group (R4C-IRG).

REFERENCES

Antelio M., Esteves M. G. P., Schneider D., & Souza J.M.D. (2012). Qualitocracy: A data quality collaborative framework applied to citizen science. Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics, 931-936. https://doi.org/10.1109/ICSMC.2012.6377847

Araújo J. L., Morais C., & Paiva J. C. (2022). Student participation in a coastal water quality citizen science project and its contribution to the conceptual and procedural learning of chemistry. Journal of Baltic Science Education. 20(6). 881-893. https://doi.org/10.33225/JBSE/21.20.881

Arazy, O., & Malkinson, D. (2021). A Framework of Observer-Based Biases in Citizen Science Biodiversity Monitoring: Semi-Structuring Unstructured Biodiversity Monitoring Protocols. Frontiers in Ecology and Evolution, 9. https://doi.org/10.3389/fevo.2021.693602

Asingizwe, D., Poortvliet, P. M., Koenraadt, C. J. M., Van Vliet, A. J. H., Murindahabi, M. M., Ingabire, C., Mutesa, L., & Feindt, P. H. (2018). NJAS - Wageningen Journal of Life Sciences, 86-87, 111-122. https://doi.org/10.1016/j.njas.2018.06.002

Ballerini, L., & Berth, S. (2021). Using citizen science data to monitor the Sustainable Development Goals: a bottom‑up analysis. Sustainability Science. https://doi.org/10.1007/s11625-021-01001-1

Baudry, J., Tancoigne, É., & Strasser, B. (2021). Turning crowds into communities: The collectives of online citizen science. Social Studies of Science, 1-26. https://doi.org/10.1177/03063127211058791

Belluigi, D. Z., & Cundill, G. (2017). Establishing enabling conditions to develop critical thinking skills: a case of innovative curriculum design in Environmental Science. Environmental Education Research, 23(7). 950-971. https://doi.org/10.1080/13504622.2015.1072802

Bolici, F., & Colella, N. A. (2019). How to Design Citizen-Science Activities: A Framework for Implementing Public Engagement Strategies in a Research Project. Lecture Notes in Information Systems and Organisation, 27, 149-162. https://doi.org/10.1007/978-3-319-90500-6_12

Bonney, R., Cooper, C., & Ballard, H. (2016). The Theory and Practice of Citizen Science: Launching a New Journal. Citizen Science: Theory and Practice, 1(1), p. 1. https://doi.org/10.5334/cstp.65

Cappa, F., Laut, J., Porfiri, M., & Giustiniano, L. (2018). Bring them aboard: rewarding participation in technology-mediated citizen science projects. Computers in Human Behavior, 89. https://doi.org/10.1016/j.chb.2018.08.017

Callaghan, C. T., Ozeroff, I., Hitchcock, C., & Chandler, M. (2020). Capitalizing on opportunistic citizen science data to monitor urban biodiversity: A multi-taxa framework. Biological Conservation, 251. https://doi.org/10.1016/j.biocon.2020.108753

Callaghan, C. T., Watson, J. E. M., Lyons, M. B., Cornwell, W. K., & Fuller, R. A. (2021). Conservation birding: A quantitative conceptual framework for prioritizing citizen science observations. Biological Conservation, 253. https://doi.org/10.1016/j.biocon.2020.108912

Castell, N., Grossberndt, S., Gray, L., Fredriksen, M. F., Skaar, J. S., & Høiskar, B. A. K. (2021). Implementing Citizen Science in Primary Schools: Engaging Young Children in Monitoring Air Pollution. Frontiers in Climate, 3. https://doi.org/10.3389/fclim.2021.639128

Cerquides, J., Mülâyim, M. O., Hernández-González, J., Shankar, A. R., & Fernandez-Marquez, J. L. (2021). A conceptual probabilistic framework for annotation aggregation of citizen science data. Mathematics, 9(8). https://doi.org/10.3390/math9080875

Chase, S. K., & Levine, A. (2016). A framework for evaluating and designing citizen science programs for natural resources monitoring. Conservation Biology, 30(3), 456-466. https://doi.org/10.1111/cobi.12697

Constant, N., & Roberts, L. (2017). Narratives as a mode of research evaluation in citizen science: Understanding broader science communication impacts. Journal of Science Communication, 16(4), 1-18. https://doi.org/10.22323/2.16040203

Coulson, S., & Woods, M. (2021). Citizen Sensing: An Action-Orientated Framework for Citizen Science. Frontiers in Communication, 6. https://doi.org/10.3389/fcomm.2021.629700

Eagles-Smith, C. A., Willacker, J. J., Nelson, S. J., Flanagan Pritz, C. M., Krabbenhoft, D. P., Chen, C. Y., Ackerman, J. T., Campbell Grant, E. H., & Pilliod, D. S. (2020). A national-scale assessment of mercury bioaccumulation in United States national parks using dragonfly larvae as biosentinels through a citizen-science framework. Environmental Science and Technology, 54(14), 8779-8790. https://doi.org/10.1021/acs.est.0c01255

ECSA (European Citizen Science Association) (2015). Ten Principles of Citizen Science. Berlin. https://doi.org/10.17605/OSF.IO/XPR2N

European Commission (2016). Open innovation, open science, open to the world: A vision for Europe. European Commission, Directorate-General for Research and Innovation.

European Commission (2020). Citizen science and citizen engagement: achievements in Horizon 2020 and recommendations on the way forward, (N, Delaney, editor, Z, Tornasi, editor) Publications Office. https://doi.org/10.2777/05286

Fischer, H., Cho, H., & Storksdieck, M. (2021). Going beyond hooked participants: The nibble-and-drop framework for classifying citizen science participation. Citizen Science: Theory and Practice, 6(1), https://doi.org/10.5334/cstp.350

García-Holgado, A., García-Peñalvo, F. J., & Butler, P. (2020). Technological ecosystems in citizen science: A framework to involve children and young people. Sustainability (Switzerland), 12(5), 1-16. https://doi.org/10.3390/su12051863

Garriga, J., Piera, J., & Bartumeus, F. (2017). A Bayesian framework for reputation in citizen science. CEUR Workshop Proceedings, 1960. http://ceur-ws.org/Vol-1960/paper6.pdf

Goehrke, S. (2020). Making Made Right: This Czech Company Guides Global 3-D Printing Pandemic Response. Forbes. https://www.forbes.com/sites/sarahgoehrke/2020/03/28/making-made-right-this-czech-company-guides-global-3-d-printing-pandemic-response/?sh=21a75e36624b

Groot, B., & Abma, T. (2022). An ethics framework for citizen science and public and patient participation in research. BMC Medical Ethics, 23(1). https://doi.org/10.1186/s12910-022-00761-4

Haklay, M. (2018). Participatory citizen science. In S. Hecker, M. Haklay, A. Bowser, Z. Makuch, J. Vogel, & A. Bonn (Eds.), Citizen Science: Innovation in Open Science, Society and Policy. UCL Press. https://doi.org/10.2307/j.ctv550cf2.11

Hecker, S., Bonney, R., Haklay, M., Hölker, F., Hofer, H., Goebel, C., Gold, M., Makuch, Z., Ponti, M., Richter, A., Robinson, L., Iglesias, J.R., Owen, R., Peltola, T., Sforzi, A., Shirk, J., Vogel, J., Vohland, K., Witt, T., & Bonn, A. (2018). Innovation in Citizen Science – Perspectives on Science-Policy Advances. Citizen Science: Theory and Practice, 3(1), p. 4. https://doi.org/10.5334/cstp.114

Hennig, S., Hölbling, D., Ferber, N., & Tiede, D. (2019). Framework and components for citizen science projects – the citizenmorph project [Rahmenkonzept und komponenten für citizen-science-projekte – das projekt citizenmorph]. AGIT- Journal fur Angewandte Geoinformatik, 5, 2-13. https://doi.org/10.14627/537669001

Hiller, S. E., Whitehead, A., & Kitsantas A. (2019). Enhancing stem motivation through citizen science programs: A framework for educators and researchers. Enhancing STEM Motivation through Citizen Science Programs, 4-26.

Hultquist, C., & Cervone, G. (2017). Citizen monitoring during hazards: validation of Fukushima radiation measurements. GeoJournal, 83, 189–206. https://doi.org/10.1007/s10708-017-9767-x

Hyvönen, E., Rantala, H., Ikkala, E., Koho, M., Tuominen, J., Anafi, B., Thomas S., Wessman A., Oksanen E., Rohiola V., Kuitunen J., & Ryyppö, M. (2021). Citizen science archaeological finds on the Semantic Web: the FindSampo framework. Antiquity, 95(382), e24. https://doi.org/10.15184/aqy.2021.87

Irwin, A. (2018). No PhDs needed: how citizen science is transforming research. Nature. https://doi.org/10.1038/d41586-018-07106-5

Katapally, T. R. (2019). The SMART framework: Integration of citizen science, community-based participatory research, and systems science for population health science in the digital age. JMIR mHealth and uHealth, 7(8). https://doi.org/10.2196/14056

Li, E., Parker, S. S., Pauly, G. B., Randall J. M., Brown B. V., & Cohen, B. S. (2019). An Urban Biodiversity Assessment Framework That Combines an Urban Habitat Classification Scheme and Citizen Science Data. Frontiers in Ecology and Evolution, 7.https://doi.org/10.3389/fevo.2019.00277

Lotfian, M., Ingensand, J., & Brovelli, M. A. (2020). A framework for classifying participant motivation that considers the typology of citizen science projects. ISPRS International Journal of Geo-Information, 9(12). https://doi.org/10.3390/ijgi9120704

McCurdy, A., & Vinogradova-Shiffer, N. (2021). Advancing Ocean Science Through Open Science and Software on the Cloud. Marine Technology Society Journal, 55(3), pp. 146-147(2). Marine Technology Society. https://doi.org/10.4031/MTSJ.55.3.25

Miranda, J., Ramírez-Montoya, M. S., & Molina, A. (2021). Education 4.0 Reference Framework for the Design of Teaching-Learning Systems: Two Case Studies Involving Collaborative Networks and Open Innovation. Smart and Sustainable Collaborative Networks 4.0. Springer. https://doi.org/10.1007/978-3-030-85969-5_65

MSCA (2022). MSCA 25 ans: Next-generation MSCA: opening a new era for change. Marie Sklodowska-Curie Actions. PSL Université Paris. https://psl.eu/msca-conference-next-generation-msca-opening-new-era-change

Nardi, F., Cudennec, C., Abrate, T., Allouch, C., Annis, A., Assumpção, T., Aubert, A. H., Bérod, D., Braccini, A. M., Buytaert, W., Dasgupta, A., Hannah, D. M., Mazzoleni, M., Polo, M. J., Sæbø, Ø., Seibert, J., Tauro, F., Teichert, F., Teutonico, R., Uhlenbrook, S., Wahrmann Vargas, C., & Grimaldi, S. (2021). Citizens AND Hydrology (CANDHY): conceptualizing a transdisciplinary framework for citizen science addressing hydrological challenges. Hydrological Sciences Journal. https://doi.org/10.1080/02626667.2020.1849707

Nesbit, P. R., Boulding, A. D., Hugenholtz, C. H., Durkin, P. R., & Hubbard, S. M. (2020). Visualization and sharing of 3D digital outcrop models to promote open science. GSA Today, 30(6). https://doi.org/10.1130/GSATG425GW.1

Pandya, R. E. (2012). A framework for engaging diverse communities in Citizen science in the US. Frontiers in Ecology and the Environment, 10(6), 314-317. https://doi.org/10.1890/120007

Paul, J. D., Bee, E., & Budimir, M. (2021). Mobile phone technologies for disaster risk reduction. Climate Risk Management, 32. https://doi.org/10.1016/j.crm.2021.100296

Peel, J. (2020). The 'rights' way to democratize the science-policy interface in international environmental law? A reply to Anna-Maria Hubert. The European Journal of International Law, 31(2). https://doi.org/10.1093/ejil/chaa042

Ramírez-Montoya, M. S., Castillo-Martínez, I. M., Sanabria-Z, J., & Miranda, J. (2022). Complex Thinking in the Framework of Education 4.0 and Open Innovation — A Systematic Literature Review. Journal of Open Innovation: Technology, Market, and Complexity, 8(4). https://doi.org/10.3390/joitmc8010004

Reed, J., Rodriguez, W., & Rickhoff, A. (2012). A framework for defining and describing key design features of virtual citizen science projects. ACM International Conference Proceeding Series, 623-625. https://doi.org/10.1145/2132176.2132314

Resnik, D. B., Elliott, K. C., & Miller, A. K. (2015). A framework for addressing ethical issues in citizen science. Environmental Science and Policy, 54, 475-481. https://doi.org/10.1016/j.envsci.2015.05.008

Robinson, K. L., Luo, J. Y., Sponaugle, S., Guigand, C., & Cowen, R. K. (2017). A Tale of Two Crowds: Public Engagement in Plankton Classification. Front. Mar. Sci. 4:82. https://doi.org/10.3389/fmars.2017.00082

Ruiz-Mallén, I., Riboli-Sasco, L., Ribrault, C., Heras, M., Laguna, D., & Perié, L. (2016). Citizen Science: Toward Transformative Learning. Science Communication, 38(4), 523-534. https://doi.org/10.1177/1075547016642241

Sarango-Lapo, C. P., Mena, J., & Ramírez-Montoya, M. S. (2017). Teachers’ Digital Skills training by using the Educational Innovation based on Evidence methodology (EIBE). In J. Mena, A. García-Valcárcel, F. García-Peñalvo, M. Martín (Eds.), Search and Research: Teacher Education for Contemporary Contexts. Editions Universidad de Salamanca. 1005–1014.

Sarango-Lapo, C. P., Mena, J. J., & Ramírez-Montoya, M. S. (2021). Evidence-based educational innovation model in the framework of Education 4.0: promoting digital information competencies for teachers. Sustainability, 13(18), 10034. https://doi.org/10.3390/su131810034

Schwab, K. (2015). The Fourth Industrial Revolution. World Economic Forum.

Spasiano, A., Grimaldi, S., Braccini, A. M., Nardi, F. (2021). Towards a transdisciplinary theoretical framework of citizen science: results from a meta-review analysis. Sustainability (Switzerland), 13(14). https://doi.org/10.3390/su13147904

Strasser, J., Baudry, J., Mahr, D., Sanchez, G., & Tancoigne, E. (2019). "Citizen Science"? Rethinking Science and Public Participation. Science & Technology Studies, 32(2), 52-76. https://doi.org/10.23987/sts.60425

Stubbs, J., Cardone, R., Packard, M., Jamthe, A., Padhy, S., Terry, S., Looney, J., Meiring, J., Black, S., Dahan, M., Cleveland, S., & Jacobs, G., (2021). Tapis: An API Platform for Reproducible, Distributed Computational Research. In K. Arai K. (Ed.), Advances in Information and Communication. FICC 2021. Advances in Intelligent Systems and Computing, vol 1363. Springer, Cham. https://doi.org/10.1007/978-3-030-73100-7_61

Tancoigne, E. (2017). Four things Twitter tells us about "Citizen Science" (and 1,000 things it doesn't). http://citizensciences.net/2017/01/26/4-things-twitter-tells-us-about-citizen-science/

Teng, W., & Albayrak, A. (2017). Framework for processing citizen science data for applications to NASA earth science missions. IGTF 2017 - Imaging and Geospatial Technology Forum 2017, ASPRS Annual Conference. https://ntrs.nasa.gov/citations/20170002319

UNDAF (2016). Theory of Change Concept Note. United Nations Development Group Latin America and the Caribbean. https://unsdg.un.org/sites/default/files/16.-2016-10-18-Guidance-on-ToC-PSG-LAC.pdf

UNDAF (2017). Theory of Change UNDAF Companion Guidance. United Nations Sustainable Development Group. https://unsdg.un.org/sites/default/files/UNDG-UNDAF-Companion-Pieces-7-Theory-of-Change.pdf

UNEP (2019). The untapped potential of citizen science to track progress on the Sustainable Development Goals. UN Environment Programme. https://www.unep.org/news-and-stories/story/untapped-potential-citizen-science-track-progress-sustainable-development

UNESCO (2021a). UNESCO Recommendation on Open Science. United Nations Educational, Scientific and Cultural Organization. https://unesdoc.unesco.org/ark:/48223/pf0000379949.locale=en

UNESCO (2021b). UNESCO SCIENCE REPORT: The race against time for smarter development. UNESCO, pp. 12-16. https://unesdoc.unesco.org/ark:/48223/pf0000377447?13=null&queryId=2a545ebd-33c2-4993-8e0a-93af21336ebd

Van Kleek, M., & Simperl, E. (2017). From Crowd to Community: A Survey of Online Community Features in Citizen Science Projects. CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 2137-2152. https://doi.org/10.1145/2998181.2998302

Warnecke, D., Wittstock, R., & Teuteberg, F. (2019). Benchmarking of European smart cities – a maturity model and web-based self-assessment tool. Sustainability Accounting, Management and Policy Journal, 10(4), 654-684. https://doi.org/10.1108/SAMPJ-03-2018-0057

Welvaert, M., & Caley, P. (2016). Citizen surveillance for environmental monitoring: combining the efforts of citizen science and crowdsourcing in a quantitative data framework. SpringerPlus, 5(1), https://doi.org/10.1186/s40064-016-3583-5

Wiggins, A., & Crowston, K. (2011). From Conservation to Crowdsourcing: A Typology of Citizen Science. Proceedings of the 44th Hawaii International Conference on System Sciences. https://doi.org/10.1109/HICSS.2011.207

Yang, D., Wan, H. Y., Huang, T.-K., & Liu, J. (2019). The role of citizen science in conservation under the telecoupling framework. Sustainability (Switzerland), 11(4). https://doi.org/10.3390/su11041108

Zourou, K., & Tseliou, A. (2020). Academia permeating society through Citizen Science: Recommendations for Higher Education Institutions. INOS consortium. In T. Gibson, S. J. Murray, S. Erdelez, B. A. Disney, & B. Greenspan (2018), Digital humanities, libraries, and crowdsourcing: Foundations of digital textual technologies. Proceedings of the Association for Information Science and Technology, 55(1), 808-810. https://doi.org/10.1002/pra2.2018.14505501126