MONOGRÁFICO

Learning about Careers: Open data and Labour Market Intelligence

Aprendizaje sobre carreras: datos abiertos e inteligencia del mercado laboral

Graham Attwell 1
University of Warwick, Reino Unido
Deirdre Hughes 2
University of Warwick, Reino Unido

Learning about Careers: Open data and Labour Market Intelligence

RIED. Revista Iberoamericana de Educación a Distancia, vol. 22, no. 1, 2019

Asociación Iberoamericana de Educación Superior a Distancia

“Los textos publicados en esta revista están sujetos a una licencia “Reconocimiento-No comercial 3.0” de Creative Commons. Puede copiarlos, distribuirlos, comunicarlos públicamente, siempre que reconozca los créditos de la obra (autor, nombre de la revista, instituciones editoras) de la manera especificada en la revista.”

Received: 05 July 2018

Accepted: 23 July 2018

How to reference this article: Attwell, G., Hughes, D. (2019). Learning about Careers: Open data and Labour Market Intelligence. RIED. Revista Iberoamericana de Educación a Distancia, 22(1), pp. 81-106. doi: https://doi.org/10.5944/ried.22.1.22289

Abstract: Decisions about learning and work have to be placed in a particular spatial, labour market and socio-cultural context – individuals are taking decisions within particular ‘opportunity structures’ and their decisions and aspirations are further framed by their understanding of such structures. This article examines ways in which learning about careers using open data and labour market intelligence can be applied. An illustrative case study of the LMI for All project in the UK shows the technical feasibility of designing and developing such systems and a model for dissemination and impact. The movement towards Open Data and increasingly powerful applications for processing and querying data has gathered momentum. This combined with the need for labour market information for decision making in increasingly unstable labour markets have led to the development and piloting of new Labour Market Information (LMI) systems, involving multiple user groups. Universal challenges exist given the increasing use of LMI, especially in job matching and the rapidly expanding use of open source data in differing education and employment settings. We highlight six emergent issues that have to be addressed so that open data and labour market intelligence can be applied effectively in differing contexts and settings. We conclude by reflecting on the urgent need to extend the body of research and to develop new methods of co-constructing in innovative collaborative partnerships.

Keywords: careers, educational technology, work, open data, LMI, labour market.

Resumen: Las decisiones sobre el aprendizaje y el trabajo deben ubicarse en un contexto particular espacial, de mercado laboral: los individuos toman decisiones dentro de “estructuras de oportunidad” particulares y sus decisiones y aspiraciones se enmarcan dentro de su comprensión de tales estructuras. Este artículo examina formas en las que se puede aplicar el aprendizaje sobre carreras con datos abiertos e inteligencia del mercado laboral. Un estudio de caso ilustrativo del proyecto LMI for All en el Reino Unido muestra la viabilidad técnica del diseño y desarrollo de dichos sistemas y un modelo para su difusión e impacto. La tendencia hacia los datos abiertos y las aplicaciones cada vez más poderosas para procesar y consultar datos han cobrado impulso. Esto, combinado con la necesidad de información sobre el mercado laboral para la toma de decisiones en mercados laborales cada vez más inestables, ha llevado al desarrollo y pilotaje de nuevos sistemas de información del mercado laboral (LMI), que involucran a múltiples grupos de usuarios. Existen desafíos universales debido al uso cada vez mayor de LMI, especialmente en la asignación de empleos y el uso en rápida expansión de datos abiertos en diferentes entornos de educación y empleo. Destacamos seis temas emergentes que deben abordarse para que los datos abiertos y la inteligencia del mercado laboral puedan aplicarse de manera efectiva en diferentes contextos y entornos. Concluimos reflexionando sobre la urgente necesidad de ampliar el cuerpo de investigación y desarrollar nuevos métodos de co-construcción en asociaciones de colaboración innovadoras.

Palabras clave: carreras, tecnología educativa, trabajo, datos abiertos, LMI, mercado laboral.

We live in a digital world in which learning and work have been transformed by new technologies. Yet, labour markets are increasingly volatile. The idea of a job for life no longer exists. Technology is not only changing occupations with some disappearing and new occupations emerging but also the content of work. At the same time, the effects of the world economic crisis continue to be felt, with persistently high levels of youth unemployment in many countries, including in south Europe.

A McKinsey report (Manyika et al., 2017) found that about half the activities people are paid to do globally could theoretically be automated using currently demonstrated technologies. Yet, it also found that very few occupations—less than 5 percent—consist of activities that can be fully automated. However, in about 60 percent of occupations, at least one-third of the constituent activities could be automated, implying substantial workplace transformations and changes for all workers. Automation and artificial intelligence (AI) is predicted to lift productivity and economic growth, but many people will need to upgrade skills and/or switch occupations.

Set alongside this, in the UK an estimated 700,000 workers are set to retire in the next ten years in the infrastructure and logistics sector alone, leading to a potential skills crisis (HM Government, 2017). More widely across the UK, skills gaps are felt acutely in a number of key sectors. In the NHS, 10 per cent of vacancies are unfilled (Forster, 2017). In the construction industry, more than 80 per cent of employers report skills shortages as a chronic problem and 59 per cent of construction firms believe that the sector’s workforce will not have the required skills to cater for the industry’s future needs and development (London Councils, 2018). In London alone, “28,300 London employers report that not all their employees have the right skills for the job. Almost a quarter (23 per cent) of all vacancies in London are due to a lack of applicants with the right skills, while almost half of firms (42 per cent) are not confident they will be able to recruit people with the higher-level skills their organisation needs over the next five years. The number of cases where employers have been unable to fill a vacancy due to skills shortages in the capital has also more than doubled since 2011 – rising from 14,000 to 37,000” (London Councils, 2018). The OECD also highlights concerns about skills mismatch. This has been accentuated by the recent crisis, but “they go back to at least the 1970s, when increases in the supply of graduates in the United States seemed to outstrip demand” (OECD, 2018).

DECISIONS AND ABOUT LEARNING AND WORK

Decisions about learning and work have to be placed in a particular spatial, labour market and socio-cultural context – individuals are taking decisions within particular ‘opportunity structures’ and their decisions and aspirations are further framed by their understanding of such structures (Roberts, 2015). Within academic literature, there is a significant body of research examining young people’s decision making styles, processes, and strategies. For example, Harren (1979) identified three decision-making styles: rational, avoidant, and dependent. The sample on which this typology was derived was restricted to college students and was conducted nearly four decades ago. However, the typology still resonates, for example, the rational style is often perceived as an active and planned approach to decision-making. The avoidant style is characterised by failure to attain and process information and the postponement of career decisions. The dependent style involves transferring responsibility for decisions to external sources, such as parents, teachers and significant others. The rational decision-making style is often viewed as a systematic approach that yields information relevant to supporting decisions. This has been found to be associated with career maturity (Janeiro & Marques, 2010), cognitive information processing (Sampson et al., 1999), career decidedness (Ferrari et al., 2012), and problem-solving efficacy (Bullock-Yowell et al., 2012). However, there is no conclusive evidence that the rational style is associated with superior decision- making outcomes, nor is this common behaviour. Gati et al. (2012) suggest using the profiles of 11 different decision dimensions as follows:

1. Information gathering reflects the degree of involvement in the collection and organisation of information

2. Information processing refers to the extent of analysis of career information

3. Locus of control is the degree of one’s perceived control over career opportunities

4. Effort invested in the process reflects the time and effort devoted to career decision making

5. Procrastination is the delay in involvement in decision-making tasks

6. Speed of making the final decision reflects the time needed to make a final career decision

7. Consulting with others refers to the extent to which help is sought from other people during the decision-making process

8. Dependence on others is the extent of reliance on others for making the career decision

9. Desire to please others reflects attempts to satisfy the expectations of others

10. Aspiration of an ideal occupation is the desire to find a perfect occupation

11. Willingness to compromise refers to flexibility in one’s career aspirations.

A literature review from adult education, and on adults who return to learning, provides insight to multiple social, cultural and economic factors associated with adults’ decision making when it comes to engagement in learning (Hughes et al., 2016). Gender, ethnicity and socio-economic factors all strongly affect these choices ( Cardoso & Marques, 2008; van Tuijl & van der Molen, 2015).

In all these situations access to detailed up to date information about the labour market is vital. This can include information about employment trends, data on the structure of the labour market and the way it functions, information about different occupations and training courses available as well as job opportunities and data focusing on equality and diversity (Attwell, 2017). Moreover, there is a growing evidence base which supports the use of online career and labour market support tools (Bimrose, 2016; Goss & Hooley, 2015; Nota, Santilli & Soresi, 2016).

Labour Market Information is used by a wide variety of different professionals. These include careers counsellors, employment advisors, education and training providers, researchers and those responsible for planning and policy. But its use is not limited to professionals. Young people thinking about their careers, people considering starting up businesses, unemployed people or those thinking about a career change all need reliable labour market information. Understanding labour market information requires a different set of skills, for both students and advisers (Bimrose, Barnes & Attwell, 2010). Accurate and individualised information (in particular on wage returns) may prevent students from creating false expectations about their choices and labour market outcomes (CEDEFOP, 2016a).

The movement towards Open Data and increasingly powerful applications for processing and querying data has gathered momentum. This combined with the need for Labour Market Information for decision making in increasingly unstable labour markets has led to the development and piloting of LMI systems in many countries, involving multiple user groups including for use on careers advice, guidance and counselling. This development parallels the increasing use of data systems, for example Learning Analytics, in education.

In the next section of this paper we examine definitions of LMI and the scope of LMI systems. This is followed by a case study of the development and implementation of the LMI for All database and API in the UK. Section four examines issues arising from the design and use of LMI systems, many of which have wider implications for the use of data applications across education.

DEFINITIONS: LABOUR MARKET INFORMATION AND ANALYSIS AND LABOUR MARKET INTELLIGENCE

There are varying definitions of Labour Market Information (LMI) and Labour Market Information Systems (LMIS) and the degree of scope of such systems as well as of integration. Woods & O’Leary (2006) indicate:

“Labour Market Information includes any quantitative or qualitative information and intelligence on the labour market that can assist labour market agents in making informed plans, choices, and decisions related to business requirements, career planning and preparation, education and training offerings, job search, hiring, and governmental policy and workforce investment strategies.”

Labour Market Information can be contrasted to Labour Market Intelligence: Labour Market Intelligence is any quantitative or qualitative facts, analysis or interpretation about the past, present or future structure and workings of the Labour Market and the factors that influence it (Attwell, 2017).

Labour Market Information is needed to inform users about:

• economic and labour market conditions,

• education, qualifications, training and skills,

• current and future demand and supply of Labour and jobs; and

• vacancies and recruitment.

LMI includes both Labour Market Information (descriptive data such as statistics or survey results) and Labour Market Intelligence (analysis, interpretation, conclusions and policy recommendations).

Another definition is provided by the International Labor Organisation (ILO) in its report entitled ‘Current Practices in Labour Market Information Systems Development for Human Resources Development Planning in Developed, Developing and Transition Economies’ (Mangozho, 2003):

“Any information concerning the size and composition of the Labour Market or any part of the Labor Market, the way it or any part of it functions, its problems, the opportunities which may be available to it, and the employment-related intentions or aspirations of those who are part of it (Thuy, Hansen, & Price, 2001).”

Chernyshev & Standing (1997) point to the importance of Labour Market analysis:

“A useful information system is one that provides the job seeker as well as the policy maker with information to assess the state of the Labor Market from their respective perspectives…it takes Labor Market statistics and other relevant facts and information and converts them into answers to questions posed by decision makers at all levels of the Labor Market.”

The LMI Institute (undated) draws attention to the importance of workforce analysis and of support mechanisms for users of LMI:

“While data are an integral part of the LMI system, developing workforce intelligence from Labor Market Information requires the integration of data elements and analysis, followed by the provision of those insights through an efficient information delivery system and a user support network. All of these factors are considerations that are integral to creating a truly comprehensive view of Labour market information.”

Labour Market Information Systems

Once more, definitions of Labour Market Information Systems vary. One focus is to examine the information provided through a system. Although different user groups will have different specific needs, the core provision of a LMIS could include:

• Information on the demand for different occupations and their changing work content.

• Information on occupations that are disappearing and emerging as a result of technological changes and structural changes in an economy.

• The employability and career opportunities of young people entering the Labour market, that is information on the school to work transition.

• Future skill requirements of the economy.

• Recurrent manpower shortage areas and analysis of the causes and consequences of such occurrences.

• Information on workplace learning as well as other forms of skills development practices, i.e. participation and investments in training.

• Information on the likely effects of economic as well as other policies on specific target groups, i.e. women, youth, migrant workers and disabled people. (EmployID, 2017)

Another way to view LMIS is in terms of the technology deployed. This is seen in the move towards real time LMIS deploying new technologies including Big Data and algorithmic processes for analysing supply, demand and skills for job vacancies in the Labour Market. However, Martin (2011), in his report for the International Organisation for Migration (IMO), draws attention to organisational and institutional arrangements, rather than technology or data sources. He reports: “A Labour Market Information System (LMIS) is the set of institutional arrangements, procedures and mechanisms by which a country can assure that all relevant labour market information is collected, shared and channelled to relevant institutions to be analysed and processed into labour market matching and policy planning processes”. He highlights: “An LMIS is primarily a “system”, that is, a group or combination of dynamically interrelated, interdependent and interacting elements forming a structure with a common purpose. As far as the elements of this system are concerned, they are institutions, private sector and civil society actors, and the different components of the LMIS.”

To some extent there is a tension between seeing a LMIS for collecting, analysing and disseminating data as a series of organisational and institutional arrangements and as a change process. Martin (2011) proposes ten features for an ‘ideal’ LMIS:

Table 1. An ideal Labour Market Information System (Martin, 2011)
Labour supply information Labour statistics National qualifications register Register of nationals residing abroad Register of foreign residents
Labour demand information Labour market needs analysis Register of international job offers
Labour market matching Register of job seekers Register of job offers Information provided by private placement agencies
Integration, analysis and dissemination Labour market observatory

But at the same time, he indicates “the truly fundamental element of any LMIS is the institutional framework establishing a rational division of competences among relevant institutions, operational mechanisms for inter-institutional coordination, outreach mechanisms for interaction with private sector actors (be they research centres or social partners) and, most importantly, a fluent circulation of information across institutions”. Martin (2011) argues that an outstanding feature of LMISs is that they should be seen as a process rather than as a ready-made structure. Fundamentally, they are evolutionary and have to accommodate changing reality and national and international policy contexts, and also the changing priorities of involved actors and institutions.

There may also be a tension between the view of an LMIS as primarily informing research and policy and an LMIS as an instrument for providing Labour Market information directly to citizens. The European Union has developed a set of quality standards for Labour Market Information (Cedefop, 2016a). These include the requirement for LMI to be pedagogical in design, accurate, up-to-date, take into account users’ needs and to be comprehensible. They point to the need for co- ordination in collection and distribution. The promotion of staff competences in the use of LMI should be supported and LMI should be available through multiple channels.

The following diagram illustrates the basic structural elements of an LMIS including the institutions and functions to be performed, actors and required information flows.

Figure 1. Labour Market Information System: institutions, information flows, actors and components (Martin, 2011)
Labour Market Information System: institutions, information
        flows, actors and components (Martin, 2011)

The definitions outlined above are essentially ideal type representations, with actual LMIS reflecting policy and strategy concerns, institutional arrangements and the availability of data in a particular country or region. They also reflect the evolution of LMIS over time and the different priorities and perceived uses and users of LMIS.

Proposals for developing an LMIS need to take into account all the different dimensions included in the definitions. These dimensions can be summarised as:

• Data collection, processing and dissemination

• Analysis and development of answers to questions

• Institutional arrangements, procedures and mechanisms

• Fluent circulation of information across different institutions

• Multiple channels for different user needs, taking into account pedagogical design

• Promotion of staff competence in use of LMI

• Ongoing monitoring and evaluation.

LMI FOR ALL

LMI helps careers advisers, Work Coaches and other careers and employment practitioners to understand the dynamic interplay between ‘supply’ and ‘demand’ in fast-changing labour markets (EmployID, 2018). LMI also helps policy makers and planners develop strategies for the labour markets of the future. Additionally, LMI helps course designers and curriculum developers to consider the skills that will be needed in the future for those embarking on education and training programmes. Overall, technology has served as one means to remove potential barriers and limitations in connecting and delivering career and employment support services to young people and adults. One way that the UK government has addressed the relationship between government and non-governmental careers and labour market information is to underpin emerging websites and careers resources is though the LMI for All (http://www.lmiforall.org.uk/) Application Programming Interface (API). This research and development work is currently funded by the Department for Education (DfE). LMI for All brings together government sources of information and makes them available in a form that other organisations can use. The O*Net resource in the USA (https://www.onetonline.org/) fulfils a similar function to LMI for All in the UK of providing underpinning national LMI. In this section, we present a case study of the UK LMI for All project to help illustrate some technological advances being made in the use of real time LMI data. We then move on to discuss lessons learned and implication for policy, research and practice.

The idea underpinning LMI for All is to design and develop a comprehensive careers labour market information (LMI) database that links and opens up career- focused LMI. The intention is to optimise access to, and use of, core national data sources that can be used by developers to create websites and applications to support individuals make better decisions about learning and work. This unique approach has been created as an open data project, which supports the wider UK government agenda to encourage use and re-use of government data sets (Cedefop, 2016b).

The development of LMI for All has extended over the period 2012 to 2018. A prototype developed in early 2012 tested the feasibility of an LMI database to develop targeted applications and websites. A key imperative is to find ways of presenting LMI in a more engaging way and targeted at specific audiences, in order to make LMI more widely deployed and to play a more active role in informing the operation of labour markets. In successive iterations (as part of a pilot project from 2012-2015) the feasibility of developing a comprehensive career LMI database that exploits open data sources for mainstreaming into service provision was tested successfully. For example, a series of bespoke Hack and Modding days were delivered in partnership with a specialist ICT communications organisation bringing creative developers, designers and industry experts together to solve real LMI challenges. This also supported the testing of the careers database API.

In the pilot stage (2013-2015), the research team brought together common and consistent information from a range of national sources and surveys – drawing on the detailed knowledge of LMI sources of members of the research team. A further implementation stage of research from 2015-2017 supported further development, including updating of the technical infrastructure, updating of existing data sets, exploration and assessment of new data sets, as well as continued dissemination of the LMI for All service.

To deliver LMI for All, a research team from the Institute for Employment Research at Warwick University worked in partnership with a technical team from a Wales based small and medium sized enterprise (SME), Pontydysgu. By linking and opening up careers focussed LMI, the web portal provides a rich data source to improve the effectiveness and efficiency of organisations involved with, and/ or directly providing services that support individuals in making better informed decisions about learning and work. Most importantly, this data source is freely available for any third-party developers wishing to harness its potential in their own particular operational context. The purpose of these applications is determined and developed by these third-party developers.

The ‘LMI for All’ database contains the following key labour market indicators, which for the first time are available from a single access point:

• Employment

• Projected employment levels

• Future job openings (replacement needs)

• Weekly pay

• Changes in pay

• Weekly hours

• Occupational descriptions;

• Skills, Abilities, Interests and Knowledge (based on US O*NET data);

• Unemployment rates

• Current vacancies

• Census data (details of geographical location of jobs and travel to work distances);

• First destinations of graduates.

• The LMI data generally covers the following dimensions/characteristics:

• 369 detailed occupational categories (SOC2010 4-digit level);

• 75 detailed industries (roughly equivalent to SIC2007 2 dig level);

• Employment status (full-time, and part-time employees and self-employment);

• Highest qualification held (9 levels of the National Qualification Framework);

• Countries and English regions within the UK; and

• Gender.

Stakeholder engagement and activities

The overall aim for LMI for All from its inception to its continued development to date has been to create a data tool that developers would use to create products to support individuals in making better decisions about learning and work. Since 2013, a range of stakeholder engagement activities have been undertaken to raise awareness and understanding of the LMI for All resource among key audiences, but importantly to raise demand by publicising the completed data tool. Stakeholder engagement activities have covered a range of different categories of stakeholders and partners Stakeholder engagement activities have focused on:

• Motivating, engaging and supporting third parties to develop applications or enhance existing applications in order to increase the reach of LMI for All among end-users;

(Careerometer);• Motivating and supporting third-parties to implement the LMI for All free widget (Careerometer);

• Disseminating the LMI for All service, current users and widget through various dissemination activities (e.g. conference presentations, training workshops, use of social media and publications);

• Driving end-user traffic (including careers guidance professionals, teachers and other stakeholders) to existing websites and applications that offer access to LMI for All data.

The ultimate success of LMI for All is contingent on uptake. Consequently, considerable effort has been expended on engagement activities to motivate, encourage and support the integration of the data tool into practice with a range of stakeholders; to gather lessons learned from stakeholders relating to the process of integration into practice; and to increase awareness of LMI for All across the broad community of careers practice through dissemination activities (DfE, 2018). Whilst the primary activity has been across the UK, presentations have been made to audiences across Europe and in Canada and Chile.

There are a number of websites and web interfaces that have been developed throughout the lifetime of the project and targeted at a range of users, including careers guidance professionals, public employment services (PES) work coaches, students in compulsory and post-compulsory education, schools and colleges, third sector professionals and local, regional and national policy makers. The LMI for All

system does not allow the easy identification of third party users, but it is thought that there are over 70 different applications using LMI for All data.

Figure 2. (below) shows the daily number of queries between in June 2018

Figure 2. No. of queries of LMI for All database
No. of queries of LMI for All
        database

The LMi for All widget

A key challenge is often the ability of some organisations to develop applications. They frequently lack developer skills or even knowledge of what skills they require and also lack the resources to contract programmers. To overcome this LMi for All has developed ‘a widget’, which can easily be embedded in websites with no technical knowledge required. The widget, which is configurable, displays up to 3 ‘cards’. Each card displays information about an occupation including:

• Annual, weekly and hourly pay and working hours.

• Projected changes in the workforce for the occupation.

• Sectors of employment or the occupation.

• Occupational profile including relevant qualifications.

Figure 3. The Careerometer widget
The Careerometer widget

Some UK case studies

The following abridged case studies illustrate how LMI for All data and the widget is being used in practice. These are briefly presented in vignettes to demonstrate the diversity of approaches.

KareerHub

In August 2016, a grant from InvestNI and the European Commission was provided to develop this platform, drawing on innovative developments in the fields of (a) online information management systems; (b) the application of ICT in differing education settings; and (c) careers education, information, advice and guidance. KareerHub applies technological solutions to inform and support individuals with their career decision-making in fast changing and often-unpredictable education and labour markets. The use of reliable real time data combined with innovative teaching and learner support materials helps brings alive options and opportunities.

Initially technologies that could be utilised for the platform were researched. For example, the team engaged with ElasticSearch, Apache Solr and other technologies. The second development phase involved researching the relevant datasets available through LMI for All and looking at how others were using LMI for All. The team has successfully completed the design and development of KareerHub “Krawlers” which can be pointed at appropriate education and training organisations across the UK. This is a digital solution supporting enhanced career decisions for young people. It provides a space to access real time information about courses in further and higher education and about apprenticeships, placing this alongside reliable Labour Market information. This involves drawing information into a single, easy-to-use place; structuring high quality information, including LMI; applying a geographic lens to the information; including apprenticeship information from local employers; and providing costs, qualifications and other course information. The diagram below illustrates how this works in practice.

Figure 4. Kareer Hub
Kareer Hub

The digital technology and algorithm can push up-to-date live information to the end-user on the basis of chosen searches and filters from across the UK, rather than a user having to painstakingly search for and pull information to themselves from a variety of sources. Searching the web and storing information for later use and reference is time-consuming and comparing diffuse information across many differing formats is difficult. KareerHub integrates LMI for All into a simple-to-use careers information package which delivers it direct into the classroom, onto pupils and parents tablets/smartphones thus making it usable by all, rather than a few specialist users.

icould

icould is a small charity funded by a number of sources, including philanthropy. This charitable organisation was an early adopter of LMI for All data, using it to underpin career videos and to visualise useful job information using charts and icons. It is based on the idea that someone could watch the story of someone talking about their job and then scroll down to the labour market information data for more information.

The approach uses real voices to help individuals learn more about the world of work. Using video interviews, the free-to-use website offers an insight into the day to day roles and career pathways of people working in a range of jobs. icould includes over 1,000 careers video stories that provide a real view of what it is like to work in particular jobs. Articles on a range of topics are available from people sharing their experiences, together with advice from industry experts, for example, information for young people and parents on key educational decision points and career-related topics (such as making job applications, CV writing, interviews and networking). There are also articles on a range of sectors, which describe potential future opportunities. This is used within schools as part of careers education lessons, as well as people on an individual basis. In 2017, icould had over 140,000 unique users per month, with users spending an average of six minutes per visit. It is used by a wide range of education establishments and individuals, mainly in the UK.

For each video, LMI information is provided, including average salary, average weekly hours, past unemployment, predicted employment, top 10 industries for the job, employment status of workforce, occupational description, qualifications required for job, tasks, employment by region, gender profile of the workforce and the skills important for the job. The LMI for All data are also being used as part of the regional and national webpages to show the most popular jobs and the jobs with the highest number of vacancies. Providing labour market information in a user friendly way is seen as significantly adding to the careers story featured in each video.

Department for Work & Pensions (DWP)

Due to recent changes and new strategies, the role and function of Public Employment Services (PES) have changed (PES Network, 2018). It is argued that the social function of PES is transforming into work-focused gateways to welfare systems, providing a more tailored service delivery. PES practitioners, therefore, need to operate differently, in roles and capacities that require them to adopt, and adapt to, a new professional identity formation. To assist with this process, DWP developed an LMI for All application, namely ‘Coach Central’ (a LMI dashboard for practitioners) that was rolled out as a resource for various staff groups in May 2017. The process of developing and piloting Coach central took place between 2014 – 2017 through the support of a European funded EmployID project. Coach Central is based on LMI for All and local data from the Office for National Statistics.

LMI is essential for PES staff (both work coaches and employment advisers):

• to help claimants take decisions on jobs, careers and training;

• to familiarise themselves on the structure of the labour market and on-going changes. For example, it can be used by employment advisers in understanding needs and opportunities in different sectors;

• to be used for labour market planning and for understanding the need for future training provision;

• to help plan future strategies and priorities for services for planning; and

• to help those responsible for employer liaison in understanding sectors and industries, prior to meetings with employers (EmployID, 2018).

At an organisational level, access to LMI is needed to enhance co-operation between PES employees in different roles. The key to unlocking the transformational potential of Coach Central by PES organisations is based on how it is integrated within the individual and organisational practices. Both require an understanding of the meanings of data for future labour market structures and for jobs and careers for claimants. An essential ‘top-down, bottom-up’ process was used to secure buy- in for the online course and LMI for All application. Key members of the Learning and Development unit and senior members of staff were involved at all stages of the process of identifying, then designing learning needs. A pilot massive open online course (MOOC) within DWP has shown promising results. The Coach Central application has now been rolled out across the entire organisation. It is being provided as a resource for anyone in DWP (not just practitioners), but its usage is not compulsory. The principle of providing DWP practitioners with some guidance and support regarding the interpretation and usage of LMI in their work with claimants is currently being discussed. Some crucial issues were identified during the development of Coach Central and solutions were tested. For example:

• ICT privacy and security policies constrained the use of technology, which resulted in creative technological solutions (such as negotiating for a closed course where DWP staff could comment and have private forums) were piloted. Within the organisation, hosting the platform for Coach Central has been an ongoing issue because of the challenges of privacy and security policies and the restrictions placed on the use of ICT.

• Initial forecasts for the resources involved in the development of Coach Central were underestimated. While it has been a more resource intensive process, the future benefits and efficiencies in the course have been realised.

• The lack of local data has been a major criticism of Coach Central. Even though the Office of National Statistics provides local Labour Market data, this is based on broad industry groupings and not on occupations. The sample size in the major surveys is not sufficient to produce meaningful data for each occupation at a local level. Two promising solutions are to use web scrapers to augment data with other sources and/or augment official data with data gathered from ‘the crowd’ – for instance from local PES employees.

• The problem of lack of data also applies to lack of job vacancy notifications. PES job vacancies tend to be skewed towards low paid and public sector jobs. One potential answer would seem to be to use web scrapers to gather data although this may have copyright implications.

The Help Build London App

Apps offer considerable benefits as they can drive information direct to users and have the potential to be combined with location aware technologies to provide bespoke information. For example, a UK careers and employment specialist company responsible for delivering a range of services to over 500,000 young people and adults each year explored ways of providing LMI for All data via an app to raise awareness about career opportunities in London, particularly in the construction sector. Its user base is wide ranging from young people in schools, vulnerable young people and those not in education, employment or training (NEET), plus adults and offenders. The ‘Help Build London’ app is designed to complement campaigns and initiatives that aim to attract, inform and retain a talented workforce for the construction and built environment sector. This also encourages users who are wedded to one occupation to consider others that may fit their skills or that have better employment prospects or are in high demand in London. Research undertaken by Prospects had found that 90% of their customer base has access to a mobile phone; most of which are smart phones.

It is not intended that the app will be used on its own. Instead, Prospects’ career advisers will support people to access and use the app during a careers session, as well as respond to careers and learning questions arising from its use. Telephone advisers will also encourage customers to download the app and support them in exploring the data.

Issues encountered during the development of the ‘Help Build London’ app included determining the focus of the app, refining initial ideas, resourcing technical skills and managing the development within time and budgetary constraints.

The LMI for All datasets have been selected as they are key to answering some of the questions users will have about the construction sector. Data included in the ‘Help Build London’ app includes:

• Will there be future job opportunities in the construction sector?

• What skills do I need to work in construction?

• What jobs are there in construction?

• Employment (historical, projected and replacement demand) Skills, abilities, interests data from O*NET.

Figure 5. Help Build London
Help Build London

A client who is interested in the construction sector in London, enters the app and selects two skills to find two potential job options (such as painters and decorators, production managers, electricians, etc.) in construction to compare. These options are based on a selection of skills. The user can then find out more about each job option, such as data on how these trades are growing across London, pay and hours, and required qualificators. A chart displaying the Working Futures data on projected employment and employment (replacement demands) is shown. A user can also access a job profile, ring to make an appointment with a specialist careers adviser and access the Build London jobs board to access current vacancies.

DISCUSSION

The rapid expansion of technology including the growth of open data and peripatetic devices including smartphones, apps, tablets and wearables (e.g. smart watchs) has resulted in new methods of accessing LMI and learning about careers. However, the LMI for All case study and other research undertaken by the authors with Public Employment Services in Europe and in Chile raises a number of issues. Some of these are about the availability of accurate and relevant data. Others relate to how best to meet differing stakeholder needs, for example, careers services and other related support services. Moreover, there are more general concerns about data literacy and understandings of data and its increasing usage in education and employment sectors. Governance, provenance and quality are also key issues that have to be addressed. The concept of accurate job matching is contested. Overall, further research is needed expand upon these findings. In this section we briefly discuss some of these key issues.

Availability of relevant data

A big problem in the UK (and also in many other countries) is linking labour market and educational data. Labour Market data is classified by occupations, educational data is usually classified by subject or discipline. In some countries there are clear qualifications required for employment in different occupations, but in other countries the link is much weaker. Yet one of the most frequently asked questions by individuals is what courses they should pursue to enter a particular occupation. One approach to this is to use destination data from school or university - i.e. a student who followed this course ended up in this job or occupation. Yet such raw data may disguise a large number of personal, social and educational variables.

Indeed, there is a wider issue when we talk about skills and employment. Because we have no accurate measure for knowledge and skills, we use qualifications and level of qualifications as a proxy for learning and the acquisition of knowledge and skills (OECD, 1997). This may lead to serious inaccuracies in studies and surveys of workforce skills. In the UK, there is a further issue in that the Labour Force Survey only records the highest qualification gained by an individual – thus potentially failing to record lower level vocational qualifications.

A further issue with LMI systems (LMIS) is that they frequently lack sufficient local data. This is due to sample sizes of surveys. Yet local labour markets vary considerably within countries, especially with a tendency for regional clusters in some sectors. The DWP case study above has attempted to use LMI dashboards that encourage practitioners to create and capture where possible local knowledge including vacancy information.

Differing needs

Different LMI is needed for different users. Colleges and universities use LMI for planning courses and curricula. But although some of the data needed is in common, using LMI for choosing a career or a course requires different data. Furthermore, individuals will need different LMI at various ages. Younger school students may just want to browse different possible occupations and find out more about them. Older students will require more focused information about where such a course might take them in terms of their career. This has implications not just for data availability, but also in terms of interface design. In this regard the LMI for All open database and API is an approach to allowing the design of multiple interfaces and applications. It still requires knowledge and skills both in user interface and application design and an understanding of careers guidance, labour markets and education and training if applications are going to meet the needs of end-users. Notably, even where information is of a high quality it often does not get used as individuals lack the time, skills and knowledge to make effective use of such information (Grubb, 2002). In many cases, individuals can often find themselves overwhelmed by the volume of information available and as a result decide not to use this information (Sampson et al, 1999). The over simplistic implementation of LMI resources which assume that the provision of information is all that is required and where scant attention is given to how to integrate this within training and professional development is the most common factor in failure to meet differing stakeholders’ needs (Kettunen & Sampson, 2018).

Data literacy

Data literacy is the ability to read, understand, create and communicate data as information (Wikipedia). Accepting literacy as a general concept, data literacy focuses on the competencies involved in working with data. Making sense of labour market data is not simple and there are doubts whether end-users have sufficient data literacy to deal with complex data, however good the design of applications. There is increasing debate and clarification around the skills and competencies that careers practitioners need in order to make successful use of online technologies as part of their practice (Kettunen, Sampson & Vuorinen, 2015; Kettunen & Sampson, ibid).

This raises the important issue of whether applications should be designed to be used with a careers counsellor or should be developed to be used by young people and job seekers themselves. However, the experience of the EmployID project, working with Public Employment Services in Europe, suggests that even experienced employment advisers and work coaches may not have the required data literacy to fully make sense of the data. Developing Labour Market Intelligence through sense and meaning making requires training. While in most countries there is an increasing focus on developing online access to Labour Market data, there does not appear to be a corresponding effort in providing training in how to make meanings and use that data.

Governance, provenance and quality

Who should govern Labour Market Information Systems is an issue in many countries. The provenance and quality of the data is a key issue. Even when high quality and reliable data is provided it is important that information is provided about the source of that data and how the data has been processed. Certainly, the development of LMIS requires collaboration between education and training organisations and economic and labour market bodies. This can be hard to achieve. Ideally LMIS would be the responsibility of broader bodies involving all interested parties, employers, trade unions, teachers, trainers, careers counsellors, employment advisers, work coaches, data producers, educational organisations etc.

Job Matching

The European Union has been setting up its own skills policy (Education & Training 2020) to address inter-alia, the challenges set by the digital revolution in societies, labour market, education system and different other policy fields. “Skills, competences, and qualifications that people need change over time. To deal with these changes people need to be equipped with a variety of basic skills … including career management skills and access to updated and validated labour market intelligence through the lifespan-”. (Education & Training 2020). Competence- based job matching is a method to find the best match between jobseekers and vacant jobs by comparing knowledge, skills and competences of the candidate with the requirements of the employer (ESCO, 2018). New database technologies and the use of algorithms can make online job matching relatively simple. However, Bimrose (undated) cites Mitchell & Krumboltz who criticise the usefulness of job matching in current labour market conditions. Matching, they say, assumes a degree of stability in the labour market. Osipow & Fitzgerald (1996) also highlight the failure of the theory to address the issue of change in environments and individuals. Additionally, they draw attention to problems inherent with the theory’s associated measures for gender but regard the most serious limitation to be its failure to explain the process of personality development and its role in vocational selection (Osipow & Fitzgerald, ibid). There is a danger that with increasing sophistication of automation and AI embedded in LMI systems linked to job matching, there will be seen as less need for human forms of careers advice and guidance. The use of AI and applications utilising machine learning may seem to provide a ‘better fit’ but remain open to the criticism levelled above. A further issue is the lack of transparency in the algorithms for job matching. With algorithms being applied in a broadening range of domains, it is crucial to ensure that machine-generated decisions are as fair and unbiased as possible, especially when they affect human lives. Olhede Rodrigues (2017) highlight “the problem is that much of what an algorithm does is hidden from view – inscrutable to those whose data it feeds on”.

Research and Evaluation

There is a general lack of research in how to use LMI in careers and employment support services. In part, sophisticated LMI systems are relatively new. Research also requires boundary crossing bringing together careers theory, labour market theory, ICT and user experience design specialisms. But research is extremely important if we are to better understand how LMI data is being used and what the potential issues and drawbacks are. This also requires detailed evaluation and the publication and sharing of evaluation results by LMI providers.

CONCLUSIONS

In this paper we have explained the economic and social background to the development of Labour Market Information systems that can support more effective career learning and decision making. The illustrative case study of the LMI for All project in the UK shows the technical feasibility of designing and developing such systems and a model for dissemination and impact. The Department for Education in England has now granted a continued extension of the research and development work (2018 – 2020). Meanwhile, there remain universal challenges given the increasing use of LMI, especially in job matching and the rapidly expanding use of open data and big data in education and employment settings. While technologies play a key role in LMI systems and processes, they also include the actors and institutions within the system. The evidence points to a number of findings which can be turned into lessons for practice. Socio technical developments require design and development and steering and that in turn requires an understanding of the underpinning educational and careers guidance theories and practices. The use of online LMI is not a replacement for professional careers information, advice and guidance. But it does require investment in training and development for practitioners and managers. LMI systems have to be embedded within a wider range of careers and employment support services if they are to increase individuals’ access to and awareness of labour market trends now and in the future. The effective use of LMI online resources using a variety of media and face-to-face channels should be integrated into planned careers programmes and is likely to be more effective for some individuals when use, at least initially, is mediated and supported by trained careers professionals. Finally, if ICT and LMI tools and resources are to become spaces for learning and reflection in support of individuals’ career construction, there is clearly an urgent need to extend the body of research and to develop new methods of co-constructing in innovative collaborative partnerships.

REFERENCES

Attwell, G. (2017). Developing an Integrated Labour Market Information System in Chile, unpublished Consultancy Report, December 2017.

Bimrose, J. (undated). Traditional theories, recent developments and critiques, Coventry: University of Warwick, Institute for Employment Research (IER). Retrieved from https://warwick.ac.uk/fac/soc/ier/ngrf/effectiveguidance/improvingpractice/theory/traditional/

Bimrose, J. (2016). Constructivism in online career counselling. In M. McMahon, Career Counselling: Constructivist Approaches, (210-221).

Bimrose, J., Barnes, S. A., & G. Attwell (2010). An Investigation into the Skills Needed by Connexions Personal Advisers to Develop Internet-Based Guidance, Reading: CfBT Education Trust and Warwick University, Institute for Employment Research.

Bimrose, J., Hughes, D., & Barnes, S. A. (2011). Integrating new technologies into careers practice: Extending the knowledgebase. London: UK Commission for Employment and Skills.

Bullock-Yowell, E., Katz, S.P., Reardon, R.C., & Peterson, G.W. (2012). The Role of Negative Career Thinking and Career Problem Solving Efficacy in Career Exploratory Behaviour. Professional Counselor, 2(2), 102-114.

CEDEFOP (2016a). Labour market information and guidance. Cedefop Research Paper, 55. Publications Office, Luxemboug, doi: http://dx.doi.org/10.2801/72440

CEDEFOP (2016b). Improving career prospects for the low-educated: the role of guidance and lifelong learning. Cedefop Research Paper, 54, Luxembourg: Publications Office.

Chernyshev, I., & Standing, G. (1997). Statistics for Emerging Labor markets in Transition Economies: A Technical Guide on Sources, Methods, Classifications and Policies (ILO Studies Series).

Cardoso, P., & Marques, J.F. (2008). Perception of career barriers: The importance of gender and ethnic variables. International Journal of Educational / Vocational Guidance, 8, 49-61.

Department for Education (2018). Labour market information (LMI) for all: Stakeholder Engagement and Usage, Data and Technical Developments, Research report, London: Department for Education. Retrieved from https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/714565/Labour_market_information_for_all.pdf

EmployID (2017). The Changing World of Work Massive Open Online Course. Retrieved from https://mooc.employid.eu/the-changing-world-of-work/labour-market-information-lmi/

EmployID (2018). EmployID Year 4 Online Book. Retrieved from https://employid.eu/sites/default/files/y4book.pdf

ESCO (2018). Competence-based Job Matching, Brussels: European Commission, European Skills / Competences, qualifications and Occupations. Retrieved from https://ec.europa.eu/esco/portal/escopedia/Competence-based_job_matching

European Commission (2017). Development of Skills, Education and Training, Brussels. Retrieved from https://ec.europa.eu/education/policy/strategic-framework/skills-development_en

Forster, K. (2017). NHS faces ‘unprecedented workforce crisis’ as vacancies rise 10% in last year, The Independent, 25 July 2017. Retrieved from http://www.independent.co.uk/news/health/nhs-staff-vacancies-rise-10-per-cent-2017-86000-nurses-midwives-doctors-recruitment-crisis-brexit-a7858961.html

Ferrari, L., Nota, L., & Salvatore, S. (2012). Evaluation of an Intervention to Foster Time Perspective and Career Decidedness in a Group of Italian Adolescents, Career Development Quarterly, 60(1), 82-96.

Gati, I., Gadassi, R., & Mashiah-Cohen, R. (2012). Career decision-making profiles vs. styles: Convergent and incremental validity, Journal of Vocational Behavior 81(1), 2-16, doi: 10.1016/j. jvb.2012.03.004.

Goss, S., & Hooley, T. (2015). Symposium on online practice in counselling and guidance (Editorial). British Journal of Guidance and Counselling, 43(1), 1-7.

Grubb, W. N. (2002). Who am I: The inadequacy of career information in the information age. Paris: Organisation for Economic Cooperation and Development.

Harren, V. A. (1979). A model of career decision-making for college students. Journal of Vocational Behaviour, 14(2), 119-133.

HM Government (2017). Industrial Strategy: Building a Britain Fit for the Future, London. Retrieved from https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/664563/industrial-strategy-white-paper-web-ready-version.pdf

Hughes, D., Adriaanse, K., & Barnes, S-A. (2016). Adult Education: Too Important to be Left to Chance – International Literature Review, Coventry: University of Warwick, Institute for Employment Research, October 2016

Janeiro, J. N., & Marques, J. F. (2010). Career coping styles: Differences in career attitudes among secondary school students. International Journal of Educational and Vocational Guidance, 10, 35-48. doi: 10.1007/sl0775-009-9170-317.

Kettunen, J., Sampson, J. P., & Vuorinen, R. (2015). Career practitioners’ conceptions of competency for social media in career services. British Journal of Guidance & Counselling, 43(1), 43-56.

Kettunen, J., & Sampson, J. P. (2018). Challenges in implementing ICT in career services: perspectives from career development experts. International Journal for Educational and Vocational Guidance, 1-18.

LMI for All (2012). Key Findings from the careers LMI feasibility study, Coventry: University of Warwick, Institute for Employment Research (IER). Retrieved from http://www.lmiforall.org.uk/wp-content/uploads/2015/07/2012_LMI_for_All_summary.pdf

LMI for All (2015). UK Commission for Employment and Skills LMI for All: Developing a Careers LMI Database: Final Report, July 2015, Coventry: University of Warwick, Institute for Employment Research (IER). Retrieved from http://www.lmiforall.org.uk/wp-content/uploads/2015/07/2012-15_LMI_for_All_Final_report_v020715.pdf

LMI for All (2016). LMI For All: Technical, Stakeholder Engagement and Data Development Services Final Report 2015-16, Coventry: University of Warwick, Institute for Employment Research (IER). Retrieved from http://www.lmiforall.org.uk/about-lmi-for-all/

LMI Institute, (undated). Defining Labor Market Information (LMI) and LMI Customers, Arlington, VA, Retrieved from http://www.lmiontheweb.org/WhatWeDo/Publications/downloads/2014-06-27_-_Defining_Labor_Market_Information.pdf

London Councils (2018). Skills Gap in London, London: London Councils. Retrieved from https://www.londoncouncils.gov.uk/our-key-themes/economic-development/adult-skills-0/bridging-skills-gap/skills-gap-london

Mangozho, N. (2003). Current Practices in Labor Market Information Systems Development for Human Resources Development Planning in Developed, Developing and Transition Economies. Skills Working Paper, 13, ILO, Geneva.

Manyika, J., Chui, M., Miremadi, M., Bughin, J., George, K., Willmott, P., & Dewhurst M. (2017). Harnessing automation for a future that works. Retrieved from https://www.mckinsey.com/global-themes/digital-disruption/harnessing-automation-for-a-future-that-works

Martin, I. (2011). Labour Market Information Systems and Labour Migration Information in Six Developing Countries: The challenge of integration, International Organization for Migration, Brussels.

Mitchell, L. K., & Krumboltz, J. D. (1996). Krumboltz’s learning theory of career choice and counseling. In D. Brown, L. Brooks, & Associates, Career choice and development (3rd ed.), (233-280). San Francisco: Jossey-Bass.

Nota, L., Santilli, S., & Soresi, S. (2016). A life- design-based online career intervention for early adolescents: Description and initial analysis. The Career Development Quarterly, 64(1), 4-19.

Olhede, S., & Rodrigues, R. (2017). Fairness and transparency in the age of the algorithm, London: Royal Statistical Society. Retrieved from https://rss.onlinelibrary.wiley.com/doi/full/10.1111/j.1740-9713.2017.01012.x

Osipow, S.H., & Fitzgerald, L. F. (1996). Theories of Career Development (4th Edn), Needham Heights, Massachusetts, Allyn & Bacon.

OECD (2018). Working It Out: Career Guidance and Employer Engagement, Paris: Organisation for Economic Cooperation and Development, July 2018, p.21.

PES Network (2017). Annual Report of the European Network of Public Employment Service January – December 2018, Luxembourg: European Commission.

Roberts, K. (2015). Implications for careers policy. In J. Bimrose, M. McMahon & M. Watson, Women’s Career Development Throughout the Lifespan: An international exploration, London: Routledge.

Sampson, J. P., Jr., Lenz, J. G., Reardon, R. C., & Peterson, G. W. (1999). A cognitive information processing approach to employment problem solving and decision making. The Career Development Quarterly, 48(1), 3-18.

Thuy, P. Hansen, E., & Price. D. (2001). The Public Employment Service in a Changing Labour Market. Geneva: International Labour Office.

van Tuijl, C., & Walma van der Molen, J. H. (2015). Study choice and career development in STEM fields: an overview and integration of the research. International journal of technology and design education, 26(2), 159-183. doi: 10.1007/s10798-015-9308-1.

Woods, J. F., & O’Leary, C. J. (2006). Conceptual Framework for an Optimal Labour Market Information System: Final Report, Upjohn Institute Technical Report No. 07-022. Kalamazoo, Michigan: W.E. Upjohn Institute for Employment Research.

Author notes

1 Graham Attwell: Is Director of the Wales based research and development SME, Pontydysgu and Associate Professor at the University of Warwick. His research interests include the use of technology for learning and for sharing knowledge and the teacher development for using ICT. He is interested in how open data and Lmi can be used to support decision making round education and training and careers. He leads the technical development of the UK government supported LMI for All database and API.
email: grahamattwell@pontydysgu.org
2 Deirdre Hughes: Is an international and national expert in career development policies, research and practice. She is an Associate Fellow at the University of Warwick, Institute for Employment Research (IER) and Director of DMH Associates in England. She has worked on the UK LMI for All initiative since its inception. Cedefop has appointed Deirdre as a UK expert in lifelong guidance and she is an adviser to the Organisation for Economic Co-operation and Development (OECD). She is a prolific writer, researcher and policy adviser.
email: deirdre.hughes3@btinternet.com

Enlaces refback

  • No hay ningún enlace refback.





RIED. Revista Iboeroamericana de Educación a Distancia (La Revista Iberoamericana de la Educación Digital)



Director/Editor:
Lorenzo García Aretio
UNED, Facultad de Educación
C/ Juan del Rosal, 14
28040 Madrid (Spain)
ried@edu.uned.es

ISSN: 1138-2783
E-ISSN: 1390-3306
Depósito Legal: M- 36.279 -1997
Edita: Asociación Iberoamericana de Educación Superior a Distancia (AIESAD)
Madrid (España)


Reconocimiento NoComercial (by-nc): Se permite la generación de obras derivadas siempre que no se haga un uso comercial. Tampoco se puede utilizar la obra original con finalidades comerciales.


Síguenos en: 

https://2.bp.blogspot.com/-wtzwURZeg6I/V_y8vM5DmdI/AAAAAAABKKQ/y_fW6U2dW3cOLG6z-tUwJ9u1Pwt9ltXHACLcB/s320/blogger_b_logo.jpg  https://4.bp.blogspot.com/-Q3lAzaCezXA/V_TZ0BTuIkI/AAAAAAABKF4/wP8QRQVCPiQnk0sE7nEDnZHY5F03AOjbgCLcB/s200/twitrer_120%2B%25281%2529.jpg   https://4.bp.blogspot.com/-4So1RLxqN7Q/VHMWABdXX9I/AAAAAAAAb4E/mV00Ac5Gm-Q/s1600/fb_icon_325x325.png    https://1.bp.blogspot.com/-S7ecZmnt3os/Vzmf77J7EfI/AAAAAAABEYc/g3MJ_0z_noUtAiLS7MRRHXgzOkGbZbfUACLcB/s200/scholar_logo_lg_2011.gif


Colaboran con RIED:

https://2.bp.blogspot.com/-VKcDNIR3Sqk/V_aPanb6P0I/AAAAAAABKIA/XSdUeendX2wJ_afKOCIIkxkZjW0ZnT0vACLcB/s320/logoCUED.jpg       https://3.bp.blogspot.com/-wxw5W-VCRGA/WAnp69yeyuI/AAAAAAABKgo/LHi490KturcyZQE7KnlK2ZT9taWEUXkgQCLcB/s320/logo-AM2.01.png    Alteridad