Towards better tools to measure social and emotional skills

by Anna Choi
Analyst, Directorate for Education and Skills

Koji Miyamoto
Analyst, Directorate for Education and Skills

Common sense and hard evidence point to the significant impact of socio-emotional skills such as perseverance and responsibility on children's lifetime success. Empowered children are much more likely to finish college, maintain healthy lifestyles and be happy. Both parental and teacher experiences as well as emerging studies also underlie that social and emotional skills can be particularly malleable from childhood until adolescence.

*Sample limited to white males with at least a high school diploma

The OECD report: “Skills for progress report” shows that American high school students who were at the highest decile of social and emotional skills distribution are 4 times more likely to self-report completing college than those who are in the middle decile (median). Needless to say, a growing number of evidence indicate how these skills can have lasting positive effects on a wide-range of outcomes, such as life satisfaction, emotional health, and well-being (see the report for further evidence and review).

Moreover, we have started to understand that some of these social and emotional skills that drive children's lifetime success are malleable and can be developed during childhood and adolescence. Yet currently, there appears to be only a few countries and school districts that have wide-ranging policies in place to foster social and emotional skills. This may in part be due to the fact that existing evidence doesn’t yet provide sufficient details on what works, for which skills, for whom, when and under what conditions. This may come as a surprise since social and emotional skills are not necessarily difficult to define or measure than cognitive skills for which we have good evidence-base. Perhaps we simply have not paid as much attention to conceptualise measures of conscientiousness or leadership skills as algebra or reading comprehension?

Why is the evidence base still limited? An important reason is likely to be the lack of reliable and well-defined measures of the range of social and emotional skills that matter for people's lives. The most popular measures we currently use are children's self-reports or ratings by parents and teachers. While these measures can provide valuable information, they also can be subject to a variety of biases including acquiescence, social desirability, faking and reference groups. There are a range of methods that are designed to account for these biases (e.g., anchoring vignettes and forced choices), but they have not been extensively tested. There are also measures that are arguably designed to directly capture these skills (e.g., performance tests and experimental games), but they have also not been subject to extensive tests.

The OECD's Longitudinal Study of Social and Emotional Skills in Cities is addressing the measurement challenges by developing valid and reliable measures of social and emotional skills that are comparable across different cultural contexts. This study will explore a variety of methods to measure these skills to better understand their development during childhood and adolescence as well as the learning contexts that could help drive this process. The OECD will spend 2016-19 on developing measurement instruments, which will be followed by the longitudinal follow-up of primary and secondary school children in grade 1 and 7 in several major cities around the world. In parallel, the OECD is working on various projects designed to understand how different learning contexts (such as family, teachers, school, and community) can help improve children's social and emotional skills using existing longitudinal data sets.

By developing robust measurement tools and longitudinal data through the Longitudinal Study of Social and Emotional Skills in Cities, we can help not only students, parents, and teachers, but also employers and society at large. Through this study, the students can have a better picture about their capabilities and their development over time. The parents can better understand how the home learning contexts related to the development of these skills and how other learning contexts are coherent with those at home. Teachers can use different measurement tools to define and assess student's social and emotional skills and provide insights on how to embed pedagogies into existing classes and curriculum to teach these skills. This study can also help school administrators, policymakers, and community leaders to better learn about how various learning contexts in schools and communities can work together and enhance these skills. Moreover, results from this study can inform employers about the types of skills the future employees may bring and enable companies to better prepare training programmes and adapt the workplace. Finally, the society as a whole can benefit from improvements via reduced inequality, happier and more responsible citizens.

Would a development of perfect metrics and evidence-base necessarily lead to a wide stakeholder engagement in social and emotional learning? For this to happen, we also need to conceptualise these skills in a way that educators can better understand and relate them into ongoing instructional systems and social and emotional learning practices.

For more information:
Skills for Social Progress: The Power of Social and Emotional Skills
OECD working paper: Fostering and Measuring Skills: Improving Cognitive and Non-Cognitive Skills to Promote Lifetime Success, by Tim Kautz et al.
Chart source: Skills for progress report (OECD, 2015)

How can the Netherlands move its school system “from good to great”?

by Montserrat Gomendio
Deputy Director, Directorate for Education and Skills

Activities undertaken by lower secondary teachers at least once per month,
OECD Teaching and Learning International Survey (TALIS) 2013

A new OECD review of the Netherlands education system offers a roadmap towards excellence. Netherlands 2016: Foundations for the Future, based on data from both PISA and the Survey of Adult Skills, confirms that the country already enjoys a high-quality and highly equitable education system. But it also identifies areas that need to be improved as the country moves its education system, in the words of Dutch Minister of Education, Culture and Science Jet Bussemaker, “from good to great”.

The Dutch school system is highly stratified, and uses early tracking extensively. For a long time the Netherlands has made this complex school system work well for students: students performed well at school, socio-economic status had a relatively weak impact on performance, and were readily employable when they completed their schooling  (the number of young people who are neither employed nor in education or training is among the lowest across OECD countries). Our analysis shows large differences in performance within educational tracks, and a large degree of overlap in literacy and numeracy performance between tracks. This implies that students in different tracks are equipped with more similar levels of skills than is observed in other countries, probably due to the existence of “bridge classes” and “scaffolding diplomas” which allow for greater flexibility among the different curricula. But evidence points to a worrying trend towards making the system more rigid, which could lead to less movement between tracks and an erosion of the equity levels that the system enjoys today.

The complexity of these issues calls for a coherent policy response. Netherlands 2016, the first report of its kind since the late 1980s, proposes making student selection more objective by giving more prominence to an objective national test; limiting secondary schools' autonomy in selecting students into different educational tracks; and making the system more permeable to ensure that students progress more smoothly through the education system. The latter calls for various measures, including the alignment of curricula of different tracks, more personalised teaching and learning, promoting larger secondary schools that offer all education tracks through financial incentives.

More efforts should also be made to attract talented and motivated people to the teaching profession especially since many teachers in the Netherlands are approaching retirement age. A more systematic approach to the professional development of teachers is needed. The OECD Teaching and Learning International Survey (TALIS) 2013 shows that collaborative working and learning among teachers is not well established in the Dutch school system, yet these practices have proven essential for improving the quality of teaching. These findings stand at odds with the country’s ambitions to develop its schools into learning organisations.

The system has achieved a good balance between a large degree of school autonomy and efficient accountability mechanisms. However, given the extent of school autonomy, more effort should be invested in training school principals.

The quality of early childhood education and care should also be improved. Although participation rates are high, most parents use childcare facilities fewer hours a week than parents in most other OECD countries do. A national curriculum framework, higher staff qualifications and more staff training are needed to ensure all early childhood education and care services are of high quality and deliver good outcomes for children, and long-term benefits for Dutch society as a whole.

Netherlands 2016: Foundations for the Future
Teaching and Learning International Survey (TALIS) country note for the Netherlands
Chart source: © OECD

No gain without (some) pain

by Bonaventura Francesco Pacileo
Statistician, Directorate for Education and Skills

When Tim Duncan, captain of the the US National Basketball Association’s San Antonio Spurs, was spotted wearing a T-shirt saying “4 out of 3 people struggle with math”, everyone realised that he was counting himself among those who have a hard time with fractions, making the joke even funnier. What is less funny, though, is that PISA 2012 results show that more than one in four 15-year-old students in OECD countries are only able to solve mathematics problems where all relevant information is obvious and the solutions follow immediately from the given stimuli.

As a professional basketball player, Tim Duncan would probably agree that hard work is a prerequisite for attaining individual goals. Working hard is also important in education. According to this month’s PISA in Focus and the recently published report Low-performing Students: Why They Fall Behind and How to Help Them Succeed, most low-performing students share a common trait: they lack perseverance.

In most PISA-participating countries and economies, when students are asked to solve problems requiring some effort, low-performing students are more likely to report that they give up easily. Across OECD countries, 32% of low-performing students reported that they give up easily when confronted with a difficult mathematics problem compared to only 13% of top performers. Differences between the two groups are largest in Jordan, Portugal, Qatar, the Slovak Republic and the United Arab Emirates. This might lead us to conclude that these struggling students are largely responsible for their own academic failures, since they have ultimate control over how much effort they invest in their schoolwork.

But evidence from PISA tells another narrative: low-performing students may be less engaged at school because they believe their efforts do not pay off. This disengagement is obvious when students are asked about the returns to their efforts. While 81% of top performers agreed that they feel “prepared for mathematics exams”, only 56% of low performers agreed with that statement. Low-performing students seem to quit studying when they see their work as an unproductive and unprofitable waste of time. But at the same time, low-performing students often engage in activities that require numeracy skills. Perhaps surprisingly, they are actually more likely to play chess or to be members of a mathematics club.

The good news is that these kinds of activities may be exactly what could help low-performing students develop better study habits. PISA finds that interest in mathematics is greater among students who do mathematics as an extracurricular activity compared to students who do not, and this positive association is stronger among low-performing students. These additional learning opportunities, which could help students gain self-confidence and find enjoyment in mathematics, could be exploited to narrow performance gaps among students.

As Tim Duncan would put it, students need a proper training court where they can learn how to become champions.

Photo credit: Net Ball just before hitting the rim of the hoop @Shutterstock

Going beyond education policies – how can PISA help turn policy into practice?

by Andreas Schleicher
Director, Directorate for Education and Skills

How are policy makers in the United States using data to help districts maximise their impact? And, what tools do districts need to work together in order to build stronger communities?  The Every Student Succeeds Act (ESSA) in the United States has transferred a great deal of autonomy to states and districts. These local authorities are now responsible for transforming state and federal policies into strategies and practices that guide teaching and learning in the classroom. This allocated autonomy creates opportunities for states and districts to collaborate, but also adds an element of the unknown, since most decisions used to be taken at the federal level. Data are crucial to understanding the effect policies have on education systems at a local level. But, collecting the right kind of data can be challenging. 

The OECD’s triennial report on the state of education, the Programme for International Students Assessment (PISA), can be a good starting point. Tracking trends over time, PISA allows policy makers to link data on student learning outcomes with data on students’ backgrounds and attitudes towards learning and on key factors that shape their learning, in and outside of school. PISA examines the relationships with low performance, what makes schools successful, including autonomy and accountability, and teacher-student collaboration.

A new online course developed by EdPolicy Leaders online aims to unlock PISA to reveal what is possible in education for US policy makers and education leaders. We hope it will enable educators to leverage the experience of the world’s leading school systems to improve policies and practice.

Understanding PISA results is a first step towards defining ways in which PISA data can be used to identify strengths and weaknesses of the U.S. education system. What then follows is creative thinking about what education leaders, schools, teachers, parents and students themselves can do to support policy actions that ensure every student is equipped with the skills necessary to achieve their full potential and participate in an increasingly interconnected global economy.

There is no single combination of policies and practices that will work for everyone, everywhere. PISA’s unique contribution to the education debate is that it shows policy makers and education leaders what’s possible and shares evidence of the best policies and practices across the globe.

For the next round of PISA in 2018, the OECD will take the assessment a step further and work with member countries to build a new framework which goes beyond testing students on their cognitive abilities. PISA will also examine if schools are helping students develop trust and respect, and are preparing them to be able to collaborate with others of different cultural origins.

PISA online course: How Do We Stack Up? Using OECD'S PISA to Drive Progress in U.S. Education
PISA 2012 Key Findings
PISA 2015 Assessment and Analytical Framework
OECD proposal for a framework to assess Global Competence in PISA 2018

Time, working and learning

by Viktoria Kis
Analyst, Directorate for Education and Skills

Seven years is the right length for apprenticeships – thought Queen Elizabeth I of England as she lifted her feather to sign the Statute of Apprentices in 1563. Seven years would ensure that everyone benefits: apprentices would receive good training and masters would gain from their apprentices’ labour – although it must be admitted that back then, many apprentices died before finishing their training or ran away from masters who starved them.

Today policy makers, employer and employee representatives have different considerations in mind, but the dynamics of costs and benefits matter just as much. Those dynamics need to be built into the design of apprenticeships and other work-based learning to make it attractive to both employers and learners. A new OECD study funded by the UK Commission for Employment and Skills, entitled Work, train, win: Work-based learning design and management for productivity gains casts a spotlight on these issues.

At the beginning of work-based learning programmes employers make an investment. This pays off later on when, after receiving high quality training, skilled trainees achieve higher productivity and contribute to production. That final period when trainees are more productive than they cost is essential, as it helps employers recoup their initial costs. But if it is too long, then trainees will find it unattractive. Of course, not all occupations are the same. For example an apprentice in retail can quickly become productive so a work-based learning scheme for this occupation should be shorter, while a person training to be an industrial mechanic typically needs more time to become competent at their job and longer duration would be appropriate.

What exactly trainees do while in the workplace also affects the balance of costs and benefits for both parties. A restaurant benefits both when an apprentice cook peels potatoes (unskilled work) and when they bake a soufflé (skilled work), but gains no immediate benefits when the would-be cook is doing practice exercises that are non-productive, even though they are developing their skills. The good news is that there is often room to build learning into productive work, in ways that benefit the firms and are neutral for the trainees. For example, after observing their supervisor a trainee might practice the skill either through simulations or by doing real work. They improve their skills either way, but doing real work also generates benefits for the firm. Indeed research found that German firms with apprentices reduced the share of non-productive activities by half between 2000 and 2007, and increased the share of productive work – and they did that while maintaining training quality.

The scope for learning through productive work does vary across occupations. An apprentice cook can have a go at their first beurre blanc on day one, but a would-be electrician must undertake substantial training before touching the wires. But whenever possible, learning should take place as part of productive activities and rigorous assessments at the end of the scheme can verify that learning has taken place – if an apprentice electrician is able to correctly install a branch circuit in front of an examiner, there will be no doubt about it.

Putting this into practice requires management capacity within firms, so that they can allocate trainees and supervisors to tasks that meet the twin goals of learning and production. In countries and sectors with a tradition of work-based learning firms have much tacit knowledge (as many employers used to be apprentices themselves) and there is a surrounding infrastructure, such as training for trainers and instructional resources. Developing the infrastructure and enhancing firms’ ability to manage work-based learning is a big job, but well worth the effort and not just for those involved in a work-based learning scheme. Keeping a workplace up-to-date means dealing with new machines, materials and software, so firms that know how to support learning while getting on with productive activities will have a competitive edge.

When Queen Elizabeth I put down her feather, the law she signed remained in place for 250 years. Today policy and practice regarding work-based learning changes much more rapidly – but the main challenge of getting the design of work-based learning schemes right remains just as important as it was in her day.

OECD Education Working Paper: Work, train, win: work-based learning design and management for productivity gains, by Viktoria Kis
Find out more about Work-based learning
Photo credit: Vector freehand linear monochrome drawing of ancient pen and inkwell @Shutterstock