Методика преподавания языка | Филологический аспект: Методика преподавания языка и литературы Методика преподавания языка и литературы №06 (29) Ноябрь 2024 - Декабрь 2024

Интерактивные контрольно-измерительные материалы как средство диагностики иноязычной устной дискурсивной компетенции обучающихся средней школы

Петухова Инна Сергеевна
магистрант института международного образования, Московский педагогический государственный университет РФ, г. Москва, Innsenk@yandex.ru
Смирнова Наталья Владимировна
канд. филол. наук, доцент кафедры английского языка и цифровых образовательных технологий, Московский педагогический государственный университет, РФ, г. Москва, Natalia22L2016@yandex.ru

Аннотация: В данной статье рассматривается важность диагностики дискурсивной компетенции (ДК) учащихся средней школы в говорении с помощью интерактивных заданий, о чем свидетельствует отсутствие признанной стратегии диагностической оценки данной компетенции. Цель исследования – создать интерактивную стратегию диагностического оценивания с последующим информативным фидбеком для обучающихся и учителей и проверка ее влияния на формирование ДК учащихся 9 класса в устной речи на английском языке. Статья содержит обоснование применения интерактивных заданий в диагностике навыков устной речи, а также описывает методологическую базу создания диагностических языковых КИМ. Представлены положительные результаты проведенного исследования.
Ключевые слова: Дискурсивная компетенция (ДК), языковая диагностическая оценка, информативная обратная связь, интерактивное оценивание, вспомогательное обучение

Petukhova Inna Sergeevna
Master student of Institute of International Education, Moscow State Pedagogical University, Russia, Moscow
Smirnova Natalia Vladimirovna
Cand. Sci. (Philology), assistant professor of English language and IT technologies department, Moscow State Pedagogical University, Russia, Moscow

Abstract: This paper examines the importance of diagnosing secondary school students' discursive competence (DC) in speaking through interactive tasks, as evidenced by the lack of a recognised diagnostic assessment strategy for this competence. The aim of the study is to create an interactive diagnostic assessment strategy with subsequent informative feedback for learners and teachers and to test its impact on the formation of 9th grade students' DC in English speaking. The article provides a rationale for the use of interactive tasks in the diagnosis of oral language skills and describes the methodological basis for the creation of diagnostic language assessment materials. The positive results of the conducted research are presented.
Keywords: Discursive competence (DC), language diagnostic assessment, informative feedback, interactive assessment, remedial learning

Modernisation processes in the education system of Russia place high demands on the model of a secondary school graduate. In view of the contemporary educational paradigm, one of the most crucial goals in teaching a foreign language is the development of foreign language communicative competence. There are several models of communicative competence (D. Hymes, 1972; M. Canale & M. Swain, 1980; I.A. Zimnyaya, 2004; A.A. Verbitsky, 2013), and most of them include discursive competence (DC) as an important component.

DC formation is crucial in a foreign language acquisition as it enables learners not only to understand a language but also confidently and competently interact with other speakers and navigate social situations by adequately comprehending and producing oral and written discourse. Acquiring a sound level of DC in speaking, therefore, is key in managing communication in foreign language effectively. This consideration places necessary on Russian educators to provide methodological support for the formation of DCs in English lessons at school, with the primary step being the diagnosis.

Thorough analysis of the scope of research on the topic of DC formation reveals the absence of adequate generally accepted system of oral DC diagnosis, and lack of real diagnostic test in foreign language teaching in general. While there is no shortage of formative and summative language tests assessing speaking abilities, one cannot find a diagnostic test to adequately evaluate a learner’s strengths and weaknesses when it comes to creating spoken discourse in social contexts. Without a comprehensive approach to foreign language DC diagnosis, it is impossible to monitor whether Russian educators meet the objectives in EFL curriculum, set by the Federal State Standard of Education (FSSE) – the formation of foreign language communicative competence [12, p. 44-45].

The above outlined research gap justifies the aim of our research to design and implement a science-based strategy of interactive diagnostic assessment of oral DC as well as to gain understanding of to what extent it enhances both teaching and learning. We hypothesise that the diagnosis of EFL learners’ oral DC will lead to the enhancement of the latter if (1) the diagnosis is based on interactive assessment materials, (2) the diagnosis is followed by the intervention stage informed by the interpretation of the diagnostic results.

Approaching the task of forging a comprehensive diagnostic strategy of oral DC formation in secondary school learners, it is crucial, first, to define the components of the construct of foreign language oral DC in order to specify the criteria of its mastery and inform the diagnostic scale design. Second, the research has to be grounded in the theory and practice of foreign language diagnostic assessment.

Discourse has systemic characteristics, the understanding of which and the ability to use them in situations of speech communication allows for the formation of discursive competence. Thus, discursive competence should be developed through the process of comprehension and learning the main systemic characteristics of discourse such as register, type (modus) and genre, functional style, formality [5, p.473-474]. Therefore, we have considered those views on the definition and component structure of discursive competence that are consistent with the systemic discourse features.

In Russian liguodidactics, the term ‘discursive competence’ has been elaborated by numerous researches (N.P. Golovina, 2004; Popova, 2011; I.A.Yevstigneeva, 2013; A.G. Gorbunov, 2014). The present research adheres to the view on DC of I.A. Yevstigneeva., who claims that discourse is a bidirectional process which involves an author and a recipient. Accordingly, discursive competence is the ability to carry out this process, i.e., the ability to encode and decode information using a foreign language according to its lexical, grammatical, and syntactic norms, as well as taking stylistic, genre, sociocultural, psychological and emotional factors into account, using cohesion and coherence means to achieve a communicative goal. [6, p.24]

In line with the above definition is the approach of the researcher to dissecting DC into four constituent components:

• the textual component - the ability to organise a sequence of sentences in such a way that they form a coherent text with all its inherent properties.
• the tactical component - the ability to analyse the communicative situation and, based on this, select the appropriate, adequate and most optimal means and ways of achieving the communicative intention of the subject in the given linguistic community.
• the genre component - the knowledge of genre rules and the ability to organise a discourse in accordance with the rules of a particular genre chosen to achieve the subject's communicative intention in a given context.
• the strategic component is considered as the subject's ability to clearly realise his or her communicative intention and plan the communicative event. [6, p.25]

The accepted component structure facilitates the selection of optimal tasks for assessment and the creation of the diagnostic scale.

The paradigm shift towards communicative approach [7, p. 54] in language teaching and testing led to a greater emphasis on meaningful tasks and authentic materials in language curricula. As a result, language learners' communicative competence development has become a top priority in many foreign language curricula, including the Federal State Standard of Education (FSSE).

This paradigm shift, which prioritised the teaching of effective communication over the teaching of discrete language systems, prompted language assessment creators to move beyond a narrow focus on language structures and consider the social, cultural, and contextual aspects of language use. Today, test designers globally strive to create exams that mimic real-life communication, where test-takers engage in authentic interactions with a clear purpose, mirroring the roles they would assume in everyday situations [8, pp.36-38].

Despite the wide implementation of communicative approach in teaching, assessment of language learners’ competences in Russian comprehensive schools is still mainly carried out using traditional methods of assessment, such as written tests and oral examinations, even though they may not accurately reflect a learner's communicative abilities. Communicative assessment of speaking, on the other hand, implies the use of interactive activities that aim to provide a more comprehensive and authentic picture of EFL learners' speaking abilities and can be more engaging for both the learner and the assessor. 

Interactive assessment involves tasks that require extended stretches of interactive discourse, either between an interviewer and interviewee (tasks like interviews, games) or between test-takers (e.g., discussions, information-gap, role-plays), to evaluate learners' ability to engage in authentic and natural language use. These tasks often involve interpersonal interactions, which aim to build connections and convey attitudes, as well as transactional interactions, which aim to convey information and achieve specific goals, such as obtaining information about opening hours of a museum, where a concert is going to be, an assignment submission deadline. Effective communication in both types of interactions demands more than just conveying the literal meaning of words; speakers must also consider the context, tone, and purpose of their message [9, pp.130-132].

Interactive methods of assessment have proven to have numerous benefits for EFL learners, including more accurate assessment of speaking skills, increased learner engagement and motivation, improved opportunities for feedback and self-assessment, development of critical thinking, problem-solving, and communication skills and a chance to enhanced language use and fluency [9, p.130].

Concerning the format of interactive assessment, pair tasks seem to be efficient in terms of organization. Moreover, they can elicit authentic peer-to-peer interaction by requiring participants to collaborate on tasks that involve information asymmetry: each one of the speakers does not know exactly what their partner will tell next. For instance, one partner might describe a graph while the other draws it, or they might jointly construct a story from shared but incomplete information. [10, p.88] Less structured tasks, such as discussing a topic from multiple perspectives, also encourage genuine interaction. Unlike examiner-examinee interviews, pair tasks distribute speaker roles, providing a more comprehensive assessment of examinees' interaction skills.

In the development of the assessment procedure, we adhered to the structure outlined in the book ‘Assessing Speaking’ by S. Luoma: in the initial phase of developing speaking assessments, it is imperative to analyse the specific types of speaking skills that are relevant to the assessment context, taking into account both social and situational needs. Furthermore, it is essential to recognise the interactive nature of speaking when designing rating criteria and assessment procedures. Examinees should be rewarded when they reiterate or mirror the phrases and structures of their interlocutors, as well as for their ability to develop topics by referencing previous contributions and building upon them, as this reflects their proficiency in engaging collaboratively with other speakers as well as maintain discourse in dialogue communication [10, p.88-92].

L. Bachman and A. Palmer in their book ‘Language Assessment in Practice’ state that the principles of foreign language assessment are deeply rooted in the educational and social contexts in which they are used. The consequences of assessment decisions will have a direct impact on individuals, programs, and institutions, influenced by the values, laws, regulations, and policies that may regulate the assessment process [8, p.211]. As such, it is essential for test developers and stakeholders to consider the potential outcomes of assessment and decision-making for various stakeholders involved. This recognition underscores the importance of responsible assessment practices that prioritize the promotion of positive consequences for educational systems and society as a whole.

In case of the present research, the stakeholders of assessment are learners and their parents, and teachers. To better identify the intended outcomes of the language diagnosis, we address the publication ‘Diagnosing diagnostic language assessment’ by South Korea based scholar Y.-W. Lee, who discusses the connection between diagnosis in EFL and learning. The scholar conceptualises the diagnostic language assessment as ‘the combination of diagnosis, diagnostic feedback and remedial learning/teaching’ [11, p.308].

The first stage of diagnosis, as Y.-W. Lee claims, aims to identify learners' strengths and weaknesses while pinpointing the causes of deficiencies. It begins with designing and validating fine-grained diagnostic tools, and using those for evaluating learners' skills or knowledge and analysing patterns of performance. Iterative testing may follow to refine result.

The second stage reports diagnostic feedback to stakeholders (who can be teachers, learners, parents, school), in form of quantitative and qualitative data. Feedback should include recommendations for remedial activities to address deficiencies.

Finally, the third stage focuses on aiding learners to overcome weaknesses and achieve learning objectives. Developers analyse responses to diagnostic tests, refer to necessary resources, and conduct parallel assessments, while teachers integrate remedial activities into curricula and programs. Learners are encouraged to participate in self-motivated activities for improvement [11, p.10].

Following Y.W. Lee’s guidelines, and in line with the research started out with assessment task and criteria selection. Based on the component structure of oral DC, the EFL learning outcomes for secondary level of education stated in FSSE, and the data about optimal interactive assessment activities, the following three types of tasks have been selected for the diagnostic assessment of 9-graders’ DC in speaking in one of comprehensive schools of Moscow:

Task Type 1: Interview (Paired gap-fill): Two test takers take on the roles of interviewer and interviewee. The interviewer uses a survey questionnaire to gather information from the interlocutor, such as family size and advice on resolving issues with parents. If misunderstandings arise, test-takers must also request clarification. This task combines elements from the Cambridge KET speaking task and the basic state exam (BSE). Unlike the BSE, this task involves a direct interaction between two people rather than a person and a recording.

Task Type 2: Simulated situation (decision task): Here, two (or three) speakers evaluate a list of alternatives, discussing their pros and cons to reach a consensus. Participants suggest options and respond to one another. This task differs from role-play activities as speakers express their genuine opinions without assuming distinct roles. Cambridge exam PET employ similar interactive speaking assessments that focus on interaction ability rather than language accuracy.

Task Type 3: Role-plays: Students engage in prompted role-play conversations that emulate real-life scenarios from topics covered in their syllabus (e.g., buying tickets, discussing films). This format demonstrates their dialogic communication skills.

In order to develop rating criteria for the selected tasks and, importantly, to evaluate the extent to which they fully enable examinees to demonstrate their oral DC skills, the researcher has organised both (tasks and indicators of mastery) into a side-by-side consolidated table where it can be seen which specific indicators can be tested with each task.       

Table 1. Interactive assessment tasks

As can be seen from the data above, the three tasks, though assessing particularly the same number of criteria, nevertheless, differ in targeted subcompetences. It can also be inferred, that the first task is intentionally lower in difficulty, which is explained by the necessity to provide students even at A1 [CEFR, 2018] level of English to showcase their DC in speaking at sentence level. Most classes in Russian state schools still experience the problem of mixed ability groups in EFL classes, so alongside intermediate and even upper-intermediate level [13] English learners, students with elementary level and below can still be found in 9^th form.

The criteria of oral DC formation outlined in table 1, though dissected into more discrete components, served as descriptors for an analytic diagnostic scale. To evaluate the reliability of the assessment procedure, inter-reliability and intra- reliability have been tested.

According to L. Bachman and A. Palmer, diagnosis in language teaching and learning is used to point out learner's strengths and shortcomings in a certain language competency or skills in order to inform and facilitate their learning [8, p.212]. This means that teachers either may need to make adjustments to the curriculum by including the work on a problematic area identified as the result of diagnosis, or certain alterations of the instructional methods and/or activities.

Results

The present work reports on and discusses the methodology and findings of the study conducted to validate the beneficial effects of interactive diagnostic assessment materials as means of informing and aiding the process of foreign language oral DC creation among secondary school students. As the hypothesis put forward in this study claims, the experimental group, which will receive detailed feedback informing their curriculum and self-study based on diagnostic assessments of their oral DC in English, will show greater improvement in their oral DC mastery compared to the control group, which will not receive such feedback.

Thus, the following research variables are under study:

• Independent variable: providing detailed feedback and action plans based on oral DC interactive diagnostic materials
• Dependent variables: the level of oral DC in English;

The current research was based in the state school number 1512 of Moscow named after Alia Moldagulova during the first trimester of the academic year 2023-2024 – the study lasted almost three months. The sample involved 28 students of the 9^th grade, who belonged to the same class but were equally divided in two groups during their English lessons, each group being taught by their constant teacher for no less than 2 years. These two groups had been randomly assigned as experimental and control since they bear no critical difference in the average English level of the students, or any other characteristic significant for the study. Both the experimental and control groups are characterised as mixed ability, with the prevailing majority of the participants having reached B1 level of English.  As was found with help of needs analysis, the students are putting special effort into the mastery of their speaking, an important feature of which is oral DC development. Around half of the students in each group share a common opinion that their weakest abilities in speaking are grammar and fluency. Since it was not possible to perform probability sampling, the design of the study is attributed to quasi-experimental.

The special condition in the experimental group was the implementation of the diagnostic strategy of foreign language oral DC formation based on the use of interactive assessment materials, which implied providing the learners with the detailed diagnostic feedback, and tailored recommendations on self-learning based on the feedback. Another part of intervention was in-class remedial activities, which took place on 6 lessons throughout the trimester. By contrast, the control group did not undergo any intervention into the standard EFL curriculum.

To measure the alteration of the dependant variables both quantitative and qualitative data analysis methods have been employed. Firstly, we administered and analysed pre-test and post-tests in the form of identical oral DC diagnosis based on interactive assessment material. Secondly, we obtained qualitative data by collecting the learners’ feedback with open-ended questionnaires, and teachers’ feedback through semi-structured interviews.

The analysis of the obtained quantitative data proved the hypothesis since the level of DC in speaking in the experimental group has increased in the result of remedial classroom activities and self-learning facilitated by the diagnostic feedback (see Appendix 1.). The mean score of pre-test in the control group is 9.7 compared to that of post-test - 9.86, which does not make a significant contrast. At the same time, the results of the experimental group show a greater increase in the mean score: 10.29 in post-test compared to 8.79 in pre-test.

The qualitative data analysis revealed the learners’ feedback to the diagnostic strategy with detailed feedback and recommendations for improvement is generally positive. The teachers have also expressed their satisfaction with the format of assessment and the results of the experiment. Thus, we believe that the diagnostic strategy reviewed in this article should be widely implemented at the secondary level of education in Russian schools to gauge and enhance students’ oral DC in English.

1. Hymes D. On communicative competence. Sociolinguistics. Pride JB, Holmes J, editors. Harmondsworth: Penguin Books, 1972. – 269 p.
2. Canale М, Swain M. Theoretical bases of communicative approaches to second language teaching. Appl Linguistics. – 1980;1:1–47. DOI: 10.1093/applin/1.1.1
3. Zimnyaya, I.A. Klyuchevye kompetentnosti kak rezul'tativno - celevaya osnova kompetentnostnogo podhoda v obrazovanii / I.A. Zimnyaya. - M.: IC PKPS, 2004.-42 p.
4. Bachman L. F., Palmer A. S. The construct validation of some components of communicative proficiency //TESOL quarterly. – 1982. – Т. 16. – №. 4. – PP. 449-465.
5. Van Dijk T. A. Critical discourse analysis //The handbook of discourse analysis. – 2015. – PP. 466-485
6. Yevstigneeva I.A. Formirovanie diskursivnoj kompetencii studentov yazykovykh vuzov na osnove sovremennykh internet-tekhnologiy // Language and Culture. – 2013. – №1. – PP. 21-25
7. Verbitsky, A.A. Lichnostnyj i kompetentnostnyj podhody v obrazovanii: problemy integracii / A.A. Verbickij, O.G. Larionova. - M.: Logos, 2013. - 336 p.
8. Bachman L., Palmer A. Language assessment in practice: Developing language assessments and justifying their use in the real world. – Oxford University Press, 2022. – 564 p.
9. Hatipoğlu Ç. Testing and assessment of speaking skills, test task types, and sample test items //Language assessment and test preparation in English as a foreign language (EFL) education. – 2021. – PP. 119-173.
10. Luoma S. Assessing speaking //Cambridge University Press. – 2009. – 236 P.
11. Lee Y.-W. Diagnosing diagnostic language assessment //Language Testing. – 2015. – V. 32. – №. 3. – PP. 299-316.

Official papers
12. Federal'nyj gosudarstvennyj obrazovatel'nyj standart osnovnogo obshchego obrazovaniya. Utverzhden prikazom Ministerstva prosveshcheniya RF ot 31.05.2021 № 287. 129 s. URL: https://fgos.ru/fgos/fgos-ooo (10.09.2023)..2023
13. Common European Framework of Reference for Languages: learning, teaching, assessment. Companion volume with new descriptors. — Strasbourg: Council of Europe, 2018. — P. 235. URL: https://rm.coe.int/cefr-companion-volume-withnew-descriptors-2018/1680787989 (18.09.2023)