Evaluation report_5 Herbartis Project by partners Project, partnership, knowledge and skills, activities and outputs
Evaluation report_0 Strategy and tools for evaluation of Herbartis Project’s activities Introduction A set of evaluation activities were planned for Herbartis Project, in order to measure its achievements and impact next to different target groups involved. During the project, some of the proposed activities were subject to adjustments, especially on their chronology, but all expected topics and objectives were tackled by the evaluation tools created for the project. A shared monitoring document was built for an ongoing evaluation of the project by the partners and a set of surveys were applied to students, teachers, staff and other participants involved in the project. Reviewing the plan Looking at the EVALUATION PLAN: The IMPACT of a project (that is: the effect that the implemented project activities have in the participants, practices, organisations and systems) can be measured as follows:
- INDIVIDUALS: Increase of competences, change of attitudes, improvement in the learning process
- ORGANISATIONS: improvement in the management, education/curricular framework, the practice of the organisation.
- OTHERS: lasting effects, innovation transfer, synergies creation, influence how the decisions are made, networking.
Examples of impacts are:
- Effective access to educational resources of the adult disadvantaged population. - Increased motivation of adult people for the education. - Implementation of foreign languages’ learning experiences in real contexts. - Creation of support networks of teachers and trainers of adult people. - Elaboration of specific materials and training. - Creation of recognition instruments of the non-formal learning, and their integration in the adult
people curriculum. Herbartis project should meet the following achievements and impact:
- Acquisition of knowledge or experiences by the participants, target public or any active stakeholder in the education community.
- Improvement in the competences and achievements.
- Higher cultural awareness - Better linguistic knowledge.
In order to achieve that, the partners should share responsibilities to:
- Control the proper implementation and the quality of the different activities at the end of them. This information will be very useful to correct mistakes and adopt changes in the Project Implementation Action Plan.
- Evaluate the achievements and impact of intangible results according to indicators, by elaborating and implementing satisfaction questionnaires addressed to participants.
- Elaborate an evaluation report for project activity, to be checked and approved by the Steering Committee. At the end of the project, a final evaluation report will be jointly elaborated.
To allow improving the skills (technical, entrepreneurial, linguistic and ICT) of teachers and participating learners, and gain experience in blended training methodology, these Intangible results were proposed for Herbartis:
- IR1: Improved policy awareness of sector stakeholders, education providers and policymakers thanks to reports and guidelines book.
- IR2: Increased skills of participant learners in the training activities (blended course). - IR3: Increased skills of staff by elaborating the curricula and training materials. - IR4: Knowledge and experience gained by participants (learners, staff and sector stakeholders)
in the transnational training activities and networking activities. - IR5: Better language skills of learners in the transnational training activities and networking
activities. - IR6: Improved cultural awareness in the different transnational training activities.
Surveys would be addressed to the partners at the end of every activity, in order to know if: results have been achieved, the methodology has been properly implemented, partners are satisfied with the project monitoring and the environment is comfortable and productive. 5 surveys were proposed (in chronologic order):
a. Kick-off meeting, Sector’s analyses and new curricula (O1-A1, O1-A2, O1-A3) and online course’s texts’ development (O4-A1).
b. Intermediate meeting, E-learning platform, Syllabus (O3) and starting of blended course. c. Blended course’ implementation (A6), Training videos editing (O4-A2), Training activities
within the blended mobilities of adult learners and related training materials (C1, C2, C3, C4, O4-A3, O4-A4, O4-A5).
d. Training‘s analyses and editing of Guidelines (O1-A4, O1-A5, O1-A6). e. Guidelines and project results’ dissemination (A4, E1) and Project’s final revision.
Satisfaction surveys would be addressed also to learners and stakeholders participating in training activities:
A. partners’ staff (satisfaction of quality of curricula and training materials) => IR3 B. learners of the online course (satisfaction on course development and quality of training
material) => IR2. C. practical workshops’ participant learners => IR2.
D. participants (learners, teachers and stakeholders) in the virtual collaboration space => IR4, IR5.
E. participants (learners, teachers and stakeholders) in the training activities of the transnational mobilities (satisfaction on the quality of activities, knowledge and language skills acquired) => IR4, IR5, IR6.
F. technical seminars and disseminating events’ attendees (stakeholders) => IR1. Proposed indicators of achievement:
Measurement of impact Expected indicator
Type of monitoring Objective
Number of evaluation reports
5 Survey at the end of activities a, b, c, d, e Quality of project’s activities
Degree of satisfaction of partner’s staff
Good
Survey at participating staff at the end of the activities (A)
Increased skills of staff by elaborating the curricula and training materials
Number of new subjects learnt per participating staff
3
Degree of satisfaction of courses’ learners
Good
Survey at registered learners at the end of the blended course (B)
Increased skills of participant learners in training activities. To achieve the professional development of adult learners by developing useful tools and methods, improving the offer of high-quality learning opportunities, and validating the non-formal learning and career guidance
Degree of satisfaction of learners on online course texts
Good
Degree of satisfaction on practical workshops
Good Survey at participating learners at the end of the workshops (C)
Degree of satisfaction of users of virtual collaboration space (networking activities)
Good Survey at learners, stakeholders and staff at the end of the activity (D)
Knowledge and experience gained by participants (learners, staff and sector stakeholders) in the transnational training activities and networking activities
Degree of satisfaction of participants in transnational training activities
Good Survey at learners, stakeholders and staff at the end of the activity (E)
Degree of satisfaction of attendees to Technical seminars
Good Survey at attendees during the event (F).
Degree of satisfaction of language comprehension
Good Survey at learners at the end of the activities (D, E)
Better language skills of learners in the transnational training activities and networking activities.
Number of new words learnt per participant
10
Degree of satisfaction of attendees of Multiplier event
Good Survey at attendees during the event (F).
Improved awareness of sector stakeholders, education providers and policymakers thanks to reports and guidelines book
Number of surveys 7 Surveys at end of activities: -O1/A3 and O4 (A)
Quality of project’s activities
-On-line course (B) -Practical workshops (C) -Virtual networking activities (D) - Transnational training activities (E) During the events: -Technical seminars (F) - Multiplier event (F)
Number of participating users in surveys
152 (60 learners,
12 staff, 80 stakeholders)
-Learners (online course, practical workshops, virtual networking, transnational training activities) -Staff (curricula and training material, virtual networking, transnational training activities) -Stakeholders (virtual networking, transnational training activities, events)
Data obtained from satisfaction surveys
To list to measure impact
Feed-back from end-users, stakeholders, policymakers
Impact on participants
Explaining the options As said before, all items of the project have been evaluated and all target groups have been addressed by evaluation tools, even if not exactly in the planned format. Topics under evaluation were grouped thematically. Three types of evaluation tools were created:
- 1 monitoring document - 4 online satisfaction surveys - Satisfaction surveys on paper
A monitoring google document was built for ongoing control of the execution of the project by the partners: time schedule and results could be permanently compared to the plan. Surveys were applied after the execution of the activities:
- 3 Online evaluation surveys on project transnational meetings applied to staff (after each meeting)
- Paper evaluation surveys on seminars and on final multiplier event delivered to all participants* at the end of the activities * not only learners and staff
- 1 Online evaluation survey on transnational activities applied to participants – includes seminars, technical visits to farms and transnational workgroups (at the end of the transnational mobilities)
- 1 Online evaluation survey on the blended course applied to learners – includes on-line lessons, virtual meetings with stakeholders and face-to-face workshops (at the end of the course)
- 1 Online evaluation survey on the project applied to staff – includes management of the project,
partnership, knowledge and skills, all activities and intellectual outputs (at the end of the project): The results
Measurement of impact
Scale (very good-good-acceptable-deficient)
Type of monitoring Objective (compared to
previewed)
Number of evaluation reports
6 Surveys at the end of activities Quality of project’s activities see all Evaluation Reports
Degree of satisfaction of partner’s staff
Good (4,1 / 5)
Online Survey addressed to participating staff at the end of the project
Increased skills of staff in elaborating the curricula and training materials And many other subjects - confirmed: see Evaluation Report_5
Number of new subjects learnt per participating staff
Many: languages, countries realities,
digital, management, knowledge on MAPs, building of b-learning courses and others
Degree of satisfaction of courses’ learners
Good (3,2 / 4) Online Survey addressed to learners at the end of the blended course
Increased skills of participant learners in training activities; Professional development of adult learners by developing useful tools and methods, improving the offer of high-quality learning opportunities. – confirmed: see Evaluation report_4
validating the non-formal learning and career guidance – work in progress : see Guidelines document
Degree of satisfaction of learners on online course texts
Good (3,4 / 4)
Degree of satisfaction on practical workshops
Good (3,4 / 4)
Surveys addressed to learners at the end of the blended course (includes workshops) + surveys on paper applied to other participants
Degree of satisfaction of users of virtual collaboration space (networking activities)
Very good (3,5 / 4) Online Surveys addressed to learners and staff at the end of the transnational activities
Knowledge and experience gained by participants (learners, staff and sector stakeholders) in the transnational training activities and networking activities – confirmed: see Evaluation report_3
Degree of satisfaction of participants in transnational training activities
Very good (3,6 /4) Online Surveys addressed to learners and staff at the end of the transnational activities
Degree of satisfaction of attendees to Technical seminars
Good (3,4 / 4)
Surveys at attendees during the event (on paper) + online survey of transnational activities addressed to learners and staff
Degree of satisfaction of language comprehension
Effective communication among participants: Students
Very good (3,5 / 4) Online Surveys addressed to learners and staff at the end of the transnational activities and of the project
Better language skills of learners in the transnational training activities and networking activities. – confirmed, in spite of difficulties felt sometimes: see Evaluation report_3 and Evaluation Report_5
Number of new words learnt per participant
The number wasn’t determined, but
participants testified their improvement in foreign languages
staff: Good (4,1 / 5)
Degree of satisfaction of attendees of Multiplier event
Good (3,4 / 4) Survey (on paper) addressed to participants during the event.
Improved awareness of sector stakeholders, education providers and policymakers thanks to reports and guidelines book – confirmed see Evaluation Report_3
Number of surveys
6 (all previewed topics were evaluated in the
surveys)
Online surveys at the end of: - each of the 3 transnational meetings - transnational activities (Seminars, transnational networking, visits) - blended course (Online course, Practical workshops, Virtual networking activities) - the project (management, partnership, skills, activities and intellectual outputs) + Surveys applied during the events: -Technical seminars - Multiplier event
Quality of project’s activities – positively or very positively evaluated by all target groups _ see all Evaluation Reports
Detailed results for each survey will be presented in the following reports. Conclusions Regarding the evaluation process itself, it is possible to conclude that the tools and methodologies designed for the evaluation of HERBARTIS project cover all proposed topics and objectives. The average grade given to the evaluation process – also subject to evaluation by the partners - is Good: 4,0 / 5. Regarding the results achieved with the project, they are globally most satisfactory, in the opinion of all target groups. The project was very successful, all proposed objectives and results – quantitative, qualitative - being accomplished. As the several following reports show, all topics under evaluation were positively or very positively evaluated by the majority of respondents. Only one problem was effectively relevant and was mentioned by the partners. That was to handle the scarce budget and the limitation of time established in the approval decisions on the project, but still maintaining the activities and products foreseen in the first proposal. The common interest in the project and the good working environment created among partners were key factors for successful results.
Evaluation report_1 The Monitoring Document Introduction Complementarily to a set of folders on google drive, to store all documents of the project, a monitoring document was built in google excel and shared with all partners for the ongoing management of the project. The purpose of this document was to enable a collective awareness and use of all relevant data gathered during the project and a permanent comparison of obtained results to the planned results, helping decision making and elaboration of reports. The Document The document is organised in 4 different parts:
- Repository of implementation plan documents and contacts of partners,
- GANT timetable of the project
- Planned and obtained results every six months
- Data related to website and other means of communication of the project
Results Due to lack of experience with this kind of documents, not all partners have used it often from the beginning. It required development of skills in the use of shared excel documents. But the proposal of the document was well received and the results are positive. The document proved to be a useful tool for project management and ongoing evaluation by a transnational partnership, being shared online. It helps the team focus on the objectives and milestones that need to be accomplished by the project, and know, at any time, if something must be corrected. When currently filled, it can provide useful and detailed information for the elaboration of reports.
Evaluation report_2 Transnational Meetings Introduction During Herbartis Project, 3 transnational meetings were organized, with participation of staff from partners’ organisations:
- kick-off meeting - Racconigi (Italy), 26th September 2015
- intermediate meeting - Moura (Portugal), 26th-28th April 2016
- final meeting - Barcelona (Spain), 26th-27th June 2017 After each of these meetings, an online feedback survey was applied to the participants (same form, possibility to choose meeting to evaluate): https://docs.google.com/forms/d/e/1FAIpQLSeHM4XCUUxlb3-5Xae5JcaxCl8WY7nxdSOVeIN9lSH6IrUpVw/viewform
The results 24 answers were received in total for the 3 meetings (8 for each meeting). The overall opinions produced the following results. LEGEND: KOM (Kick-off meeting); IM (Intermediate meeting); FM (Final meeting)
how productive was the meeting?
KOM IM FM Total
extremely productive 2 4 6
mostly productive 6 4 8 18
neutral 0
a little productive 0
not at all productive 0
Total 8 8 8 24
TOTAL (3 meetings)
N/A strongly agree
agree neutral disagree strongly disagree
TOTAL
the meeting objectives were clearly communicated in advance of the meeting
15 9 24
the meeting objectives were met 15 9 24
follow-up actions resulted from the meeting 9 13 2 24
owners of follow-up actions were assigned 11 12 1 24
the meeting leader(s) effectively moderated the meeting
1 16 7 24
meeting attendees had an opportunity to participate 1 15 8 24
the right people were invited to the meeting 1 15 8 24
the meeting was the appropriate length of time 8 15 1 24
the meeting location fit all the attendees comfortably 17 6 1 24
the speakers were easily heard 17 7 24
the presentation was easily seen 18 6 24
the accommodation, food and the social element were satisfactory
17 6 1 24
Total 3 173 106 6 0 0 288
Kick-off meeting (KOM)
N/A strongly agree
agree neutral disagree strongly disagree
TOTAL
the meeting objectives were clearly communicated in advance of the meeting
5 3 8
the meeting objectives were met 4 4 8
follow-up actions resulted from the meeting 2 4 2 8
owners of follow-up actions were assigned 3 4 1 8
the meeting leader(s) effectively moderated the meeting
6 2 8
meeting attendees had an opportunity to participate 7 1 8
the right people were invited to the meeting 7 1 8
the meeting was the appropriate length of time 1 7 8
the meeting location fit all the attendees comfortably 6 2 8
the speakers were easily heard 5 3 8
the presentation was easily seen 7 1 8
the accommodation, food and the social element were satisfactory
6 2 8
Total 0 59 34 3 0 0 96
Intermediate meeting (IM)
N/A strongly agree
agree neutral disagree strongly disagree
TOTAL
the meeting objectives were clearly communicated in advance of the meeting
6 2 8
the meeting objectives were met 8 8
follow-up actions resulted from the meeting 7 1 8
owners of follow-up actions were assigned 6 2 8
the meeting leader(s) effectively moderated the meeting
1 7 8
meeting attendees had an opportunity to participate 1 5 2 8
the right people were invited to the meeting 1 7 8
the meeting was the appropriate length of time 4 4 8
the meeting location fit all the attendees comfortably 7 1 8
the speakers were easily heard 8 8
the presentation was easily seen 7 1 8
the accommodation, food and the social element were satisfactory
8 8
Total 3 80 13 0 0 0 96
Final meeting (FM)
N/A strongly agree
agree neutral disagree strongly disagree
TOTAL
the meeting objectives were clearly communicated in advance of the meeting
4 4 8
the meeting objectives were met 3 5 8
follow-up actions resulted from the meeting 8 8
owners of follow-up actions were assigned 2 6 8
the meeting leader(s) effectively moderated the meeting
3 5 8
meeting attendees had an opportunity to participate 3 5 8
the right people were invited to the meeting 1 7 8
the meeting was the appropriate length of time 3 4 1 8
the meeting location fit all the attendees comfortably 4 3 1 8
the speakers were easily heard 4 4 8
the presentation was easily seen 4 4 8
the accommodation, food and the social element were satisfactory
3 4 1 8
Total 0 34 59 3 0 0 96
To the question “How productive was the meeting?”, all the answers were positive or very positive: 75% “mostly productive” and 25% “extremely productive”, distributed as follows by the 3 meetings:
It is possible to present these average numbers for the items under evaluation:
- The meeting objectives were clearly communicated in advance of the meeting: KO (4,6), IM (4,8), FM (4,5)
- The meeting objectives were met: KOM (4,5), IM (5), FM (4,4) - Follow-up actions resulted from the meeting: KOM (4), IM (4,9), FM (4) - Owners of follow-up actions were assigned: KOM (4,25), IM (4,8), FM (4,3) - The meeting leader(s) effectively moderated the meeting: KOM (4,8), IM (5), FM (4,4) - Meeting attendees had an opportunity to participate: KOM (4,9), IM (4,7), FM (4,4) - The right people were invited to the meeting: KOM (4,9), IM (5), FM (4,1)
- The meeting was the appropriate length of time: KOM (4,1), IM (4,5), FM (4,3) - The meeting location fit all the attendees comfortably: KOM (4,8), IM (4,9), FM (4,4) - The speakers were easily heard: KOM (4,6), IM (5), FM (4,5) - The presentation was easily seen: KOM (4,9), IM (4,9), FM (4,5)
0% 20% 40% 60% 80% 100%
the meeting objectives were clearly communicated in advanceof the meeting
the meeting objectives were met
follow-up actions resulted from the meeting
owners of follow-up actions were assigned
the meeting leader(s) effectively moderated the meeting
meeting attendees had an opportunity to participate
the right people were invited to the meeting
the meeting was the appropriate length of time
the meeting location fit all the attendees comfortably
the speakers were easily heard
the presentation was easily seen
the accomodation, food and the social element were satisfatory
Transnational meetings: contents, organisation and logistics
N/A
stronglyagree
agree
neutral
disagree
stronglydisagree
- The accommodation, food and the social element were satisfactory: KOM (4,8), IM (5), FM (4,3)
Conclusions Transnational meetings were very satisfactory in the opinion of the participants. There were no negative opinions in any of the evaluated topics. The lowest average rate was related to the length of the meetings, in the sense that they should be a little longer for all issues to be discussed. Even so, the opinions were positive. The intermediate meeting reached the best scores in almost all the topics. The opportunity to visit some farms may have been relevant for that. Also, it happened at a very relevant moment of the project, to plan the preparation of the b-learning course.
Results 50 participants from the 4 courses answered the evaluation report on the transnational activities. The results are as follows. Organisation of the transnational events
N/A 1
lowest 2 3
4 highest
evidence of clear planning of activities 16 34
realistic timescales 3 25 22
evidence in the mobility program of real synergy with the overall objectives of the project
3 20 27
evidence of partners from the 4 countries sharing roles and responsibilities during the event
3 23 24
Total 9 84 107
Effectiveness of content and appropriate range and balance of activities
N/A 1
lowest 2 3
4 highest
relevant mixture of activities e.g. icebreaking activities, didactic sessions, workshops, social activities, free time
1 2 22 25
attention to practical details and catering 2 18 30
suitability of the working venues (welcome meeting, seminar, workgroups) 3 14 33
quality of overnight accommodation 4 1 4 15 26
evidence of special requirements (dietary for example) being met 1 1 24 24
effective communication amongst participants 1 1 20 28
opportunity to meet people and companies relevant for future collaboration 1 16 33
total 6 2 14 129 199
Evaluation of technical visits to farms and other stakeholders in MAP sector
N/A 1
lowest 2 3
4 highest
quality of the program of visits (appropriate selection of producers and other stakeholders to visit)
1 4 16 29
interesting and useful information provided 5 15 30
quality of communication (ability to hear/see and to understand what was being said/shown)
4 5 18 23
total 5 14 49 82
Evaluation of Seminar(s)
N/A 1
lowest 2 3
4 highest
quality of the program of the seminar(s) 1 4 19 26
interesting and useful information provided 1 1 6 14 28
quality of the communication (ability to hear/see and to understand what was being said/shown)
1 5 5 17 22
total 3 6 15 50 76
The transnational activities were very positively evaluated by the participants. The 2 highest grades represent 92% of notes given in the several topics of the evaluation, and the best score (level 4) was selected in more than half of all “votes”. These high scores were obtained in all 4 countries: sum of “3” and “4” grades represent between 85% and 97% of the total. This is a result of very positive opinions on all transnational activities that were evaluated, as the following graphics illustrate.
Organisation of the transnational events
1% 2%6%
37%
55%
NA 1 2 3 4
total preferences
6% 6% 6% 5%
32%
50%40% 46% 42%
68%
44%54% 48% 54%
evidence of clear planningof activities
realistic timescales evidence in the mobilityprogram of real synergy
with the overall objectivesof the project
evidence of partners fromthe 4 countries sharing
roles and responsabilitiesduring the event
total
NA 1 2 3 4
Effectiveness of content and appropriate range and balance of activities
Evaluation of technical visits to farms and other stakeholders in MAP sector
2% 8% 2% 2%
2%
2% 1%4% 4% 6%
8%
2% 2% 2%4%
44%36% 28%
30%48%
40%32%
37%
50%60% 66%
52% 48%56%
66%57%
relevant mixtureof activities e.g.
icebreakingactivities,didactic
sessions,workshops,
social activities,free time
attention topractical details
and catering
suitability of thethe working
venues(welcomemeeting,seminar,
workgroups)
quality ofovernight
accomodation
evidence ofspecial
requirements(dietary for
example) beingmet
effectivecommunication
amongstparticipants
opportunity tomeet people and
companiesrelevant for
futurecollaboration
total
NA 1 2 3 4
2% 8% 3%8% 10%
10%9%
32% 30%
36%33%
58% 60%46%
55%
quality of the program of visits(appropriate selection of
producers and other stakeholdersto visit)
interesting and usefull informationprovided
quality of communication (abilityto hear/see and to understandwhat was being said/shown)
total
NA 1 2 3 4
Evaluation of Seminars
This positive evaluation occurs in all 4 mobilities. Average scores, in all items analysed are:
- Program of activities (very good 3,6): Portugal_3,9; France_3,3; Spain_3,5; Italy_3,5
- Networking-virtual collaboration space (very good 3,5): Portugal_3,5; France_3,0; Spain_3,6; Italy 3,7
More detailed data specifically on Seminars can be found in this report (one of the intellectual outputs previewed for the project): https://herbartis.files.wordpress.com/2017/09/herbartis-seminars-proceedings.pdf An evaluation survey (paper) was also delivered to participants (many of them were from outside the project) in the end of the Seminar in Spain. 53 people attended the technical seminar and their satisfaction rate was as following (1 bad, 2 regular, 3 good, 4 very good):
- The length of the seminar is adequate: 3,7 - The general content of the seminar has satisfied my expectations: 3,7 - The contents of the seminar are applicable to my working area: 3,6 - The organisation of the seminar has been correct: 3,6 - Global valorisation of the seminar: 3,7
2% 2% 2% 2%0% 2%10% 4%8%
12%10%
10%
38% 28%34%
33%
52% 56%44% 51%
quality of the program of theseminar(s)
interesting and usefull informationprovided
quality of the communication(ability to hear/see and to
In Portugal, an online survey was also sent to all participants in that came from outside of the project to the Transnational Seminar, but only 7 answers were obtained. Although very favourable, were not considered to be relevant, because of the small number of answers. Average grades were:
- Format of the seminar: 3,9
- Facilities and logistic aspects: 3,9
- Quality of interventions/ interest of subjects: 3,7
- Usefulness for your work: 3,3
- Opportunity to share doubts and opinions: 4
- Global satisfaction with the meeting: 4
Also in the Herbartis final multiplier event the participants were asked to give their opinion on these aspects (average grades):
- Length of the seminar is adequate: 3,2 - The general content has satisfied my expectations: 3,4 - The seminar contents are applicable to my working area: 3,2 - The organisation of the seminar has been correct: 3,6 - Global valorisation of the seminar: 3,4
Conclusions In general, the transnational HERBARTIS’ activities were evaluated positively, in all aspects. The most relevant issue for worst scores was language understanding. In spite of efforts to have all presentations written in English (during the seminars), some participants complained about the level of comprehension of the contents provided. Having analysed all the information, it is possible to conclude that a relevant number of HERBARTIS’ students couldn’t speak English (the official language of the course) very well, fact that had relevant impact on the evaluation. Some difficulty occurred also during the visits to farms, where the spoken language was mainly (sometimes exclusively) the local one. In spite of the similarities between the languages of the 4 countries, the communication wasn’t always easy. Even so, the participants were very satisfied with the activities.
Evaluation report_4 B-learning course Introduction All students from the 4 countries were asked to give their opinion on the national activities included in the blended course:
- Information, follow-up (organisers)
- Communication (with organisers, with other students)
- online lessons and tutoring
- e-learning platform
- weekly challenges
- virtual meetings with MAPs stakeholders
- face-to-face workshops
An online survey was built for that: https://docs.google.com/forms/d/e/1FAIpQLSeDc8Nms-krnQy0iI1FkF0601F-0QuSMQvBs5urU1SfZocD0w/viewform
clearness of instructions (course, platform, activities...) 2 25 17 44
follow-up (information, support) by the course's national organisers 5 15 24 44
Possibility to interact with other participants (teachers, students, producers...)
1 19 24 44
herbartis moodle platform 1 6 20 17 44
Total 1 21 123 124 264
Evaluation of the structure and contents of the on-line course
N/A 1
lowest 2 3
4 highest
Total
your overall satisfaction with the on-line course 1 22 21 44
adjustment of the curriculum to the objectives of the course 4 17 23 44
interest of contents provided 17 27 44
usefulness of contents provided 18 26 44
format and page design 6 24 14 44
timescale of the modules 1 4 26 13 44
support from teachers whenever asked 1 16 27 44
examples, links, exercises, clues for further study provided 1 19 24 44
the weekly challenge 5 27 12 44
Total 1 24 189 187 396
Evaluation of the virtual meetings with MAPs producers
N/A 1
lowest 2 3
4 highest
Total
interest/ pertinence of program (themes and invited participants) 13 1 14 16 44
interest and usefulness of information gathered 14 13 17 44
possibility to participate: ask questions, give opinions 14 1 1 9 19 44
use of Skype for the meeting 12 3 4 13 12 44
Total 53 4 6 52 64 176
Evaluation of face-to-face workshops
N/A 1
lowest 2 3
4 highest
Total
interest/ pertinence of themes 13 11 20 44
interest and usefulness of information 13 14 17 44
possibility to participate actively 13 1 8 22 44
Workshop facilities 13 1 9 21 44
clearness of presentations, clarification of doubts 13 1 12 18 44
Total 65 3 57 98 220
The students have a good opinion on the activities of the blended course, as the graphics show.
Evaluation of the b-learning course as a whole
Evaluation of the structure and contents of the on-line course
0,7%5,5%
43,5%50,3%
1 2 3 4
overall satisfaction level with blended course
15% 7% 5% 11%2%
14%
52% 41%57%
34%43%
45%
43%52%
39%55% 55%
39%
the blended courseas a whole
mix of activities(course on-line +
workshops + virtualmeetings)
clearness ofinstructions (course,
platform,activities...)
follow-up(information,
support) by thecourse's national
organisers
possibility to interactwith other
participants(teachers, students,
producers...)
herbartis moodleplatform
1 2 3 4
12% 9% 14% 9%2% 2%
11%
50% 39%39% 41%
55% 59%
36% 43%
61%
48% 52%61% 59%
32% 30%
61% 55%
27%
your overallsatisfactionwith the on-line course
adjustment ofthe
curriculum tothe objectivesof the course
interest ofcontentsprovided
usefulness ofcontentsprovided
format andpage design
timescale ofthe modules
support fromteacherswhenever
asked
examples,links,
exercices,clues for
further studyprovided
the weeklychallenge
1 2 3 4
Evaluation of the virtual meetings with MAPs producers
Evaluation of face-to-face workshops
The average grades (from bad 0 to 4 very good) obtained in the 4 countries are:
- Blended course as a whole (good 3,2): Portuguese_3,5; French_2,6; Spanish_3,6; Italian_3,2 (best: professional and high knowledge; worst: lack of interactive tools)
- On-line lessons (good 3,4): Portuguese_3,5; French_2,8; Spanish_3,7; Italian_3,4 (best: complete information; worst: more workload than expected)
- Virtual meetings with local stakeholders (good 3,4):
3,33% 9,38%3% 3%
13%
45%43% 30%
41%
52% 57% 63%
38%
interest/ pertinence of program(themes and invited
participants)
interest and usefulness ofinformation gathered
possibility to participate: askquestions, give opinions
use of Skype for the meeting
1 2 3 4
0% 0% 3% 3% 3%
35%45%
26% 29%39%
65%55%
71% 68%58%
interest/ pertinence ofthemes
interest andusefulness ofinformation
possibility toparticipate actively
Workshop facilities clearness ofpresentations,
clarification of doubts
1 2 3 4
Portuguese_3,6; French_2,5; Spanish_3,7; Italian_3,7 (best rated: knowing local experiences; less rated: connexion problems with Skype);
- Face-to-face workshops (good 3,4): Portuguese_3,9; French_2,8; Spanish_3,5; Italian_3,2 (best: technical information; improve: more time to deep).
Conclusions The students thought that the b-learning course’s activities were very good: their overall evaluation was 93.8% positive, with 50.3% of all evaluated topics ranked by "4", the highest grade. The most appreciated activities were not the same in the 4 countries. Face-to-face workshops received the best grade in Portugal, while Spanish students preferred the on-line lessons and the virtual meetings with local stakeholders. Italian students also rated these meetings best. For the French students, the online lessons were better rated than the rest of national activities. “Time” was seen as the major problem in all countries: the course should have lasted longer, or the online lessons should be shorter. It was very difficult to read all delivered information and to fulfil every challenge within the proposed time.
Evaluation report_5 Herbartis Project by partners Introduction An online survey was sent to the partners in the end of the project to know if results have been achieved, the methodology has been properly implemented, partners are satisfied with the project monitoring and environment is comfortable and productive. https://docs.google.com/forms/d/e/1FAIpQLSewtCqTRJYB5jU_i5ijK6FCYhmA9b4WjMtFdIHcuUHyD9YQMg/viewform#responses
Results 12 people from the staff of the partners – from the 4 countries - answered the survey. The general opinion is, like in all other surveys, positive or very positive. Only in one case, regarding financial aspects, almost all participants exposed their unsatisfaction. The project
N/A strongly disagree
disagree neutral agree strongly agree
Total
the project was well designed/ planned activities make sense
7 5 12
the project objectives were achieved 9 3 12
the right beneficiaries (students, participants) were involved
5 7 12
all planned activities were successfully executed 2 7 3 12
approved budget was adequate to the project 7 4 1 0 12
there are good perspectives to continue developing the project in the future
2 7 3 12
Total 7 4 5 35 21 72
The Partnership
N/A strongly disagree
disagree neutral agree strongly agree
Total
the present partnership was adequate for this project? 8 4 12
partners showed commitment to the project? 9 3 12
communication between partners was effective 9 3 12
the relation amongst partners was built on trust and positive attitudes
6 6 12
leadership and decision making processes were good 7 5 12
roles, resources and responsibilities were clearly defined
1 1 6 4 12
there are good perspectives that this partnership may continue working together in the future
6 6 12
Total 1 0 0 1 51 36 84
Competences/ skills
N/A strongly disagree
disagree neutral agree strongly agree
Total
competences on projects management improved 1 2 5 4 12
the project was an opportunity to contact with different realities in other rural territories
5 7 12
ideas, knowledge and experiences were shared 6 6 12
international contacts and networking were amplified 7 5 12
new competences in the use of technology were developed
1 1 8 2 12
news subjects were learnt (namely with preparation of on-line course)
2 6 4 12
skills in preparation and management of b-learning courses
1 2 6 3 12
foreign languages use and/or comprehension improved 2 7 3 12
Total 3 0 0 9 50 34 96
Activities and intellectual outputs
N/A totally
unsatisfied unsatisfied neutral satisfied
extremely satisfied
Total
the project as a whole 9 3 12
analysis of the profession (lists of specifications, existing courses, characterisation of stakeholders and their needs of training)
1 10 1 12
curriculum of herbal craft production 1 9 2 12
on-line course lessons 10 2 12
on-line course platform 2 8 2 12
virtual meetings with stakeholders 1 2 7 2 12
face-to-face workshops 1 9 2 12
international seminars 10 2 12
international workgroups 1 2 7 2 12
international technical visits 9 3 12
the guidelines 1 10 1 12
communication tools (website, facebook, press reviews...)
Looking at the graphics, the referred only exception to the most common pattern – quite well visible - comes with the statement “approved budget was adequate to the project”, to which 91% of the participants disagreed or strongly disagreed. The general opinion is that the approved budget (with cuts to the applied proposal) was too short for the work that had to be done for the project.
Conclusions The project was seen as a very successful one, by the partners. Aside from the financial issue, they testified their satisfaction with the project’s objectives and general results, the partners involved, the skills developed and the activities and materials produced. The best rated topics were those related to the partnership: its constitution and the good work environment, and to the skills developed, in terms of awareness of differences and similarities between countries, new knowledge, new technical skills directly linked to the project (like building a b-learning course) and improvement in understanding of foreign languages. A relevant note is the fact that most of the participants really would like to continue with the project and to have other opportunities to work together again within this partnership.
