Searching for "online evaluation"
company or group who is doing online program evaluation?
this information is extracted from the Blend-Online discussion list
Colleagues,
Do you know any company or group who is doing online program evaluation? Our school is seeking a consulting group to come to review our online programs and identify areas relate to online learning that we need to improve.
Thanks,
Carrie Halpin, Ph.D. Professor/Instructional Designer & Technologist eLearning & Instructional Technology (eLIT) Virginia Western Community College 3095 Colonial Ave. SW, Roanoke, VA 24015 Office: Brown Library 102 Phone: 540-857-6636 Fax: 540-857-6138 Email: chalpin@virginiawestern.edu
iDesign will do both of those things using a fee-for-service model versus a revenue-sharing model like Academic Partnerships or 2U. I have no personal experience with any OPM, but iDesign is the only one I know of that offers that ala carte type service.
+++++++++++
more on evaluations regarding online teaching in this IMS blog
https://blog.stcloudstate.edu/ims?s=online+evaluation
Tobin, T. J., Mandernach, B. J., & Taylor, A. H. (2015). Evaluating Online Teaching: Implementing Best Practices (1 edition). San Francisco, CA: Jossey-Bass.
- 5 measurable faculty competencies for on line teaching:
- attend to unique challenges of distance learning
- Be familiar with unique learning needs
- Achieve mastery of course content, structure , and organization
- Respond to student inquiries
- Provide detailed feedback
- Communicate effectively
- Promote a safe learning environment
- Monitor student progress
- Communicate course goals
- Provide evidence of teaching presence.
Best practices include:
- Making interactions challenging yet supportive for students
- Asking learners to be active participants in the learning process
- Acknowledging variety on the ways that students learn best
- Providing timely and constructive feedback
Evaluation principles
- Instructor knowledge
- Method of instruction
- Instructor-student rapport
- Teaching behaviors
- Enthusiastic teaching
- Concern for teaching
- Overall
8. The American Association for higher Education 9 principle4s of Good practice for assessing student learning from 1996 hold equally in the F2F and online environments:
the assessment of student learning beings with educational values
assessment is most effective when it reflects an understanding of learning as multidimensional, integrated and revealed in performance over time
assessment works best when the programs it seeks to improve have clear, explicitly stated purposes.
Assessment requires attention to outcomes but also and equally to the experiences that lead to those outcomes.
Assessment works best when it is ongoing, not episodic
Assessment fosters wider improvement when representatives from across the educational community are involved
Assessment makes a difference when it begins with issues of use and illumines questions that people really care bout
Assessment is most likely to lead to improvements when it is part of the large set of conditions that promote change.
Through assessment, educators meet responsibilities to students and to the public.
9 most of the online teaching evaluation instruments in use today are created to evaluate content design rather than teaching practices.
29 stakeholders for the evaluation of online teaching
- faculty members with online teaching experience
- campus faculty members as a means of establishing equitable evaluation across modes of teaching
- contingent faculty members teaching online
- department or college administrators
- members of faculty unions or representative governing organizations
- administrative support specialists
- distance learning administrators
- technology specialists
- LMS administrators
- Faculty development and training specialists
- Institutional assessment and effectiveness specialists
- Students
Sample student rating q/s
University resources
Rate the effectiveness of the online library for locationg course materials
Based on your experience,
148. Checklist for Online Interactive Learning COIL
150. Quality Online Course Initiative QOCI
151 QM Rubric
154 The Online Insturctor Evaluation System OIES
163 Data Analytics: moving beyond student learning
- # of announcments posted per module
- # of contributions to the asynchronous discussion boards
- Quality of the contributions
- Timeliness of posting student grades
- Timelines of student feedback
- Quality of instructional supplements
- Quality of feedback on student work
- Frequency of logins
- 180 understanding big data
- reliability
- validity
- factor structure
187 a holistics valuation plan should include both formative evaluation, in which observations and rating are undertaken with the purposes of improving teaching and learning, and summative evaluation, in which observation and ratings are used in order to make personnel decisions, such as granting promotion and tenure, remediation, and asking contingent faculty to teach again.
195 separating teaching behaviors from content design
+++++++++++++++++
more on online teaching in this IMS blog
https://blog.stcloudstate.edu/ims?s=online+teaching
Mariya P. Ivancheva, Rebecca Swartz, Neil P. Morris, Sukaina Walji, Bronwen J. Swinnerton, Taryn Coop & Laura Czerniewicz (2020) Conflicting logics of online higher education, British Journal of Sociology of Education, DOI: 10.1080/01425692.2020.1784707
https://www.tandfonline.com/action/showCitFormats?doi=10.1080%2F01425692.2020.1784707&area=0000000000000001
The advent of massive open online courses and online degrees offered via digital platforms has occurred in a climate of austerity. Public universities worldwide face challenges to expand their educational reach, while competing in international rankings, raising fees and generating third-stream income. Online forms of unbundled provision offering smaller flexible low-cost curricular units have promised to disrupt this system. Yet do these forms challenge existing hierarchies in higher education and the market logic that puts pressure on universities and public institutions at large in the neoliberal era? Based on fieldwork in South Africa, this article explores the perceptions of senior managers of public universities and of online programme management companies. Analysing their considerations around unbundled provision, we discuss two conflicting logics of higher education that actors in structurally different positions and in historically divergent institutions use to justify their involvement in public–private partnerships: the logic of capital and the logic of social relevance.
Unbundling – the disaggregation of educational provision and its delivery, often via digital technologies
Luc Boltanski and Laurent Thévenot’s (2006) framework of different orders of justification, connecting them to the sociological literature on institutional logics
We suggest that more explicit and nuanced national and institutional policies need to be produced around unbundled provision, which are cognisant of emerging trends in and dangers to the evolution of unbundling at public universities.
Unbundling the traditional university ‘bundle’ affects not only property, services and facilities, but also administration, evaluation, issuing credentials and even teaching (Wallhaus 2000, 22). This process involves separating educational provision (e.g. degree programmes) into component parts (e.g. courses) for delivery by multiple stakeholders, often using digital approaches (Swinnerton et al. 2018). Universities can unbundle on their own, offering individual credit-bearing modules outside bounded disciplinary curricula, or in partnership with OPM providers, offering MOOCs or credit-bearing courses or programmes. Proponents of unbundling suggest that the disaggregation of television and music production and its re-aggregation as on-demand digital content like Netflix or Spotify could represent a template for universities (Craig 2015; McIntosh 2018).
The introduction of market logic into the sector happens even if higher education is a stratified positional pseudo-market with scarce excludible resources only available to groups with access to a few prestigious institutions; its outcomes and value are difficult to measure in purely economic terms
Under accelerated marketisation, Tomlinson (2018, 714 and 724) argues, higher education is reduced to the latter frame and measured in terms of income generation, employability, consumption and performativity. Building on this framework, and relating it to unbundling, we identify the emergence of two organisational logics of higher education: the logic of social relevance and the logic of capital.
Institutional logics are ‘supra-organizational patterns of activity by which individuals and organizations produce and reproduce their material subsistence … [and] symbolic systems, ways of ordering reality… rendering experience of time and space meaningful’ (Friedland and Alford 1991, 243). Unlike new institutionalism, which remained focused on processes of institutional isomorphism or the replacement of a static single logic by another, the institutional logics perspective offers a more dynamic multi-level view: a plurality of logics coexist in complex interrelations within organisational fields like higher education
https://www.facebook.com/groups/onlinelearningcollective/permalink/591411374822898/
Hi colleagues, My provost just put out a set of expected guidelines for instructors in online classes that emphasize expectations around discussion forums (I pasted them below). These discussion forum expectations are very narrowly defined. I am needing group-think on references that might help me put together some “best practice” alternatives. If an article or other resource comes to mind, please share!
Online Faculty Expectations
Weekly Required (all weeks)
• Faculty will demonstrate their presence in the class 5 days per week
• Respond to all students’ (who post on-time) primary discussion post if you have 9 or fewer students (1/2 of students if you have 10 or more).
• Faculty with larger courses should take special care to post to different students each week.
• Faculty who provide a weekly zoom lecture need only post on the board two other times (on two different days for a total of two other posts).
• Provide individual feedback (posted in the feedback section of the gradebook) for all discussion grades within a reasonable timeframe for students to complete subsequent assignments.
responses:
Dayna Henry I balk at the admin trying to tell us what to do. At the same time, I am very angry with colleagues who did not actually offer anything in the way of virtual learning when we went online in spring. It’s hard to balance academic freedom with faculty who don’t care to learn/offer a new way of learning (for your institution). I also recognize the admin was not in their F2F courses either and likely the slacking was occurring there too. The problem is the students LOVE these folks for giving them an easy A/pass.
Cathy Curran For years I have said that administrators need to teach at least one each year or every other year. My Dean has been out of the classroom for over 20 years, the Provost for over 25 and the Chancellor has never taught. They have zero clue how to build or implement and online class. They keep making mandates that to those of us who do actually teach seem absurd. We know the “count and classify” nonsense never works but it is the same argument they use for numerical evaluations of teaching effectiveness: it is objective. The decisions they are making do not make instruction better they are all about power and control, they need us to “prove” that we are doing our job and somehow logging into the LMS five days a week does that. Sad really really sad. Well you know some do and other become administrators…
++++++++++++++++++
more on online discussions in this IMS blog
https://blog.stcloudstate.edu/ims?s=online+discussions
https://www.edsurge.com/news/2018-06-12-the-number-of-students-taking-in-online-courses-is-quickly-rising-but-perceptions-are-changing-slowly
The Babson Survey Research Group, an organization that tracks online enrollment, notes that between 2012 and 2016 the percent of online enrollment in universities increased 17.2 percent while overall enrollment decreased. But that expansion doesn’t necessarily correlate with how the public perceives the quality of online courses, historically questioned for its lack of rigor and limited measurable learning gains.
A Gallup poll conducted back in 2015, found that 46 percent of Americans “strongly agree” or “agree” that online colleges and universities offer a high-quality education—up 30 percent from when the poll was conducted in 2011.
However, researchers caveat these findings, noting that these perception changes happen within particular pockets and are sometimes the result of strategic practices, such as universities not listing the medium of learning on student transcripts.
The last academic leader perception survey released by the Babson Research Group was in 2016.
“We’ve had more and more of the group in the middle that said, ‘I’m not sure’ move into a pro online learning stance,” says Seaman, speaking of the academic leaders he surveyed in the past. “The negative group [those who viewed online learning negatively] had not wavered at all. The positive group did not waiver at all, but we had a steady migration flow of academic leaders in the middle.”
Lowenthal has also researched student perceptions of online learning in the past, finding that learners tend to give such courses more negative evaluations than in-person courses. He says that the findings may represent the lack of experience some educators have teaching in online classrooms. He expects that to change over time, noting that good teachers in person will eventually become good teachers online.
++++++++++++
more on online learning in this IMS blog
https://blog.stcloudstate.edu/ims?s=online+learning
Gunawan, F. (2018). GAMIFICATION ANALYSIS AND IMPLEMENTATION IN ONLINE LEARNING. ICIC Express Letters, 12(12), 1195–1204.
Khan [14] has introduced the eight-dimensional elearning framework, a detailed self assessment instrument for institutions to evaluate the readiness and the opportunity of their e-learning classes to grow.
institutional, management, technological, pedagogical, ethical, interface design, resource support, and evaluation. Institutional refers to the administrative and academic part of the system. Management refers to the quality control, budget, and scheduling. Technological refers to the infrastructure, hardware, and software. Pedagogical refers to analysis, organization and learning strategies. Ethical refers to ethical, legal, and social and political influences. Interface design refers to the user interface, accessibility, and design content. Resource support refers to career services, journals, and online forums. Finally, the evaluation refers to the assessment of learners and educators.
gamification – definition
Modern gamification term was first introduced by
Nick Pelling in 2002 [15]. Gamification is a concept that implements the game components
into the non-game contents such as education, marketing, administration, or even software
engineering [16]. These components include points, badges, leaderboards, and quests.
Each of them serves the purpose to increase the level of user engagement in the learning
process.
three components of engagement: cognitive, behavioral, and emotional [19].
Weakest students more likely to take online college classes but do worse in them
Protopsalt is is a professor at George Mason University, where he directs Center for Education Policy and Evaluation. He previously served as a senior official in the U.S. Department of Education.
The paper, “Does Online Education Live Up to Its Promise? A Look at the Evidence and Implications for Federal Policy,” was also written by Sandy Baum, an economist at the Urban Institute, a nonprofit research organization.
At four-year universities, students with high grades often did just as well in an online course, but those with low grades suffered more. Another 2017 study of students at a for-profit university which offers both in-person and online classes found that students who took an online class not only got lower grades in that class but also in future classes. Online students were more likely to drop out of college altogether than similar students who attended in-person classes.
The question is whether we should keep expanding online learning, with generous federal subsidies, to the most vulnerable students before colleges have tested and proven they can educate them adequately outside the classroom.
+++++++++++
more on online learning in this IMS blog
https://blog.stcloudstate.edu/ims?s=online+learning
Advancing Online Education in Minnesota State
Advancing Online Education – Full Report-1s94jfi
Defining Online Education
The term “online education” has been used as a blanket phrase for a number of fundamentally different educational models. Phrases like distance education, e-Learning, massively open online courses (MOOCs), hybrid/blended learning, immersive learning, personalized and/or adaptive learning, master courses, computer based instruction/tutorials, digital literacy and even competency based learning have all colored the definitions the public uses to define “online education.”
online education” as having the following characteristics:
- Students who enroll in online courses or programs may reside near or far from the campus(es) providing the course(s) or program.
- A student’s course load may include offering where attendance is required in person or where an instructor/students are not required to be in the same geographic location.
- Students may enroll in one or more individual online course offerings provided by one or more institutions to that may or may not satisfy degree/program requirements.
- Student may pursue a certificate, program, or degree where a substantial number of courses, perhaps all, are taken without being in the same geographic location as others.
Organizational Effectiveness Research Group (OERG),
As the workgroup considered strategies that could advance online education, they were asked to use the primary and secondary sources listed above to support the fifteen (15) strategies that were developed
define a goal as a broad aspirational outcome that we strive to attain. Four goal areas guide this document. These goal areas include access, quality, affordability and collaboration. Below is a description of each goal area and the assumptions made for Minnesota State.
- Access
Over twenty percent of existing Minnesota State students enroll in online courses as a way to satisfy course requirements. For some students, online education is a convenient option; for others, online is the only option available
- Quality
The Higher Learning Commission (HLC) accreditation guidelines review the standards and processes institutions have in place to ensure quality in all of educational offerings, including online.
There are a number of ways in which institutions have demonstrated quality in individual courses and programs including the evaluation of course design, evaluation of instruction and assessment of student
- Affordability
a differential tuition rate to courses that are offered online. If we intend to have online education continue to be an affordable solution for students, Minnesota State and its institutions must be good stewards of these funds and ensure these funds support online education.
Online education requires different or additional services that need to be funded
transparency is important in tuition setting
- Collaboration
Distance Minnesota is comprised of four institutions Alexandria Technical & Community College, Bemidji State University, Northland Community & Technical College, and Northwest Technical College) which collaborate to offer student support services, outreach, e-advising, faculty support, and administrative assistance for online education offerings.
Strategies
strategies are defined as the overall plan used to identify how we can achieve each goal area.
Action Steps
Strategy 1: Ensure all student have online access to high quality support services
students enrolled in online education experiences should have access to “three areas of support including academic (such as tutoring, advising, and library); administrative (such as financial aid, and disability support); and technical (such as hardware reliability and uptime, and help desk).”
As a system, students have access to a handful of statewide services, include tutoring services through Smarthinking and test proctoring sites.
Strategy 2: Establish and maintain measures to assess and support student readiness for online education
A persistent issue for campuses has been to ensure that students who enroll in online course are aware of the expectations required to participate actively in an online course.
In addition to adhering to course expectations, students must have the technical competencies needed to perform the tasks required for online courses
Strategy 3: Ensure students have access to online and blended learning experiences in course and program offerings.
Strategy 4: These experiences should support and recognize diverse learning needs by applying a universal design for learning framework.
The OERG report included several references to efforts made by campuses related to the providing support and resources for universal design for learning, the workgroup did not offer any action steps.
Strategy 5: Expand access to professional development resources and services for faculty members
As online course are developed and while faculty members teach online courses, it is critical that faculty members have on-demand access to resources like technical support and course assistance.
5A. Statewide Faculty Support Services – Minnesota State provide its institutions and their faculty members with access to a centralized support center during extended hours with staff that can assist faculty members synchronously via phone, chat, text/SMS, or web conference
5C. Instructional Design and Technology Services – Establish a unit that will provide course design and instructional technology services to selected programs and courses from Minnesota State institutions.
Quality
Strategy 1: Establish and maintain a statewide approach for professional development for online education.
1B. Faculty Mentoring – Provide and sustain faculty mentoring programs that promote effective online pedagogy.
1C. Professional development for support staff – including instructional designers, D2L Brightspace site administrators and campus trainers, etc.)
+++++++++++
more on online education in this IMS blog
https://blog.stcloudstate.edu/ims?s=online+education
The Online Discussion group for Blended and Online Learning leads an interesting discussion on course evaluations; here are the highlights:
Perceptions-of-Online-Learning–Analysis-of-Online-Course-Evaluations-tbz6om
When we first started in 1999, we included ~10 questions in addition to our standard questions that were different for online courses. This information was particularly useful as we grew our online offerings (i.e. Would you take another online course. 93-5% answered yes consistently. How would you rate the level of interactivity between you and the instructor? Between you and the other students?) These were administered via SurveyMonkey because there were no online evaluation services back then.
Now we have a single evaluation that is administered to all students regardless of the delivery format (online, hybrid, blended, F2F or intensive) The questions were designed to be relevant regardless of the delivery format. All of these evaluations are administered online…which has its downsides (e.g. response rate is less especially compared to what was captured in F2F classes in the past.) We continue to explore ways to increase the response rate.
Reta Chaffee Director of Educational Technology-Academic Affairs Granite State College 25 Hall Street Concord, NH 03301 (603) 513-1350
+++++++++++++++++++++
On Behalf Of Krajewski, Scott
Sent: Friday, February 17, 2017 1:00 PM
To: BLEND-ONLINE@LISTSERV.EDUCAUSE.EDU
Subject: Re: [BLEND-ONLINE] Course Evaluations
Hi Hala, We have a standard online evaluation form for all courses. We do add 3 questions to the sports courses but otherwise we’re 100% standardized. We have a ton of info at
http://inside.augsburg.edu/ctl/resources/augsburg-resources-and-support/course-evaluations/
++++++++++++++++++++
You might find this study (or the related literature) helpful — http://patricklowenthal.com/publications/Student-Perceptions-of-Online-Learning–Analysis-of-Online-Course-Evaluations.pdf
Patrick Patrick R. Lowenthal | Associate Professor Educational Technology, Boise State University http://www.patricklowenthal.com
+++++++++++++++++++++++
On Behalf Of Rob Gibson
Sent: Friday, February 17, 2017 8:39 AM
To: BLEND-ONLINE@LISTSERV.EDUCAUSE.EDU
Subject: Re: [BLEND-ONLINE] Course Evaluations
We use the IDEA evaluation framework combined with CampusLabs as the delivery engine.
- IDEA is a well-established evaluation process dating back to the 1970s.
- The CampusLabs delivery process (new as of about 2 years ago) provides students with a single URL to complete their evaluations – on-campus or on-line. Mobile friendly.
- It uses the same base evaluation criteria across the university. (That’s how IDEA is able to substantiate reliability and validity.) IDEA is matched against a national database using a CIP code. Hence, faculty can gather comparative data of their course against other similar courses in the university, or at the national level.
- While each department uses the same basic framework, there are modification that can be made. For example, custom questions can be added to the eval (these fall outside the scope of the comparative data) and the learning objectives can be modified by course, department, school, college. We have one School that has custom learning objectives for each course in their program. Objectives are set using a 3 point Likert scale.
Very easy to set up a survey administration. Data is retrievable within 48 hours after close.
+++++++++++++
more on online learning in this IMS blog
https://blog.stcloudstate.edu/ims?s=online+education
Discussion on the EDUCAUSE Blended and Online Learning Group’s listserv
Question:
develop anonymous mid-course student evaluations allowing students to reflect on course and progress and informing instructor about what is working or not in the course.
Answers:
– what is working well for you in the course?
– what is not working well for you in the course?
krajewsk@AUGSBURG.EDU
++++++++++++++++++
- What is helping you learn?
- What is hindering your learning?
- What suggestions do you have to make the course better for you, your peers, or the instructor?
Katie Linder Research Director Extended Campus, Oregon State University 4943 The Valley Library Corvallis, Oregon 97331 Phone 541-737-4629 | Fax 541-737-2734 Email: kathryn.linder@oregonstate.edu
++++++++++++++++
At the University of Illinois, we have been using Informal Early Feedback as a way to gauge information from our students to help improve the courses before the end. Here are a couple of links to our site. The first is the main page on what IEF is and the second is the question bank we offer to faculty. This is a starting point for them, then we meet with those who want to work on tweaking them for their specific needs.
* About IEF: https://citl.illinois.edu/citl-101/measurement-evaluation/teaching-evaluation/ief
* Question Bank: https://citl.illinois.edu/citl-101/measurement-evaluation/teaching-evaluation/ief/ief-question-bank
If you have any questions at all, don’t hesitate to ask.
Sol Roberts-Lieb Associate Director, Center for Innovation in Teaching and Learning Pedagogy Strategy Team and Industry Liaison UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN
++++++++++++++++++
more on student evaluations in this IMS blog:
https://blog.stcloudstate.edu/ims?s=student+evaluation