company or group who is doing online program evaluation?
this information is extracted from the Blend-Online discussion list
Do you know any company or group who is doing online program evaluation? Our school is seeking a consulting group to come to review our online programs and identify areas relate to online learning that we need to improve.
Carrie Halpin, Ph.D. Professor/Instructional Designer & Technologist eLearning & Instructional Technology (eLIT) Virginia Western Community College 3095 Colonial Ave. SW, Roanoke, VA 24015 Office: Brown Library 102 Phone: 540-857-6636 Fax: 540-857-6138 Email: firstname.lastname@example.org
iDesign will do both of those things using a fee-for-service model versus a revenue-sharing model like Academic Partnerships or 2U. I have no personal experience with any OPM, but iDesign is the only one I know of that offers that ala carte type service.
Tobin, T. J., Mandernach, B. J., & Taylor, A. H. (2015). Evaluating Online Teaching: Implementing Best Practices (1 edition). San Francisco, CA: Jossey-Bass.
5 measurable faculty competencies for on line teaching:
attend to unique challenges of distance learning
Be familiar with unique learning needs
Achieve mastery of course content, structure , and organization
Respond to student inquiries
Provide detailed feedback
Promote a safe learning environment
Monitor student progress
Communicate course goals
Provide evidence of teaching presence.
Best practices include:
Making interactions challenging yet supportive for students
Asking learners to be active participants in the learning process
Acknowledging variety on the ways that students learn best
Providing timely and constructive feedback
Method of instruction
Concern for teaching
8. The American Association for higher Education 9 principle4s of Good practice for assessing student learning from 1996 hold equally in the F2F and online environments:
the assessment of student learning beings with educational values
assessment is most effective when it reflects an understanding of learning as multidimensional, integrated and revealed in performance over time
assessment works best when the programs it seeks to improve have clear, explicitly stated purposes.
Assessment requires attention to outcomes but also and equally to the experiences that lead to those outcomes.
Assessment works best when it is ongoing, not episodic
Assessment fosters wider improvement when representatives from across the educational community are involved
Assessment makes a difference when it begins with issues of use and illumines questions that people really care bout
Assessment is most likely to lead to improvements when it is part of the large set of conditions that promote change.
Through assessment, educators meet responsibilities to students and to the public.
9 most of the online teaching evaluation instruments in use today are created to evaluate content design rather than teaching practices.
29 stakeholders for the evaluation of online teaching
faculty members with online teaching experience
campus faculty members as a means of establishing equitable evaluation across modes of teaching
contingent faculty members teaching online
department or college administrators
members of faculty unions or representative governing organizations
administrative support specialists
distance learning administrators
Faculty development and training specialists
Institutional assessment and effectiveness specialists
Sample student rating q/s
Rate the effectiveness of the online library for locationg course materials
Based on your experience,
148. Checklist for Online Interactive Learning COIL
150. Quality Online Course Initiative QOCI
151 QM Rubric
154 The Online Insturctor Evaluation System OIES
163 Data Analytics: moving beyond student learning
# of announcments posted per module
# of contributions to the asynchronous discussion boards
Quality of the contributions
Timeliness of posting student grades
Timelines of student feedback
Quality of instructional supplements
Quality of feedback on student work
Frequency of logins
180 understanding big data
187 a holistics valuation plan should include both formative evaluation, in which observations and rating are undertaken with the purposes of improving teaching and learning, and summative evaluation, in which observation and ratings are used in order to make personnel decisions, such as granting promotion and tenure, remediation, and asking contingent faculty to teach again.
195 separating teaching behaviors from content design
Defining Online Education
The term “online education” has been used as a blanket phrase for a number of fundamentally different educational models. Phrases like distance education, e-Learning, massively open online courses (MOOCs), hybrid/blended learning, immersive learning, personalized and/or adaptive learning, master courses, computer based instruction/tutorials, digital literacy and even competency based learning have all colored the definitions the public uses to define “online education.”
online education” as having the following characteristics:
Students who enroll in online courses or programs may reside near or far from the campus(es) providing the course(s) or program.
A student’s course load may include offering where attendance is required in person or where an instructor/students are not required to be in the same geographic location.
Students may enroll in one or more individual online course offerings provided by one or more institutions to that may or may not satisfy degree/program requirements.
Student may pursue a certificate, program, or degree where a substantial number of courses, perhaps all, are taken without being in the same geographic location as others.
Organizational Effectiveness Research Group (OERG),
As the workgroup considered strategies that could advance online education, they were asked to use the primary and secondary sources listed above to support the fifteen (15) strategies that were developed
define a goal as a broad aspirational outcome that we strive to attain. Four goal areas guide this document. These goal areas include access, quality, affordability and collaboration. Below is a description of each goal area and the assumptions made for Minnesota State.
Over twenty percent of existing Minnesota State students enroll in online courses as a way to satisfy course requirements. For some students, online education is a convenient option; for others, online is the only option available
The Higher Learning Commission (HLC) accreditation guidelines review the standards and processes institutions have in place to ensure quality in all of educational offerings, including online.
There are a number of ways in which institutions have demonstrated quality in individual courses and programs including the evaluation of course design, evaluation of instruction and assessment of student
a differential tuition rate to courses that are offered online. If we intend to have online education continue to be an affordable solution for students, Minnesota State and its institutions must be good stewards of these funds and ensure these funds support online education.
Online education requires different or additional services that need to be funded
transparency is important in tuition setting
Distance Minnesota is comprised of four institutions Alexandria Technical & Community College, Bemidji State University, Northland Community & Technical College, and Northwest Technical College) which collaborate to offer student support services, outreach, e-advising, faculty support, and administrative assistance for online education offerings.
strategies are defined as the overall plan used to identify how we can achieve each goal area.
Strategy 1: Ensure all student have online access to high quality support services
students enrolled in online education experiences should have access to “three areas of support including academic (such as tutoring, advising, and library); administrative (such as financial aid, and disability support); and technical (such as hardware reliability and uptime, and help desk).”
As a system, students have access to a handful of statewide services, include tutoring services through Smarthinking and test proctoring sites.
Strategy 2: Establish and maintain measures to assess and support student readiness for online education
A persistent issue for campuses has been to ensure that students who enroll in online course are aware of the expectations required to participate actively in an online course.
In addition to adhering to course expectations, students must have the technical competencies needed to perform the tasks required for online courses
Strategy 3: Ensure students have access to online and blended learning experiences in course and program offerings.
Strategy 4: These experiences should support and recognize diverse learning needs by applying a universal design for learning framework.
The OERG report included several references to efforts made by campuses related to the providing support and resources for universal design for learning, the workgroup did not offer any action steps.
Strategy 5: Expand access to professional development resources and services for faculty members
As online course are developed and while faculty members teach online courses, it is critical that faculty members have on-demand access to resources like technical support and course assistance.
5A. Statewide Faculty Support Services – Minnesota State provide its institutions and their faculty members with access to a centralized support center during extended hours with staff that can assist faculty members synchronously via phone, chat, text/SMS, or web conference
5C. Instructional Design and Technology Services – Establish a unit that will provide course design and instructional technology services to selected programs and courses from Minnesota State institutions.
Strategy 1: Establish and maintain a statewide approach for professional development for online education.
1B. Faculty Mentoring – Provide and sustain faculty mentoring programs that promote effective online pedagogy.
1C. Professional development for support staff – including instructional designers, D2L Brightspace site administrators and campus trainers, etc.)
When we first started in 1999, we included ~10 questions in addition to our standard questions that were different for online courses. This information was particularly useful as we grew our online offerings (i.e. Would you take another online course. 93-5% answered yes consistently. How would you rate the level of interactivity between you and the instructor? Between you and the other students?) These were administered via SurveyMonkey because there were no online evaluation services back then.
Now we have a single evaluation that is administered to all students regardless of the delivery format (online, hybrid, blended, F2F or intensive) The questions were designed to be relevant regardless of the delivery format. All of these evaluations are administered online…which has its downsides (e.g. response rate is less especially compared to what was captured in F2F classes in the past.) We continue to explore ways to increase the response rate.
Reta Chaffee Director of Educational Technology-Academic Affairs Granite State College 25 Hall Street Concord, NH 03301 (603) 513-1350
We use the IDEA evaluation framework combined with CampusLabs as the delivery engine.
IDEA is a well-established evaluation process dating back to the 1970s.
The CampusLabs delivery process (new as of about 2 years ago) provides students with a single URL to complete their evaluations – on-campus or on-line. Mobile friendly.
It uses the same base evaluation criteria across the university. (That’s how IDEA is able to substantiate reliability and validity.) IDEA is matched against a national database using a CIP code. Hence, faculty can gather comparative data of their course against other similar courses in the university, or at the national level.
While each department uses the same basic framework, there are modification that can be made. For example, custom questions can be added to the eval (these fall outside the scope of the comparative data) and the learning objectives can be modified by course, department, school, college. We have one School that has custom learning objectives for each course in their program. Objectives are set using a 3 point Likert scale.
What suggestions do you have to make the course better for you, your peers, or the instructor?
Katie Linder Research Director Extended Campus, Oregon State University 4943 The Valley Library Corvallis, Oregon 97331 Phone 541-737-4629 | Fax 541-737-2734 Email: email@example.com
At the University of Illinois, we have been using Informal Early Feedback as a way to gauge information from our students to help improve the courses before the end. Here are a couple of links to our site. The first is the main page on what IEF is and the second is the question bank we offer to faculty. This is a starting point for them, then we meet with those who want to work on tweaking them for their specific needs.
The eCampus Quality Instruction Program (eQIP) is an online faculty development program developed to train faculty in designing and teaching fully online courses.
What is the best way to design and develop high- quality online courses and support faculty as they teach online?
Given faculty’s competing priorities and limited time, we contend that it is important for institutions, and specifically faculty developers, to analyze how much time faculty are spending in online faculty development activities as well as which parts are taking the most (or least) time. (p. 5)
A successful online faculty development program must include pedagogical support, technology support, and design and development support (Baran & Correia, 2014) that overcome obstacles about time, expertise, and motivation of faculty (Henning, 2012).(p. 17)
is it possible that the Iranian government realized the evolution of social media and his respective obsolescence and this is why they freed him prematurely?
Blogs were gold and bloggers were rock stars back in 2008 when I was arrested.
The hyperlink was a way to abandon centralisation – all the links, lines and hierarchies – and replace them with something more distributed, a system of nodes and networks. Since I got out of jail, though, I’ve realised how much the hyperlink has been devalued, almost made obsolete.
Nearly every social network now treats a link as just the same as it treats any other object – the same as a photo, or a piece of text. You’re encouraged to post one single hyperlink and expose it to a quasi-democratic process of liking and plussing and hearting. But links are not objects, they are relations between objects. This objectivisation has stripped hyperlinks of their immense powers.
At the same time, these social networks tend to treat native text and pictures – things that are directly posted to them – with a lot more respect. One photographer friend explained to me how the images he uploads directly to Facebook receive many more likes than when he uploads them elsewhere and shares the link on Facebook.
Some networks, like Twitter, treat hyperlinks a little better. Others are far more paranoid. Instagram – owned by Facebook – doesn’t allow its audiences to leave whatsoever. You can put up a web address alongside your photos, but it won’t go anywhere. Lots of people start their daily online routine in these cul-de-sacs of social media, and their journeys end there. Many don’t even realise they are using the internet’s infrastructure when they like an Instagram photograph or leave a comment on a friend’s Facebook video. It’s just an app.
A most brilliant paragraph by some ordinary-looking person can be left outside the stream, while the silly ramblings of a celebrity gain instant internet presence. And not only do the algorithms behind the stream equate newness and popularity with importance, they also tend to show us more of what we have already liked. These services carefully scan our behaviour and delicately tailor our news feeds with posts, pictures and videos that they think we would most likely want to see.
Today the stream is digital media’s dominant form of organising information. It’s in every social network and mobile application.
The centralisation of information also worries me because it makes it easier for things to disappear.
But the scariest outcome of the centralisation of information in the age of social networks is something else: it is making us all much less powerful in relation to governments and corporations. Surveillance is increasingly imposed on civilised lives, and it gets worse as time goes by. The only way to stay outside of this vast apparatus of surveillance might be to go into a cave and sleep, even if you can’t make it 300 years.
Higher Education institutions use course evaluations for a variety of purposes. They factor in retention analysis for adjuncts, tenure approval or rejection for full-time professors, even in salary bonuses and raises. But, are the results of course evaluations an objective measure of high quality scholarship in the classroom?
Badges are a mechanism to award ‘micro-credits’ online. They are awarded by an organization for an individual user, and can be either internal to a website or online community, or use open standards and shared repositories.
In open online learning settings, badges are used to provide incentives for individuals to use our resources and to participate in discussion threads.
The IBM skills gateway is an example of how open badges can be leveraged to document professional development. EDUCAUSE microcredentialing program offers 108 digital badges in five categories (community service, expertise development, presentation and facilitation, leadership development, awards).
Open Badge Initiative and “Digital Badges for Lifelong Learning” became the theme of the fourth Digital Meaning & Learning competition, in which over 30 innovative badge systems and 10 research studies received over $5 million in funding between 2012 and 2013.
Standardization is the key to creating transferability and recognition across contexts
badges awarded for participation are valued less meaningful than skill-based badges. For skill-based badges, evidence of mastery must be associated with the badge along with the evaluation criteria. Having a clear purpose, ensuring transferability, and specifying learning objectives were noted by the interviewees as the top priorities when implementing badge offerings in higher education contexts.
Sheryl Grant is a senior researcher on user experience at OpenWorks Group, a company that focuses on supporting educational web applications and mobile tools, including credentialing services. Prior to her current position, Dr. Grant was Director of Alternative Credentialing and Badge Research at HASTAC. She was part of the team that organized the ‘Badges for Lifelong Learning Competition’.
advice o offer for the design and implementation of digital badges. She stressed that badge systems need to be designed in a participatory manner together with the target audience who is supposed to receive them. This will allow for fair, realistic and transparent criteria. Another crucial aspect is the assessment portion: Who will make verify that the badge credentials are issued correctly? While badges can offer additional motivation, they can also diminish motivation and create a ‘race to the bottom’ if they are obtained too easily. Specifically, Dr. Grant advised to use badges to reward exceptional activities, and acknowledge students who want to go above and beyond. She also gave guidelines on when to avoid issuing badges, i.e., activities that are already graded and activities that are required.
All current UNC badging pilots used the platform cred.ly for issuing badges. An alternative is the Mozilla Open Badge backpack follow-up Badgr. The European platform Badgecraft is another repository with a fairly broad user base. The badge wiki project offers a comprehensive list with implementation details for each platform: Badge Platforms (Badge Wiki). (23 platforms)
Designing Effective Digital Badges (https://www.amazon.com/Designing-Effective-Digital-Badges-Applications/dp/1138306134) is a hands-on guide to the principles, implementation, and assessment of digital badging systems. Informed by the fundamental concepts and research-based characteristics of effective badge design, this book uses real-world examples to convey the advantages and challenges of badging and showcases its application across a variety of contexts.
An interactive discussion on the Innovating Pedagogy 2019 report from The Open University
About the Guest
Rebecca is a senior lecturer in the Institute of Educational Technology (IET) at The Open University in the UK and a senior fellow of the Higher Education Academy. Her primary research interests are educational futures, and how people learn together online and I supervise doctoral students in both these areas.
Rebecca worked for several years as a researcher and educator on the Schome project, which focuses on educational futures, and was also the research lead on the SocialLearn online learning platform, and learning analytics lead on the Open Science Lab (Outstanding ICT Initiative of the Year: THE Awards 2014). She is currently a pedagogic adviser to the FutureLearn MOOC platform, and evaluation lead on The Open University’s FutureLearn MOOCs. She is an active member of the Society for Learning Analytics Research, and have co-chaired many learning analytics events, included several associated with the Learning Analytics Community Exchange (LACE), European Project funded under Framework 7.
Rebecca’s most recent book, Augmented Education, was published by Palgrave in spring 2014.
Mor, Y., Ferguson, R., & Wasson, B. (2015). Editorial: Learning design, teacher inquiry into student learning and learning analytics: A call for action. British Journal of Educational Technology, 46(2), 221–229. https://doi.org/10.1111/bjet.12273
Hansen, C., Emin, V., Wasson, B., Mor, Y., Rodriguez-Triana, M., Dascalu, M., … Pernin, J. (2013). Towards an Integrated Model of Teacher Inquiry into Student Learning, Learning Design and Learning Analytics. Scaling up Learning for Sustained Impact – Proceedings of EC-TEL 2013, 8095, 605–606. https://doi.org/10.1007/978-3-642-40814-4_73