Center for Digital Education (CDE)
real-time impact on curriculum structure, instruction delivery and student learning, permitting change and improvement. It can also provide insight into important trends that affect present and future resource needs.
Big Data: Traditionally described as high-volume, high-velocity and high-variety information.
Learning or Data Analytics: The measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.
Educational Data Mining: The techniques, tools and research designed for automatically extracting meaning from large repositories of data generated by or related to people’s learning activities in educational settings.
Predictive Analytics: Algorithms that help analysts predict behavior or events based on data.
Predictive Modeling: The process of creating, testing and validating a model to best predict the probability of an outcome.
Data analytics, or the measurement, collection, analysis and reporting of data, is driving decisionmaking in many institutions. However, because of the unique nature of each district’s or college’s data needs, many are building their own solutions.
For example, in 2014 the nonprofit company inBloom, Inc., backed by $100 million from the Gates Foundation and the Carnegie Foundation for the Advancement of Teaching, closed its doors amid controversy regarding its plan to store, clean and aggregate a range of student information for states and districts and then make the data available to district-approved third parties to develop tools and dashboards so the data could be used by classroom educators.22
Tips for Student Data Privacy
Know the Laws and Regulations
There are many regulations on the books intended to protect student privacy and safety: the Family Educational Rights and Privacy Act (FERPA), the Protection of Pupil Rights Amendment (PPRA), the Children’s Internet Protection Act (CIPA), the Children’s Online Privacy Protection Act (COPPA) and the Health Insurance Portability and Accountability Act (HIPAA)
— as well as state, district and community laws. Because technology changes so rapidly, it is unlikely laws and regulations will keep pace with new data protection needs. Establish a committee to ascertain your institution’s level of understanding of and compliance with these laws, along with additional safeguard measures.
Make a Checklist Your institution’s privacy policies should cover security, user safety, communications, social media, access, identification rules, and intrusion detection and prevention.
Communicate, Communicate, Communicate
Students, staff, faculty and parents all need to know their rights and responsibilities regarding data privacy. Convey your technology plans, policies and requirements and then assess and re-communicate those throughout each year.
“Anything-as-a-Service” or “X-as-a-Service” solutions can help K-12 and higher education institutions cope with big data by offering storage, analytics capabilities and more. These include:
• Infrastructure-as-a-Service (IaaS): Providers offer cloud-based storage, similar to a campus storage area network (SAN)
• Platform-as-a-Service (PaaS): Opens up application platforms — as opposed to the applications themselves — so others can build their own applications
using underlying operating systems, data models and databases; pre-built application components and interfaces
• Software-as-a-Service (SaaS): The hosting of applications in the cloud
• Big-Data-as-a-Service (BDaaS): Mix all the above together, upscale the amount of data involved by an enormous amount and you’ve got BDaaS
Use accurate data correctly
Define goals and develop metrics
Eliminate silos, integrate data
Remember, intelligence is the goal
Maintain a robust, supportive enterprise infrastructure.
Prioritize student privacy
Develop bullet-proof data governance guidelines
Create a culture of collaboration and sharing, not compliance.
more on big data in this IMS blog:
Big Data and Privacy
April 17, 2014
Big data has been generating big hype for a while. In January, the White House jumped into the fray, launching a big data and privacy review. CDT participated in all three public workshops convened in connection with the review and submitted written comments.
CDT’s Big Data and Privacy Comments
In our comments, we focused on three main areas: applying the Fair Information Practice Principles (FIPPs) to both private sector and government big data programs; exploring technical measures such as de-identification to safeguard privacy; and reforming existing privacy laws, most notably the Electronic Communications Privacy Act, to account for rapid changes in the ways that digital data is collected, stored, and used.
CDT stressed that, as entities collect more data to offer innovative products and more efficient services, they must still be guided by purpose specification, consent, security, and the other elements of the FIPPs framework.
Government and Big Data
|“Strong consensus is forming that the bulk collection of phone records should end.”
|The Administration says that it will end its bulk collection of telephony metadata, although the details of what will replace it remain unsettled. Meanwhile, CDT is pointing out that the laws the government has used to justify bulk collection are not limited just to phone records. Instead, they could be used to justify collection of location data, Internet browsing history, financial records, and more. CDT has been vocal in advocating the end of all forms of bulk collection, and we endorse the USA FREEDOM Act as the best legislation to do just that.
A report from the White House review is due before the end of April, but it is expected to present more questions than answers. In this complex and unsettled space, CDT will continue to work with companies and other stakeholders to develop workable approaches that will protect privacy while pursuing the benefits promised by advanced data analytics.
Check Out CDT’s New Website
CDT has launched a totally revamped website: http://www.cdt.org. It has a fresh new look and tools that should make our content more easily accessible. Thanks to our partners at iStrategy Labs for their creative and technical efforts on the new site.
Thursday, April 11, 11AM-1PM, Miller Center B-37
We invite the campus community to a presentation by three vendors of Classroom Response System (CRS), AKA “clickers”:
11:00-11:30AM Poll Everywhere, Mr. Alec Nuñez
11:30-12:00PM iClikers, Mr. Jeff Howard
12:00-12:30PM Top Hat Monocle Mr. Steve Popovich
12:30-1PM Turning Technologies Mr. Jordan Ferns
links to documentation from the vendors:
Top Hat Monocle docs:
|Questions to vendor: email@example.com
- 1. Is your system proprietary as far as the handheld device and the operating system software?
The site and the service are the property of Poll Everywhere. We do not provide handheld devices. Participants use their own device be it a smart phone, cell phone, laptop, tablet, etc.
- 2. Describe the scalability of your system, from small classes (20-30) to large auditorium classes. (500+).
Poll Everywhere is used daily by thousands of users. Audience sizes upwards of 500+ are not uncommon. We’ve been used for events with 30,000 simultaneous participants in the past.
- 3. Is your system receiver/transmitter based, wi-fi based, or other?
- 4. What is the usual process for students to register a “CRS”(or other device) for a course? List all of the possible ways a student could register their device. Could a campus offer this service rather than through your system? If so, how?
Student participants may register by filling out a form. Or, student information can be uploaded via a CSV.
- 5. Once a “CRS” is purchased can it be used for as long as the student is enrolled in classes? Could “CRS” purchases be made available through the campus bookstore? Once a student purchases a “clicker” are they able to transfer ownership when finished with it?
N/A. Poll Everywhere sells service licenses the length and number of students supported would be outlined in a services agreement.
- 6. Will your operating software integrate with other standard database formats? If so, list which ones.
Need more information to answer.
- 7. Describe the support levels you provide. If you offer maintenance agreements, describe what is covered.
8am to 8pm EST native English speaking phone support and email support.
- 8. What is your company’s history in providing this type of technology? Provide a list of higher education clients.
Company pioneered and invented the use of this technology for audience and classroom response. http://en.wikipedia.org/wiki/Poll_Everywhere. University of Notre Dame
South Bend, Indiana
University of North Carolina-Chapel Hill
Raleigh, North Carolina
University of Southern California
Los Angeles, California
San Diego State University
San Diego, California
King’s College London
London, United Kingdom
Fayetteville State University
Fayetteville, North Carolina
New Brunswick, New Jersey
Texas A&M University
College Station, Texas
University of Illinois
- 9. What measures does your company take to insure student data privacy? Is your system in compliance with FERPA and the Minnesota Data Practices Act? (https://www.revisor.leg.state.mn.us/statutes/?id=13&view=chapter)
- 10. What personal data does your company collect on students and for what purpose? Is it shared or sold to others? How is it protected?
Name. Phone Number. Email. For the purposes of voting and identification (Graded quizzes, attendance, polls, etc.). It is never shared or sold to others.
- 11. Do any of your business partners collect personal information about students that use your technology?
- 12. With what formats can test/quiz questions be imported/exported?
Import via text. Export via CSV.
- 13. List compatible operating systems (e.g., Windows, Macintosh, Palm, Android)?
Works via standard web technology including Safari, Chrome, Firefox, and Internet Explorer. Participant web voting fully supported on Android and IOS devices. Text message participation supported via both shortcode and longcode formats.
- 14. What are the total costs to students including device costs and periodic or one-time operation costs
Depends on negotiated service level agreement. We offer a student pays model at $14 per year or Institutional Licensing.
- 15. Describe your costs to the institution.
Depends on negotiated service level agreement. We offer a student pays model at $14 per year or Institutional Licensing.
- 16. Describe how your software integrates with PowerPoint or other presentation systems.
Downloadable slides from the website for Windows PowerPoint and downloadable app for PowerPoint and Keynote integration on a Mac.
|17. State your level of integration with Desire2Learn (D2L)?Does the integration require a server or other additional equipment the campus must purchase?Export results from site via CSV for import into D2L.
- 17. How does your company address disability accommodation for your product?
We follow the latest web standards best practices to make our website widely accessible by all. To make sure we live up to this, we test our website in a text-based browser called Lynx that makes sure we’re structuring our content correctly for screen readers and other assisted technologies.
- 18. Does your software limit the number of answers per question in tests or quizzes? If so, what is the max question limit?
- 19. Does your software provide for integrating multimedia files? If so, list the file format types supported.
Supports image formats (.PNG, .GIF, .JPG).
- 20. What has been your historic schedule for software releases and what pricing mechanism do you make available to your clients for upgrading?
We ship new code daily. New features are released several times a year depending on when we finish them. New features are released to the website for use by all subscribers.
- 21. Describe your “CRS”(s).
Poll Everywhere is a web based classroom response system that allows students to participate from their existing devices. No expensive hardware “clickers” are required. More information can be found at http://www.polleverywhere.com/classroom-response-system.
- 22. If applicable, what is the average life span of a battery in your device and what battery type does it take?
N/A. Battery manufacturers hate us. Thirty percent of their annual profits can be contributed to their use in clickers (we made that up).
- 23. Does your system automatically save upon shutdown?
Our is a “cloud based” system. User data is stored there even when your computer is not on.
- 24. What is your company’s projection/vision for this technology in the near and far term.
We want to take clicker companies out of business. We think it’s ridiculous to charge students and institutions a premium for outdated technology when existing devices and standard web technology can be used instead for less than a tenth of the price.
- 25. Does any of your software/apps require administrator permission to install?
- 26. If your system is radio frequency based, what frequency spectrum does it operate in? If the system operate in the 2.4-2.5 ghz. spectrum, have you tested to insure that smart phones, wireless tablet’s and laptops and 2.4 ghz. wireless phones do not affect your system? If so, what are the results of those tests?
- 27. What impact to the wireless network does the solution have?
Depends on a variety of factors. Most university wireless networks are capable of supporting Poll Everywhere. Poll Everywhere can also make use of cell phone carrier infrastructure through SMS and data networks on the students phones.
- 28. Can the audience response system be used spontaneously for polling?
- 29. Can quiz questions and response distributions be imported and exported from and to plaintext or a portable format? (motivated by assessment & accreditation requirements).
- 30. Is there a requirement that a portion of the course grade be based on the audience response system?
Fall 2011 Student Response System Pilot
NDSU has been standardized on a single student response (i.e., “clicker”) system for over a decade, with the intent to provide a reliable system for students and faculty that can be effectively and efficiently supported by ITS. In April 2011, Instructional Services made the decision to explore other response options and to identify a suitable replacement product for the previously used e-Instruction Personal Response System (PRS). At the time, PRS was laden with technical problems that rendered the system ineffective and unsupportable. That system also had a steep learning curve, was difficult to navigate, and was unnecessarily time-consuming to use. In fact, many universities across the U.S. experienced similar problems with PRS and have since then adopted alternative systems.
A pilot to explore alternative response systems was initiated at NDSU in fall 2011. The pilot was aimed at further investigating two systems—Turning Technologies and iClicker—in realistic classroom environments. As part of this pilot program, each company agreed to supply required hardware and software at no cost to faculty or students. Each vendor also visited campus to demonstrate their product to faculty, students and staff.
An open invitation to participate in the pilot was extended to all NDSU faculty on a first come, first serve basis. Of those who indicated interest, 12 were included as participants in this pilot.
Pilot Faculty Participants:
- Angela Hodgson (Biological Sciences)
- Ed Deckard (AES Plant Science)
- Mary Wright (Nursing)
- Larry Peterson (History, Philosophy & Religious Studies)
- Ronald Degges (Statistics)
- Julia Bowsher (Biological Sciences)
- Sanku Mallik (Pharmaceutical Sciences)
- Adnan Akyuz (AES School of Natural Resource Sciences)
- Lonnie Hass (Mathematics)
- Nancy Lilleberg (ITS/Communications)
- Lisa Montplaisir (Biological Sciences)
- Lioudmila Kryjevskaia (Physics)
The pilot included three components: 1) Vendor demonstrations, 2) in-class testing of the two systems, and 3) side-by-side faculty demonstrations of the two systems.
After exploring several systems, Instructional Services narrowed down to two viable options—Turning Technologies and iClicker. Both of these systems met initial criteria that was assembled based on faculty input and previous usage of the existing response system. These criteria included durability, reliability, ease of use, radio frequency transmission, integration with Blackboard LMS, cross-platform compatibility (Mac, PC), stand-alone software (i.e., no longer tied to PowerPoint or other programs), multiple answer formats (including multiple choice, true/false, numeric), potential to migrate to mobile/Web solutions at some point in the future, and cost to students and the university.
In the first stage of the pilot, both vendors were invited to campus to demonstrate their respective technologies. These presentations took place during spring semester 2011 and were attended by faculty, staff and students. The purpose of these presentations was to introduce both systems and provide faculty, staff, and students with an opportunity to take a more hands-on look at the systems and provide their initial feedback.
In the second stage of the pilot, faculty were invited to test the technologies in their classes during fall semester 2011. Both vendors supplied required hardware and software at no cost to faculty and students, and both provided online training to orient faculty to their respective system. Additionally, Instructional Services staff provided follow-up support and training throughout the pilot program. Both vendors were requested to ensure system integration with Blackboard. Both vendors indicated that they would provide the number of clickers necessary to test the systems equally across campus. Both clickers were allocated to courses of varying sizes, ranging from 9 to 400+ students, to test viability in various facilities with differing numbers of users. Participating faculty agreed to offer personal feedback and collect feedback from students regarding experiences with the systems at the end of the pilot.
In the final stage of the pilot, Instructional Services facilitated a side-by-side demonstration led by two faculty members. Each faculty member showcased each product on a function-by-function basis so that attendees were able to easily compare and contrast the two systems. Feedback was collected from attendees.
Results of Pilot
In stage one, we established that both systems were viable and appeared to offer similar features, functions, and were compatible with existing IT systems at NDSU. The determination was made to include both products in a larger classroom trial.
In stage two, we discovered that both systems largely functioned as intended; however, several differences between the technologies in terms of advantages and disadvantages were discovered that influenced our final recommendation. (See Appendix A for a list of these advantages, disadvantages, and potential workarounds.) We also encountered two significant issues that altered the course of the pilot. Initially, it was intended that both systems would be tested in equal number in terms of courses and students. Unfortunately, at the time of the pilot, iClicker was not able to provide more than 675 clickers, which was far fewer than anticipated. Turning Technologies was able to provide 1,395 clickers. As a result, Turning Technologies was used by a larger number of faculty and students across campus.
At the beginning of the pilot, Blackboard integration with iClicker at NDSU was not functional. The iClicker vendor provided troubleshooting assistance immediately, but the problem was not resolved until mid-November. As a result, iClicker users had to use alternative solutions for registering clickers and uploading points to Blackboard for student viewing. Turning Technologies was functional and fully integrated with Blackboard throughout the pilot.
During the span of the pilot additional minor issues were discovered with both systems. A faulty iClicker receiver slightly delayed the effective start date of clicker use in one course. The vendor responded by sending a new receiver, however it was an incorrect model. Instructional Services temporarily exchanged receivers with another member of the pilot group until a functional replacement arrived. Similarly, a Turning Technologies receiver was received with outdated firmware. Turning Technologies support staff identified the problem and assisted in updating the firmware with an update tool located on their website. A faculty participant discovered a software flaw in the iClicker software that hides the software toolbar when disconnecting a laptop from a second monitor. iClicker technical support assisted in identifying the problem and stated the problem would be addressed in a future software update. A workaround was identified that mitigated this problem for the remainder of the pilot. It is important to note that these issues were not widespread and did not widely affect all pilot users, however these issues attest to the need for timely, reliable, and effective vendor support.
Students and faculty reported positive experiences with both technologies throughout the semester. Based on feedback, users of both systems found the new technologies to be much improved over the previous PRS system, indicating that adopting either technology would be perceived as an upgrade among students and faculty. Faculty pilot testers met several times during the semester to discuss their experiences with each system; feedback was sent to each vendor for their comments, suggestions, and solutions.
During the stage three demonstrations, feedback from attendees focused on the inability for iClicker to integrate with Blackboard at that time and the substantial differences between the two systems in terms of entering numeric values (i.e., Turning Technologies has numeric buttons, while iClicker requires the use of a directional key pad to scroll through numeric characters). Feedback indicated that attendees perceived Turning Technologies’ clickers to be much more efficient for submitting numeric responses. Feedback regarding other functionalities indicated relative equality between both systems.
Based on the findings of this pilot, Instructional Services recommends that NDSU IT adopt Turning Technologies as the replacement for the existing PRS system. While both pilot-tested systems are viable solutions, Turning Technologies appears to meet the needs of a larger user base. Additionally, the support offered by Turning Technologies was more timely and effective throughout the pilot. With the limited resources of IT, vendor support is critical and was a major reason for exploring alternative student response technologies.
From Instructional Services’ standpoint, standardizing to one solution is imperative for two major reasons: cost efficiency for students (i.e., preventing students from having to purchase duplicate technologies) and efficient utilization of IT resources (i.e., support and training). It is important to note that this recommendation is based on the opinion of the Instructional Services staff and the majority of pilot testers, but is not based on consensus among all participating faculty and staff. It is possible that individual faculty members may elect to use other options that best meet their individual teaching needs, including (but not limited to) iClicker. As an IT organization, we continue to support technology that serves faculty, student and staff needs across various colleges, disciplines, and courses. We feel that this pilot was effective in determining the student response technology—Turning Technologies—that will best serve NDSU faculty, students and staff for the foreseeable future.
Once a final decision concerning standardization is made, contract negotiations should begin in earnest with the goal of completion by January 1, 2012, in order to accommodate those wishing to use clickers during the spring session.
Appendix A: Clicker Comparisons
Turning Technologies and iClicker
Areas where both products have comparable functionality:
- Setting up the receiver and software
- Student registration of clickers
- Software interface floats above other software
- Can use with anything – PowerPoint, Websites, Word, etc.
- Asking questions on the fly
- Can create questions / answers files
- Managing scores and data
- Allow participation points, points for correct answer, change correct answer
- Reporting – Summary and Detailed
- Uploading scores and data to Blackboard (but there was a big delay with the iClicker product)
- Durability of the receivers and clickers
- Free software
- Offer mobile web device product to go “clickerless”
Areas where the products differ:
Main Shortcomings of Turning Technology Product:
- Costs $5 more – no workaround
- Doesn’t have instructor readout window on receiver base –
- This is a handy function in iClicker that lets the instructor see the %’s of votes as they come in, allowing the instructor to plan how he/she will proceed.
- Workaround: As the time winds down to answer the question, the question and answers are displayed on the screen. Intermittently, the instructor would push a button to mute the projector, push a button to view graph results quickly, then push a button to hide graph and push a button to unmute the projector. In summary, push four buttons quickly each time you want to see the feedback, and the students will see a black screen momentarily.
- Processing multiple sessions when uploading grading –
- Turning Technologies uses their own file structure types, but iClicker uses comma-separated-value text files which work easily with Excel
- Workaround: When uploading grades into Blackboard, upload them one session at a time, and use a calculated total column in Bb to combine them. Ideally, instructors would upload the grades daily or weekly to avoid backlog of sessions.
Main Shortcomings of iClicker Product:
- Entering numeric answers –
- Questions that use numeric answers are widely used in Math and the sciences. Instead of choosing a multiple-choice answer, students solve the problem and enter the actual numeric answer, which can include numbers and symbols.
- Workaround: Students push mode button and use directional pad to scroll up and down through a list of numbers, letters and symbols to choose each character individually from left to right. Then they must submit the answer.
- Number of multiple choice answers –
- iClicker has 5 buttons on the transmitter for direct answer choices and Turning Technologies has 10.
- Workaround: Similar to numeric answer workaround. Once again the simpler transmitter becomes complex for the students.
- Potential Vendor Support Problems –
- It took iClicker over 3 months to get their grade upload interface working with NDSU’s Blackboard system. The Turning Technology interface worked right away. No workaround.