Sunday, 8 October 2017

Requirements for Data Informed Personal, and Personalised eLearning and eTeaching

Requirements for Data Informed Personal, and Personalised eLearning and eTeaching
Poh-Sun Goh
1st draft on 9 October 2017 @0415am (last update on 20 November 2017 @ 0615am)

Using data to guide (e)teaching and (e)learning from Poh-Sun Goh

above from
Goh, P.S. Learning Analytics in Medical Education. MedEdPublish. 2017 Apr; 6(2), Paper No:5. Epub 2017 Apr 4.

above from
quoted from
Goh, P.S. Learning Analytics in Medical Education. MedEdPublish. 2017 Apr; 6(2), Paper No:5. Epub 2017 Apr 4.

"We can only meaningfully measure based on theory" - David Williamson Shaffer
above quote from Slide 48 from 

"We value what we measure, rather than measure what we value" - Jonathan Martin
above quote from Slide 5 from

above from

"...not everything that can be counted counts, and not everything that counts can be counted"
William Bruce Cameron (1963)

"Can we tell from your digital profile if you're learning?"
above quote from
Simon Buckingham Shum

"As learning analytics data provides a snapshot of how engaged students are and how they are performing, this could be considered a useful indication of where excellent teaching is taking place"
above quote from
From Bricks to Clicks - The Potential of Data and Analytics in Higher Education, report by the Higher Education Comission, on 26 January 2016.

Hervatis V, Loe A, Barman L, O'Donoghue J, Zary N A Conceptual Analytics Model for an Outcome-Driven Quality Management Framework as Part of Professional Healthcare Education JMIR Med Educ 2015;1(2):e11 DOI: 10.2196/mededu.4789

Siemens G, Gasevic D, Haythornthwaite C, Dawson S, Shum SB, Ferguson R. Knowl Creat Diffus Util. 2011 Jul 28. Open Learning Analytics: an integrated & modularized platform

The Society for Learning Analytics Research (SoLAR) defines learning analytics as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” (Long, Siemens, Conole, & Gašević, 2011).

Long, P. D., Siemens, G., Conole, G., & Gašević, D. (Eds.). (2011). Proceedings of the 1st International Conference on Learning Analytics and Knowledge (LAK’11). New York, NY, USA: ACM.

"It would be nice if all of the data which sociologists require could be enumerated because then we could run them through IBM machines and draw charts as the economists do. However, not everything that can be counted counts, and not everything that counts can be counted."
William Bruce Cameron in 1963 text “Informal Sociology: A Casual Introduction to Sociological Thinking”

It's not that I'm so smart, it's just that I stay with problems longer.
Albert Einstein
Read more at:

If you are out to describe the truth, leave elegance to the tailor.
Albert Einstein
Read more at:

The monotony and solitude of a quiet life stimulates the creative mind.
Albert Einstein
Read more at:

"Assessment is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time. The adoption of learning analytics too must be informed not only by what can be measured but also by what cannot. There will be limits in what learning analytics can do. In this vein, Siemens and Long have appropriately acknowledged that learning "is messy" and have warned that with learning analytics, "we must guard against drawing conclusions about learning processes based on questionable assumptions that misapply simple models to a complex challenge."5 The message here is important: not every aspect of learning can be captured by the powerful tool that analytics promises to be. Sometimes learning is ineffable! Therefore, multiple methods for assessing learning should be employed, including assessments that function as learning opportunities to support students' deep integration of knowledge, their personal development, and (hopefully!) their transformation over time."
"Assessment is most likely to lead to improvement when it is part of a larger set of conditions that promote change. As this principle states, assessment alone changes very little; likewise, learning analytics cannot act alone in radically disrupting and transforming education. Assessment (when done well) is about the authentic and deep understanding and improvement of teaching and learning. Analytics is about using the power of information technology to see patterns of success (or failure) in learning. Combining the two might actually produce the seeds of transformation—a powerful inquiry into what supports authentic, deep, transformative learning for students."
above quotes from

Ferguson, Rebecca (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6) pp. 304–317.

Educational Data Mining and Learning Analytics Ryan S.J.d. Baker, Teachers College, Columbia University George Siemens, Athabasca University

Ferguson, Rebecca and Buckingham Shum, Simon (2012). Social learning analytics: five approaches. In: 2nd International Conference on Learning Analytics & Knowledge, 29 Apr - 02 May 2012, Vancouver, British Columbia, Canada, pp. 23–33.

Leitner P., Khalil M., Ebner M. (2017) Learning Analytics in Higher Education—A Literature Review. In: Peña-Ayala A. (eds) Learning Analytics: Fundaments, Applications, and Trends. Studies in Systems, Decision and Control, vol 94. Springer, Cham

Atherton, Mirella & Shah, Mahsood & Vazquez, Jenny & Griffiths, Zoe & Jackson, Brian & Burgess, Catherine. (2017). Using learning analytics to assess student engagement and academic outcomes in open access enabling programmes. Open Learning: The Journal of Open, Distance and e-Learning. 1-18. 10.1080/02680513.2017.1309646.

Applying Learning Analytics for Improving Students Engagement and Learning Outcomes in an MOOCS Enabled Collaborative Programming Course Lu, Owen H. T.; Huang, Jeff C. H.; Huang, Anna Y. Q.; Yang, Stephen J. H. Interactive Learning Environments, v25 n2 p220-234 2017

Stewart, Courtney (2017) "Learning Analytics: Shifting from theory to practice.," Journal on Empowering Teaching Excellence: Vol. 1 : Iss. 1 , Article 10. DOI: 10.15142/T3G63W

Saqr M, Fors U, Tedre M. How learning analytics can early predict under-achieving students in a blended medical education course. Med Teach. 2017 Jul;39(7):757-767. doi: 10.1080/0142159X.2017.1309376. Epub 2017 Apr 19. PubMed PMID: 28421894.

Conde M.Á., Hérnandez-García Á., J. García-Peñalvo F., Séin-Echaluce M.L. (2015) Exploring Student Interactions: Learning Analytics Tools for Student Tracking. In: Zaphiris P., Ioannou A. (eds) Learning and Collaboration Technologies. LCT 2015. Lecture Notes in Computer Science, vol 9192. Springer, Cham









How do we start? Sate and directions of learning analyics adoption. / Gasevic, Dragan; Dawson, Shane; Pardo, Abelardo. Oslo, Norway : International Council for Open and Distance Education, 2016.

'...Digital “footprints” (or trace data) about user interactions with technology have been recorded since the very introduction of the Internet and web-based software systems ... ...Over time, the value of such digital traces has been recognized as a promising source of data about student learning ...'
above quote from
Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59 (1), 64–71.

Siemens, G. (2013). Learning Analytics The Emergence of a Discipline. American Behavioral Scientist, 57(10), 1380– 1400.

Siemens, G., Dawson, S., & Lynch, G. (2014). Improving the Quality and Productivity of the Higher Education Sector - Policy and Strategy for Systems-Level Deployment of Learning Analytics. Canberra, Australia: Office of Learning and Teaching, Australian Government.

Rogers, T., Gašević, D., & Dawson, S. (2016). Learning analytics and the imperative for theory driven research. In C. Haythornthwaite, R. Andrews, J. Fransma, & E. Meyers (Eds.), The SAGE Handbook of E-Learning Research, 2nd edition (pp. 232–250). London, UK: SAGE Publications Ltd.

"...The amount of activity and time online for the group of most successful students was mostly below the class average. These learners were interpreted as highly effective with good prior knowledge and strong study skills. The findings of the Kovanović et al. were corroborated in several studies reported by Lust at al. (Lust, Elen, & Clarebout, 2013; Lust, Vandewaetere, Ceulemans, Elen, & Clarebout, 2011).

Kovanovic, V., Gašević, D., Dawson, S., Joksimovic, S., & Baker, R. S. (2016). Does Time-on-task Estimation Matter? Implications on Validity of Learning Analytics Findings. Journal of Learning Analytics, 2(3), 81–110. https://

Kovanović, V., Gašević, D., Dawson, S., Joksimović, S., Baker, R. S., & Hatala, M. (2015). Penetrating the Black Box of Time-on-task Estimation. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 184–193). New York, NY, USA: ACM.

Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89.

"Analytics-based tools designed to construct feedback for students, among other key points, are more effective when they adopt a task-specific language and provide guidance while prompting dialogue between students and instructors (Boud & Molloy, 2013; O’Donovan, Rust, & Price, 2016).

Studies making use of learning analytics methods, by examining trace data to extract learning
strategies followed by students, reveal that students have a high tendency to exhibit performance-oriented behaviors – i.e., focusing on summative assessments deemed to contribute to grades (Lust et al., 2013; Pardo, Jovanović, Dawson, Gašević, & Mirriahi, 2016).

Pardo, A., Jovanović, J., Dawson, & Gašević, D. (2016). Using Learning Analytics to Scale the Provision of Personalised Feedback. Submitted for Publication to British Journal of Educational Technology.

Pardo, A., Jovanović, J., Dawson, S., Gašević, D., & Mirriahi, N. (2016). Exploring student interactions with preparation activities in a flipped learning experience. Submitted for Publication to Computers & Education.

UTA “Big Data Questions”
❖ How will big data and new models provide a more complex understanding of the learner in higher education today?
❖ How can universities use big data to improve student success (retention and successful progress to graduation)?
❖ Can higher education develop new, more multivariate models of student engagement? How might these models drive faculty, staff, and coaches to improve student cognitive and social presence in formal coursework?
❖ How can we better understand learners of diversity and personalize the educational experience for engagement and success?"
above from Slide 12 (presentation below)

Google image search for "visualising learning analytics"

Google image search for "data visualisation process'

Our aim as teachers will always be (is) to encourage, support, facilitate, and augment learning. This includes the provision of learning resources, which may be created or curated (recommended); designing generic and customised learning pathways; and providing timely, and ideally real-time feedback on learning and performance, both in-person, and using technology.

To move and keep up with the trends and preferences of our mobile, technology enabled students, and workforce; who are time-pressed, attention challenged (with multiple competing online / mobile resources and activities, not to mention distractions), who are also very focused on the relevance, usefulness, and impact of learning resources and learning activities (as are faculty); requires teachers to increasingly rely on online, mobile, and interactive (for learning and feedback) platforms. These not only provide access to content and learning pathways (24/7, as required, just-in-time), they facilitate and encourage (ideally) active, and social (interactive) learning; which is increasingly public (online), and documented. Allowing us (as teachers, administrators, and students) to track attendance/participation, access to learning resources, record and review interactive class discussions, record and build on personal notes and reflections on learning (which are shared as part of the group learning experience), document learning outcomes digitally, and online (public, semi-public, private – yet “live” permanently within cloud and networked storage systems and platforms). Our individual learning paths, footprints and trails are potentially, and increasingly recorded digitally, and online; as are our intermediate, and “final” stage-wise learning milestones, summits, and portfolios. As eLearning instructors, we can design these instructional pathways to guide our students. We can also learn from analogies, and experiences from other fields, for example the sales funnel in online marketing. To draw parallels with Miller's pyramid (awareness levels), Kirkpatrick levels, and Bloom's Taxonomy. Using technology to replicate the classroom experience online; to blend online with face to face learning; to flip the classroom.

George E Miller (1919 - 1998)

"Constructive alignment"
(Prof John Biggs, 1999)

"First we get the objectives straight, what the students have to do. Then we decide how to get them to do it. Assessment serves a double purpose: it checks the quality of learning, and for students, it defines what is to be learnt.
Biggs, JB (1999)
What the student does: Teaching for quality learning at university

"At a high level, metrics are simply quantifiable measures of how you’re will people’s lives be improved if your efforts are successful?"
Julie Zhuo in Medium

Jennifer Pei-Ling Tan, Elizabeth Koh. (2017) Situating learning analytics pedagogically: towards an ecological lens. Learning: Research and Practice 3:1, pages 1-11.

Outcome(s) and Competency-based medical education

"Time on task"

It takes time to learn anything worthwhile. To accumulate knowledge and skills. To integrate this new learning, and be able to, and be confident applying this in the workplace, and real life settings.

This is the difference between undertaking a program of training, and formal courses, compared with short symposia and workshops, or an isolated lecture. Formal training programs gives students time, space, and a place to learn. On a regular basis. This promotes a cumulative increase in learning. Combining theoretical learning with practical case studies integrates basic principles with practice points, and promotes transfer of learning from the classroom to the real world. Online learning programs should include elements from traditional classroom practices which facilitate learning. This includes scheduled time to review the learning material, to work on applying what is learnt by working on assessments and assignments, individually and by learning collaboratively with peers; as well as provide opportunities for timely feedback from instructors."
above from
Goh, P.S. A series of reflections on eLearning, traditional and blended learning. MedEdPublish. 2016 Oct; 5(3), Paper No:19. Epub 2016 Oct 14.

"The aim of this paper is to demonstrate that PBL has the potential to prepare students more effectively for future learning because it is based on four modern insights into learning: constructive, self-directed, collaborative and contextual. These four learning principles are described and it is explained how they apply to PBL."
above from
Dolmans DH, De Grave W, Wolfhagen IH, van der Vleuten CP. Problem-based learning: future challenges for educational practice and research. Med Educ. 2005 Jul;39(7):732-41. Review. PubMed PMID: 15960794.

White C, Bradley E, Martindale J, Roy P, Patel K, Yoon M, Worden MK. Why are medical students 'checking out' of active learning in a new curriculum? Med Educ. 2014 Mar;48(3):315-24. doi: 10.1111/medu.12356. PubMed PMID: 24528466.

"A key finding was that the students preferred a variety of different learning formats over an "all or nothing" learning format. Learning format preferences did not necessarily align with perceptions of which format led to better course exam performance. Nearly 70% of respondents wanted to make their own decisions regarding attendance. Candid responses to open-ended survey prompts reflected millennial preferences for choice, flexibility, efficiency, and the ability to control the pace of their learning, providing insight to guide curricular improvements."
Pettit RK, McCoy L, Kinney M. What millennial medical students say about flipped learning. Adv Med Educ Pract. 2017 Jul 20;8:487-497. doi:10.2147/AMEP.S139569. eCollection 2017. PubMed PMID: 28769600; PubMed Central PMCID: PMC5529113.

Students learn (devote time, attention, and effort to) what is assessed. Students will attend class, or log on, if this is mandated to graduate (and obtain their learning certificate, increasingly learning credential verified by Blockchain). Students will participate in physical classrooms, and online; work in groups (both online and offline) as a training requirement. The reason we record class attendance (traditionally and digitally), log and track training hours, take notes, work collaboratively, and submit assignments and projects is that students, and we (as teachers, and educational scholars) know empirically and with evidence (from observations and the literature) that time, participation, engagement, and active, constructive, self-directed, collaborative, contextual and reflective processes anchor (deep) learning, which transfers to practice.

What are the pros and cons of students creating, and leaving behind increasing, and potentially public, and permanent digital footprints, learning trails, and milestones?

For teachers, having access to dynamic, real-time (near real-time) data on content access, and usage; as well as the volume, and intensity of individual, and class engagement with the learning process, as well as visibility of intermediate and learning outcomes (think notes, reflections, questions, discussion threads, assignments, projects, portfolio items) has always been helpful. To allow refinement and customisation of content provision, and the learning / training process. The point of contention, and debate is how much of this information should be made public, and accessible to individual students, cohorts of students (for comparison of performance – rates and milestones for example; for benchmarking, or peer comparative/competitive behaviour awareness purposes); other teachers further along the learning path, administrators, and outside parties (including employers).

How much of cohort data should be anonymized? How long should data be kept? How much, and what data should be available for access, and by whom? What data, in what format, and after what processing, evaluation and judgement should be made accessible, and permanent records made of? It can be argued that though the performance, and training outcomes are what really matter; yet the personal notebooks, training logs, coaching and teaching feedback, peer discussions and feedback records, opinions and judgements of instructors (both formal and informal) are all useful not only to an individual student, but also for an employer in the future. Think letters of reference, and phone calls / emails / digital communications with referees.

We recommend data be collected of participation, engagement, and outcomes. With cohort data anonymized (to allow benchmarking, without overt one to one comparison to mitigate overly competitive behaviour). That personal data be fully accessible, in transparent manner. Individual process data have a shelf-life confined to the duration of a learning program and course; with hours logged, and learning outcomes as digital portfolio artifacts kept as achievement milestones, together with information of potential referees (for the future).

"...Enter, Engaged Time .... the Attention Web, measuring visitor activity by tracking scrolling, mouse movements or keystrokes –  .....know how many people are actively consuming .. content and for how long ... Engaged Time correlates with .... likelihood to return to ... site as well as reading comprehension"
above quote from

"...big data refers to the recurring accumulation of large amounts of data, including personal data, from a variety of sources, which are subject to automatic processing by computer algorithms and advanced data-processing techniques in order to generate certain correlations, trends and patterns (big data analytics)..."
above quote from
DRAFT REPORT on fundamental rights implications of big data: privacy, data protection, non-discrimination, security and law-enforcement (2016/2225(INI))

Mobile, Social, Cloud and Data. We are increasingly spending our time online; consuming, creating, collaborating and sharing; producing digital trails and artefacts; which can be, and is tracked, often without our knowledge, awareness, or even expressed consent. The Pros of the "free" internet, comes often with the Cons of loss of privacy, and risks of misuse of personal information. How can students, educators and administrators take advantage of the reach, and utility of online, networked learning, with data stored on the cloud, while mitigating risks to data and privacy?

Privacy. Data Security. Responsible Access/Use. Awareness. Digital literacy. Thoughtful use. Confidence. Public networks and platforms + "Free" commercial/ad driven networks and platforms vs Private/Institutional Intranets and Learning Management Systems

What are some of the key requirements necessary to take full advantage of an eLearning/Technology enhanced learning platform and process?

How can we take advantage of the flexibility, low cost (often "free"), scale and reach of public networks and platforms (for example Instagram, WhatsApp, WeChat, Facebook, Blogger, LinkedIn), and combine this with the greater privacy, rule based use and governance of private (password, dual-key, biometric secured) and institutional networks, which are still potentially vulnerable to potential data breeches, data misuse, hacking, and data loss/theft?

Do we respond by not participating? or restricted/limited participation? creating silos, both online, and some off-line? Not using digital formats? Going offline?

We believe one sensible, and safe approach is to combine, or blend the use of fully public, semi-private, and private digital platforms and approaches, some off-line, some online.

We can learn from our current cautious, and informed use of digital medical records by our healthcare systems; and translate strategies and approaches to the use of student and trainee records and data (learning and training logs, data streams and artefacts). We can also learn from financial practice and the finance industry, in their use of encryption and blockchain for example, at an institutional and system level.

An aware, informed, careful, layered approach, with public, accessible, anonymised data (or fully consented for public dissemination), and separate, indexing, labelling and captioning, and explanatory information layers, with key information (for privacy, or intellectual property reasons) both stored separately, behind biometric and if necessary multiuser ID require secure access, and if necessary, even offline, or unwritten and unrecorded.

Google search for "big data and privacy concerns"

Google Scholar "big data and privacy concerns"

Google Scholar "big data and privacy concerns" (since 2017)

Broadening marketing’s contribution to data privacy
Ferrell, O.C. J. of the Acad. Mark. Sci. (2017) 45: 160.

Hsinchun Chen , Roger H. L. Chiang , Veda C. Storey, Business intelligence and analytics: from big data to big impact, MIS Quarterly, v.36 n.4, p.1165-1188, December 2012

European parliament adopts resolution on big data (2017)

"...big data refers to the recurring accumulation of large amounts of data, including personal data, from a variety of sources, which are subject to automatic processing by computer algorithms and advanced data-processing techniques in order to generate certain correlations, trends and patterns (big data analytics)..."
above quote from
DRAFT REPORT on fundamental rights implications of big data: privacy, data protection, non-discrimination, security and law-enforcement (2016/2225(INI))

"It's super important to acknowledge that connecting everyone and giving everyone the ability to share is not necessarily always a good thing," Mosseri said during a conversation at the CUNY Graduate School of Journalism. "I believe it will create more good than bad overall."
above quote from

“There's a line between abuse and misinformation, and most of these companies for a while, and including Twitter, were more focused on abuse,” said Ev Williams, cofounder of Twitter and CEO of Medium. “I think the misinformation thing is something that's come up really in the last year much more dramatically.”
above quote from

"At a time when political misinformation is in ready supply, and in demand, “Facebook, Google, and Twitter function as a distribution mechanism, a platform for circulating false information and helping find receptive audiences,” said Brendan Nyhan, a professor of government at Dartmouth College (and occasional contributor to The Times’s Upshot column)... ...
For starters, said Colleen Seifert, a professor of psychology at the University of Michigan, “People have a benevolent view of Facebook, for instance, as a curator, but in fact it does have a motive of its own. What it’s actually doing is keeping your eyes on the site. It’s curating news and information that will keep you watching.”
above quotes from


"Getting information off the Internet is like taking a drink from a fire hydrant"
Mitchell Kapor

Confronting the privacy and ethical risks of Big Data (Financial Times, published on 25 September 2013, accessed on 10 October 2017)

"...making incredibly cool sites that seamlessly connect billions of people to their friends as well as to a global storehouse of knowledge..." and
"If you are not paying for it, you’re not the customer; you’re the product being sold"
quote in article below
Silicon Valley is Not Your Friend (NYTimes, Sunday Magazine, 15 October 2017)

"...Perhaps Google is betting that the popularity of smart speakers has made the public less uneasy about a device that constantly listens to them. And Google says the phone isn't recording anything and all the listening and processing is done on the device, using a downloaded music catalog. But some people will, rightly, still have privacy concerns..."
quote in article below

" restore some sense of balance over our relationship with digital technology, and the best way to do that is with analog: the ying to digital’s yang..."

Sport's best of the best, as are the rest of us, are suffering from digital deterioration courtesy of our irresistible digital devices. "We've found in the last five to six years there's been an overall decline in the visual motor skill level of elite players," she says. "The eyes were never designed to work on small devices and, because of that, we're really abusing how we should be using our eyes.
"When you look at your phone there is limited eye movements happening and everything is pretty static. "When we're on digital devices we really have limited attention span so our ability to concentrate and pay attention to a specific task is deteriorating. "Any system you don't use effectively or abuse deteriorates so we think that every person in the world should be doing some form of eye/brain training to prevent the decline of the system. "Looking up to the furthest point you can focus on helps, and therefore looking up now and again would be a start. However you need to specifically train that system."

"Deep Learning is a new area of Machine Learning research, which has been introduced with the objective of moving Machine Learning closer to one of its original goals: Artificial Intelligence."
above quote from

" the individual level, Singaporeans should acquire, grow and apply skills at a deep level..."
above quote from

Making the Lives of Cybercriminals and Spies Harder Online (NY Times, Oct 11, 2017)

(as of 18 October 2017 @ 1019am)