Course Review: Data Analysis by Jeff Leek

I took Data Analysis on Coursera in the spring 2013. It was offered again in October 2013, but it may have been supplanted by the new Data Science specialisation, which starts in just a couple of days.

tl/dr: 5/5 don't miss out.

Overview:

This is a very well-run course that builds on Roger Peng's Computing for Statistical Analysis (in R) course. Professor Leek does a very good job of introducing tools and habits for 'reproducible research' including suggesting ways to organize the data munging problem for reproducibility. Professors Peng and Leek, both of John's Hopkins School of Medicine, are leaders in this area, and discuss the challenges and progress on their excellent blog, Simply Statistics. They are also at the core of the newly offered Data Science specialization on Coursera, which offers a nice series of courses that build on each other and give good detail in the various aspects of data analysis. It's also by far the most accessible data science study track available.

Reproducible research is very important for statistical science in general, and promises to be increasingly important over time. Consequently, this course is really a great opportunity. Professional programmers eventually learn the value of unit testing, documentation, and other approaches that make their code more useable. Professors Peng and Leek are at the forefront of developing the professional statistical analysis standards that will become the TDD+version control+issue tracking of data analysis. In other words: you must take this course if you are interested in professional level data analysis. Really. Well, actually, you could take all nine courses…

The Data Analysis course is 8 weeks long. In the first offering, it had weekly lectures of about 2 hours as well as weekly homework assignments that could add up to several hours. As usual, the Coursera recommendation of 3-5 hours / week is on the low side for most students with little experience. Additionally, there are two longer, data analysis assignments with real (web based) data. Additionally, the due date for the first analysis is just after many of the better statistical analysis tools are introduced in lecture. Both analyses are peer-graded by 5 other students and the mean of the central 3 scores is used. Peer grading may change in different offerings of the course as this is one area where Jeff Leek was not sure how the course should be organised. There's a full discussion of this in a Simply Statistics podcast. Also, as one might expect from statisticians, there's data! an interactive graph of completion rates for various online programs. This course had a completion rate of 5.4%, and many of the students active on the forums had done a lot of data analysis before.

Recommendations:

Make sure to set aside time for the data analysis assignments. These will easily take 8-10 hours and could take up to 20 depending on how obsessive you become. The data analysis grade is based on a peer-review, so your reviewers may not understand your analysis, even if it is perfectly correct. There was grousing about this on the forums, but I think the point is clear: the peer graders are not experts in data analysis, and you should never write a report aimed at experts in data analysis, so take it into account when you are writing the report. I had trouble with this on the second data analysis. It involved classifying accelerometer data by action performed (walk, run, walk up stairs, etc). The data had been expanded into a set of features including some filtering, Fourier analysis and other transformations. In my analysis, I used the language of signal processing and sampling to describe this feature set, but this language was not familiar to the peer reviewers, so they didn't realise that I was describing the spacing and number of data points very precisely.

How to run a MOOC: a student perspective

Over the course of the past year, I've followed nine MOOC's on Coursera. I've also used other online learning tools including Code Academy and Khan Academy among others. I've enjoyed this period of learning and updated a number of skills, which I hope will be useful in the future.

The MOOC's I've taken have ranged from the wonderfully organised Machine Learning course taught by Coursera founder Andrew Ng to the extravagantly disorganised Startup Engineering course which was primarily led by another Stanford lecturer, statistician, and cofounder of Counsyl, Balaji S. Srinivasan. As more professors become involved in the MOOC phenomenon and try to gain audiences on YouTube and other social media, I thought I'd write up some of my experiences as a student and make some recommendations, or maybe it's a wish-list.

In most MOOCs, the course staff appear to be under-prepared for the demands of the platform. This is a recurring theme, particularly with newly offered classes, so if you are thinking of offering a MOOC, please, please have a beer with someone who has run one and get the full story. It is clearly not an easy thing to do, particularly when the student numbers get large. Coursera courses regularly have > 100,000 students. Jeff Leek's and Roger Peng's post mortem of the course 'Data Analysis' might be a good place to start.

In general, MOOC's tend to have a better student experience when the professor has taught the course material many times before and is not straying too far into new territory. This is particularly true for a first attempt at MOOC's. Bill Howe's course 'Introduction to Data Analysis' was not a particularly good student experience. I think this was because he tried to add too much to what he had taught before. It's better to be focussed and have only 50,000 happy, learning students than to try to do too much and have 120,000 frustrated, failing students. At least to me. You can plan to change an assignment in the second offering to incorporate new technology.

Recommendations:

• Lectures: Have every lecture planned very well ahead of time, preferably before the course even starts. Lectures of 7-10 minutes work well. Some students like them longer, but others don't. Leave yourself plenty of time for technology hiccups -- estimate the time it will take, then multiply x2 and change the units. Hours become days. 
• Resources: If you see 3 -5 similar forum threads running simultaneously, each with > 200 comments of people trying to help each other out, you have failed to get the message across clearly in lecture. A few links to additional introductory material can do wonders.
• Forums: These will be going 24/7. There will always be complaints about the level of the assignment or the language used or something. Some of it will also be interesting discussion that you want to stay on top of - 24/7. Keeping up with the forums can be daunting, so a community TA or two can be useful. Forum organisation is important. As the course progresses, important threads get buried, and useful information is often buried at the bottom of a long thread. It's useful to have a section of the forum for each separate assignment as well as a section for software issues, platform issues, deadline problems and general discussion. A TA who can summarize important points regularly and point up useful posts if very helpful.
• Extra interactions: Some students find local Meetups or study groups to be invaluable. 
• Community TA's: These people are volunteers. Most community TA's appear to be more interested in interacting with the more advanced students and furthering their own learning than in supporting students who are having difficulties. Please review comments made by your community TA's. A few will fall into using snark to glorify themselves. The best ones will highlight useful forum contributions and links to help other students.
• Assignments: Students have different expectations of assignments. One of the main advantages of an online system is instant feedback. I like assignments that contain questions of different difficulty levels. This lets me identify where I could spend more time and also how solid my knowledge is. I use the feedback from incorrect answers, so having two chances is useful, but a bit stressful. Having five chances is better. It's fine if the maximum number of points gained is reduced when more than two chances are used (e.g. automatic 20% reduction at the 3rd submission). This can be invaluable for international students who may have difficulty interpreting the questions. 
• Deadlines: MOOC students are often unable to accommodate deadlines. Reasons vary. For me, I can put in 8-10 hours / week when my kids are in school, but during Fall break week, I might manage 2-3 hours. Other students have occasional work deadlines, or a long-planned vacation. Most MOOC students are managing to set aside a few hours for studying from otherwise busy lives, but those lives occasionally interfere. One useful approach is to have 10 late days that can be applied at any time. This means that if I join the MOOC late and miss the 1st deadline by 2 days, I can use two of my late days. If I have to miss a deadline because much of my weekend was taken up throwing a birthday party for my 5 year old, I can use another one. If the online system we were supposed to use is too overloaded and breaks down, students can apply late days to shift the deadline to a time when the system is less busy (and therefore functioning). This gives flexibility and responsibility to the students, which is really nice. Some teachers disagree, though.
• Timezones: Your students will not only come from all over the world, as in a modern classroom, but will actually be all over the world. This means that you must be aware of time differences. Time zones for deadlines and for release of new lectures makes a difference, but more importantly if you require students to do an online collaboration of some sort, allow them to log in at different times. Some students can find hours in the middle of the day, while others only find them late at night, and those times are staggered all around the globe. Consider grouping time-zone regions so students can choose to participate at a convenient time. (Hint: 2am in China is not convenient.)
• SNAFU's:  Things won't go as planned the first time. It will reflect better on you as a teacher and on your institution if you can adjust as needed. Jeff Leek had to drop a code reproducibility portion of his grading, and Antonio Rangel had to drop an experimental interactive market. Please be flexible in your use of new technology. By all means, try it out, but be aware of student needs, which will vary -- not everyone has an American credit card, which some web services require for registration, even if nothing will be charged. 

It will be interesting to see how MOOC's develop. These new online platforms are effective for learning, and education as we know it is clearly changing. So, as soon as my 10 year old finishes his decimal math on Khan Academy, I'm back to Financial Accounting.