Online class on Statistical Learning

Trevor Hastie and Robert Tibshirani are teaching an online class on Statistical Learning starting this week.

The first week is introduction and overview, so it's not too late to join up.

They've also published a new book, An Introduction to Statistical Learning, as a more accessible companion to their widely revered The Elements of Statistical Learning. Like it's older sibling, the new book is availabe for free download as a PDF.

The class overlaps a bit with Andrew Ng's Machine Learning class, but I'm looking forward to a different perspective, new material on penalized regression, resampling methods and non-linear fitting and random forest, and more practice.

The Statistical Learning class is taught with examples in R, which is great.

Amir Sadoughi is starting a community driven solution guide to the exercises.

If you prefer Python, some folks at Boston startup DataRobot is planning to follow the class with a series of blog posts that show how "statistical learning techniques presented in the course can be applied using tools from the Python ecosystem: “numpy”, “scipy”, “pandas”, “matplotlib”, “scikit-learn”, and “statsmodels”". Awesome!

Those interested may also like Yaser Abu-Mostafa's MOOC Learning from Data which ran "live" last year but is now available in "take at your own pace" mode. I haven't taken it, but have heard glowing recommendations. Students for that course produced a truely impressive solutions guide with code in R, Python, Octave, Haskell and several other languages.

For those who want and have a budget for the in-person experience Hastie and Tibshirani are teaching a 2 day seminar in Palo Alto on March 20-21.

Scaling higher education

In the fall of last year, over 100,000 students signed up for Andrew Ng's Machine Learning Class and more than 12,000 of them completed the course. Sebastian Thrun and Peter Norvig taught Artificial Intelligence with similarly impressive numbers.

I was one of the thousands in the Machine Learning class. I had so much fun with that, I also took Daphne Koller's Probabilistic Graphical Models. That one was a quite a bit harder, covering some fairly advanced stuff at least for my few remaining brain cells. But, I finished! For the PGM class, 6702 took the first quiz and 1441 took the final - pretty good retention for such a ball-buster of a class.

This spring, at least two new companies offering online courses were founded. Andrew Ng and Daphne Koller founded Coursera. Partnering with professors from Princeton, Penn, University of Michigan, and Berkeley, they've broadened their course catalog from a base in computer science to include classes in history, mathematics and even poetry. Sebastian Thrun, founder of Udacity, calls his approach University 2.0 and speaks of "democratizing higher eduction" and "empowering students" especially in the developing world where access to higher education is more limited. Almost three quarters of Coursera's students are outside the US, in countries like Brazil, Britain, India and Russia.

The classes are surprisingly fun. The formula boils down to two key elements:

• short segments
• interaction

In the mold of Khan academy, the lectures are broken into short segments of 10 to 15 minutes, which fit nicely into busy schedules. Short quizzes test the student's understanding. The courses have social aspect, as well. Online forums provide a place for questions and a sense of camaraderie while struggling through difficult concepts. Meetups and study groups have sprung up in several cities across the world.

The programming exercises are where the real fun begins. Students write code that implements the crux of an operation, filling in the blanks in provided boilerplate code. Grading works a bit like unit testing. Progressing through the assignment by getting tests to pass gives gratifyingly immediate feedback. Completing an assignment results in working code for handwriting recognition, spam classification, image processing or recognizing an action from kinect position sensing data.

Thomas Friedman says, Let the revolution come:

Welcome to the college education revolution. Big breakthroughs happen when what is suddenly possible meets what is desperately necessary.

With the cost of tuition rising, and public funding falling, the timing might be right for some disruptive innovation in higher education. And, the skills on offer are in high demand. One proposed business model is to offer classes for free and charge employers for access to the data.

Refactoring the university classroom to function at internet scale meshes with the building momentum behind open access journals that some are calling an Academic Spring as well as with citizen science projects like Galaxy Zoo.

Increasing openness in academics, in both teaching and research, can only reduce friction in the process of transferring technology from the lab to production and may help engage the public with science. Look for lots of interesting developments in the next few years, as technology knocks a new door into the ivory tower.

More

• Harvard and M.I.T. Team Up to Offer Free Online Courses
• Inside Higher Ed asks Who Takes MOOCs?
• From Silicon Valley, A New Approach To Education from NPR
• Solving College With Big Data
• Can Free Online Courses Transform the Higher Education Industry? Published: June 20, 2012 in Knowledge@Wharton
• Opening Ivy League Universities To The Masses
• It'll be interesting to see if (when) they come out with a writing class, whether they can effectively create machine assisted methods for learning to write.
• Twilight of the Lecture Eric Mazur on new interactive teaching techniques. The trend toward “active learning” may overthrow the style of teaching that has ruled universities for 600 years.

Updates

Coursera continues to generate lots of news, signing up 12 new universities, including the University of Washington (yay, UW!), attracting the attention of Bill Gate, and being described as The Single Most Important Experiment in Higher Education by the Atlantic and The Beginning of the End for Traditional Higher Education by Fortune and Reshaping Education on the Web by the NYT.

• The Trouble With Online Education
• The Revolution: Top Ten Disruptors of Education
• I'm stoked about Functional Programming in Scala taught by Martin Odersky
• The Simply Statistics folks on Why we are teaching massive open online courses (MOOCs) in R/statistics for Coursera
• The Siege of Academe: For years, Silicon Valley has failed to breach the walls of higher education with disruptive technology. But the tide of battle is changing. A report from the front lines.

Stanford Machine Learning class

Stanford is offering a free online version of it's Machine Learning class taught by Andrew Ng. Study groups are popping up everywhere. Cool!

The class officially starts Monday, October 10th, but the first few lectures are up already, broken into bite size pieces of 10 minutes or so. What I've seen so far is at a basic level, covering a course introduction and terminology. Ng then posses a linear regression problem.

We want to find a line y = ϴ₀ + ϴ₁ x such that we minimize the squared error between our line and our data points.

The solution is our first learning algorithm, gradient descent.

More about the Machine Learning class

The real class at Stanford is: CS229. Exercises are to be done in Octave. Recommended reading includes the usual suspects:

• Pattern Recognition and Machine Learning, Christopher Bishop
• Machine Learning, Tom Mitchell
• The Elements of Statistical Learning, Hastie, Tibshirani and Friedman

Several of the Primers in Computational Biology series would probably make for good supplementary material.

There are threads related to the class on Quora and Reddit, for whatever that's worth. Also, see some good resources for learning about machine learning.

Leroy Hood on a career in science

Leroy Hood, the founder of the Institute for Systems Biology, where I've worked for 3 years now, wrote up some career advice for scientists last year. It probably applies fairly well to any professional.

I leave students (and even some of my colleagues) with several pieces of advice. First, I stress the importance of a good cross-disciplinary education. Ideally, I suggest a double major with the two fields being orthogonal-say, biology with computer science or applied physics. Some argue that there is insufficient time to learn two fields deeply at the undergraduate level.

I argue that this is not true. If we realize that many undergraduate courses now taught are filled with details that are immediately forgotten after the course is finished, we must then learn to teach in an efficiently conceptual manner. As I noted above, as an undergraduate at Caltech I had Feynman for physics and Pauling for chemistry, and both provided striking examples of the power of conceptual teaching.

Second, I argue that students should grow accustomed to working together in teams: In the future, there will be many hard problems (like P4 medicine) that will require the focused integration of many different types of expertise.

Third, I suggest that students acquire an excellent background in mathematics and statistics and develop the ability to use various computational tools. Fourth, I argue that a scholar, academic, scientist, or engineer should have four major professional objectives: (a) scholarship, (b) education (teaching), (c) transferring knowledge to society, and (d ) playing a leadership role in the local community to help it become the place in which one would like one’s children and grandchildren to live.

Fifth, with regard to the scientific careers of many scientists-they can be described as bellshaped curves of success-they rise gradually to a career maximum and then slowly fall back toward the base line. To circumvent this fate, I propose a simple solution: a major change in career focus every 10 or so years. By learning a new field and overcoming the attendant insecurities that come from learning new areas, one can reset the career clock. Moreover, with a different point of view and prior experience, one can make fundamental new contributions to the new field by thinking outside the box. Then the new career curve can be a joined series of the upsides of the bellshaped curve, each reinvigorated by the ten-year changes.

Finally, science is all about being surrounded by wonderful colleagues and having fun with them, so I recommend choosing one’s science, environment, and colleagues carefully. I end this discussion with what I stressed at the beginning-I am so fortunate to have been surrounded by outstanding colleagues who loved science and engineering. Science for each of us is a journey with no fixed end goal. Rather, our goals are continually being redefined.

ISB recently topped 300 employees and, as of early 2009, had a budget of $55 million. Dr. Hood turned 70 in 2008.

• Hamming on you and your research

Functional Programming

I've harbored a secret desire to learn Haskell for a few years now. Simon Peyton-Jones is one of the key people behind Haskell. His web site at MSR has tons of papers, a tutorial on concurrent programming in Haskell, and a video lecture of A taste of Haskell. There's also a Simon Peyton-Jones podcast at SE-Radio.

What is Haskell

Haskell is a programming language that is

• purely functional
• lazy
• higher order
• strongly typed
• general purpose

Why should I care?

Functional programming will make you think differently about programming

• Mainstream languages are all about state
• Functional programming is all about values

Whether or not you drink the Haskell Kool-Aid, you'll be a better programmer in whatever language you regularly use

I should read a Haskell book or two, and, in related functional goodness, I keep reading how great Practical Common Lisp is. I also need to fulfill my quest to finish SICP. I've read the first three chapters twice, doing the examples once in Scheme and again in OCAML. I've read chapter 4 on interpreters. I need to work through the examples in that chapter and take in the final fifth chapter.

Bioinformatics as a Queryable Knowledge Map: the Pygr Project

Pygr is a hypergraph database in Python with applications in bioinformatics written by Christopher Lee, a faculty member at UCLA. There's a 30 minute video of talk about Pygr and a bunch of other resources on the Lee Lab website and Lee's thinking bioinformatics blog.

Thesis: Hypergraphs are a general model for bioinformatics and Python’s core models are already a good model of Bioinformatics Data

• Sequence: protein and nucleic acid sequences 
• Mapping / Graphs: alignment, annotation 
• Attributes: schema, i.e. relations between data 
• Namespace (import): the ontology of all bioinformatics data 

Pygr aims to show that these Pythonic patterns are a general and scalable solution for bioinformatics.

The general idea is not entirely different from the data types behind Gaggle, especially in the emphasis on basic data structures without a heavy semantic component.

Dr. Lee is also writing a textbook on probabilistic inference.

Lecture videos online

You can become a scholar these days just by sitting in front of your computer watching video lectures.

• Structure and Interpretation of Computer Programs: Videos of Abelson and Sussman's classic lectures
• Donald Knuth lectures online
• ResearchChannel
• Ars Digita University
• UW's Professional Master's Program in Computer Science lecture videos.

A blog called FreeScienceOnline has a better course catalog than most universities.

• UC Berkeley's youtube collection
• MIT's public events videos and Open Courseware
• Videolectures.net lots of computer science, machine learning, and bioinformatics
• SE-Radio Software Engineering Radio - the podcast for professional developers
• parleys.com geek oriented video lectures

Updates

• IEEE new media
• Hidden Video Courses in Math, Science, and Engineering from the data wrangling blog
• Must-see TED talks
• A Brief History of Mathematics
• MIT's Calculus Revisited

A couple discussions about computer science lecture videos on stack exchange.

More Updates

Life-hacker has a post on how to Get a free college education online.