Intro to GitHub (for scientists)

The problem

I talked to a young woman yesterday who is a bio-engineering postdoc at Stanford. She has some code that she'd like to 'upload' to GitHub. She admitted that, well, actually, she hadn't managed to get any of her code onto her GitHub account yet, and she looked so overwhelmed and dejected that I felt bad. I know that frustration.
So here is yet another blog post to try and help with the learning curve that is git. I'm writing this for the scientist who has written some code and wants to share it on GitHub. Most scientists write code in order to accomplish a particular task, and thus are not familiar with professional programming practices, including writing documentation, unit testing, SCM (source control management) and version control. There are great benefits to learning these techniques, and their usefulness is becoming more and more apparent to academics as research relies more and more on computer programs. In fact, the table of contents for the journal Nature Methods just arrived in my inbox with a lead editorial on reproducible research. I quote:

Nature Methods strongly encourages researchers to take advantage of the opportunity that code repositories, such as GitHub, provide to improve a software tool before submission. Even if others do not examine and test the code, the act of preparing the code and necessary documentation for deposit and use by others will help avoid delays in publication.

In short, if you are coding and publishing work derived from your code, the process of uploading your code for collaboration will help bring it up to a publishable quality. And yes, it's even more important than making the figures pretty.

GitHub == collaboration

Firstly, if you just want to upload some code, you need to take step back, a deep breath and 'think different'. There is no 'upload' button on GitHub for a good reason. It is not built for uploading code and leaving it to rot on a server, but for fostering collaboration between programmers. Thus, in order to share your code on GitHub, you first need to get it ready for collaboration. To do that, you need to set up version control. This is somewhat more complicated than finding the 'track changes' setting in Microsoft Word, but it is also far more useful.
As you try to think different, keep in mind that GitHub was built with a particular set of workflows in mind. Those workflows have to do with managing a code-base that is constantly being updated by multiple contributors. GitHub tries to ease the difficulties of this type of collaboration by bringing several things together.

• version control -- similar to 'track changes' in a word processor, but for whole projects
• cloud-based storage -- simultaneously backup and share your code
• user accounts -- keep it private, share with a team or make it public as your needs change
• social -- there's messaging, so you can talk, argue, document, discuss.... collaborate!

But version control is the main thing that sets GitHub apart from any number of social sharing platforms, and makes it so powerful for people who code. To use GitHub effectively you therefore need to understand the basics of version control with git. It's hairy and scary at first, but no worse than... well, ok, it is worse than a lot of things, but sometimes a learning curve is the price you pay for really capable tools. So, put a set of bookmarks in your browser, make a cheatsheet and keep it handy. A local cheatsheet with a good searchable title does wonders. If you're thinking 'Oh, I'll just use it once' or 'I'll remember', well, git is for professionals. Are you a professional?

Git == version control

Version control is to 'track changes' for a text document, as Superman is to Tarzan; as Microsoft Word is to TextEdit, as New York City is to Detroit. Firstly, with version control, you are tracking changes to an entire project, not a single document. Changes are tracked line by line with comments and attribution through time as the project grows, changes, and splits; as subroutines develop into full projects of their own; as new owners take control of the code base. It is flexible and thorough and reliable. You should learn to use it.
Use case: You've got some code you developed for your thesis that you want to upload to GitHub. Maybe someone else will find it useful. There are GUI front ends to git, which may help with many tasks, but git was designed to be run from the command line, and this simple use case it not too difficult to master at that level, so let's just go for it.

Download and install git

Walk through the steps at github set-up. Today, they suggest that you download their native app, but note that this only manages part of the workflow. The steps you have to do to get git working still involve the terminal. Be brave. The steps are:

1. get a GitHub account
2. download and install git
3. link your local system to your GitHub account
   
   • tell git your name, email, GitHub login information
   • set up security keys so that GitHub knows you are you

Prepare your codebase

There are a couple of adjustments you probably want to make before releasing your code in the wild. GitHub recommends that all code comes with a License, a Readme and a .gitignore file.

• License If your code comes with a license, it's easier for collaborators to re-use the code and build on it. Specifically, it makes clear what they are allowed to do. It may not be important to you, but your code will be easier to share if you make it clear what the rules are.
• Readme
  You have to explain your work at some point. If you do it in a file named README, GitHub will automatically put it on the front page of the repository. This is very helpful for anyone trying to understand what you did and why. The README can be just a text file. If it is in markdown, perhaps with additional flavoring GitHub will render it with headings and styles, which is much nicer for the reader and not difficult for the writer.
• .gitignore
  Some files don't need to be tracked. For instance, some old Mac directories contain .DS_Store files with directory display information for the Finder app. That doesn't need to be part of the repository. So here's my .gitignore for an old MatLab project:
  

    $cat .gitignore
    .DS_Store 
  

  Pretty simple. You might also add *.log or tmp/ to the .gitignore file, depending on your context. Basically, any files that are automatically generated or updated on compile should not be tracked.

Make a local repository

The project you want to get onto GitHub probably consists of a directory or directory tree containing a series of text files, and possibly some image files or data files. In order for git to track changes to this project, you have to put these files into a repository. This is simple to do once you've prepared it for sharing.
Find your command line. On a Mac, you can use the Terminal app. Navigate to the directory holding your project. If you have never ever used the command line before, this might be challenging. If you want to dive in, you can certainly do so with three little commands: ls, cd, pwd. You can look at the man pages for these commands by typing, for example $man ls, or you could try a crash course in using the command line.
Once you get to your project directory, type:

$git init   

You should see a message from git, something like:

Initialized empty git repository in /Users/suz/programming/octave/OrX/.git/

This initialises an empty repository, which looks like a file named .git. You can check that it's there by typing

$ls -a

Git gives some feedback about what it has done, but I often find it useful to check with

$git status

after each command to see what has happened.
Now we can get the project into the repository. To do this, first type

$git add .   

This prepares git to add your files to the repository, a process known as 'staging'.
Note: The . tells git to stage all the files in the directory tree to the repository. This is very handy when we add files because they don't all have to be specified by name. On the other hand, it isn't ideal, because there are often binary files or .log or even image files that update automatically. You won't want to keep track of those changes. Fortunately, git will automatically look for the .gitignore file we already prepared to get the list of exceptions.

Ready for some commitment? Type:

$git commit -m 'initial commit' 

You should get a full list of the files that git is committing to the repository. Check the status when it's done and you should give a reassuring message:

# On branch master
nothing to commit (working directory clean)

Success! now your project is actually in the repository and git can track any changes to the files. The repository will also keep all the messages that you put with each commit. Always use the -m flag and take the time to add a meaningful message.

Congratulations! Your code is now in a git repository, under version control. You are ready to collaborate.

Share your work

Make a repository on GitHub

1. Log into your GitHub account
2. Create a new repository on GitHub
   On your profile page, in the 'Repositories' tab is a list of repositories that you've contributed to. At the upper right should be a bright green 'New' button.
   
   
3. Follow the directions in the form, adding an informative description so others know what treasure they have found.
   

Congratulations! You have a GitHub repository to share your code from!

Link the repositories

In git terminology, the current state of your code is the head of a branch. By default, the initial commit is called the 'master' branch. You can make other local branches, and probably should to try out new features. You can also make remote branches. At this point, your new GitHub repository is essentially an empty remote branch. By custom, this branch will be referred to as 'origin'. To point git to it, type (on one line), with your appropriate username and project title:

$git remote add origin https://github.com/username/project.git

This command translates roughly as "Dear git; Please be so kind as to add a connection to a remote repository. I will be referring to the remote repository as 'origin' in our future correspondence. It can be found at https://...... Thank you for your kind assistance in this matter. Sincerely, yours truly, esq."

Upload your code

Ok, ok, there is no upload on GitHub, but it is payoff time. Once you have a local repository linked to a remote repository, you can just push the code from one to the other.

$ git push -u origin master

Translation: "Dear git; Please push the recent changes I committed to my local repository, known as master, into the remote repository known as origin. Also, please be aware that I'd like this to be the default remote repository, sometimes referred to as 'upstream'. Thank you again for your kind assistance. I am forever in your debt. Sincerely, Yours truly, esq. and, etc."

Success!!! You have now successfully pushed your code to GitHub.

Or at least I hope you were successful. If not, if you've tried to follow this post and the directions at GitHub and you still feel lost, there is more help out there. Many universities are running Software Carpentry bootcamps to help students and faculty develop more professional programming skills. The skills taught aim to improve software collaboration and impart the skills needed to carry out reproducible research. Two key tools they teach are version control with git and collaboration via GitHub.
Live long and collaborate!

NHS hack day thoughts

What a glorious weekend! The sun finally came out, and while my family was out enjoying the countryside and biking to the beach, I spent it in central London at the NHS Hackday.

Not that I'm complaining. I had a great time. There were some really talented people there, and some very committed parents, doctors, programmers and just plain technically minded people. It was a really interesting weekend.

The hackers present were a pleasant mix of odd-ball grad student types, young doctors, programmers, developers and random 'IT' people, all with an interest in trying to contribute something to the efficiency, ability and smooth running of the NHS. The elephant in the room is whether any of these projects will ever become useful. Several applications had very useful ideas. The winning app 'Waitless' aimed to provide an SMS service in which people could send an SMS and get information about the distance to local NHS services and the likely waiting time once they get there. This way, someone with an earache could make an informed choice to go to their local walk-in clinic instead of the A&E department depending on wait times, opening hours, and distance.

The progress made on these apps over the course of the weekend was astounding, with several nearly becoming usable services, and certainly good proof of principle demos by the time of presentation at 3pm on Sunday. It is truly amazing what a team of 8 can get done in a weekend with modern developer tools, available APIs, open source software, and online services. Wow.

And me? well, not so much...
I ended up working on the aptly named FAIL project: 'Fatal Accident Inquiry Learning', which attempted to apply machine learning techniques to Scottish Fatal Accident Inquiry (FAI) Reports. Unfortunately, I spent most of the first day struggling to get nltk and various supporting technologies installed on my system and most of the second day learning the very basics of working with these technologies.  Carl had somewhat more success in getting some snippets of the reports into Carrot2, but the results were less than impressive.

Challenges:
1) I need more experience with Python
Everything I know about Python, I learned from Code Academy and from my homework for 'Intro to Data Science' on Coursera. The homework was a pretty good introduction for this project, as it involved sentiment analysis of a tweet stream. I was able to do some basic filtering and use the structure of the base homework code. There were some difficulties in translating the learning from the homework analyses of short tweets to these much larger, richer records, and I struggled to create a workflow between parts of the analysis.

2) I need more time with natural language processing
The FAIs are long legal text documents. It is possible to extract text from them, but it isn't easy. The text has some consistent elements, but is not in a consistent form. Some dates are 'day month year' format, while others are given as 'month day, year' and still others are 'day of the month of year'. This makes it somewhat difficult to even extract basic information such as the age of the victim. It should be possible to get this by using a grammar with NLTK, but well, I didn't manage to come to grips with it in 45 minutes I gave to it. Perhaps not surprising. Similarly with bi-grams, tri-grams, collocations and point mutation importance (PMI) to collect information on phrases and unusual words. I learned a lot, but wasn't able to put much of it into use. Yet.

...hopefully I'll find some time to try out topic modelling on this data at some point.

3) Relevance of the project
Ideally, we'd like to analyse these reports and make some inferences that are relevant for the NHS and that could lead to improvements in quality of care. Unfortunately, the data set we are looking at is not like hospital episode statistics -- it is not a statistic. Although there were some 1652 fatal accidents in Scotland in 2011, only 28 FAI reports were published that year. Our dataset consists of the 82 such published inquiry results from the last few years. Some inquiries are published long after the incident, but this indicates that inquiries are held for less than 2% of fatal accidents.

Inquiries can be called whenever there are unusual circumstances. They are required in some circumstances, such as when a death occurs in custody. By definition, then, these accidents are the outliers. Some of them are candidates for the 'Darwin Awards': tragedies begot by stupidity. Others are simply tragedies.

The Scottish authorities hold these inquiries with an eye toward preventing further accidents, and such investigations do have impact on our daily lives. Protocols for how often the highway lines are repainted, police guidelines for how people in custody are transported, and yes, even those ubiquitous labels: this is not a toy; not for children under 3 yrs of age; do not play on or around. FAIs are the fault-checking analyses that lead to health and safety advice.

So we tried various approaches to extracting information and comparing text in the reports, but ultimately we did not come up with a truly compelling use case for the data or inferences from it.

Observations on the data:

• Accident statistics are interesting reading.
• Each of these accidents is a story of its own. 
• Men are in more fatal accidents than women. For all ages, nearly 65% of the accident victims are men.  For men < 65 years of age, the ratio climbs to nearly 75%. 
• Fatal accidents are more common in older people. In 2011, 57% of the male accident victims were over 65 yo. At the same time, among female victims, 76% were over 65 yo. 
• In younger age groups, poisoning is the most common cause of accidental fatalities. In 2011, there were no poisonings in children < 15 yo. The statistics include alcohol poisoning.
• Falls are the most common fatal accident type for people over 65 years, and over 60% of the victims are women, showing a sharp contrast with all other accident types and ages. 

Observations on the hack day:

• don't forget your coffee cup
• wander around and see what different groups are doing  -- don't wait until the pub afterwards to find the person with a degree in computational linguistics!
• what's your goal? if it's social, be social. If it's coding, join a group
• you will probably learn more from a larger team with more varied skills
• how competitive do you want to be? 
• a video of a good use case is impressive
• it's often more efficient to ask for help

 

Big Data Hackathon London: A few lessons learned

I spent most of my weekend at the Big Data Hackathon London. I'm not hard-core, and I didn't pull an all nighter, but then, even the winning visualisation team said the code written between 4-7 am was rubbish. Better to get some sleep. This was my first ever 'hackathon', and part of the fun was just observing the phenomenon.

The basics: 

The hackathon was organised by Data Science London, and I found out about it through the Data Science Meetup group. I highly recommend this group if you are interested in learning about current methods and tools in Data Science. Their meetings are very interesting and educational, but you have to be very quick with the RSVP -- there's a lot of competition for the limited spaces. As always, the organisers did a great job. I didn't manage to take home any of the swag or awards, but I certainly drank my share of the coffee. And the whole weekend was totally free. Well done!

The hackathon took place at The Hub Westminster, which was a very nice, light, open space. The talk space holds about 100 or so people, and there is desk space and stand-up area where the food is served for milling around and meeting people. The space is well organized with good systems for internet and power. A pleasure to work in.

• Lesson learned: Bring your own mug to cut down on waste 

The hackathon had three different categories of challenges that teams could submit.

1. data science challenge 
2. data visualization challenge 
3. free-style data challenge 

Most people who came did not have a team lined up. The winning team in the data visualisation challenge got together when two of them carried signs around saying 'Node.js' and 'd3'. The other two thought this was a good idea, and a winning team was formed. One of the team members later said that their goal had just been to improve their javascript skills. The visualisation was quite lovely, and should be showing up in a 'major UK publication' someday soon.

• Lesson learned: MongoDB + Node.js + d3 = powerful stuff 
• Lesson learned: Connect a team through the technology you want to learn 

The hackathon started out with a presentation on Microsoft Azure and the suggestion that we use a free trial account (good for 3 months) to do our analysis.

After the talk, someone asked about setting up R on the system. I approached them after the talk, and that was the beginnings of a team. Our team, 'State of the A[R]t' set up an Azure account, and we were able to get R working on a Ubuntu virtual machine without much difficulty. Wenming Ye's blog was helpful for this. It's probably even more helpful if you want to use Python. The Kaggle assessment of the data science submissions relied on the ROCR package, and this relied on gplot, which required us to build R 3.0 from code. Fortunately, one of our team was ace at this and we had it running quite quickly. Meanwhile, the rest of us were looking at the data.

• Lesson learned: Technology is broad and deep. Someone will like doing the parts you hate. Let them do it. (I have to re-learn this continuously. I try to do too much on my own.) 

The hackathon has a tight schedule. There were talks all afternoon, and if I had gone to all the talks, I would not have made much progress with the data. However, missing all the talks probably wasn't the best strategy either. Next time, I'll try to keep my head up and look around for which talks are truly interesting. Talks were presented by the hackathon sponsors, so highlighted their newest technologies. I can only hope that the talks will be posted so I can catch up with the parts I missed.

• Lesson learned: It's about learning. Think about what are the learning opportunities today? Will the talks be available tomorrow? 

On Sunday, the data analysis winners each gave a brief indication of what they did. We ended up 89th overall, and we only did that well because one of my team-mates took a careful look at the original benchmark code. I don't think we were alone in this, as 15 teams finished within 0.00001 of us. Re-assuringly, though, we were working along similar lines to the winning team.

• Lesson learned: Find a good starting place. 

The benchmark was not quite the simple logistic regression we expected from the description. We would have done much better if we had taken the time to look at the code for the benchmark as a first step.

• Lesson learned: Work efficiently -- write functions or scripts for each step. 

At 12:32 on Sunday, I had a model that resembled the winning model. Maybe it would have done better than 89th, but I didn't get a chance to find out. It took me too long to make the model into a submittable prediction! I should have anticipated this, because the 1st submission also took ages. If I had written some of the steps into functions, it could have been much faster, and the team would have done better.

Overall, it was great fun. I met some lovely people and I learned a lot. Coursera's offerings, including Jeff Leek's 'Data Analysis' and Roger Peng's 'Computing for Data Analysis' gave me a good background for taking part in this event. Hopefully, learning some Network Analysis and a bit of Scala will prove useful for the next one.