Hacking big data – there’s never been a better time

Photo by Georgia Moloney

There has never been a better time to work in data analysis. While the term “data science” has been around since the 1960s, it’s only in the last ten years that the job of “data scientist” has emerged as “The sexiest job of the 21st Century” according to the Harvard Business Review. All of our major universities offer data science courses and job finding web sites show hundreds of open positions looking for staff with these skills.

Recent advances in easy to use software tools for data analysis and visualisation have now been joined by widely available, often open source, Machine Learning toolkits. Having humiliated humankind by beating our greatest Go player, Google AlphaGo has now retired but the point is clear – Artificial Intelligence systems are delivering results that seem so futuristic as to be described as “magic”.

The traditional business model of old media, in which classified advertising provided the “rivers of gold” needed to fund the investment in journalism is gone. We’ve witnessed a US political win supported by the spouting of “alternative facts” that have gone unchallenged.

To make a case in such a way that it is difficult to simply reject as “fake news” we need to base our arguments on data. Often that data comes from government and thankfully, here in Australia, we have a strong culture of open government that continues to be supported by our political leaders and senior public servants.

This year’s GovHack event, running from September 7-9, provides an opportunity for contestants to team up and make creative use of available government data. Publicly creating knowledge out of data also encourages government to make more data available in the future and provides feedback on the format and usefulness of existing data.

Past contestants have gone on to commercialise their projects and have taken new skills back to their workplaces. 

Rain Parrot”, a prize winner from 2014 is still going strong and is regularly the top paid weather app in the Apple App Store.

Chamonix, which has won three GovHack awards, has developed Healthi which is used by over 1,000 consumers and health professionals. The idea for Healthi originated when the Australian Government opened a digital portal for third party providers to connect to the My Health Record System.

The Sydney Citizen Journalism Meetup was founded to lobby for better data after Rosie Williams created the first ever implementation of a searchable federal budget inspired by an early GovHack project theopenbudget.org.

More Australian data than ever before will be available this year.

Traditional free tools for data analysis such as R and python’s pandas, SciPy and NumPy are now joined by easily accessible tools for machine learning such as sklearn, Google’s TensorFlow and many others.

Participation in this year’s GovHack can be a launching pad for a highly sought after career, and valuable skills and experience can be taken back to the workplace, or provide key insights about life in Australia.

For more info see http://govhack.org


Written by Peter Marks for GovHack.
Peter Marks is a software developer and technology analyst.
He is a regular contributor to ABC Radio National and blogs at