Introducing BARNABY

A language sentiment model for Australian political news

No — not that Barnaby!

About BARNABY

Barnaby is the Auspolstats Ranking of News Associated Bias for You. Simple right?

For those who are still not following, BARNABY is a machine learning model trained specifically to determine the sentiment of Australian political news headlines. We classify news headlines into one of three categories:

• Positive — The headline frames a positive interpretation of the news.
  e.g. Government shines with new tax cuts to benefit millions.
• Negative — The headline frames a negative interpretation of the news.
  e.g. Government’s new tax cuts are leaving an unfair burden on young Australians.
• Factual — The headline is purely factual.
  e.g. The government announces new tax cuts.

By performing this classification across hundreds of thousands of AusPol news articles we’re able to give you insights into how the media covers Australian politics.

About AusPolStats

AusPolStats is an aggregator of Australian political news stories. Since 2018, we’ve collected the details of hundreds of thousands AusPol news articles. We believe this makes us the largest database of Australian political news.

Each day we tweet out insights from the day’s political news (AusPolStats). You can also deep dive into our database at auspolstats.com.

What about some results?

Here’s some insights from BARNABY looking at how sentiment has varied over time, between parties and between media organisations.

About the Model

BARNABY is a neural network trained using Tensorflow. We use the open-sourced Universal Sentence Encoder from TensorFlow Hub as the embedding layer in our model and then train two dense layers using AusPol headlines from our database which we hand classified.

BARNABY comprises two models. The first classifies a news headline as either factual or opinionated. The second model then classifies an opinionated headline as either positive or negative.

We trained and saved BARNABY to our server which also runs our database. Each day when we find new articles we run BARNABY and add the results to our database of articles.

Model Validation

Prior to developing BARNABY we used Google’s language model to evaluate the sentiment of news articles. By comparing these Google results to BARNABY we have been able to validate our model.

We have found good correlation between the two models, with a general shift towards more negative results when using BARNABY. Digging into the details we believe that BARNABY is performing more accurately and that this negative shift is reflective of the real sentiment of these articles. The graph below illustrates this shift in sentiment between the two models split by news source.

We also did spot checking on articles which flipped from positive to negative or vice versa. We think that the examples below are good illustrations of BARNABY providing better insight than the Google sentiment analysis.

• Aust on cusp of making big virus headway (Google -0.8, BARNABY 0.8)
• Save the Children: Convert US$60 billion of debt into fund to fight Covid-19 in worlds poorest countries (Google -0.8, BARNABY 0.8)
• Four bins might help, but to solve our waste crisis we need a strong market for recycled products (Google -0.8, BARNABY 0.8)
• Virus stimulus package to be unveiled soon (Google -0.8, BARNABY 0.7)
• Morrison has stepped out of Fight Club and into a Liberal PM’s fever dream (Google 0.8, BARNABY -0.98)
• The Morrison Governments Spin Addiction (Google 0.9, BARNABY -0.95)
• WA coronavirus deaths rise to six as Artania patient and overseas traveller die in Perth hospitals (Google 0.7, BARNABY -0.86)
• 70 complaints against real estate agents on early release super: ASIC (Google 0.8, BARNABY — 0.87)
• Big parts of the Great Barrier Reef are dying (Google 0.8, BARNABY -0.93)
• Great in a crisis, Australia cannot lose it in recovery (Google 0.7, BARNABY -0.94)
• Angus Taylor can’t resist a deal, even amid a global pandemic (Google 0.7, BARNABY -0.92)
• Economic collapse faster than Great Depression (Google 0.6, BARNABY -0.67)

Want to Collaborate with Us?

We’re always open to collaborating with new people and sharing our data. If you’re interested hit us up on Twitter or at michael@dimension.media