Statistical Illiteracy from the Star

With all the talk of “fake news”, a real concern is what might better be described a “sloppy news” – news which is ostensibly credible, but which falls apart with a modcium of critical thinking.  Consider this piece in the Star “LGBTQ Students Saddled with More Debt, Poll shows”.   The Star reports the results of a recent Forum poll which purports to show that:

 A new poll by Forum Research, believed to be the first Canadian survey on student debt to ask about sexual orientation, shows the LGBTQ+ community is harder hit. Members are more likely to rack up greater student debt, take on a second job to pay it off, and make lifestyle changes because of it.

The conclusion is that LGBTQ students are more heavily burdened by student loans, presumably because of additional barriers that arise as a result of their LGBTQ status.  Interesting, if true.  But is it true? Well, maybe, but the poll doesn’t really support the conclusion.  There are a number of reasons why the data might not support the conclusion that LGBTQ students are more heavily burdened with student loans than their peers as a result of their sexual orientation.

First, according to the poll, LGBTQ students are more likely to pursue a graduate degree than non-LGBTQ students – 21% vs. 16%.  This might explain, for example, why LGBTQ students are more likely to have very high levels of debt. It is not surprising that people who get more education, graduate with more debt – nor is such added debt due to one’s sexual orientation, it’s a function of the cost of more education.  Unfortunately, the poll doesn’t control for level of eduation – so doesn’t tell us if the higher levels of debt for LGBTQ students arises from their sexual orientation or the fact that they get more education.  If, for example, the poll told us that the average student debt of LGBTQ graduates with a graduate degree was higher than the average student debt of LGBTQ graduates with a similar degree, that might be helpful.  But it doesn’t.

Second, the poll isn’t a poll of recent graduates, its a poll of all Canadians.  So it’s comparing the amount of student loans of people who graduated today with people who graduated in 1960.  The problems with this are multiple and obvious.  Tuition was lower in 1960 then it is now (certainly in nominal terms, probably also in real terms) – graduating with $10,000 in student debt in 1960 would be far more onerous then than graduating with $10,000 in student loans now, but the poll presents them as being equally onerous – and far fewer people (and fewer socially disadvantaged people) went to university in 1960 (meaning that fewer people would have had to borrow to go to university).  This sort of comparison of nominal levels of student debt is useless if you’re trying to assess actual student debt burdens, and failing to take into account the changing economic background of students and changing tuition fees, makes it worse).

Moreover, and this is important to the story, older Canadians – i.e., the people who graduated from university in the 1960’s – are less likely to identify as LGBTQ then the current cohort of university graduates.  So part of of the result we’re seeing here is not attributable to LGBTQ graduates bearing a greater burden than non-LGBTQ graduates, but younger graduates – who are more likely to identify as LGBTQ (15% for graduates in the 16-24 year old cohort, vs. 4% in the 65+ cohort) – bearing a greater (nominal, at least) burden than older graduates.  And this shows up in the data – 46% of the graduates who identified as LGBTQ graduated in the 2010s.  For non-LGBTQ graduates, the same numbers is 24%.

So the problem with the poll is it’s an apples-to-oranges comparisons – it’s not comparing the student loan experiences of otherwise similarly situated LGBTQ graduates with non-LGBTQ graduates, it’s comparing the experience of LGBTQ graduates who are more likely to have gone on to graduate education and to have graduated in the last 20 years, with non-LGBTQ graduates who are less likely to have gone on to the graduate education and more likely to have graduated when tuition fees were materially lower, 30 or 40 years ago.   It doesn’t tell us anything, per se, about the impact of LGBTQ status on student loan experiences.

And that’s unfortunate.   It’s an interesting question, one which could probably be answered by a carefully designed survey – perhaps one looking at the current crop of graduates, broken down by level of education.  But instead, we have this sloppy, careless poll, which can be thoroughly discredited with a few minutes of critical thinking.

This is a common practice in modern media, with the Star being no worse an offender than others – dubious polls and studies are routinely reported by credulous journalists and media organizations.   One suspects this is due more to ignorance of statistics and a lack of training about how to think about them – never attribute to malice that which can be explained by incompetence and all that- but, in the era of “fake news”, is it asking too expect respectable newspapers and journalists to think before they print?

This also looks very bad on Forum – there are a host of methodological problems with this study beyond the two discussed above.  No doubt, they’re happy just to get media mentions from credulous reporers, but this sort of study reflects poorly on their competence.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s