While focusing on Coursera for my artefact, I observed inclusions/exclusions, tendencies to privilege “tech” courses from Western institutions, and experienced the complexities of researching algorithmic systems (Kitchin 2017). Does Silicon Valley-based Coursera – with its vision of life transformation for all – subscribe to Silicon Valley’s notion of ‘progress’ while blind to its exclusions? Is it another example of an attempt to ‘revolutionise’ education, as Williamson (2017) details, framed by neoliberal and commercial agendas yet presented as an objective and universal ‘learning experience’?
I reflect on the ‘black box’ metaphor, yet ‘algorithmic systems are not standalone little boxes, but massive, networked ones with hundreds of hands reaching into them’ (Seaver 2013: 10) and we ‘need to examine the work that is done by those modelling and coding [them]’ (Beer 2017: 10). Thus, rather than stripping individual algorithms of their wider social and political context, we should ‘unpack the full socio-technical assemblage’ (Kitchin 2017: 25) and examine the complex ‘human-machinic cognitive relations’ (Amoore 2019: 7), ‘entanglement of agencies’ (Knox 2015) and the implications for education in an era of ‘datafication’ (Knox et al. 2020).
‘These algorithmic systems are not standalone little boxes, but massive, networked ones with hundreds of hands reaching into them, tweaking and tuning, swapping out parts and experimenting with new arrangements…we need to examine the logic that guides the hands…’ (Seaver 2013: 10)
Early on this block, I played with a range of different algorithms, looking at the recommendations served up by my existing activity. I have created a “recommendations feed” for my lifestream, and also made general notes and screenshots in number of posts.
Later on, and in thinking specifically about digital education, and tying the activity with our MOOC exploration, I first looked at my FutureLearn recommendations, and finally, focused on Coursera – where I was able to provide false data for my “profile” and “learning plan” and see the alterations in recommended courses.
My artefact: Algorithmic systems: entanglements of human-machinic relations
You can see the Coursera ‘algorithmic play’ posts in a lifestream feed, and also through this CodePen (which brings in the same feed but presents, ranks and orders it slightly differently). How might consuming the feeds through different ranking systems affect your perception, and how might the pre-amble surrounding it have changed your conclusions and what you click on?
You can also access the “code behind” the CodePen. However, even though access is granted to this code, it will not necessarily be easy to make sense of it (Kitchin 2017: 20; Seaver 2013). While presented as ‘open’ source, is it really ‘open’ and who for? What exclusions might this create?
The CodePen (while slightly misleadingly called an “Educational Algorithm”) is a very basic attempt to present the feed of Coursera recommendations in a different fashion for comparison, while provoking some thought around algorithmic systems in general. There is no complex processing, just a little reordering. It also does not store any data (information entered is used within the browser to rank articles, but not stored in an external database), and the text descriptions are fictional – it is just for visual/demonstration purposes.
Reflections on the CodePen
Inspired by Kitchin (2017: 23), I have written a few words reflecting on my experiences, while acknowledging the limitations of the subjectiveness of this approach.
The CodePen was adapted (“forked“) from an existing CodePen, which provided the majority of the base upon which (with my very limited skills!) I could tweak very slightly to add a basic feed of the Coursera recommendations by pasting in another bit of code (which turned out to be old and broken) and then ultimately another bit of example code (this is from a Google-“led” project). It is very much a case of different bits of code “hacked” together, full of little “bugs” and “glitches” and not particularly well designed or written! I was keen to strictly limit the time spent on it, although I know much time could be spent tidying and refining it.
Presumably, similar time constraints, albeit with more resources/testing, affect development of, say, Facebook/Google etc. algorithms and lead to mistakes. After all, Facebook’s internal motto used to be ‘move fast and break things’, although this race to create a “minimum viable product” and “disrupt” (regardless of the consequences) is increasingly criticised.
In any case, this CodePen is like a snapshot of an experimental “work in progress” (which others are welcome to “fork”, use and adapt) and brings to mind the messy, ontogenetic, emergent, and non-static nature of algorithmic systems (Kitchin 2017: 20-21).
Ultimately, my aim is to raise some questions about some details of this process and, since most of the code was “stuck together” from bits of free and ‘open’ source code, how the culture of the individuals/teams is significant too. As Seaver (2013: 10) puts it…
‘…when our object of interest is the algorithmic system, “cultural” details are technical details — the tendencies of an engineering team are as significant as the tendencies of a sorting algorithm…’
I am experimenting with inputting false information about myself in Coursera, in order to see the difference in algorithmic recommendations. Here is how I described myself…
False data provided to Coursera
… and here are some recommendations provided after entering the above data…
Recommendations provided by Coursera
The top listed courses are exclusively technology-based and “offered by” Google, and appear to have no direct connection to my listed industry “Health and Medicine”…
While my explorations were very limited here, in some ways this seems fairly consistent with my experiences of using certain (but not all) MOOC or educational course/video sites (and even more general “apps”). As soon as you step outside of the area of computer science, the range of courses is more limited, despite the sites themselves being presented as general educational sites. In looking to change my “learning plan” options (which change your profile and recommendations) revealed the “default” or “suggested” text, presented before you enter your own profile options:
Setting your Coursera “learning plan”
You can see the results of my profile/”learning plan” alterations here. However, at this stage of deciding my profile options, the “software engineer” who works in “tech” seems to be the “default” starting point here. This is all perhaps no surprise given that Coursera was set up by Stanford computer scientists; as often seems the way, the developers build something for themselves (ensuring a seamless user experience for their own circumstances) and then only later branch out.
One example outside of education here is the online bank Monzo, whose early customer base was ‘95% male, 100% iPhone-owning, and highly concentrated in London’ (Guardian 2019). This description mirrors the co-founder Tom Blomfield, as he himself admits:
‘Our early customer was male, they lived in London, they were 31 years old, they had an iPhone and worked in technology. They were me. I’ve just described myself. Which has huge advantages, right? It’s very easy to know what I want.’ (The Finanser 2017)
While Monzo does claim to have a focus on social inclusion (This is Money 2019), why is this always seemingly secondary to building the app, gaining users (similar to themselves) and getting investors on board? Should social inclusion, whereby apps are designed for all users in a democratic fashion where everyone has a say, not be inherent in the very beginning planning, design and development processes? There may be a place here for considering platform cooperativism, inclusive codesign and participatory design approaches here (see Beck 2002; Scholz and Schneider 2016; West-Puckett et al. 2018).
Coming back to education, if Coursera have taken a similar approach as Monzo to designing their platform and building up their catalogue of courses, it is perhaps concerning that who do not mirror the designers and developers may be left excluded and on the margins.
‘The importance of inclusive codesign has been one of the central insights for us. Codesign is the opposite of masculine Silicon Valley “waterfall model of software design,” which means that you build a platform and then reach out to potential users. We follow a more feminine approach to building platforms where the people who are meant to populate the platform are part of building it from the very first day. We also design for outliers: disabled people and other people on the margins who don’t fit into the cookie-cutter notions of software design of Silicon Valley.’
Trebor Scholz (P2P Foundation 2017)
‘…it is worth reflecting on what one means by ‘self learning’ in the context of algorithms. As algorithms such as deep neural nets and random forests become deployed in border controls, in one sense they do self-learn because they are exposed to a corpus of data (for example on past travel) from which they generate clusters of shared attributes. When people say that these algorithms ‘detect patterns’, this is what they mean really – that the algorithms group the data according to the presence or absence of particular features in the data.
Where we do need to be careful with the idea of ‘self learning’, though, is that this is in no sense fully autonomous. The learning involves many other interactions, for example with the humans who select or label the training data from which the algorithms learn, with others who move the threshold of the sensitivity of the algorithm (recalibrating false positives and false negatives at the border), and indeed interactions with other algorithms such as biometric models.’
The field of AI is extremely broad and somewhere in there appears Machine Learning (ML), the automated learning of machines. This system is widely used these days and it basically consists of computers learning. In other words, these machines’ performance improves with experience.
‘Do algorithms compute beyond the threshold of human perceptibility and consciousness? Can ‘cognizing’and ‘learning’ digital devices reflect or engage the durational experience of time? Do digital forms of cognition radically transform workings of the human brain and what humans can perceive or decide? How do algorithms act upon other algorithms, and how might we understand their recursive learning from each other? What kind of sociality or associative life emerges from the human-machinic cognitive relations that we see with association rules and analytics?’
…and, as I explore these ‘human-machinic cognitive relations’, look beyond the polished “app” user interfaces and reflect on how algorithms (despite how they are presented) are far from objective or neutral (Kitchin 2017: 18). I turn my attention to investigating discrimination and bias (Noble 2018 and #AlgorithmsForHer)…
I also investigate the notion of ‘data colonialism’ (Knox 2016: 14), rethink the relation between algorithms and power (Beer 2017) and look to the future of what this might all mean in an educational context (Knox et al. 2020; Williamson 2017).
Neither Karam nor Odili know for sure what role automated decision-making played in their visa rejections, but in both cases computerised errors played a part and a decision seemed to be made primarily based on nationality. This level of service from the Home Office doesn’t come cheap.
I’ve taken screenshots of some further play with Google autocomplete below… are existing views and biases being reproduced in the suggested terms, or is it more complex than that (see Noble 2018)?
I wonder what the results might be if I was to search these terms from a different location (see Kitchin (2017: 21) and Mahnke and Uprichard (2014))?
‘Why is detroit…’ Google autocomplete‘Why is england…’ Google autocomplete‘Why is britain…’ Google autocomplete
A revealing look at how negative biases against women of color are embedded in search engine results and algorithms
Run a Google search for “black girls”—what will you find? “Big Booty” and other sexually explicit terms are likely to come up as top search terms. But, if you type in “white girls,” the results are radically different. The suggested porn sites and un-moderated discussions about “why black women are so sassy” or “why black women are so angry” presents a disturbing portrait of black womanhood in modern society.
In Algorithms of Oppression, Safiya Umoja Noble challenges the idea that search engines like Google offer an equal playing field for all forms of ideas, identities, and activities. Data discrimination is a real social problem; Noble argues that the combination of private interests in promoting certain sites, along with the monopoly status of a relatively small number of Internet search engines, leads to a biased set of search algorithms that privilege whiteness and discriminate against people of color, specifically women of color.
Through an analysis of textual and media searches as well as extensive research on paid online advertising, Noble exposes a culture of racism and sexism in the way discoverability is created online. As search engines and their related companies grow in importance—operating as a source for email, a major vehicle for primary and secondary school learning, and beyond—understanding and reversing these disquieting trends and discriminatory practices is of utmost importance.
An original, surprising and, at times, disturbing account of bias on the internet, Algorithms of Oppression contributes to our understanding of how racism is created, maintained, and disseminated in the 21st century.
Machine learning systems are profoundly influenced by the methods of data collections and labelling that are used in their creation. Yet there has been a lack of research into the processes of how training data is constructed and used.
