Week nine and algorithmic systems: unpacking the socio-technical assemblage

Photo by Federico Beccari on Unsplash
Algorithmic systems as ‘massive, networked [boxes] with hundreds of hands reaching into them’ (Seaver: 2013: 10)? (PhotoFederico BeccariUnsplash.)
Having just published my artefact Algorithmic systems: entanglements of human-machinic relations, I have started commenting on others’ artefacts, finding insightful (but sobering) explorations of the algorithmic systems of FacebookNetflix and others. Facebook’s foray into ‘personalized learning platforms’ and the resistance against it is very relevant as I reflect on the issues of surveillance and automation in education (Williamson 2019).

While focusing on Coursera for my artefact, I observed inclusions/exclusions, tendencies to privilege “tech” courses from Western institutions, and experienced the complexities of researching algorithmic systems (Kitchin 2017). Does Silicon Valley-based Coursera – with its vision of life transformation for all – subscribe to Silicon Valley’s notion of ‘progress’ while blind to its exclusions? Is it another example of an attempt to ‘revolutionise’ education, as Williamson (2017) details, framed by neoliberal and commercial agendas yet presented as an objective and universal ‘learning experience’?

I reflect on the ‘black box’ metaphor, yet ‘algorithmic systems are not standalone little boxes, but massive, networked ones with hundreds of hands reaching into them’ (Seaver 2013: 10) and we ‘need to examine the work that is done by those modelling and coding [them]’ (Beer 2017: 10). Thus, rather than stripping individual algorithms of their wider social and political context, we should ‘unpack the full socio-technical assemblage’ (Kitchin 2017: 25) and examine the complex ‘human-machinic cognitive relations’ (Amoore 2019: 7), ‘entanglement of agencies’ (Knox 2015) and the implications for education in an era of ‘datafication’ (Knox et al. 2020).

View references

‘Algorithmic play’ artefact – ‘Algorithmic systems: entanglements of human-machinic relations’

‘These algorithmic systems are not standalone little boxes, but massive, networked ones with hundreds of hands reaching into them, tweaking and tuning, swapping out parts and experimenting with new arrangements…we need to examine the logic that guides the hands…’ (Seaver 2013: 10)

Early on this block, I played with a range of different algorithms, looking at the recommendations served up by my existing activity. I have created a “recommendations feed” for my lifestream, and also made general notes and screenshots in number of posts.

Later on, and in thinking specifically about digital education, and tying the activity with our MOOC exploration, I first looked at my FutureLearn recommendations, and finally, focused on Coursera – where I was able to provide false data for my “profile” and “learning plan” and see the alterations in recommended courses.

Some initial conclusions

Many of the “recommendation engines” I played with, such as SoundCloud, Spotify and Twitter, led me into what I perceived to be a “you loop“, “filter bubble” or “echo chamber”. Google’s autocomplete showed some signs of reproducing existing biases, perhaps amplifying dominant views held by other users. Google may also be making assumptions based on data they hold about me, which may have skewed YouTube search results, although it would be interesting to compare results with other users or in different locations, an approach Kitchin (2017: 21) discusses. I have written a little about ethical concerns and my Google data also.

Moving my focus to Coursera, in its recommendations they appeared to privilege courses with a information technology/computer science focus, although the range of available courses is clearly a factor too. In any case, the founders’ background in computer science, and education at a Western university, shone through despite attempts to tweak my “profile” and “learning plan” (I have written a little about the “profile” ethical issues also). This appears to be a common theme in “apps” or websites” developed by Western companies, and the design processes utilised (whereby products are developed first for the “profile” of the founder(s), and secondarily for others) arguably creates exclusions for some (and a strong sense of “this wasn’t designed for me”) and inclusions for others (notably switching my profile to “software engineer” produced a wealth of “relevant“ results which I could tailor to my choosing).

My artefact: Algorithmic systems: entanglements of human-machinic relations

You can see the Coursera ‘algorithmic play’ posts in a lifestream feed, and also through this CodePen (which brings in the same feed but presents, ranks and orders it slightly differently). How might consuming the feeds through different ranking systems affect your perception, and how might the pre-amble surrounding it have changed your conclusions and what you click on?

View the CodePen

‘Algorithmic systems: entanglements of human-machinic relations’
‘Algorithmic systems: entanglements of human-machinic relations’

You can also access the “code behind” the CodePen. However, even though access is granted to this code, it will not necessarily be easy to make sense of it (Kitchin 2017: 20; Seaver 2013). While presented as ‘open’ source, is it really ‘open’ and who for? What exclusions might this create?

The CodePen (while slightly misleadingly called an “Educational Algorithm”) is a very basic attempt to present the feed of Coursera recommendations in a different fashion for comparison, while provoking some thought around algorithmic systems in general. There is no complex processing, just a little reordering. It also does not store any data (information entered is used within the browser to rank articles, but not stored in an external database), and the text descriptions are fictional – it is just for visual/demonstration purposes.

Reflections on the CodePen

Inspired by Kitchin (2017: 23), I have written a few words reflecting on my experiences, while acknowledging the limitations of the subjectiveness of this approach.

The CodePen was adapted (“forked“) from an existing CodePen, which provided the majority of the base upon which (with my very limited skills!) I could tweak very slightly to add a basic feed of the Coursera recommendations by pasting in another bit of code (which turned out to be old and broken) and then ultimately another bit of example code (this is from a Google-“led” project). It is very much a case of different bits of code “hacked” together, full of little “bugs” and “glitches” and not particularly well designed or written! I was keen to strictly limit the time spent on it, although I know much time could be spent tidying and refining it.

Presumably, similar time constraints, albeit with more resources/testing, affect development of, say, Facebook/Google etc. algorithms and lead to mistakes. After all, Facebook’s internal motto used to be ‘move fast and break things’, although this race to create a “minimum viable product” and “disrupt” (regardless of the consequences) is increasingly criticised.

In any case, this CodePen is like a snapshot of an experimental “work in progress” (which others are welcome to “fork”, use and adapt) and brings to mind the messy, ontogenetic, emergent, and non-static nature of algorithmic systems (Kitchin 2017: 20-21).

Ultimately, my aim is to raise some questions about some details of this process and, since most of the code was “stuck together” from bits of free and ‘open’ source code, how the culture of the individuals/teams is significant too. As Seaver (2013: 10) puts it…

‘…when our object of interest is the algorithmic system, “cultural” details are technical details — the tendencies of an engineering team are as significant as the tendencies of a sorting algorithm…’

…and, given that a large portion of the code came from a Google-led project (and Twitter too), how might the tendencies of those teams have created inclusions/exclusions? Furthermore, as the focus of my artefact is Coursera, whose founders both have experience working at Stanford University and Google, how might the tendencies there have guided Coursera and, subsequently, my CodePen?

Finally, given that Coursera presents itself as a universal educational space, where its vision is ‘a world where anyone, anywhere can transform their life by accessing the world’s best learning experience’, what are the implications of this for digital education? In my initial explorations, my perceptions are that computer science and information technology disciplines from Western universities are prioritised by Coursera, in general and through their algorithmic systems. However, further research is needed to, as Kitchin (2017: 25) puts it, ‘unpack the full socio-technical assemblage’.

View references

Coursera recommendations (based on false data) – what inclusions and exclusions are apparent?

I am experimenting with inputting false information about myself in Coursera, in order to see the difference in algorithmic recommendations. Here is how I described myself…

False data provided to Coursera
False data provided to Coursera

… and here are some recommendations provided after entering the above data…

Recommendations provided by Coursera
Recommendations provided by Coursera

The top listed courses are exclusively technology-based and “offered by” Google, and appear to have no direct connection to my listed industry “Health and Medicine”…

While my explorations were very limited here, in some ways this seems fairly consistent with my experiences of using certain (but not all) MOOC or educational course/video sites (and even more general “apps”). As soon as you step outside of the area of computer science, the range of courses is more limited, despite the sites themselves being presented as general educational sites. In looking to change my “learning plan” options (which change your profile and recommendations) revealed the “default” or “suggested” text, presented before you enter your own profile options:

Setting your Coursera "learning plan"
Setting your Coursera “learning plan”

You can see the results of my profile/”learning plan” alterations here. However, at this stage of deciding my profile options, the “software engineer” who works in “tech” seems to be the “default” starting point here. This is all perhaps no surprise given that Coursera was set up by Stanford computer scientists; as often seems the way, the developers build something for themselves (ensuring a seamless user experience for their own circumstances) and then only later branch out.

One example outside of education here is the online bank Monzo, whose early customer base was ‘95% male, 100% iPhone-owning, and highly concentrated in London’ (Guardian 2019). This description mirrors the co-founder Tom Blomfield, as he himself admits:

‘Our early customer was male, they lived in London, they were 31 years old, they had an iPhone and worked in technology. They were me. I’ve just described myself. Which has huge advantages, right? It’s very easy to know what I want.’ (The Finanser 2017)

While Monzo does claim to have a focus on social inclusion (This is Money 2019), why is this always seemingly secondary to building the app, gaining users (similar to themselves) and getting investors on board? Should social inclusion, whereby apps are designed for all users in a democratic fashion where everyone has a say, not be inherent in the very beginning planning, design and development processes? There may be a place here for considering platform cooperativism, inclusive codesign and participatory design approaches here (see Beck 2002; Scholz and Schneider 2016; West-Puckett et al. 2018).

Coming back to education, if Coursera have taken a similar approach as Monzo to designing their platform and building up their catalogue of courses, it is perhaps concerning that who do not mirror the designers and developers may be left excluded and on the margins.

Conversely, an inclusive codesign approach may have produced different results. As Trebor Scholz (P2P Foundation 2017) explains:

‘The importance of inclusive codesign has been one of the central insights for us. Codesign is the opposite of masculine Silicon Valley “waterfall model of software design,” which means that you build a platform and then reach out to potential users. We follow a more feminine approach to building platforms where the people who are meant to populate the platform are part of building it from the very first day. We also design for outliers: disabled people and other people on the margins who don’t fit into the cookie-cutter notions of software design of Silicon Valley.’
Trebor Scholz (P2P Foundation 2017)

Michael saved in Pocket: ‘The politics of artificial intelligence: an interview with Louise Amoore’ (Open Democracy, Louise Amoore and Krystian Woznicki, 2018)

Excerpt (Louise Amoore)

‘…it is worth reflecting on what one means by ‘self learning’ in the context of algorithms. As algorithms such as deep neural nets and random forests become deployed in border controls, in one sense they do self-learn because they are exposed to a corpus of data (for example on past travel) from which they generate clusters of shared attributes. When people say that these algorithms ‘detect patterns’, this is what they mean really – that the algorithms group the data according to the presence or absence of particular features in the data.

Where we do need to be careful with the idea of ‘self learning’, though, is that this is in no sense fully autonomous. The learning involves many other interactions, for example with the humans who select or label the training data from which the algorithms learn, with others who move the threshold of the sensitivity of the algorithm (recalibrating false positives and false negatives at the border), and indeed interactions with other algorithms such as biometric models.’

View full article

Week eight: critically playing with algorithms

This week, while commenting on micro-ethnographies, I began the ‘algorithmic play’ activity, adding new lifestream feeds including Vimeo and Deezer and, inspired by ‘Show and Tell: Algorithmic Culture’ (Sandvig 2014) and Noble’s (2018) Algorithms of Oppression, played with Google search algorithms including their autocomplete

'Is edtech...' Google autocomplete‘Is edtech…’ Google autocomplete

I also discovered (some) of what Google “knows” about me, collected ‘algorithmic play’ notes/screenshots and recorded algorithm ‘recommendations’ from…

I reflect on questions from Amoore (2019: 7)

‘Do algorithms compute beyond the threshold of human perceptibility and consciousness? Can ‘cognizing’and ‘learning’ digital devices reflect or engage the durational experience of time? Do digital forms of cognition radically transform workings of the human brain and what humans can perceive or decide? How do algorithms act upon other algorithms, and how might we understand their recursive learning from each other? What kind of sociality or associative life emerges from the human-machinic cognitive relations that we see with association rules and analytics?’

…and, as I explore these ‘human-machinic cognitive relations’, look beyond the polished “app” user interfaces and reflect on how algorithms (despite how they are presented) are far from objective or neutral (Kitchin 2017: 18). I turn my attention to investigating discrimination and bias (Noble 2018 and #AlgorithmsForHer)…

I also investigate the notion of ‘data colonialism’ (Knox 2016: 14), rethink the relation between algorithms and power (Beer 2017) and look to the future of what this might all mean in an educational context (Knox et al. 2020; Williamson 2017).

View references

Michael saved in Pocket: ‘Price and prejudice: Automated decision-making and the UK government’ (Open Democracy, Charlotte Threipland and Oscar Rickett, 2020)


Neither Karam nor Odili know for sure what role automated decision-making played in their visa rejections, but in both cases computerised errors played a part and a decision seemed to be made primarily based on nationality. This level of service from the Home Office doesn’t come cheap.

View full article

Google autocomplete and bias?

I’ve taken screenshots of some further play with Google autocomplete below… are existing views and biases being reproduced in the suggested terms, or is it more complex than that (see Noble 2018)?

I wonder what the results might be if I was to search these terms from a different location (see Kitchin (2017: 21) and Mahnke and Uprichard (2014))?

'Why is detroit...' Google autocomplete
‘Why is detroit…’ Google autocomplete
'Why is england...' Google autocomplete
‘Why is england…’ Google autocomplete
'Why is britain...' Google autocomplete
‘Why is britain…’ Google autocomplete

Michael saved in Pocket: ‘Algorithms of Oppression’ (Noble 2018)

Description (NYU Press)

A revealing look at how negative biases against women of color are embedded in search engine results and algorithms

Run a Google search for “black girls”—what will you find? “Big Booty” and other sexually explicit terms are likely to come up as top search terms. But, if you type in “white girls,” the results are radically different. The suggested porn sites and un-moderated discussions about “why black women are so sassy” or “why black women are so angry” presents a disturbing portrait of black womanhood in modern society.

In Algorithms of Oppression, Safiya Umoja Noble challenges the idea that search engines like Google offer an equal playing field for all forms of ideas, identities, and activities. Data discrimination is a real social problem; Noble argues that the combination of private interests in promoting certain sites, along with the monopoly status of a relatively small number of Internet search engines, leads to a biased set of search algorithms that privilege whiteness and discriminate against people of color, specifically women of color.

Through an analysis of textual and media searches as well as extensive research on paid online advertising, Noble exposes a culture of racism and sexism in the way discoverability is created online. As search engines and their related companies grow in importance—operating as a source for email, a major vehicle for primary and secondary school learning, and beyond—understanding and reversing these disquieting trends and discriminatory practices is of utmost importance.

An original, surprising and, at times, disturbing account of bias on the internet, Algorithms of Oppression contributes to our understanding of how racism is created, maintained, and disseminated in the 21st century.

View book