The distiller’s art…

• 

It’s been a couple of months since the last post on this blog so it’s about time I wrote another instalment – but what have I been doing since Christmas? As we are now firmly into analysis stages of our Realist literature review looking at the factors involved in the sustainability of community-based groups and activities for people affected by dementia, it’s difficult to convey the (sometimes seemingly endless) sifting, organising and synthesising work that’s going on in any kind of interesting or understandable way.

But as we start to see a slow trickle of distilled data emerge that is looking promisingly like it could be the basis of something approaching final results, I’ve begun to see what I’m doing as just that – the work of a distiller, ever-refining the brew I concocted in previous months on this project (apologies to people who actually work in the production of alcoholic beverages – I’m quite sure I’m hideously mixing up terminology all over the place here).

My point is, though, that we seem to have passed our sources and data through various types of processing again and again and again now, each analysis slightly different and ever more fine-grained. As this is all new to me, I suspect if I did it all again I could bypass some bits, or at least streamline the work a little – but as it stands I can justify why each step along the way was necessary, or at least helpful, to prepare for the next, on the way to where we want to end up.

At this stage the data we’re looking at is significantly refined after having been sifted, scrutinised and categorised in various ways, but it’s still directly rooted in the various sources – academic papers, evaluation reports, magazine articles, information booklets, conference presentations etc. – that we have collected.

So, if you’ll bear with me, here is another laboured metaphor for what we have been doing in conducting (this part of) the Realist Review process.

The SCI-Dem distilling process:

• 

Note: Good practice recommends a 10% sample at most of these stages should be checked independently by a second member of the team and notes compared and discussed, to ensure quality and consistency.

1. Gather thousands of papers through formal searching, remove duplicates, then screen so that only those that look potentially useable are left.
2. Pour these into a big XL spreadsheet container, and add all the other tasty-looking articles and papers you’ve found through informal searching methods outside of your formal search.
3. Now quality control these by taking a closer look at the relevance and rigour of each. Remove the ropier stuff, and ones that shouldn’t be there at all, and transfer the rest to the metaphorical mash tun that is NVivo software for qualitative analysis.
4. With your articles brewing in NVivo, categorise all the data in each that is relevant to your research question(s) so that it is organised and batched into themes/factors. Brewing tip: You should let theme/factor categories bubble up from the data themselves but you can also add pre-cooked theme/factor categories that arose when consulting stakeholders previously, to see if they work and to aid the process.
5. Now distil: Summarise the (themed) data from each source and look for anything that might be a potential context, mechanism or outcome that affects your research question(s) as you do so.
6. Now further distil: Decide upon the most salient and important outcomes that the data suggests have a bearing on your research question(s). What is most talked about? What is there most evidence for? What seems to be most impactful?
7. Further distil (again): Now rake through the summarised data again and highlight everything else that appears to have a bearing on your decided-upon outcomes, for each outcome.
8. And further distil (yet again!): Take the stuff that appears to have a bearing on each outcome and restate all of it in terms of cause and effect. Distilling tip: You are looking for contexts (circumstantial factors) and mechanisms (processes that might be triggered by those circumstantial factors) that might lead to your outcomes, but if this proves mind-melting, there is a half-way step that some use – “if-then” statements.

If-then statements

What are these? These are basically statements of what appears to be leading to what in your data –e.g. “IF this thing happens, THEN this is what can follow” or “IF this isn’t done in these circumstances, THEN this is likely to happen”. We used them as loose theoretical constructs from the data, not quite as detailed and pinned down as the final Realist-style “Context + Mechanism = Outcome” (CMO) configurations you are ultimately looking for but a step on the way. They do not break things down into contextual factors and mechanisms of action, but simply express how “something or other” could lead to, tends to lead to, or will lead to, a particular outcome, according to the data collected.

Not everyone uses or likes “if-then” statements, so why use them? Well, there can often be a lot of debate and head-scratching about how exactly to put together those final CMO configurations – particularly over what should be classed as contexts (background circumstances) and what should be classed as mechanisms (often-hidden processes, responses and actions) on the way to producing an outcome. Furthermore CMO configurations can get complex – multiple context elements can work together or feed in at different points to trigger multiple mechanisms on the way to an outcome. We found “if-then” statements were a simpler way of logging and surveying what we had in the data without having to get into the nitty gritty of contexts and mechanisms – in fact by writing out a list of all the simple “if-then” statements we could find (that might affect the particular outcomes we had identified as important), this helped us to get our heads around what to we had work with and is a good way of laying things out for further discussion as we approach having to construct the final CMO configurations we are ultimately aiming for.

So this is where we are at the moment, and where I will leave behind my over-laboured “distillery” metaphor (notice I already did)! I’ll keep you posted on what happens next. Now I’m off for a nice wee dram…

Advertisement