Improving Conferences

Or: The Struggle Between Good and Progress

Recently I saw a tweet from Mr Bach:

There is a much bigger problem than #paytospeak and that is that nearly all conference talks about software testing are terrible and conference organizers do nothing about it.

— James Marcus Bach (@jamesmarcusbach) October 30, 2018

I’ve not been involved with conferences much recently, but I’ve been to a few during my alive time, and translating from Bach to Me I’d say that many conferences could do more to encourage the progress of testing. At the moment conferences seem to stand between a position of neutrality, allowing whatever’s said to be said and then we all have lunch, to a position of scientific uneasiness, downplaying and discouraging creative discourse. There are good financial reasons to encourage placid calm at a conference, of course.

Most conferences I’ve been to are fine, and mostly not any more than fine. I didn’t pay out of my own pocket to get in, the food was nice, I got to network and meet people in the industry, and I saw some talks that gave me ideas of my own. I rarely find a talk with ideas both convincing and new to me, but it has happened – Iain McCowatt gave a talk once that changed the way I see tools in testing.

What many conferences do not do is dig any deeper than the surface. I should now make my position fairly clear on scientific progress as it pertains to argument. Argument, and attacking ideas in the service of good ideas, is a central tenant of scientific progress. The process requires it. As Ben Goldacre says about his book “Bad Science”:

All too often, people hoping to make science accessible fall into the trap of triumphalism, presenting science as a canon, and a collection of true facts. In reality, science is about the squabble. Every fight you will read in this book, over the meaning of some data, is the story of the scientific process itself: you present your idea, you present your evidence, and we all take turns to try and pull them both apart. This process of close critical appraisal isn’t something we tolerate reluctantly, in science, with a grudge: far from it. Criticism and close examination of evidence is actively welcomed – it is the absolute core of the process – because ideas only exist to be pulled apart, and this is how we spiral in on the truth.

Away from the newspapers and science TV shows you can see that process, very clearly, in the institutions of science. The question-and-answer session at any academic conference, after someone presents their scientific research, is often a bloodbath: but nobody’s resentful, everyone expects it, and we all consent to it, as a kind of intellectual S&M activity. We know it’s good for our souls. If the idea survives, then great; if it needs more evidence, we decide what studies are needed next and do them. Then we all come back next year, tear the evidence apart again, and have another think. Real scientists know this. Only the fakers cry foul.

— Ben Goldacre
[https://www.badscience.net/2014/10/my-new-book-is-out-today-here-is-the-introduction-hooray/]

I’ll need to sum up a partial history of scientific philosophy here, so brace yourself and please bear with me. Saying a hypothesis (“all swans are white”) is true by looking for evidence (“there’s a white swan, there’s another white swan…”) is not logically valid because it’d take only one non-white swan to prove it wrong. We can’t say “all swans are white” is true. What we could do, given we only see white swans, is to say that it’s not wrong yet and it’s been questioned enough to seem correct so far. We cannot prove things to be true in this way, just… less false. Falsifying the hypothesis (“oh, there’s a black swan in Australia”) means we throw the hypothesis away. That means that a hypothesis that has undergone rigorous, systematic and creative attempts to falsify it is a stronger hypothesis that one that has not. We attack the idea to test the idea and see if it’s a good idea. The best ideas become solid, long-standing scientific theories (gravity, evolution, etc). Ideas become stronger (or fall apart) when tested. They are anti-fragile.

We know this feeling almost intuitively from software testing. We don’t know there’s NOT a bug, all we know is that we haven’t found one given what we’ve experienced with the product so far. We call a product without problems a higher quality product, until we find a problem. The hypothesis “there’s nothing I’d call a bug in this product” is stronger in a product that has undergone rigorous, systematic and creative attempts to find problems in it.

The problem is that a conference is usually a series of one-way conversations plus a lunch break and free pen. That does not support the testing of ideas, only the conveyance of them, good, bad and ugly. A conference organiser has to take money from participants and often companies who put their name and logo on the conference. Would you want to allow free exchange of ideas under your company logo? Also there are people who don’t like argument of any kind. People who are not okay with conflict. Sometimes it’s because they don’t understand that we are all agreeing to consent to argument in the service of good ideas. Sometimes it’s because they understand and choose not to consent.

There are those who use this lack of quality control to exert themselves on the world. Advertising a product or company or themselves, or getting paid to present a poor idea because it doesn’t have to be questioned. They may use language in a lazy way or confusing way (on purpose or otherwise) so that nobody really knows what it is they’re trying to say. This means that the testing body of knowledge suffers from poor ideas, some conference speakers suffer from lack of feedback, and when someone gets up to sell you something we all suffer.

What to do? Well firstly if you’re interested in starting a conference this would be a good point to start. It’d give your conference a unique selling point and encourage all sorts of new conversations. Changing culture is not easy, especially when your conference attendees have set expectations from what you’ve previously provided.

Here are some easier starter ideas written the following in mind:

1. Conferences have an established format and audience and total revolution is not desired.
2. Not all attendees want, understand or tolerate disagreement or conflict, but we want as many kinds of people as possible to be involved.
3. People are emotional beings.

Here we go…

Ideas for Conference Holders

1. Facilitate feedback to speakers. If you cannot have a constant feedback loop of conversation then perhaps hand out little cards requesting feedback, or create an app where you can tap out anonymous feedback, including the “negative”. The results could optionally be made public by the speaker before it’s received. Make it very easy to give feedback, and make it clear to all that honest feedback en masse is required to improve things. The format of feedback is a whole other post but I quite like the old TED system: adjectives that we ascribe to the talks. Convincing, funny, inspiring, entertaining, unconvincing, boring, confusing, manipulative, useful. Maybe if the talk was of use to them or not in their work or lives. Easy levers to pull that give fast feedback and a space to write optional notes. This is great for conference speakers who are looking to identify the effectiveness of their talks, and encourage them to speak with more rigour and find ways to improve. It also encourages attendees to think critically about the talks given.
2. Have a separate room for deep discussion. Run an unconference, or have a room where people can speak freely about what they’ve heard at the talks, with the speaker optionally in attendance to field questions and take notes. People could come along with new ideas to test and present them to a group, followed by testing the idea. If you need to, put a warning on the door. It’s important for a conference to feel like a safe space for people, but also ensure a safe space for ideas – a place where constructive argument is encouraged. Make it overt and explicit that leaving at any time for any reason is permitted and not judged – some may be trying out constructive conflict for the first time. Have water and snacks to maintain hydration and blood sugar. Gather feedback on the process and iterate. The discussion would have to be facilitated in some way; only one person may speak at a time, no interruptions and everyone should have opportunity to speak if they wish including the quiet and timid. Nobody should be forced to speak. Consider recording the process to review for improvement and show people what constructive criticism in the service of ideas can look like. Encourage fair play and comradeship by making the intent and context of this explicit. Encourage deep-diving on an important topic, asking questions and refuting the bad, with judgement of ideas rather than people. I’d recommend removing name tags and not doing introductions – nobody should be here to self-promote a personal brand or put on a show.
3. Have the option, and time put aside, for a facilitated Q&A session after a talk. Have audience mics and encourage clarifying and testing questions – having a feedback system would keep feedback out of the Q&A section.
4. Make it known to your sponsors the nature of your support for scientific thinking and the benefits of the free exchange of ideas. Having financial support that accepts disagreement and controversy will make you comfortable in having those at your conference – or parts of it.
5. Offer a come-down service of some kind. My Ben Goldacre quote used the analogy of a consenting S&M activity. In BDSM the come-down period afterwards is often called “aftercare” and it’s considered by many to be vital in seeing to the emotional needs of its participants and a healthy rotation back to “normal” life. It delineates play from real life and helps restrict emotions from one leaking into the other. Borrowing this idea you could provide an argument-free space with comfortable chairs, calm lighting, drinks and snacks and social-positive conversation. This will be a place to calm down, top up blood sugar and remind oneself of the objectivity of the practice. BDSM also sometimes employs a debrief – a discussion about what went well, what went badly and the emotional state of all involved. This would be a good idea to facilitate also – a reminder, via ceremony, that all involved are intellectual friends in the service of good ideas and not to carry anger or bad feelings out of the room with them. Encourage people to shake hands with their opponents and politely express their feelings if they wish.
6. Make it clear that for to your speakers to speak at your conference that their ideas may be questioned and how the format of your conference will work in that regard. If a vendor is to advertise give them an advert space to do it in. Delineate sponsorship and useful content.

Ideas for Conference Speakers

1. If you watch scientific talks (for an interesting example I recommend Doug Zongker’s brief talk on Chicken Chicken Chicken: Chicken Chicken) you will see slides that make claims are defended with evidence, sources, graphs and charts (real ones with data). Use these if appropriate – and where you cannot ask yourself how “correct” you feel you can make yourself out to be whilst maintaining your honesty.
2. If you watch scientific talks (again, see Chicken Chicken Chicken: Chicken Chicken) you will see that questions are sometimes anticipated. This shows that the speaker considered their material, questioned it themselves or had other’s do it, then prepared an answer. For any talk or debate it’s vital to see the other side of an argument or weaknesses in points and ideas. Remove the chaff from your talks and put additional appendices you can’t fit into the main body of the talk at the end for a Q&A. Question your material first, before your audience do. Find out where something is confusing, or you’ve stretched a point too far, or you’ve said something you cannot substantiate and deal with it.
3. Examine ethical use of language and evidence. You may wish to convince an audience, and you may wish to build a personal brand through entertainment, but ensure you are not using confusion or anecdote-as-argument or any other subethical mechanics. Ensure that what you’re saying is presented as opinion or is backed up by evidence. Ensure what you say is hard to misinterpret or weaponize. Speak accurately, as far as you can.
4. Collect feedback and critically appraise your own work.
5. Explicitly ask to be disagreed with, and where and how that can happen.
6. Check your ego. Your self-brand marketing might actually benefit from the humility of improvement, rather than be seen as “wrong”, especially if you point out the flaws in your own arguments and where your ideas might fail or not be useful. Plus it stops others from having to point that out for you.
7. Know what you are trying to do, and how you’re going to do it. What your mission is, what your message is.
8. Your ideas are going to be shared. Their inception and form belong to you, but the ideas themselves will belong to everyone, good and bad, strong parts and weak. Emotionally distance yourself from your ideas so that you can examine them dispassionately – and so that others can examine them without upsetting you too much. If your ideas have flaws or contexts in which they don’t work that is useful and powerful and important to know. If your idea is wrong or dangerous that’s even more important to know. Having a different perspective or missing someone else’s experience is not a personal failure, but a happenstance of misfortune from which we can all draw lessons. It does not make you a bad or ignorant person to present a flawed idea, unless you lie and manipulate to present it.

Ideas for Conference Attendees

1. Decide that you wish to get something from a conference that is honest, useful and applicable to your work. That could mean your emotional state, or tools to use, or a way of thinking to apply, or self-improvement of other kinds, but it should be useful.
2. Critically appraise all talks you see. Yes, exclaim your joy, but also consider the motives of the speaker. I like Bach’s “Huh? Really? So?” to help critical thinking.
3. When appraising a talk, providing feedback or offering criticism ensure you are criticising the idea and not the person or presentation. If you need clarity, ask for it. Maybe you’d have done the slides differently, but that’s not what this is about. Personal attacks are the second most base of logical fallacies, referred to as “ad hominem” – narrowly beating childish name-calling in last place. They don’t help the speaker, don’t help anyone in the audience, don’t help the industry and make you look bad. Try to be an adult about it. It’s okay to talk about feelings and emotion about the ideas (“the idea of using test cases makes me uneasy and scared and I don’t know why”) but not about people (“I’d pay attention, but I can’t concentrate on someone who wears those colour shoes”).
4. You don’t have to know everything. You can have a vague feeling of unease or somehow feel that an idea won’t quite work. Start from there and dig deeper – why do you feel uneasy? Explore some reasons, talk to people, see what fits. Maybe your unease will disappear! Maybe you’ll find a deeper reason to discuss.
5. Not everyone else needs to know everything. Digging deep requires explanation of all the abstractions as you go down, which requires time and patience and polite explanations. Don’t judge someone for not knowing something or not sharing your view – simply explain your perspective to them as best you can, then move on.
6. Make your wish to see higher quality content known to conference organisers.
7. Make your wish to have constructive-argument-friendly facilities at conferences or meetups known to conference organisers.
8. Seek out and attend conferences that offer some of these things already. I hear good things about Let’s Test and CAST from people whose opinion I trust.
9. Organise post-conference chats! Like a book club meet up for a friendly drink and to pull apart ideas. Talk about what you found fascinating or disingenuous or useful.
10. Usefulness deserves its own point. How will you use what you’ve learned? How will the world be any different for you? What will you do to change things now you’ve been to a conference? Write it down or explain it to someone to make it real. If a talk isn’t applicable, even indirectly, then what value does it have?
11. You may carry happiness and levity and energy away from a conference, but also a responsibility. What is required of you now to improve the world, learning what you have learned? Do you need to spread the world, or implement a change, or offer an apology, or inspire others, or change your attitude at work or at home?
12. If you know someone who’s seen a talk and brings it up ask them what they think of it. Ask them what effect it might have on their work. Force them, kindly, to evaluate the talk and reflect on possibility.

Ways to Evaluate a Talk

1. Identify what you think is the core message. Identify why you think it’s the core message.
2. What is the talk’s purpose? Was it perhaps to entertain, or educate, or spark conversation? Maybe it was to inspire or bolster us emotionally. Maybe it was a pick-n-mix story where people could draw out their own value, a bit like using an explicit model in testing such as FEW HICCUPPS – inspiration for our own ideas through triggering words and phrases. Identify the purpose.
3. Was the talk successful in delivering the core message, and has it fulfilled its purpose? What would it require from you to have it have an impact on the world?
4. What about the idea fails? Where are its weaknesses? Are the failures and weaknesses and heuristic errors enough to disregard the whole talk, or does it beat a cost-benefit analysis? Knowing the failures and weak points and caveats of an idea will help you to implement it with greater safety and intelligence, and keep you from idolatry, crusading and over-constrained views of work and life.

 

 

These are just some starters, but this is long enough already.