Improving Conferences

Or: The Struggle Between Good and Progress

Recently I saw a tweet from Mr Bach:

I’ve not been involved with conferences much recently, but I’ve been to a few during my alive time, and translating from Bach to Me I’d say that many conferences could do more to encourage the progress of testing. At the moment conferences seem to stand between a position of neutrality, allowing whatever’s said to be said and then we all have lunch, to a position of scientific uneasiness, downplaying and discouraging creative discourse. There are good financial reasons to encourage placid calm at a conference, of course.

Most conferences I’ve been to are fine, and mostly not any more than fine. I didn’t pay out of my own pocket to get in, the food was nice, I got to network and meet people in the industry, and I saw some talks that gave me ideas of my own. I rarely find a talk with ideas both convincing and new to me, but it has happened – Iain McCowatt gave a talk once that changed the way I see tools in testing.

What many conferences do not do is dig any deeper than the surface. I should now make my position fairly clear on scientific progress as it pertains to argument. Argument, and attacking ideas in the service of good ideas, is a central tenant of scientific progress. The process requires it. As Ben Goldacre says about his book “Bad Science”:

All too often, people hoping to make science accessible fall into the trap of triumphalism, presenting science as a canon, and a collection of true facts. In reality, science is about the squabble. Every fight you will read in this book, over the meaning of some data, is the story of the scientific process itself: you present your idea, you present your evidence, and we all take turns to try and pull them both apart. This process of close critical appraisal isn’t something we tolerate reluctantly, in science, with a grudge: far from it. Criticism and close examination of evidence is actively welcomed – it is the absolute core of the process – because ideas only exist to be pulled apart, and this is how we spiral in on the truth.

Away from the newspapers and science TV shows you can see that process, very clearly, in the institutions of science. The question-and-answer session at any academic conference, after someone presents their scientific research, is often a bloodbath: but nobody’s resentful, everyone expects it, and we all consent to it, as a kind of intellectual S&M activity. We know it’s good for our souls. If the idea survives, then great; if it needs more evidence, we decide what studies are needed next and do them. Then we all come back next year, tear the evidence apart again, and have another think. Real scientists know this. Only the fakers cry foul.

— Ben Goldacre

I’ll need to sum up a partial history of scientific philosophy here, so brace yourself and please bear with me. Saying a hypothesis (“all swans are white”) is true by looking for evidence (“there’s a white swan, there’s another white swan…”) is not logically valid because it’d take only one non-white swan to prove it wrong. We can’t say “all swans are white” is true. What we could do, given we only see white swans, is to say that it’s not wrong yet and it’s been questioned enough to seem correct so far. We cannot prove things to be true in this way, just… less false. Falsifying the hypothesis (“oh, there’s a black swan in Australia”) means we throw the hypothesis away. That means that a hypothesis that has undergone rigorous, systematic and creative attempts to falsify it is a stronger hypothesis that one that has not. We attack the idea to test the idea and see if it’s a good idea. The best ideas become solid, long-standing scientific theories (gravity, evolution, etc). Ideas become stronger (or fall apart) when tested. They are anti-fragile.

We know this feeling almost intuitively from software testing. We don’t know there’s NOT a bug, all we know is that we haven’t found one given what we’ve experienced with the product so far. We call a product without problems a higher quality product, until we find a problem. The hypothesis “there’s nothing I’d call a bug in this product” is stronger in a product that has undergone rigorous, systematic and creative attempts to find problems in it.

The problem is that a conference is usually a series of one-way conversations plus a lunch break and free pen. That does not support the testing of ideas, only the conveyance of them, good, bad and ugly. A conference organiser has to take money from participants and often companies who put their name and logo on the conference. Would you want to allow free exchange of ideas under your company logo? Also there are people who don’t like argument of any kind. People who are not okay with conflict. Sometimes it’s because they don’t understand that we are all agreeing to consent to argument in the service of good ideas. Sometimes it’s because they understand and choose not to consent.

There are those who use this lack of quality control to exert themselves on the world. Advertising a product or company or themselves, or getting paid to present a poor idea because it doesn’t have to be questioned. They may use language in a lazy way or confusing way (on purpose or otherwise) so that nobody really knows what it is they’re trying to say. This means that the testing body of knowledge suffers from poor ideas, some conference speakers suffer from lack of feedback, and when someone gets up to sell you something we all suffer.

What to do? Well firstly if you’re interested in starting a conference this would be a good point to start. It’d give your conference a unique selling point and encourage all sorts of new conversations. Changing culture is not easy, especially when your conference attendees have set expectations from what you’ve previously provided.

Here are some easier starter ideas written the following in mind:

  1. Conferences have an established format and audience and total revolution is not desired.
  2. Not all attendees want, understand or tolerate disagreement or conflict, but we want as many kinds of people as possible to be involved.
  3. People are emotional beings.

Here we go…

Ideas for Conference Holders

  1. Facilitate feedback to speakers. If you cannot have a constant feedback loop of conversation then perhaps hand out little cards requesting feedback, or create an app where you can tap out anonymous feedback, including the “negative”. The results could optionally be made public by the speaker before it’s received. Make it very easy to give feedback, and make it clear to all that honest feedback en masse is required to improve things. The format of feedback is a whole other post but I quite like the old TED system: adjectives that we ascribe to the talks. Convincing, funny, inspiring, entertaining, unconvincing, boring, confusing, manipulative, useful. Maybe if the talk was of use to them or not in their work or lives. Easy levers to pull that give fast feedback and a space to write optional notes. This is great for conference speakers who are looking to identify the effectiveness of their talks, and encourage them to speak with more rigour and find ways to improve. It also encourages attendees to think critically about the talks given.
  2. Have a separate room for deep discussion. Run an unconference, or have a room where people can speak freely about what they’ve heard at the talks, with the speaker optionally in attendance to field questions and take notes. People could come along with new ideas to test and present them to a group, followed by testing the idea. If you need to, put a warning on the door. It’s important for a conference to feel like a safe space for people, but also ensure a safe space for ideas – a place where constructive argument is encouraged. Make it overt and explicit that leaving at any time for any reason is permitted and not judged – some may be trying out constructive conflict for the first time. Have water and snacks to maintain hydration and blood sugar. Gather feedback on the process and iterate. The discussion would have to be facilitated in some way; only one person may speak at a time, no interruptions and everyone should have opportunity to speak if they wish including the quiet and timid. Nobody should be forced to speak. Consider recording the process to review for improvement and show people what constructive criticism in the service of ideas can look like. Encourage fair play and comradeship by making the intent and context of this explicit. Encourage deep-diving on an important topic, asking questions and refuting the bad, with judgement of ideas rather than people. I’d recommend removing name tags and not doing introductions – nobody should be here to self-promote a personal brand or put on a show.
  3. Have the option, and time put aside, for a facilitated Q&A session after a talk. Have audience mics and encourage clarifying and testing questions – having a feedback system would keep feedback out of the Q&A section.
  4. Make it known to your sponsors the nature of your support for scientific thinking and the benefits of the free exchange of ideas. Having financial support that accepts disagreement and controversy will make you comfortable in having those at your conference – or parts of it.
  5. Offer a come-down service of some kind. My Ben Goldacre quote used the analogy of a consenting S&M activity. In BDSM the come-down period afterwards is often called “aftercare” and it’s considered by many to be vital in seeing to the emotional needs of its participants and a healthy rotation back to “normal” life. It delineates play from real life and helps restrict emotions from one leaking into the other. Borrowing this idea you could provide an argument-free space with comfortable chairs, calm lighting, drinks and snacks and social-positive conversation. This will be a place to calm down, top up blood sugar and remind oneself of the objectivity of the practice. BDSM also sometimes employs a debrief – a discussion about what went well, what went badly and the emotional state of all involved. This would be a good idea to facilitate also – a reminder, via ceremony, that all involved are intellectual friends in the service of good ideas and not to carry anger or bad feelings out of the room with them. Encourage people to shake hands with their opponents and politely express their feelings if they wish.
  6. Make it clear that for to your speakers to speak at your conference that their ideas may be questioned and how the format of your conference will work in that regard. If a vendor is to advertise give them an advert space to do it in. Delineate sponsorship and useful content.

Ideas for Conference Speakers

  1. If you watch scientific talks (for an interesting example I recommend Doug Zongker’s brief talk on Chicken Chicken Chicken: Chicken Chicken) you will see slides that make claims are defended with evidence, sources, graphs and charts (real ones with data). Use these if appropriate – and where you cannot ask yourself how “correct” you feel you can make yourself out to be whilst maintaining your honesty.
  2. If you watch scientific talks (again, see Chicken Chicken Chicken: Chicken Chicken) you will see that questions are sometimes anticipated. This shows that the speaker considered their material, questioned it themselves or had other’s do it, then prepared an answer. For any talk or debate it’s vital to see the other side of an argument or weaknesses in points and ideas. Remove the chaff from your talks and put additional appendices you can’t fit into the main body of the talk at the end for a Q&A. Question your material first, before your audience do. Find out where something is confusing, or you’ve stretched a point too far, or you’ve said something you cannot substantiate and deal with it.
  3. Examine ethical use of language and evidence. You may wish to convince an audience, and you may wish to build a personal brand through entertainment, but ensure you are not using confusion or anecdote-as-argument or any other subethical mechanics. Ensure that what you’re saying is presented as opinion or is backed up by evidence. Ensure what you say is hard to misinterpret or weaponize. Speak accurately, as far as you can.
  4. Collect feedback and critically appraise your own work.
  5. Explicitly ask to be disagreed with, and where and how that can happen.
  6. Check your ego. Your self-brand marketing might actually benefit from the humility of improvement, rather than be seen as “wrong”, especially if you point out the flaws in your own arguments and where your ideas might fail or not be useful. Plus it stops others from having to point that out for you.
  7. Know what you are trying to do, and how you’re going to do it. What your mission is, what your message is.
  8. Your ideas are going to be shared. Their inception and form belong to you, but the ideas themselves will belong to everyone, good and bad, strong parts and weak. Emotionally distance yourself from your ideas so that you can examine them dispassionately – and so that others can examine them without upsetting you too much. If your ideas have flaws or contexts in which they don’t work that is useful and powerful and important to know. If your idea is wrong or dangerous that’s even more important to know. Having a different perspective or missing someone else’s experience is not a personal failure, but a happenstance of misfortune from which we can all draw lessons. It does not make you a bad or ignorant person to present a flawed idea, unless you lie and manipulate to present it.

Ideas for Conference Attendees

  1. Decide that you wish to get something from a conference that is honest, useful and applicable to your work. That could mean your emotional state, or tools to use, or a way of thinking to apply, or self-improvement of other kinds, but it should be useful.
  2. Critically appraise all talks you see. Yes, exclaim your joy, but also consider the motives of the speaker. I like Bach’s “Huh? Really? So?” to help critical thinking.
  3. When appraising a talk, providing feedback or offering criticism ensure you are criticising the idea and not the person or presentation. If you need clarity, ask for it. Maybe you’d have done the slides differently, but that’s not what this is about. Personal attacks are the second most base of logical fallacies, referred to as “ad hominem” – narrowly beating childish name-calling in last place. They don’t help the speaker, don’t help anyone in the audience, don’t help the industry and make you look bad. Try to be an adult about it. It’s okay to talk about feelings and emotion about the ideas (“the idea of using test cases makes me uneasy and scared and I don’t know why”) but not about people (“I’d pay attention, but I can’t concentrate on someone who wears those colour shoes”).
  4. You don’t have to know everything. You can have a vague feeling of unease or somehow feel that an idea won’t quite work. Start from there and dig deeper – why do you feel uneasy? Explore some reasons, talk to people, see what fits. Maybe your unease will disappear! Maybe you’ll find a deeper reason to discuss.
  5. Not everyone else needs to know everything. Digging deep requires explanation of all the abstractions as you go down, which requires time and patience and polite explanations. Don’t judge someone for not knowing something or not sharing your view – simply explain your perspective to them as best you can, then move on.
  6. Make your wish to see higher quality content known to conference organisers.
  7. Make your wish to have constructive-argument-friendly facilities at conferences or meetups known to conference organisers.
  8. Seek out and attend conferences that offer some of these things already. I hear good things about Let’s Test and CAST from people whose opinion I trust.
  9. Organise post-conference chats! Like a book club meet up for a friendly drink and to pull apart ideas. Talk about what you found fascinating or disingenuous or useful.
  10. Usefulness deserves its own point. How will you use what you’ve learned? How will the world be any different for you? What will you do to change things now you’ve been to a conference? Write it down or explain it to someone to make it real. If a talk isn’t applicable, even indirectly, then what value does it have?
  11. You may carry happiness and levity and energy away from a conference, but also a responsibility. What is required of you now to improve the world, learning what you have learned? Do you need to spread the world, or implement a change, or offer an apology, or inspire others, or change your attitude at work or at home?
  12. If you know someone who’s seen a talk and brings it up ask them what they think of it. Ask them what effect it might have on their work. Force them, kindly, to evaluate the talk and reflect on possibility.

Ways to Evaluate a Talk

  1. Identify what you think is the core message. Identify why you think it’s the core message.
  2. What is the talk’s purpose? Was it perhaps to entertain, or educate, or spark conversation? Maybe it was to inspire or bolster us emotionally. Maybe it was a pick-n-mix story where people could draw out their own value, a bit like using an explicit model in testing such as FEW HICCUPPS – inspiration for our own ideas through triggering words and phrases. Identify the purpose.
  3. Was the talk successful in delivering the core message, and has it fulfilled its purpose? What would it require from you to have it have an impact on the world?
  4. What about the idea fails? Where are its weaknesses? Are the failures and weaknesses and heuristic errors enough to disregard the whole talk, or does it beat a cost-benefit analysis? Knowing the failures and weak points and caveats of an idea will help you to implement it with greater safety and intelligence, and keep you from idolatry, crusading and over-constrained views of work and life.



These are just some starters, but this is long enough already.


3 thoughts on “Improving Conferences

  1. This all sounds absolutely fine, as long as that’s the sort of conference you expect to be attending. But a lot of conferences are there to evangelise about something the speaker has just worked out for themselves and they want to educate others about it. if the aim of the conference is ‘education’, then debate and challenge is probably not what the speaker or their attendees are looking for.

    In my time, I’ve been to a number of different sorts of conference. I was at the last Brighton TestBash and enjoyed it, because I was looking to pick brains and to network and to get the vibe of this testing community thing which I’ve only just discovered despite having been in testing for something like 25 years. But I have also attended literary science fiction conventions, where there was a mix of events – some promotional (“In this session, Elmer T. Hack is promoting his latest novel”), some celebratory (such as a Guest of Honour speech) and some actually taking the form of a debate which would meet your requirements of hypothesis > challenge > debate > conclusions > new paradigm (on a good day) – or “everyone retreats to the bar” (on a less good day). Flame wars can start there; equally, using the pretext of continuing the discussion in the bar, informal break-out groups will continue the debate, and conclusions (if any) will be continued in fanzines, possibly for months afterwards.

    I have also been to trade union conferences, where there were different models. The aim of a trade union conference is to set the union’s policies for the coming year; and that is done by a very formal mechanism of motions which are then subjected to open but moderated debate before a vote is taken. These issues will be ones that are already known to the delegates and in any event, the motions for debate will have been circulated well in advance as delegates will have been expected to have already debated these motions in their local Branches in order for those Branches to instruct their delegate on how to vote.

    This decision-making function of the conference tends to exclude wider educational matters, so the fringe of Conference will also include lectures and training events, similar to the sort of events we’ve already discussed.

    I think this points the way to squaring your circle, of holding conferences that are all things to all testers. Science fiction conventions developed the concept of the “multi-stream” convention, where different subject streams run in parallel. Attendees can mix and match as they want; they can spend all their time in gaming circles, or catching up on Real Science in presentations by Real Scientists, or attending any of the other streams; or they can pick and choose. Applying the same concept to a testing conference could mean that there is a “lecture” stream, where the audience receives wisdom and emerges better informed than when they went in; a “workshop” stream where people participate and learn that way; and a “debate” stream, where ideas are debated in the way you suggest and break-out groups continue discussion under strict debating rules (as defined by the organisers) to explore new ideas and subject them to challenge.

    This has one disadvantage – it’s much more complex than a single-stream event where people sit in an auditorium and are lectured at. It takes more organisation, and as the number of venues that can accommodate the multiple events, break-outs and slightly varying times that sessions will break at are limited, it tends to mean that conferences will get bigger, longer and more expensive. TestBash Brighton seemed to me (judging by the experience of colleagues who had been to earlier ones)to be heading in that direction; I think that is the future of the testing conference, because that is what the profession will end up needing and demanding.

    Liked by 1 person

    1. Thank you, this is exactly the sort of constructive insight I was hoping for! You gave me to pause on the complexity of even a simple idea like how to practically implement a discussion on an idea. I do stop and remember the kind of bizarre vitriol I saw on a check vs test or certification discussion and wonder how we’ll ever do better. Flame wars are my fear. I think that a lot of the success of a deep discussion will rely on the sort of people that are attracted to it, which hinges on the way it’s advertised and presented and the shape the chairs are set up in and the facilitator. I’m painfully aware that this is a filtering system for something I’d like to see populated by more kinds of idea and not less, and I go back and forth trying to think of a way of testing ideas with people who don’t like testing ideas in the way I think ideas are tested and I’m not enough to reinvent scientific inquiry.

      I’m definitely with you on the idea of expectation. I think that scrutiny helps everyone, including the scrutinised, but I don’t want anyone to find themselves under scrutiny they didn’t sign up for. The whole subject is rife with such ambivalent bipoles.

      I do like a multi-stream convention. I’d like the streams that involve scientific inquiry and deep diving on topics populated by more kinds of people. Modern science works the way it does because certain kinds of people shaped it that way. I’m excited to see what happens when artists test ideas, or people who have been historically discouraged or not permitted to join that club. What an opportunity! How to achieve that is beyond my ability, I think, but I’d love to see it. If I see an artist who is not great with conflict come to love the testing of ideas, which I think it totally possible, because I believe we can separate the conflict from anti-fragile processes, I will cry tears of joy. I also believe, as Einstein claimed, that great scientists must also be artists. I can imagine the philosophical processes behind that extending to testers and testing, perhaps more so.

      I haven’t been to TestBash lately, because I’m a media post-processor now and I can’t really warrant the expense of a testing conference, but I’m glad it’s going that way. I completely agree that the profession will need this, and knowing the hands that have held TestBash over the years and watching how it’s grown I have high expectations for it.


  2. I think that TestBash Brighton is getting close to the maximum flexibility their venue can offer. Running multiple streams might well demand a move to a larger venue and a corresponding hike in the cost.

    It can be done. Brighton was the location of a number of science fiction conventions – including two World conventions which were held at the Brighton Centre and the conference suite in the Metropole Hotel – in the 1970s and 1980s, and only stopped being used because of a bizarre fall-out between a hotel manager who was not honouring the terms of his contract and the convention organisers which has had resonances down through the years ever since and has resulted in Brighton becoming known to some in the community as “the Forbidden City”.

    It is an odd and interesting fact that the Internet and wider IT communities are the way they are in no small part because many of the people who played a part in creating them were science fiction fans. SF fandom was a sort of pre-electronic Internet, using real mail, paper fanzines and conventions. As a peer-to-peer channel of communication and education, it formed the prototype for what was to come later. So much original Netspeak came from fan terminology. And indeed, attending my first TestBash felt vey much like the early conventions I went to in the 1970s in terms of their enthusiasm, the sense of being a special and perhaps unique community, the hands-on approach to doing stuff and indeed the very demographic (adjusted, of course, for age 🙂 )

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s