30 Days of AI in Testing experience report

Publication date Read time 6 min Tags testing

In March, Ministry of Testing held 30 Days of AI in Testing challenge. I participated, and post my general impressions below.

If you have never heard of “$x days of $something” challenges, the idea is simple - there is some kind of prompt posted every day, and your goal is to write a short piece in response. You publish the work on a blog, dedicated forum or on social media. Prompts might include a task to complete, and these tasks will vary from very simple to somewhat complex. Some days majority of your time will be spent on the task itself, not writing.

I have heard about challenges like that years ago, but this was my first time attempting it.

One thing that I was definitely not prepared for is the pace. To be honest, it was absolutely exhausting. I started on a high note, fully motivated to write something every single day. At 9th day there was a technical glitch that caused a daily prompt to be published much later than usual, and I skipped that day due to scheduling problems. Only then I realized how badly I needed the break, and that 30 days straight are not going to be sustainable for me. I ended up skipping 5 days.

Finding the right rhythm and schedule was somewhat challenging, too. Prompts were published at the forum at 1 AM my local time, and arrived by email at around 11 AM. I usually checked the forum before going to sleep, to give myself some time to think about the reply. Then I would try to kick off the day by writing a reply as one of the first things in the morning. I found it much harder to fit writing in my afternoon / evening schedule. On the other hand, composing a reply could easily get out of control, creeping into my working hours. The solution I applied is not revolutionary - after few days I started to timebox myself much more aggressively. This resulted in omitting some points I thought about raising, but couldn’t carry out in satisfactory manner in reasonable time.

I participated primarily to see how others are using AI in their day-to-day work, or hope to use it in the future. The challenge definitely did push me into taking a closer look at various tools and getting some hands-on experience with them. I was able to try them in tasks that I wouldn’t come up with on my own. That gave me some idea what these tools might be good at, and where they definitely need some more work.

At the same time, I wouldn’t say it changed my overall approach to AI. I haven’t seen anything that would change my mind about the problems pullulating in the entire space, including output quality, copyright, privacy, data security and natural resource usage.

Few of the tasks involved using chatbots to handle some generic testing tasks - coming up with test ideas, generating test data, creating test code or explaining existing code. I have used Microsoft Bing Copilot for these. Every time I made a point to give it a problem that I have already faced in the past, so I can compare the output with some baseline reference material. There were some surprises, when Copilot did things I didn’t think about myself. But almost without a fail, Copilot would also omit something that I did cover. For me, that tips the balance against using LLMs for most of the tasks - I might be slower, but I produce higher quality work, gain experience and acquire some new domain-specific knowledge. These are things that are going to pay off manifold in the future. My current opinion is that by using AI systems you might be faster, but you miss opportunities to learn and upskill yourself.

One of the reasons to take part was to build connections and interact with other members. I don’t think it worked for me. Doing a task and writing a response was challenging and time consuming already, I just wasn’t able to put conversations with others on top of that. These kinds of interactions were not facilitated by the forum software, which displays a long list of messages one under another. It was especially problematic in the early days where each thread saw hundreds of responses. To be fair, I did saw more people responding to other folks later on, when activity has toned down a bit.

Unexpected benefit of taking part in the challenge was that it forced me to write. A lot. In total, I have written a bit over 8500 words (51 000 characters). For comparison, all the blog posts published so far on this website total to 42 500 words and a bit over 280 000 characters. To put things in perspective, in these 30 days I have written about 20% of what I’ve published here in 8 and a half years.

Especially in early days, the challenge kind of unlocked something in me, and I became much more prolific writer than ever before. Apart from forum posts, I published two articles on this blog, wrote notes, multiple emails and became much more active on LinkedIn or Mastodon. I even have two blog posts written that are not published yet - something that I haven’t experienced since 2012 or so. My only worry is this change might not be permanent and will fade as quickly as it appeared; that after short period of high productivity I will succumb to levels even lower than before.

The final observation I’m going to share is that Ministry of Testing forums were rather unusable in the first half of the month. While the goal of challenge is to do a small task every day, it’s only natural that not everyone can do that. It’s encouraged to write a reply to previous prompt at later date - better to do things late than to drop out. But as a result, all threads from recent days constantly received new replies. The default view of the forum, where threads are listed by time of last reply, was constantly swamped by challenge threads. Things improved after two or three weeks, when initial excitement faded and only most persistent people remained. Most of them responded daily, so previous days saw little activity and unrelated threads were easier to notice.

Honestly, I don’t know if I will ever participate in similar challenge in the future. I think it will require a topic that is highly interesting for me. If I do participate, I will definitely not go in with the goal of writing something every single day. I imagine I will try to respond to third, maybe half of all prompts. I will also give myself permission to slack and skip days from the get-go. I will definitely timebox my daily participation aggressively from the very first day. Perhaps I will focus exclusively on hands-on experience and completing the tasks, while skipping the writing part. Another approach I might try is treating that more as a networking event, focus on reading what others are publishing, ask clarifying or supplementary questions and encourage them to keep going.

I thought I’ll finish with some numbers. I have participated in 25 days, which means I have skipped 5 days. In total, I have written 51 562 characters, which gives a bit over 2000 characters per post. For reference, average article on this blog is about 6000 characters, and this article is about 7700 characters.

The chart below shows the number of replies to each day prompt, as of April 1st. Since one person could write multiple posts in each thread, this is at best a proxy variable for much more interesting survival analysis. Nonetheless, the pattern is pretty clear.

Number of responses in each day thread

Comments