Mirek Długosz personal website

Found on web: Commoncog

2025-09-17T18:24:52+02:00

Commoncog by Cedric Chin is probably the most influential resource I’ve discovered on the Internet in the last couple of years. It’s deep. It’s grounded. It’s deliberate. It’s insightful. It’s judicious. It’s nuanced. It’s well written. It’s practical. If I were stranded on a desert island and could only receive updates from five websites, Commoncog would be one of them. It’s that good.

Consider the series on putting mental models to practice. There’s a very compelling argument for limits of scientific knowledge from practical point of view in part two; and right away other types of knowledge are proposed to fill the gap. Some of them are collectively called Naturalistic Decision Making models, elaborated on in part four. Part five is a real treasure - it provides detailed, actionable advice on developing proficiency in expertise-driven decision making. The whole series gives both theoretical framework and practical guidance with a proven track record.

These themes are then echoed in The Secret at the Heart of Continuous Improvement, which covers Plan-Do-Study-Act model - something that you might have heard about before. Nonetheless, Cedric manages to really nail down the importance of how the stages support one another, and the entire model as a whole. He then delivers a knockout punch with revolutionary framing - PDSA provides a methodology for single subject studies. It’s the ultimate tool for finding and refining what works for you.

There’s also entire series on Cognitive Flexibility Theory, which tries to uncover how people are learning in ill-structured domains - and provides tools to accelerate that learning. One of them is case library, which is explained in depth in How Note Taking Can Help You Become an Expert. It acknowledges the messiness of the real world and builds on the insight that experts are pattern-matching novel situations against all the cases they experienced or heard of, identifying similarities and differences. Becoming an expert is largely about growing this body of cases you can reference.

I’m truly captivated by these ideas. Where I stand right now, I believe that one of the most beneficial learning resources for testers is building a shared case library. I’ve been modestly contributing to that in this very blog. If this is something you would be interested in and would like to work on together, just send me a message.

Meanwhile, be sure to check out existing Commoncog article base and subscribe to their newsletter or RSS feed.

Experience report: Implementing High Volume Automated Testing system

2025-08-06T18:23:20+02:00

I first heard about High Volume Automated Testing in 2017, and I’ve wanted to try it ever since. The opportunity came in 2022, when I was working on revamping a UI testing framework for software called Quipucords. This article focuses on the decision points I encountered, the alternatives I considered, and the choices I made. At the end, I share the lessons I learned from this experience.

Keep in mind that this is not intended to apply to all possible High Volume Automated Testing systems, nor is it meant to be an exhaustive list of such system design decisions. I can’t talk about alternatives I didn’t think of, I have not actually tried all the alternatives, and I was working in the context of the UI layer of a specific product.

I share links to my implementation near the end of the article. Looking at the code might help to understand some of the more abstract parts for the article.

But before we begin, let’s briefly talk about High Volume Automated Testing.

What is High Volume Automated Testing

High Volume Automated Testing (HiVAT) takes the idea that automation changes the very concept of how tests can be created, run and evaluated, and extends it to its logical conclusion. When learning about test automation, people tend to think about programming a computer to do the same things that a human would do when interacting with a software, but faster and unaffected by boredom or fatigue. HiVAT is born from the question: why stop there? If you program a computer to interact with a software, you can program it to do things that would be too costly for a human to do, or which a human could never do.

I first heard about it in 2017, from the article published by Cem Kaner in 2013, but the idea is much older than that. Kaner gave a couple of talks on the topic in 2003 and 2004. He claims to have experienced systems like that first-hand in 1980s, and in various places he mentions prior work dating back to 1970s and even 1960s. However, the phrase “High Volume Automated Testing” and the acronym “HiVAT” are relatively obscure. They don’t seem to be discussed in the community often, and I reckon most testers have never heard of them.

These days, the ideas behind HiVAT see mainstream recognition in property-based testing and fuzzing.

I highly recommend reading the Kaner article for a better explanation and full background before moving on.

Designing a HiVAT system

Kaner lists twelve techniques that he considers to be examples of HiVAT. So the very first decision to make is: do you want to implement one of already known techniques? If so, which one? If not, how is it different from established solutions?

I wanted a robot that is able to automatically and autonomously wander through the application web interface. It should click random links, fill the forms, edit and delete items where possible etc. The idea was that it should work without supervision for some time, say 15 to 30 minutes. During that time it would most likely cover all the same scenarios we explicitly covered in the standard test suite, but would do so in a random order, while canceling some operations, while repeating certain actions etc. If it encountered any problems, they would likely be caused by the specific data that was introduced or the order of actions that was taken. Ideally, it should just finish the session without any troubles.

This is one decision that I haven’t considered too comprehensively. It was kind of obvious for me that in the context I was working in, this is how HiVAT would look like.

In Python-inspired pseudo-code, the entire system may be approximated as a relatively simple loop. That loop doubles as a map of the main decisions that we have to make.

global browser_driver = create_browser_driver()

while should_still_run():
   methods = get_methods_available_at_this_place()
   chosen_method = random_choice(methods)
   method_arguments = get_arguments_required_for_method(chosen_method)
   try:
       chosen_method(method_arguments)
    except Exception:
       # could be a bug!
       handle_exception()

What are the system stopping criteria?

An autonomous testing session can last a long time, but it can’t run forever. So when should it stop? How can a system decide if it should keep going?

The obvious solution is to just let it run, basically outsourcing the decision outside of the system. It will stop when the process is killed. That can be done using kill command, by shutting down the machine, forcing a CI environment to stop execution etc. That approach has two fundamental problems: it’s not particularly user-friendly and it might bring additional complexity if you need to clean up. Signal might be received in the middle of the loop, and you don’t control when exactly it is sent.

Other approach is to let it run until the first exception, signaling a real issue. This builds on the assumption that issue might leave a system in erratic state that invalidates any later actions. The main drawback is that the faulty state might not apply to the entire system, and you lose the opportunity to observe how it operates for a longer period or under larger data volumes. I imagine that for most systems, invalid data causes them to enter a partially broken but still functional state.

You can stop the execution after a number of loop iterations. The main problem is that actions are essentially random, and they exercise the system unevenly - a simple navigation between pages has different impact than creating a report that needs to be computed in background task. If you are unlucky, even a relatively large number of iterations might end up executing mostly vain actions.

You can stop the execution after a certain time. This has similar problem to counting loop iterations, but this time it’s longer actions that take undue part of the assigned budget. It might happen that a long session did relatively few actions, because all of them happened to take a lot of time.

Finally, you can try to introduce some system-specific stopping criteria. One way would be keeping separate track of low-impact and high-impact actions, and stopping when both are exceeded. Another would be keeping track of the number of objects created.

I decided to go with loop iterations counter. That was very easy to implement and at this stage I intended to run the tool exclusively on a local machine. I assumed that after a few sessions I will be in a better position to decide if things should stay that way, or change. I wasn’t too worried about placing uneven demands on the system during execution, as these should smooth out across many sessions.

Which actions should be available to the system?

At every step, HiVAT should do “a random thing”. But what does that mean? Are there any actions that should never be selected? How can a system know what actions are available?

In user interface, a single action may be broken down into multiple steps or interactions. Think about editing an existing object, like a user address. Usually user would have to click “Edit” button, which opens a form with address fields. After modifying the form user needs to click “Save” for data to be persisted in the system. Clicking “Cancel” discards changes.

There’s no general consensus how that should be modeled by the test framework. Some people opt to provide methods for modifying a form, and a separate method for submitting changes. But that makes test code more verbose and ties tests to a specific implementation (what if form changes are saved automatically in the new version?). So other people think there should be a single method that both modifies a form and submits changes. But then the framework loses the ability to test a scenario where form is modified and user decides to cancel changes. This trade-off is explained in depth in “Going Deeper into the Page Object Model” article by Blake Norrish. On a side note, highly recommended read.

The main problem is that the solution you might prefer from the point of view of a generic testing framework might not be the same solution that you would pick for autonomous software traversing program.

In my case, I decided to do both. Page objects provided separate methods for granular actions, like changing the form and clicking “Submit”. At the same time, they also provided more abstract methods representing user intent, like “modify the address”. Usually this more abstract method would just call a sequence of methods for more specific actions. While the traditional tests used abstract methods exclusively, the system I was developing was operating only on concrete methods.

The specifics of how the main loop may obtain available actions depend on the programming language and this is something that might be worth considering before writing a single line of code. You probably want a language with rich runtime introspection capabilities, so you can take a random object and just see what methods it has available. Dynamic dispatch seems to also be a must. Language that offers interpreted runtime with dynamic typing is going to have an edge over the language that requires compilation and static typing.

The problem was relatively easy to solve in Python with type hints, which was what I used. I’m not sure if I would be able to solve it in Rust, but that might say more about my Rust expertise than the language itself.

How can a system call a method that requires arguments?

Conceptually, you can consider this an implementation detail of a previous section. Methods and functions usually require arguments. How can a system know what arguments are required? How can it differentiate valid values from invalid, and handle them appropriately?

Answer to the first question really depends on a programming language and features it offers. Similar to the previous section, languages with rich introspection capabilities will generally make it much easier than ones without. Here, however, a very strict typing will prove easier to use than a system with dynamic typing. If you need to pass “something” to a function for it to work, it helps to know the exact definition of “something”. Practically speaking, you might need to create rich library of data transfer object classes that will model arguments taken by various methods across the system. It’s even better if library can quickly generate DTOs filled with data. In case of Python, using type hints is almost required.

The second question refers not only to type validation, but also a business validation. If you have a method to delete a user, the argument of that method is not as much a string, but a string that represents the name of a user that exists in the system. Proper autonomous system needs a way of knowing if given input will pass system validation or not, and how to handle the second case.

For simplicity, I decided to stick to valid data. That changes a problem a little, as now my tool needs a way of obtaining such data, and possibly creating it on demand. Usually that can be done by manipulating the system database directly, or using APIs to manage the data. In my case it was not that much of a problem, as testing framework already had API tests and ability to obtain and create data. I only needed to wire these modules to the code I was writing.

One obvious alternative is for HiVAT to keep track of data it created (or deleted), and use that whenever existing data is needed to perform a step. That quickly gets complicated if your data has dependencies - testing software might want to perform a step that depends on specific data, but it would not be possible if it didn’t happen to create that data first. This can be overcome with larger number of iterations in a single session.

This is one of the decision points that were mostly theoretical for me. I was extending an existing testing framework written in Python, and changing programming language was not up for debate. So I didn’t need to weigh the options carefully as much as I had to figure out how to solve this specific problem in Python. As I mentioned, I decided to use type hints liberally and used introspection capabilities built into the language.

How to handle actions history?

Should the history of actions be tracked? How? Where should it be stored? Should it be displayed? How should it look like, then? How can you make the history machine-readable, if that’s desired?

This is one of the few decision points where you don’t really have to choose, but instead you can have your cake and eat it too. You can both save all the actions in more human-readable form in a log file and keep machine-readable format in memory during a session. You can serialize data from memory at the end of the session, or only if there was an issue during execution. Contrary to most of other decision points, there’s no real reason why one choice here should prevent you from also doing another.

One of the most consequential choices is deciding where to store a list of actions. You can keep it in system memory, or you can persist it on a disk. The trade-offs are well known. If you decide to keep it in the memory, then you can lose everything if the entire system crashes. You also use the computer memory, making it unavailable to other processes, but that’s probably negligible unless you run billions of actions. If you save to disk, you might need to think about ways of reading it back.

If you keep the data in system memory, you should probably model it as some kind of structured information. Depending on the programming language, you might also have an option to keep references to existing classes, instances or functions. This would make it much easier to use them if needed. You can always easily turn that into human-readable form. The alternative - keeping human-readable strings in memory - appears to be all downside and I struggle to take it seriously.

It’s important to decide when the data is going to be displayed. I can think of four options: never, in real time (streaming, display action as soon as it is taken), at the end of the session and only in the case of failure. Displaying in real time and displaying in case of failure seem like main options to consider. The risk of displaying in case of failure is that a bug in HiVAT might prevent information from being displayed at all, which is especially problematic in case of critical failure of software under test. If you decide to display information in a streaming fashion, you might end up with a lot of information at the end of session where generally nothing noteworthy happened.

Finally, some actions might involve sensitive data, like private information or passwords. They probably should be protected, so they should be hidden or filtered out before being displayed. At the same time this is an information loss that would prevent fully reproducible session replays. If this is a concern, you should probably have different ways of storing and accessing human-readable and machine-readable data.

I decided to keep the history information in memory. Actions were modeled by a special class. A test runner would print action history only in the case of failure. When printed, actions were somewhere between human-readable and machine-readable. I did not go out of my way to make them easy for humans to read, but eventually I did not need to read them programmatically either.

Since there was no sensitive data that would require additional carefulness in handling, I did not implement any kind of filtering or hiding mechanism. However, this is something I did consider briefly.

Should the system allow for a control of certain sequences?

The HiVAT system I envisioned relied heavily on randomness, but it’s not hard to think about cases where it could be beneficial to limit it. Should every session be independent from all the previous ones and purely random? Or should there be a way to re-run the previous session? Should it be possible to constrain randomness of certain parts of the session? Which ones?

Some of these questions might make sense only in the context of an answer to a question posed in the previous section, and some of answers to these questions are self-evident from the answer given in the previous section. If you want the ability to replay the entire session, or any subset of it, then you need a way to store the history of all actions in machine-readable format on a persistent storage.

The ideal outcome is that if you have a session that finds an issue, you can re-run it on a brand new instance of software with empty database. That would be a significant step towards making all the issues reproducible.

And imagine you could have a system which takes a session that finds an issue and automatically reduces it to a minimal reproducible path. One session might last 15 minutes, but it doesn’t mean that everything that happened during that time is equally important. Probably most of things could be safely skipped, as there is relatively short subpath that guarantees the occurrence of the issue.

These two scenarios were main things I was thinking about when making that decision. It’s really a binary choice - should you design a system that allows to do these capabilities, or should you just give up on the entire thing? I haven’t spent too much time considering randomness of specific choices, like always picking up the same item from the list. I feel if I went down this path, there would be a lot of decisions to be made around communicating to the system which parts should be more reproducible.

While I have not implemented a replay system, and didn’t spend any time actually thinking how a critical path finder could be built, I decided that my system should be designed in a way that makes such changes in the future possible. In practice it meant identifying possible extension points and leaving around few hooks that might help later.

How should unexpected errors be handled?

It’s almost inevitable that system encounters an issue during a sufficiently long run, either due to a problem in a software under test, or due to a bug in the testing system itself. What should happen then?

This is something you might have covered when deciding about stopping criteria. One of the options back then was that the system should stop on the first error it can’t handle.

But if you decided to stop based on a different condition, now is the time to revisit that choice. Should your system also stop on an error? Or should it press forward, unreliability of later information be damned? Or should you design more elaborate error handling code?

First two choices were already discussed in an earlier section, so I’m going to focus only on the last one. Maybe your testing system can communicate with the software under test and revert it back to a known working state? Or maybe there are some actions that you know how to cancel or revert? Or maybe you can make decisions based on the nature of the issue and context of the action - e.g. transient network errors probably can be ignored or retried.

In my case, I decided to ignore some errors that I knew can be safely ignored, but make all the others fatal. That decision goes back to an earlier choice to only print actions history in case of error. If I decided to ignore errors and continue running, then I would have to find a way to still print history at the end of the run, or change that other decision and always print the entire history. But then it would make it harder to decide if any given session encountered issues or not.

Sample implementation

The HiVAT system I implemented for Quipucords project is available at GitHub. It’s only 25 lines of code!

However, that brevity was possible thanks to a number of helpers and design decisions made along the way. The UI framework that made HiVAT possible is introduced in Camayoc PR #365.

Lessons learned

The main lesson I learned was that my HiVAT was not a good fit for the project. In Quipucords, the most interesting part happens in the backend, when software interacts with the external world. Software that randomly walks the user interface and has limited ability to trigger interactions with the external world is not particularly helpful. I think this kind of tool would be much more applicable for more self-contained systems, like ecommerce, banking or social media.

Another important lesson is that some programming languages and design approaches can make non-deterministic testing much easier to implement. It definitely helps if you can make choices of technology and framework design while thinking about that particular use-case - it might be hard to bolt onto existing code.

The kind of HiVAT I created seems to be worth considering if your system meets two requirements:

It is stateful. This kind of testing requires application to live longer than a single request, ideally longer than any individual session. Most of backend code is like that. Frontend - standing alone - not necessarily. To give you an idea, this would not be a right approach to test a tool like grep, which needs to initialize every time it is started.
Most of the data comes from the user. There are hardly any pure systems like that - everything is interconnected. Online shops are pretty close (the inventory usually needs to be loaded from outside, but after initial setup, it’s mostly user actions). Banks are also pretty close - obviously there are bank-to-bank transfers, and most banks don’t allow to do everything online, but a large part of everyday operations can be modeled this way. Things like Evernote, JIRA or Trello are largely user-generated, but they also offer various integrations with 3rd party services.

Of course if your system does not meet these two requirements, it might still be worth to consider some other kind of HiVAT.

Thanks to James Thomas for his feedback and review of an earlier draft of this blog post.

Interesting bugs: Artifact building failure caused by GitHub API rate limits

2025-06-03T18:01:24+02:00

This is a story of a memorable bug I encountered at work. As with every story of that kind, there are a few ways to look at it: it might save you some time if you encounter a similar issue; it surfaces the hidden work - work that is not visible when you only look at the output and artifacts; it demonstrates the benefits of broad knowledge. I’m sharing it as a contribution to testing case library.

I was introducing a Jenkins pipeline to run for every pull request submitted to a project repository. One of the pipeline steps was to build a deployment artifact from the version of code with proposed changes. The artifact was a container image.

The application was following a standard architecture for web apps, with a separate backend and frontend communicating through an HTTP API. Each component had a separate repository hosted on GitHub and technically had an independent release policy that could result in version numbers that do not match. At the time, whole software was shipped as a single container image comprising both frontend and backend.

The container image was nominally a backend build artifact. One step in the container building procedure was to download the frontend files. The container build script recognized a special version “latest”, which caused script to ask GitHub API for latest releases and extract version of the newest one.

Not long after the pipeline was introduced, the team noticed that jobs sometimes fail at the container building step. Logs indicated GitHub responded with HTTP status 429 Too Many Requests. This happens when GitHub thinks that the client is exceeding the API rate limits.

However, the team was small and at the most active days created no more than a few pull requests. Even at the very unlucky timing, the combined pipeline runs from all pull requests should create no more than a dozen or two API requests in an hour. GitHub claimed to allow up to 60 requests per hour and it seemed extremely unlikely that we would routinely exceed that.

It should be noted that nobody in the team reported any problems with building containers locally. Developers were doing that for months already, often multiple times a day.

No matter how unlikely it seemed that our pipeline generated more than 60 requests per hour, that’s some clue - and one that is cheap to verify. I created a spreadsheet with one column to store the date of pipeline run, and another column to indicate if an error occurred or not. Assuming that rate limiting is the root cause, we would expect failures when there are several runs in the same one-hour window. The earlier ones would succeed, and later ones would fail. Runs that are far removed in time from other runs should never fail. Of course, this pattern was not found in the data.

Timestamps of pipeline runs and whether GitHub API responded with an error. Rows in green are consistent with the model of exceeding allotted API requests, rows in red are not consistent with the model.

The next step was to learn more about the GitHub API rate limits. When reading a documentation page on that topic, I learned there are multiple pools with different limits. 60 requests per hour is for unauthenticated users, grouped by IP address. Each authenticated user gets 5000 requests per hour.

At this point I need to share a bit of information about the wider infrastructure. This is subliminal context - something that you learn at some point when working in a company. It is not strictly related to your role and position, so it’s not something that you would think about constantly; but it’s a kind of background information you can draw from when thinking about issues that you face.

Jenkins controller was running inside a container deployed on OpenShift. Most of the work - including building deployment artifacts - was done on short-lived virtual machines provisioned on OpenStack. OpenShift and OpenStack operated from internal company network. The network was set up that outgoing connections to external world were allowed, but they all were routed through a limited number of proxy servers.

That allowed me to form a hypothesis - our pipeline is rate limited because from GitHub point of view, a sizable chunk of the company appears as a single IP address and is counted against a shared limit. It’s unlikely for our Jenkins agents to generate over 60 requests during normal operation, but all the development teams sharing OpenStack resources? I can see that happening.

This was not a research project, so I wasn’t exactly interested in refuting or corroborating that hypothesis. It’s enough that when stated like that, it provides clear steps towards solving the issue. If we authenticate our GitHub API requests, that should convince GitHub to count our requests against a different pool than the one shared with other OpenStack users. It will also increase our limit almost hundredfold, from 60 to 5000.

Requests may be authenticated by including a specific header with a token value. Token management, including ensuring they are not expired, is a separate topic I will not delve into here. In my case, the hardest part was working around the amount of layers that token value needs to be passed through. Jenkins created a virtual machine on OpenStack and provided GitHub API token value in environment variable. Pipeline at one point called make build-container, which is basically a shortcut for more extensive podman build command. Containerfile copied Makefile from backend project source directory to container build environment, and called make fetch-ui there. That in turn used curl to obtain data from the GitHub API. And this is where token needed to be used.

Illustration of number of wrappers around a command that actually connects with GitHub API servers. Some wrappers are skipped for brevity, like agent.jar executable.

Child processes in Linux by default inherit the environment from their parent, so the basic solution is to ensure a specific environment variable is set by Jenkins and all the layers below indeed recognize it. Two things complicate the picture here.

First, the container build environment drops most of the surrounding environment variables. This is in part to ensure consistent and reproducible build environment. Variables that you want to use when creating a container need to be explicitly opted in.

The second thing is worse. Environment variables present in the build stage end up embedded in the final container image. Setting an environment variable fundamentally shares a value with all the people who have access to the container image. When it comes to things like access tokens, this becomes an unacceptable security liability.

Luckily, the problem was identified in the last few years and the container build tools now support secret mounting. They are special files that are available to containers at the build stage, but are exempted from final container image. It means that at one point after creating a Jenkins agent virtual machine, but before issuing an API request, the environment variable content needs to be written to a file, and that file must be passed as a secret to container build environment.

The final touch was ensuring that everything still works when an environment variable with a GitHub token is not set. Developers never experienced the problem locally and they shouldn’t be forced to maintain and set up a GitHub token in their own environment.

The final version of the solution I implemented at the time can be viewed in GitHub PR.

After merging the PR, we no longer observed pipeline failures at the container image building step caused by GitHub API rate limits. It means that the proposed solution turned out to be correct.

Understanding Python web deployment

2025-05-22T18:48:09+02:00

Trying to deploy server-side Python web application may be overwhelming. Many tutorials will gladly tell you how to run it inside gunicorn, and how to put nginx in front of it, but they usually skim over details on why all this is needed. Why do you need both nginx and gunicorn? Why gunicorn and not Waitress or daphne? What is WhiteNoise and do you still need nginx if you have it? What even is WSGI?

In this article I will share my mental model of distinct pieces and how they fit together. If I badly missed a mark somewhere, please reach out and tell me how wrong I am.

This is one of those topics that really benefits when you go through it in non-linear fashion. If you are here to learn, you might want to skim over headings and read sections in different order, or re-read selected sections out of order after reading the whole thing.

The main model

Let’s start with a high-level model of handling HTTP request.

That model follows the basic HTTP request/response cycle. Server passively waits for connections, which are initiated by clients. Client composes and sends a request. Server generates a response and sends it back, at which point connection is closed. If client decides it needs anything more, it sends new HTTP request.

The real world is complicated by HTTP/2, websockets and probably other things. HTTP/2 allows server to group multiple responses when addressing a single request. Websockets only start with HTTP request/response before switching to a completely different protocol. Some of these complexities will become relevant later.

Accepting HTTP request

HTTP may seem simple. Client prepares a text message defining a request and sends it to a server. Server prepares a text message constituting a response and sends it back. This works well for text-based content, such as HTML, XML and JSON.

But there is so much more than that. The current iteration of HTTP standard has a separate document on caching alone, which is about 30 pages long. There are three different compression algorithms, each described in detail in separate documentation. There’s an optional section about partial access to large resources. Then that whole thing may be wrapped in encrypted frames. HTTPS also involves special procedure for the initial handshake and a fair deal of cryptography. And on top of all that, you need to decide what to do with clients that do not conform to the specification.

Then, it’s somewhat rare for a single physical machine to serve single domain (website). On one hand, one machine may serve multiple domains and there must be something that can decide how exactly a specific request directed at one of managed domains should be handled. On the other hand, modern web-based applications are usually distributed across multiple physical machines and there must be something that can choose a specific machine to handle particular incoming request. These days decision is often made based on the request content, analysis of recent traffic and knowledge of internal service infrastructure.

Unfortunately, the world out there is a dangerous place. Some invalid requests are actually malicious. What if client sends hundreds of thousands of requests? What if client opens a connection and never sends any data? What if client sends less data it claims it will? These sorts of questions do not come naturally from reading the specification alone, and answers have been hard won over the years.

As you can imagine, there’s a lot of complexity, depth and lessons learned in last 30 years of HTTP usage. Most application developers don’t want to deal with any of that - they would rather focus on simple request/response cycle and only dive deep into other areas as they are relevant to problems they face. They would happily outsource everything else to some other software.

That other software is usually called HTTP server, load balancer or router. Some popular programs in that problem space are nginx, traefik and Kubernetes ingress.

It’s worth noting that many web app hosting platforms that focus on ease of use have their own setup and do not give any choice in that matter. Part of value proposition of Heroku or Fly.io is that the first stage is taken care of by dedicated professionals, and you can jump straight into step two or even three.

Translating between HTTP and programming language

Thanks to HTTP server, you have a sequence of bytes that constitute a valid HTTP request directed at your application. Your job is to produce a sequence of bytes that will constitute a valid HTTP response.

You can write application that works on these primitives. Python’s built-in HTTP server is basically that. But for most application developers, this is too low level - they don’t want to work with sequences of bytes, they want native objects and a library of helpers that make it easy to parse a request and construct a response. Reading and setting a cookie should take one line of code each.

Translating between a sequences of bytes and some kind of internal programming language objects (and back from objects to sequence of bytes when processing a response) is the main goal of the second step in the main model. The name and specifics of the approach varies by a programming language. Usually that component is called “application server”, “middleware” or “translation framework”.

“Middleware” is overloaded term. The concept is also used in traefik, an HTTP server (step number 1), and Django, a Python web application development framework (step number 3). The world is a tangled web and depending on where you draw the lines and how much you are willing to squint, a lot of things are “in the middle” between some other things.

You can think of application server as a tool written in some programming language that helps developers working in that language to work with HTTP, by translating between byte sequences and programming language constructs. There are some exceptions, which I’ll cover briefly near the end of article.

Python - WSGI

In Python community, application servers usually are designed to run WSGI-compliant applications.

WSGI is an abstract specification, published as PEP-333 and PEP-3333. It’s like a contract - for application server, it defines what it must provide to application, and what it should expect back. For application, it defines what it must provide back to application server, and what it should expect from it. The idea is that you should be able to run your application on any application server - and application server authors may write code once and support all applications. It’s an example of adapter design pattern.

Popular application servers are gunicorn, Waitress and bjoern.

Ruby - Rack

Rack is the main application server for Ruby web applications. These days it is both a concrete implementation of application server software, and de facto specification of how Ruby applications should work with application servers. unicorn and Puma are independent projects capable of hosting “Rack applications”.

Java - Jakarta EE

Jakarta EE is a bit of overloaded term that may refer to either Jakarta EE software, or Jakarta EE specification. Jakarta EE software is concrete implementation of Jakarta EE specification. As far as I understand, the software was first and later the community formalized whatever it was doing into a specification.

So Jakarta EE software is a bit similar to Rack, as it’s a specific software you can use which became a de facto standard. Jakarta EE specification is a bit similar to WSGI, because it’s just a document - the idea is that you should be able to use any application server with any application, as long as both are compliant with that document.

Most notable application servers are Tomcat and WildFly.

Node.js - built-in

Node.js has HTTP server and helpers built in. That’s because Node.js is built on top of a browser JavaScript engine, and JavaScript in a browser needs to deal with HTTP requests all the time. Node.js, as a de facto JavaScript backend platform, already does the same thing as application server does for other languages.

Rust - compiled-in

Rust does not provide HTTP helpers in standard library, and there is no generally accepted application server specification. Instead, there are multiple HTTP server libraries that solve many of the problems that your application is likely to deal with across the entire model. When you run application that uses one of these libraries, it binds to a socket and reads and writes byte sequences directly. At runtime, a single Rust application will effectively merge steps number 2 and 3 of the main model.

Popular libraries are Actix Web, axum and Rocket.

Adding value

At this point HTTP request is in the form suitable for a programming language. This is the step where majority of web application developers spend most of their time. Everything up to this point was a groundwork - things that had to be done, but it makes little difference how exactly they are done. Now we are getting close to the place that differentiates your application from all the other applications in the world.

In practice, many problems at this point are still shared and there is a room for outsourcing them. The objects and structures exposed to a programming language may still be relatively low level and a bit awkward to work with. Application likely needs to connect to one of popular database engines. A large part of application might be available to authenticated users, so there is a need for authentication framework. There are probably various levels of permissions and roles that users might have.

These problems are solved by frameworks or libraries. They differ mainly in how many of these problems they solve, and how opinionated they are in solutions they provide. Some popular frameworks are Django (Python), Flask (Python), FastAPI (Python), Quart (Python), Ruby on Rails (Ruby), Grape (Ruby), Spring (Java), Grails (Java), Apache Wicket (Java), Vaadin (Java), Apache Struts (Java), Meteor (JavaScript) or Next.js (JavaScript). In general, communities of most popular languages came up with at least one web application framework.

Noteworthy exceptions and complications

WSGI and ASGI

ASGI is another specification for Python. It serves the same role as WSGI.

Why two specifications? There are two main reasons. First, WSGI is written around HTTP request/response loop, and as such is unable to support newer protocols like websockets. Second, WSGI supports only synchronous functions, while asynchronous code is increasingly popular since it was introduced in Python 3.5. Some application developers concluded that performance gains they receive from asynchronous code is worth more for them than WSGI compatibility. ASGI was created in response to these needs.

From the main model perspective, WSGI and ASGI are functionally the same thing.

Practically speaking, gunicorn is go-to WSGI application server and uvicorn is go-to ASGI application server. Django is compatible with both WSGI and ASGI, while Flask users can easily move to Quart, developed by the same team. Some newer frameworks, like FastAPI, are exclusive to ASGI. There is also at least one application server capable of serving either WSGI or ASGI applications.

WhiteNoise

WhiteNoise is a Python package for efficient static files serving. Static files are all files not generated dynamically by a Python program, such as stylesheets, frontend scripts, images and fonts.

It can act as a generic WSGI application wrapping around a target WSGI application. You can think of it as being somewhere between step number 2 and 3 of the main model. However, it also provides facilities to easily integrate with Django, and in such scenario it belongs firmly to step number 3.

The common knowledge of Python community is that static files should not be served by Python application, but instead should be handled by an HTTP server at the earlier stage of request/response cycle. WhiteNoise rejects that notion. It promises a high performance of serving static files while making sure they are correctly cached by CDN, proxies and client. It takes over one of responsibilities of an HTTP server.

Given all the things that HTTP server does, personally I would not be bold enough to skip it and open WhiteNoise-enabled application server directly to the world. But it might be worth considering in some special cases, like inside a container image or for internal company service.

Phusion Passenger

Phusion Passenger is application server notable for two main things. First, it’s an example of application server written in different language that target application - Phusion Passenger is written in C++ and initially hosted only applications written in Ruby. Second, it’s an example of application server compatible with multiple programming languages - newer versions can also host Python applications (compatible with WSGI) and Node.js applications.

Granian

Granian is application server written in Rust, capable of hosting both WSGI and ASGI Python applications. It’s notable because it’s an example of application server written in different language than target application, but also because it’s application server supporting both Python specifications.

Spring Boot

Spring Boot supports multiple deployment models. It can be built into a package compatible with the Jakarta EE specification, intended to run on application server. But there is also an option to build it into standalone HTTP server application that directly responds to incoming HTTP requests. That option is similar to how HTTP servers are built in Rust.

Where Rust fits for me

2025-05-05T17:45:36+02:00

I started eyeing Rust around mid-2023. My first program taught me it’s not a language you can just learn on the job - you have to spend some time on fundamentals or you are going to have a really bad time. I did my part in early 2024 - I have read The Rust Book, went through Rust By Practice and rustlings. Then I was looking for reasons to keep learning Rust, aside from the fun of it, potential job security and getting to understand programming at the deeper level.

Current Rust toolchain creates statically-linked binaries that can be run without any special runtime. That binary file is everything that you need to run a program on a specific CPU architecture and specific operating system. It’s forward compatible and very likely to continue to just work in years to come. I explored that bit in an article about binaries stability.

When you put things this way, Rust is almost the exact opposite of Python. Rust produces single binary, while Python programs tend to consist of multiple files. Rust does not need runtime environment, while Python not only needs Python binary, but also a virtual environment to store all the dependencies. And while Python as a language is stable and takes backwards compatibility seriously, the entire ecosystem is notorious to suffer from bit rot. You can’t take a random Python program from 2018 and assume everything will just work after pip install -r requirements.txt. It might, but that’s very hit and miss.

That makes Rust a very good choice for things that I expect to run in the future and I want them to just work then. Weeks or months might pass between invocations, but when the time comes, I want as little friction as possible. I want to run the thing, not to deal with uninstallable dependency. At the same time, the problem must be defined well enough that I can expect to want exact same solution in years to come.

That’s why I have chosen Rust for my ebook file renamer and a tool to generate text from template file.

These days, when I encounter a task to automate and I think about best programming language to use, I tend to favor three choices.

Shell. It’s really only good for setting up environment variables needed by other tools and occasional throwaway loop. I don’t bother with Bourne Shell compatibility any more, unless I have a strong reason to. I draw the line at functions - if I need functions, I’ll reach for something else.

Python. This is very much my default choice, as this is the language I am most comfortable in. However, backwards compatibility of ecosystem and bootstrapping the environment are major concerns. It’s a natural choice if I can get away with just a standard library or I can be sure all the required packages are already there in the system, usually pulled in by some system-wide programs installed through standard operating system packages. It’s also a good choice for one-off and ad hoc programs that I don’t expect to run in the future. Finally, it might be a good choice for programs that I work on regularly, like at least once a month - this way ecosystem problems don’t pile up and can be dealt with one at a time.

Rust. Kind of fallback for cases that don’t fit well in scenarios described above. Forward-compatible statically linked binary is a good choice for situations where I want a frictionless execution of a program in the future. It works best where problem is small enough to prepare a complete solution in couple of days.

pytest: running multiple tests with names that might contain spaces

2025-04-09T20:28:38+02:00

You most certainly know that you can run a single test in entire suite by passing the full path:

PRODUCT_ENV='stage' pytest -v --critical tests/test_mod.py::test_func[x1]

This gets old when you want to run around 3 or more tests. In that case, you might end up putting paths into a file and passing this file content as command arguments. You probably know that, too:

PRODUCT_ENV='stage' pytest -v --critical $(< /tmp/ci-failures.txt)

However, this will fail if your test has space in the name (probably as pytest parameter value). Shell still performs command arguments splitting on space.

To avoid this problem, use cat and xargs:

cat /tmp/ci-failures.txt |PRODUCT_ENV='stage' xargs -n 200 -d '\n' pytest -v --critical

I always thought that xargs runs the command for each line from stdin, so I would avoid it when command takes a long time to start. But turns out, xargs is a little more sophisticated - it can group input lines into subclasses and run a command once for each subclass.

-n 200 tells xargs to use no more than 200 items in subclass, effectively forcing it to run pytest command once. -d '\n' tells it to only delimit arguments on newline, removing any special meaning from space.

PRODUCT_ENV and any other environment variables must be set after the pipe character, or exported beforehand, because each part of shell pipeline is run in a separate subshell.

After writing this article, I learned that since pytest 8.2 (released in April 2024), you can achieve the same by asking pytest to parse a file for you:

PRODUCT_ENV='stage' pytest -v --critical @/tmp/ci-failures.txt

However, everything written above still stands for scenarios where any other shell command is used in place of pytest.

Interesting bugs: peculiar intermittent failure in testing pipeline

2025-03-26T19:22:30+01:00

Over the years I have encountered my share of memorable problems. They were remarkably complex, hard to debug, completely obvious in retrospect, or plain funny. This is the story of one of them.

At the beginning, there was a suite of automated tests that I was maintaining. One day one of them failed. Not a big deal, unfortunate reality is that some of them fail sometimes for various reasons. Usually they pass when run again and we can blame unreliable infrastructure, transient networking issue or misalignment of the stars. But few days later the same test failed again. And then again. It was clear that there’s something going on and this particular test is intermittently failing. I had to figure out what is happening and how can I make the test provide the same result reliably.

(Note the choice of words here. My goal was not to make the test passing, or “green”. There might as well have been a bug in the test itself, or in the product. At this point nobody knew. The main goal was understanding the issue and making sure test is reliably providing the same result - whether it is pass or fail.)

Before we move on, there’s some relevant context that I need to share. That suite contained only UI tests. Running them all took about an hour. They were running against staging environment few times a day. The test that was failing was responsible for checking a chart which plots the data from last 30 days. There were other tests verifying other charts, sometimes using different time spans. The website used the same generic chart component in all cases. These other tests never failed.

On a high level, the failing test consisted of three main steps: request the data from last 30 days using the API, read the data from the graph on the website, and compare both. Test was considered failed if there was any difference between the data from these two sources. Python deepdiff package was used for comparison. To make it possible, data from API was locally transformed to mimic the structure returned by function responsible for reading the data from UI.

Testing infrastructure had few distinct pieces. There was a Jenkins server that triggered a test suite run at certain times of the day. Job executors were containers in a Kubernetes cluster. To facilitate UI testing, there was a Selenium Grid server with few workers hosted as virtual machines on OpenStack. Tests were running against staging environment of the product, which was also hosted on a Kubernetes cluster, but different than the one where job executors were. I believe all that was scattered across two data centers, with most of testing infrastructure being co-located, and product under test being elsewhere.

Not necessarily accurate illustration of infrastructure.

Now, let’s get back to the story.

The very first thing I did was looking into test logs. Unfortunately, differences between objects as reported by deepdiff in this particular case are not easy to read (see below). The amount of data is overwhelming, and displaying everything in single line contributes to the challenge. The log made it clear that lists returned by API and read from UI are different, but it was not immediately obvious where exactly these differences are.

>       assert not deepdiff.DeepDiff(expected_graph_data, actual_graph_data)
E       assert not {'values_changed': {"root[0]['Date']": {'new_value': '1970-01-01', 'old_value': '1970-01-02'}, "root[0]['Foo']": {'new_value': 46, 'old_value': 23}, "root[0]['Bar']": {'new_value': 60, 'old_value': 99}, "root[0]['Total']": {'new_value': 106, 'old_value': 122}, "root[1]['Date']": {'new_value': '1970-01-02', 'old_value': '1970-01-03'}, "root[1]['Foo']": {'new_value': 23, 'old_value': 26}, "root[1]['Bar']": {'new_value': 99, 'old_value': 92}, "root[1]['Total']": {'new_value': 122, 'old_value': 118}, "root[2]['Date']": {'new_value': '1970-01-03', 'old_value': '1970-01-04'}, "root[2]['Foo']": {'new_value': 26, 'old_value': 49}, "root[2]['Bar']": {'new_value': 92, 'old_value': 86}, "root[2]['Total']": {'new_value': 118, 'old_value': 135}, "root[3]['Date']": {'new_value': '1970-01-04', 'old_value': '1970-01-05'}, "root[3]['Foo']": {'new_value': 49, 'old_value': 68}, "root[3]['Bar']": {'new_value': 86, 'old_value': 60}, "root[3]['Total']": {'new_value': 135, 'old_value': 128}, "root[4]['Date']": {'new_value': '1970-01-05', 'old_value': '1970-01-06'}, "root[4]['Foo']": {'new_value': 68, 'old_value': 33}, "root[4]['Bar']": {'new_value': 60, 'old_value': 14}, "root[4]['Total']": {'new_value...ue': 25}, "root[24]['Bar']": {'new_value': 29, 'old_value': 78}, "root[24]['Total']": {'new_value': 106, 'old_value': 103}, "root[25]['Date']": {'new_value': '1970-01-26', 'old_value': '1970-01-27'}, "root[25]['Foo']": {'new_value': 25, 'old_value': 57}, "root[25]['Bar']": {'new_value': 78, 'old_value': 84}, "root[25]['Total']": {'new_value': 103, 'old_value': 141}, "root[26]['Date']": {'new_value': '1970-01-27', 'old_value': '1970-01-28'}, "root[26]['Foo']": {'new_value': 57, 'old_value': 48}, "root[26]['Bar']": {'new_value': 84, 'old_value': 18}, "root[26]['Total']": {'new_value': 141, 'old_value': 66}, "root[27]['Date']": {'new_value': '1970-01-28', 'old_value': '1970-01-29'}, "root[27]['Foo']": {'new_value': 48, 'old_value': 89}, "root[27]['Bar']": {'new_value': 18, 'old_value': 14}, "root[27]['Total']": {'new_value': 66, 'old_value': 103}, "root[28]['Date']": {'new_value': '1970-01-29', 'old_value': '1970-01-30'}, "root[28]['Foo']": {'new_value': 89, 'old_value': 61}, "root[28]['Bar']": {'new_value': 14, 'old_value': 66}, "root[28]['Total']": {'new_value': 103, 'old_value': 127}}, 'iterable_item_added': {'root[29]': {'Date': '1970-01-30', 'Foo': 61, 'Bar': 66, 'Total': 127}}}

Trying to understand this log felt daunting, so my next step was running the failing test locally, in isolation. Predictably, it passed. I didn’t have the high hopes that I will be able to reproduce the problem right away, but that was a cheap thing to try, so I think it was worth giving a shot.

At this point I decided there is no way around it and I have to better understand how API and UI responses are different. I copied the log line into editor and inserted a new line character after each },. Few more changes later I had a form that was a little easier to decipher.

Deepdiff shows the differences between elements under the same index in lists. But focusing on elements with the same date value revealed that they are fundamentally the same. Values appearing under “old_value” in one list appears as “new_value” in the other list, just under different index. I have put colored overlay on the screenshot below to make it easier to see. You can think of these lists as mostly the same, but one is shifted when compared to other; or you can say that one list has extra element added at the end, while the other has extra element added at the very beginning. Specifically, API read data from January 2nd to February 1st, but UI displayed data from January 1st to January 31st. There’s a large overlap, but deepdiff output obscured this key insight.

Deepdiff output after editing. Color overlays shows that both lists have the same data, but in different places.

At this point I had an idea what is wrong, but I had no clue why, and why it would affect only this one single test. So in the next step I wanted to see if there are any patterns to the failure. I grabbed test results from last few weeks and put them in the spreadsheet. I added columns for basic things, like the result itself, how long did it take for test to finish, date and time when test was run. To make failing tests visually distinct, I added background color to highlight them. In separate column I tagged all rows where test was running for a first time in a given day. Then I added columns representing known issues that we encountered in previous few weeks, to see if all failures fall into one of them.

While there wasn’t a clear and predictable pattern, I did notice a curious thing - if the test failed, it would fail in the first run of a given day. Subsequent runs of any day never failed. And the first run in a day always started shortly after midnight UTC.

That allowed me to construct a working hypothesis: the issue is somehow related to time and there’s only a short window when it may occur, maybe up to few hours. That window is located around midnight UTC. Such hypothesis explains why subsequent pipeline runs never failed, and why I was never successful at reproducing the issue locally - I am located east of UTC line and I would have to try running the test way outside of working hours. Of course I didn’t know if I was up to something or I was just creating complex ad hoc hypothesis that fits the data. But it directed my next step.

To corroborate the hypothesis I needed some new information, things I didn’t have before. To gather it, I have added further logging in the test suite. First, I have used Selenium JavaScript execution capability to obtain the date and time as the browser “sees” it. Then I have done the same from Python, which both drives Selenium and requests data from API. The important part is that Python code is executed directly on test runner (container in Kubernetes) and JavaScript code is executed in the browser (Selenium Grid VM on OpenStack).

diff --git package/tests/ui/test_failing.py package/tests/ui/test_failing.py
index 1234567..abcdef0 100644
--- package/tests/ui/test_failing.py
+++ package/tests/ui/test_failing.py
@@ -10,6 +10,13 @@ def test_failing_test(user_app, some_fixture):
     """
     view = navigate_to(user_app.some_app, "SomeViewName")
+    browser_time_string = view.browser.execute_script("return new Date().toTimeString()")
+    browser_utc_string = view.browser.execute_script("return new Date().toUTCString()")
+    view.logger.info(
+        "[JavaScript] Time right now: %s ; UTC time: %s",
+        browser_time_string,
+        browser_utc_string,
+    )
     expected_x_axis = get_xaxis_values()
     view.items.item_select(some_value)
     view.graph.wait_displayed()
diff --git package/utils/utils.py package/utils/utils.py
index 1234567..abcdef0 100644
--- package/utils/utils.py
+++ package/utils/utils.py
@@ -10,6 +10,14 @@ METRIC_MAP = {


 def _get_dates_range(some_param="Daily", date=None):
+    current_time = arrow.now()
+    log.info(
+        "[Python] Time right now: %s ; TZ name: %s ; TZ offset: %s ; UTC time: %s",
+        current_time,
+        current_time.strftime("%Z"),
+        current_time.strftime("%z"),
+        arrow.utcnow(),
+    )
     try:
         date = arrow.get(date)
     except TypeError:

With the above patch applied and deployed, all I needed to do was waiting for the next failure. I hoped that new logs would reveal some more information once it fails again.

That turned out to be true. JavaScript showed a date one day earlier than Python. In fact, the time in JavaScript was about 15 minutes earlier than in Python. So if test suite ran around midnight, and we got to offending test within 15 minutes of suite start, then Python would request data through API for some dates, but website in browser would think it is still the previous day, and request different set of dates. It means that the window where issue occurs is extremely small - just around 15 minutes each day.

[JavaScript] Time right now: Thu Jan 01 1970 23:58:17 GMT+0000 (Coordinated Universal Time) ; UTC time: Thu, 01 Jan 1970 23:58:17GMT
[Python] Time right now: 1970-01-02T00:14:36.042473+00:00 ; TZ name: UTC ; TZ offset: +0000 ; UTC time: 1970-01-02T00:14:36.042473+00:00

This concludes the main part of the debugging story - at this point we knew what is wrong, we knew that failure is not caused by a bug in a test or a product, and it was clear that the solution is for all machines involved in testing to reconcile date and time. It also seemed like the JavaScript shows wrong date, which might mean that the issue is with Selenium Grid machines or OpenStack instance.

I connected to all Selenium Grid machines using SSH and checked their local time using date command. They were about 15 minutes behind their wall-clock time. I assumed the difference is caused by various OpenStack and underlying infrastructure maintenance work, so I just used hwclock to force OS clock to synchronize with hardware clock and moved on with my day.

Couple of days later I connected to these machines again and noticed that the local time is behind again, but only by about 15 seconds. It looked like the local clock is drifting by about 5 seconds a day. It might not sound like much, but it also meant that it’s only a matter of time before original issue happens again. Clearly someone logging in to these machines every once in a while and resetting clock would not be a good long term solution - we needed something that can automatically keep time synchronized.

That something is called NTP and all the machines already had chrony installed. However, it didn’t seem to work correctly. While the commands succeeded and logs did not show any problems, the clock would just not change. After few frustrating hours I think I ruled out all possible causes at the operating system level and came to the conclusion that perhaps the NTP traffic to public servers is blocked by data center firewall. I reached out to OpenStack administrators for help and they told me that there is a blessed NTP server instance inside the data center that I should use. Once I configured chrony to use it as a source, everything finally worked.

This way browsers started to consistently report the same time as Python executors. That fixed the original issue and we did not observe any test failures caused by it again.

Django: data modified in pre_save signal is not saved

2025-03-12T19:09:14+01:00

When I was working on my first Django project, I encountered a problem where modifications done inside pre_save signal were not stored in the database. Back then it took me a couple of hours to figure out what is wrong and how to fix it. I’m sharing what I’ve learned to save that time someone else.

Imagine the application to track your gym workouts. At the core of application is exercise session, which knows how many repetitions you want to do and how many you have done so far. Once you do enough repetitions, session is considered complete. It could be implemented like that:

class ExerciseSession(models.Model):
    target_repetitions = models.IntegerField(null=False, blank=False)
    repetitions = models.IntegerField(null=False, blank=False, default=0)
    completed = models.BooleanField(null=False, blank=False, default=False)

Since “session is complete once you do enough repetitions” is a hard rule, it needs to be enforced. pre_save signal sounds like reasonable place to implement it, as it allows us to have many functions that modify repetitions. Signal handler might look like that:

@receiver(pre_save, sender=models.ExerciseSession)
def exercise_session_pre_save(sender, instance, raw, using, update_fields, **kwargs):
    if instance.repetitions >= instance.target_repetitions:
        instance.completed = True

With the above setup we can modify the model instance and rest assured that exercise session is marked as completed when needed:

exercise_session.repetitions = 10
exercise_session.save(update_fields=["repetitions"])

Except that… this doesn’t work. Once you retrieve this exercise session from database, completed will be set to False.

The reason for this is at intersection of pre_save signal and update_fields argument. pre_save is called early in save() call and may be used to modify the instance before data is saved in database. However, SQL query constructed later in save() call will contain only the fields listed in update_fields argument.

So when calling a save(), you need to predict that pre_save signal might modify the instance, and include possibly modified fields in update_fields list:

exercise_session.save(update_fields=["repetitions", "completed"])

The complete code used in this blog post is available in GitHub repository. See the README file for instructions how to set up and run the project.

Customizing fonts’ look with OpenType features

2025-03-02T19:36:10+01:00

Many programming fonts allow customization through OpenType features. For people who are picky about certain properties, like easily distinguishable l and I (lowercase L and uppercase i), a single toggle can make a difference between rejecting a font and starting to use it. Unfortunately, choosing the perfect font with the perfect setup is way harder than it should be. This post describes the tools that may help, including the one that I wrote recently.

Whenever I stumble upon a new font, I want to know:

how does the font generally look?
does the font allow customization of specific glyphs that bother me?
when set up properly, how does the font fare against my current setup?

But first, how do you even learn about a new font? Two proven resources are programmingfonts.org and codingfont.com. That topic is also frequently brought up on social media like Reddit in communities discussing editors, terminals and programming in general.

Let’s say that JetBrains Mono caught my attention. The demo page is nice enough, but does not give an option to type in custom text. I am not a fan of ligatures for code (the ones they say “reduces noise”), while I appreciate ligatures that “balance whitespace” - I wonder if I can disable one category? Also, I prefer slashed zero.

A few years ago, that would have been the end of it for me. The font looks nice, but there are some details I do not like. Today, I know that many of these details can often be customized through OpenType font features. But how can you learn what features the font supports, and what exactly do they change?

The best tools that I have found so far are web based: Wakamai Fondue and Bulletproof Font Tester. Other similar tools are Axis-Praxis and FontDrop!. All of them allow you to select a font file from local computer to explore its capabilities. Usually you want to select a file with “medium” or “regular” in name.

There are a few things that I like about Wakamai Fondue. It tries to display all characters affected by a given setting, so you can learn that in JetBrains Mono, “Character Variants 04” affects how letter “j” is displayed. You can toggle a setting on and off, making it easier to notice changes. It also gives you the internal name of OpenType font feature, so you know that slashed zero is accessed through a feature called zero. This becomes relevant later, when you try to apply settings on your machine.

There are also a few things that I like about Bulletproof Font Tester. It has a large collection of texts, including cases specifically designed to show each letter or common issues. It’s easier to see how specific set of font features apply to the entire text. It remembers all the fonts you have selected during the session, making comparisons easier. It also provides interface to modify text layout properties, although I never use these.

The main thing I don’t like is that there’s no way to save your settings. It’s not possible to pause your exploration and resume it another day. If you close a tab, you need to start from scratch. Comparing a new font to the one you are currently using involves setting it up in the tool - possibly multiple times. I have not found a good way to compare multiple fonts to each other. It’s also not easy to compare effect of two different sets of features on a single font.

I decided to use recent Day of Learning at Red Hat to see if I can do anything about it. The result is called Font Feature Tester and you can find it on GitHub.

At the very core is a simple command line tool written in Rust that reads a font configuration file and generates an image of provided text set with that font.

There’s also a Python script that automates the execution of the Rust tool across multiple configuration files and collects all the output images in a single directory. It’s responsible for ensuring that images are consistent and use the same text, image size and colors.

Using web tools mentioned earlier, I can find a set of font features that look promising. Then I prepare a configuration file for each font I want to try - at the very least, that would be my current font and font I consider switching to. Then I run a Python script to generate images.

I can use a standard image viewer to see how the text looks. When I want to compare multiple fonts, I can open a few images at once and put them side by side.

Images generated by my tool. I would keep one as a reference, and cycle through images in the other

That approach scales to any number of fonts and feature sets - you only need to add a new configuration file. Since all the data is written in files and somewhat automated, you can stop at any time and resume later.

One major gap is that you can’t easily move your setup between all the tools. Once you find features you like, it’s up to you to apply the same configuration in your terminal emulator, editor and desktop environment. At the end of the day, the ergonomics of my simple tool depend a lot on ergonomics of your desktop environment and image viewer.

Vision for the future

Font Feature Tester is a barely minimal viable product. It solves the very specific part of the larger problem, and manages to do that with support from other tools. The integration leaves a lot to desire. I can envision it growing into much more complete solution, so I thought it might be fun to finish this post by role playing a product manager. What is the end state of things? How can we get there?

I imagine the final version looking something like that:

Each tab would display text set in specific font with specific font features enabled. It should be easy to move between tabs using keyboard shortcuts. There should be an option to disable a tab without closing it - so you can temporarily remove a specific setup from consideration, but you can also bring it back if you change your mind. Alternatively, there could be a history of closed tabs.

Options on the left side would allow to modify the current text setup. Obviously there needs to be a way to change font and font features, text properties and text content. One of the challenges is that some of these settings sometimes apply to all tabs, while other times should apply only to current tab. It’s valid to want to increase font size for all tabs, but it’s also valid to change font size for current tab only.

It’s not clear to me how to handle a case where you want to compare images side by side. Some kind of split view is needed, possibly with tab locking.

The first step on the road towards that goal is introducing the graphical interface. For me, it makes sense to focus on single tab - just a view of image and options in the sidebar. Options can be added iteratively.

Some options may be improved over time. Font selection may start as a simple text field where the path to font is pasted. Later it may be changed to use file selection widget. Finally it can be integrated with something like fontconfig to also allow selecting fonts from these that are already installed. The same is true for OpenType font features - initially, it may be a text box, but later it should display a list of all font features available in the font, and provide a way to set a value for each feature. While most fonts support simple toggles, there are some fonts that support one of a few values for selected features. Interface should acknowledge that.

All the changes should be saved in configuration file, so program may be closed at any time. Ideally, current configuration file format would be reused, so the existing workflow could be supported while user interface is being developed.

Once the single font view is finished, it may be wrapped in a tabbed interface. At that point, the old Python tool-based workflow may be removed.

At one point, it would be great to ensure that the application works on Windows and macOS.

I don’t know how far on that journey I will go, especially since creating desktop applications is not something I find particularly appealing. But if this is something you find interesting, feel free to reach out to me!

Testing a new feature - should you start with automation or by inspecting it yourself?

2025-02-11T19:56:12+01:00

A fellow engineer submitted a question to internal mailing list. The gist is: a new feature is supposed to be released in a few months and there is no automation coverage for it. What should a person with a Quality Software Engineer title do? Should they test the feature “manually” and contribute automation later, risking it happens after a release? Or should they start writing automation right away, potentially postponing finding and reporting important issues?

My answer is below.

There are no simple recommendations and go-to answers that are going to be usually correct. It all depends.

First and foremost, what is the state of the feature in general? Are we still at the ideation phase, is any code already developed, does developer consider it done, did anyone else test it already?

Second, where does that feature fit in entire product? Is this completely new thing, or extension to something that product already does? Is this on the critical path of product usage? Did any customers ask for that, or did we internally decide it’s a good idea? Is it going to be marked as tech preview in release notes or not?

Third, what is the social context of your team? Are you the only Software Quality Engineer, or is there a larger team? What is the workload for you and others? Who develops unit tests? And are there unit tests for this feature?

Fourth, how does that feature fit your automation suite? Can you easily extend existing framework to account for the new thing, or do you need to create completely new capabilities first? How much time do you think it’s going to take?

As a tester, your main job is to provide accurate information to the team. Ideally, you would do that sooner rather than later. You should be able to identify risks and start the conversation on what the team is going to do with them. If you can, propose solutions and advocate for them. If you can and have time for that, implement solutions yourself.

Personally, I always start with understanding of the feature. What is the goal? How is it achieved? What workflow do we envision? Where are “hot” points where workflow can fail (i.e. due to unmet dependency)? Depending on the size of the feature, I would spend at least few hours ensuring that my understanding is correct and verifying my assumptions. More often than not, I will encounter questions that I can’t find answers for, and if I am (un)lucky - that nobody thought about before. I will try to do that without looking at the implementation, to not guide my thinking and to avoid biases.

Then - or while doing above - I will create a list of test ideas. I will keep it very lightweight, maybe a sentence for each idea. Usually it’s a mix of things I want to verify and things I want to find answers for.

At this point I will start thinking about the most effective ways of performing these tests and finding answers. Sometimes I will use tools to help me; sometimes these are going to be tools I create myself. At this point I don’t think about maintainability, performance, error handling etc. - it’s too early for that. Sometimes I will look into unit tests (if there are any) and see if the answer is already there. It may happen that I find unit test for a thing close enough to one I was interested in. In such case I will leverage it. That means modifying the test or creating new one based on one that I found. I might try to submit a PR with this new or modified test, so it becomes a part of unit test suite, but usually it’s not my first priority.

As I go through my list, usually I encounter things that are interesting, puzzling or noteworthy for other reasons. Often they trigger new test ideas that I did not think about earlier. I will perform them right away or add to the list, to check later.

When I decide I have seen enough, it’s time to think what tests seem important enough to include in future regression runs. It’s the act of balancing sometimes conflicting constraints - what value does the test bring in the great scheme of things? How costly it is to develop? How fast can it run? Where is the right place to run it? Usually only a fraction of test ideas are going to make it into regression suite.

Finally, obviously, I need to implement these regression tests. I will start with most important first and look for ways that other people could help with that.

So, to answer your question: neither of the two. Look for things that matter. Identify real problems and focus on letting others know about them quickly. Use tools to aid you, but remember that tool is only here to support you. Don’t repeat the work that developers have already done. See if you can leverage the work of other team members.

Understanding Linux virtualization stack

2024-11-12T23:37:46+01:00

I find Linux virtualization stack confusing. KVM? libvirt? QEMU? Xen? What does that even mean?

This post is my attempt at making sense of that all. I don’t claim it to be correct, but that’s the way I understand it. If I badly missed a mark somewhere, please reach out and tell me how wrong I am.

Virtualization primer

Virtual machine is a box inside which programs think they are running on different hardware and operating system. That box is like a full computer, running inside a computer. Virtualization is process of creating these boxes. Virtual machine is called guest, while operating system that runs virtual machine is called host.

Two main reasons to virtualize are security and resource management. Security - because if virtual machine is done correctly, then program inside it won’t even know it’s inside a virtual machine, and that means there’s a good chance it won’t escape to interfere with other programs. Resource management - so you can buy huge server and assign slice of resources to each box as you see fit, and dynamically change it as your needs shift.

Virtual machine pretends to be the entire computer, but most discussions revolve around CPU. Each CPU architecture has a specific set of instructions, and programs generally target only one of them. It is technically possible to translate instructions from one architecture to another, but that’s usually slow. Most modern CPUs provide extensions that allow virtual machines to run at speed comparable to non-virtualized installations. Using these extensions is only possible when the guest and the host target the same CPU architecture.

Conceptually, there is no significant difference between virtualization and emulation. Practically, emulation usually refers to guest thinking it has different CPU architecture than host, and to virtual machine pretending to be some very specific physical hardware - like a video game console. Virtualization in turn usually refers to specific case where guest thinks it has the same CPU architecture as the host.

KVM

KVM is part of the kernel, responsible for talking with hardware. As I mentioned before, most modern CPUs have special extensions that allow guests to achieve near-native performance. KVM makes it possible for Linux host to use these extensions.

QEMU

QEMU is weird, because it’s multiple different things.

First, it can run virtual machines and pass instructions from guest to KVM, where they will be handled by virtualization CPU extension. So you can think about QEMU as user space component for KVM.

Second, it can run virtual machines while emulating different CPU architecture. So you can think about QEMU as user space virtualization solution, which can run without KVM at all.

Third, it can run programs targeting one CPU architecture on a computer with another CPU architecture. It’s like lightweight virtualization, where only single program is virtualized.

Finally, it is set of standalone utilities related to virtualization, but not directly tied to anything in particular. Most of these programs work with disk image files in some way.

QEMU is probably the hardest part to understand, because it can do things that other parts of the stack are responsible for. This results in some overlap between different components and encountering program names in contexts where you would not expect them.

I think this overlap is mostly a historical artifact. The oldest commit in QEMU git repository is dated 2003, while KVM was included in Linux kernel in 2007. So it seems to me that QEMU was originally intended as software virtualization solution that did not depend on any hardware or kernel driver. Later Linux gained drivers for virtualization CPU extensions, but they were useless without something in user space that could work with them. So instead of creating completely new thing, QEMU was extended to work with KVM.

Technically, the journey towards understanding Linux virtualization stack could end here - you can use QEMU to create, start, stop and modify virtual machines. But QEMU commands tend to be verbose and long, so people rarely use it directly.

libvirt

I have ignored that so far, but KVM is not the only virtualization driver in kernel. There are many, some of them closed-source.

libvirt is intended as unified layer that abstracts virtual machine management for application developers. So if you would like your application to create, start, stop or delete virtual machine, you don’t have to support each of these operations on KVM, QEMU, Xen and VirtualBox - you can just support libvirt and trust it will do the right thing.

libvirt doesn’t do any work directly, and instead asks other programs to do it. System administrator is responsible for setting up the host and ensuring virtualization may work. This makes libvirt great for application developers, who can forget about details of different solutions; but most of libvirt users are in less fortunate position, because they have to take a role of system administrator.

libvirt will talk with QEMU, which in turn may pass CPU instructions to KVM. Some sources online claim that libvirt can talk with KVM directly. libvirt itself perpetuates this confusion, as KVM is listed alongside QEMU on the main page, giving the impression they are two equivalent components. But detailed documentation page makes it clear - libvirt only talks with QEMU. It can detect KVM and allow user to create “hardware accelerated guests”, but that case is still handled through QEMU.

It’s also worth noting that libvirt supports multiple virtualization drivers across multiple operating systems, including FreeBSD and macOS.

virsh

virsh is a command line interface for libvirt. That’s it, there’s nothing more to it. If it was created today, it might have been called libvirtctl.

Vagrant

Vagrant abstracts virtual machine management across operating systems. It’s similar to libvirt, in the way that it allows application developers to target Vagrant and leave all the details to someone else. And just like libvirt, Vagrant doesn’t do anything on its own, but pushes all the work to tools lower in the stack.

Vagrant primary audience is software developers. It allows to succinctly define multiple virtual machines, and then start and provision them all with a single command. These machines are usually considered disposable - they might be deleted and re-created multiple times a day. If you want to manually modify the virtual machine and keep these changes for longer period of time, Vagrant is probably not a tool for you.

On Linux, Vagrant usually works with libvirt, but may also work with VirtualBox or Xen.

VirtualBox

VirtualBox is full virtualization solution. It consists of kernel driver and user space programs, including one with graphical interface. You can think of VirtualBox as something parallel to all that we’ve discussed so far.

The main reason to use VirtualBox is ease of use. It offers simpler mental model for virtualization - you just install VirtualBox, start VirtualBox UI and create and run virtual machines. While it does require a kernel module, and kernel is notoriously bad at breaking modules that are not in tree, VirtualBox has a lot of tooling and integrations that will ensure that module is built every time you install or update kernel image. On distributions like Ubuntu you don’t have to think about this at all.

Technically speaking, libvirt supports VirtualBox, so you can use libvirt and virsh to manage VirtualBox machines. In practice that seems to be rare.

Xen

Xen is another virtualization stack. You can think about it as something parallel to KVM/QEMU/libvirt and VirtualBox. What sets Xen apart is that instead of running inside your operating system, it runs below your operating system.

Xen manages scheduling, interrupts and timers - that is, it manages who gets access to computer resources, when, and for how long. Xen is the very first thing that starts when you boot your computer.

But Xen is not an operating system. So right after starting, it starts the guest with one. That guest is special, because it provides drivers to hardware and is the only one with a privilege to talk with Xen. Only through that special guest virtual machine you can create other virtual machines, assign them resources etc.

Xen is one of the oldest virtualization solutions, with first release back in 2003. However, it was included in kernel only in 2010. So while many people prefer Xen and think favorably of its architecture, it seems to lost the popularity contest to KVM and QEMU.

Xen provides the tools required to manage guest virtual machines, but you can also manage them through libvirt.

Summary

I thought to put a visual aid in place of summary. Each column represents a possible stack - you generally want to pick one of them. Items with dashed line style are optional - you can use them, but don’t have to. Click to see the bigger image.

“Software Testing Strategies” by Heusser and Larsen - review

2024-10-22T19:54:29+02:00

This is the review of a book “Software Testing Strategies. A testing guide for the 2020s” by Matthew Heusser and Michael Larsen. The book was published in December 2023 by Packt. You can buy the ebook or printed copy directly from the publisher, at Amazon and perhaps a number of other places.

I admit I was a bit on the fence since I first heard about the book. The title sounds very promising and I know authors were involved in AST and context-driven testing. But the price seems a bit high in this part of the world. Eventually I took the plunge and bought a copy in March. I finished reading after 5 or 6 weeks. I wrote this review in April.

Things that I liked

The book is full of stories, examples and experience reports. You can easily tell that authors have actual experience with things that they talk about. This is a welcome change from a number of books and blog posts that are superficial and read as if authors had only theoretical knowledge of a thing they cover, if any. Overall, that makes for an easier reading and adds credibility. I especially appreciate that authors did not shy away from talking about their mistakes, blunders and errors. Too many experience reports are vanity in disguise.

Case studies in chapter 12 are wonderful. Testing community needs so much more of these. I dream that one day we might have a testing-focused case library. I hope to write about this more in the future, and I would be more than happy to join a group that is working towards that goal. But until these dreams and hopes come true, chapter 12 is one of the closest things available to international audience.

There were things I have learned. Pairwise testing sounds like something I should have known, but somehow I missed it so far. Model-driven testing is something I encountered recently, but it was called by a different name. It also has a surprising connection to high-volume automation I have implemented in one of the projects.

I found it thought-provoking to introduce two new schools when discussing the schools of testing in chapter 15 - “The Agile school” and “The DevOps or CD school”. The idea of testing schools themselves is a bit dated, caused a lot of controversy at the time and is now largely rejected by the community. So I’m not sure if it’s a good thing to include it in the book, but I do appreciate the effort to modernize it.

I really enjoyed the description of testing work in the very first chapter. It did an excellent job at capturing the thrill of exploration and the joy of learning something new. Even though I am fortunate enough to enjoy these feelings most of the time at my daily job, I still found that part inspiring. It has this rare and precious effect that it makes you want to put the book down and go test something.

Things I did not like

I don’t think the title suits the content well. I have expected something similar to “Design Patterns” by the Gang of Four - a list of established and emerging strategies, each with a description, probably an example, and a summary that says when a strategy can be applied beneficially, and when it’s better avoided. This book is not that.

The book is all over the place. It touches a huge number of topics, but dives deep into only a handful of them. Sometimes it’s hard to understand why certain things were included and how they relate to everything else. This makes it challenging to decide which topics are worth studying more and which can be ignored for now.

It’s unclear to me what level of experience is the reader assumed to have. For example, hexagonal architecture is introduced as early as chapter 3. But this is relatively advanced topic from software architecture area - something that most testers don’t really encounter, and developers start to think about as they prepare to move to a senior role, or later. The way it’s being described might make you think this is how most software looks like internally, which could not be further from truth.

The choice of software covered is baffling at times. Chapter 5 mentions couple of continuous integration systems. Somehow CircleCI and Travis CI were included, but GitHub Actions was not. The same chapter recommends Apache OpenOffice, which at this point of time is straight negligence. Apache OpenOffice is abandonware that barely manages to release a few bugfixes once a year or so. All the community has moved on to LibreOffice, which sees 2 major releases every year and monthly bugfix releases. If you want to recommend free and open source office suite, LibreOffice is the only one in town.

There are occasional mistakes and typos in the text. Duplicated word in one place, sentence beginning by lowercase letter in another. These are mostly negligible. The one exception is in the chapter 13: “division by zero and interpreted code crashes tend to show up as 400 errors”. I think this is a typo, but one that novices will not catch and instead learn the wrong thing. 400 errors are client-side errors; division by zero and crashes in interpreted code are most likely going to show up as 500 errors, server-side problems. I don’t really blame the authors for this and other mistakes, but it’s clear that the editor assigned by the publisher could do a better job. That’s unfortunate.

Conclusion

I want to like this book. I really do. But I am clearly not a target audience. So I’m thinking: to whom could I recommend it?

I think this might be a good book for someone early in their career, probably someone with 6 months up to a few years of experience. To readers with less experience I would recommend “Perfect Software” by Jerry Weinberg instead. More experienced folks might find “Lessons Learned in Software Testing” by Kaner, Bach and Pettichord to be more valuable. But there is a sweet spot between these two prominent works that this book seems to fill.

Perhaps this book can also help people who have been in testing for a very long time, settled a bit, feel they no longer keep up with the Joneses and want to change that. They won’t learn ins and outs of everything in the current landscape of software development, but will get the useful overview. They will encounter all the most important ideas, and some buzzwords too.

That might be a bit of a stretch, but I guess this can also be an alright book for people who consider a career in IT, and are looking for single book that would give them the general overview of how the work looks like today. Ironically, these people might want to skip some of the parts that discuss testing in depth.

AST Board of Directors elections are on!

2024-07-08T21:38:06+02:00

The Association for Software Testing yearly election process has started. We are looking for three people to join the Board.

Official announcement on AST site has all the details, including dates and formal requirements.

If you aren’t sure if you would be a good fit, you have any questions about the Board, Association or the elections, or you are looking for someone from within the organization to support your nomination, I am more than happy to help - just reach out using the usual channels.

And if you can’t run for election because you are not a member yet, now is the perfect time to join - just in time to meet formal requirements for the next year. Head on to “Become a Member” page on AST website.

How can I possibly test “all the stuff” every iteration?

2024-06-20T22:49:44+02:00

The Association for Software Testing is crowd-sourcing a book, Navigating the World as a Context-Driven Tester. The book is edited by Lee Hawkins, who posts questions on Mastodon, LinkedIn, BlueSky, Twitter, Slack, and the AST mailing list. You don’t have to be a member to give your answer to the most recent prompt, “How can I possibly test »all the stuff« every iteration?”. Mine is below.

Depends on what you mean by “all the stuff”.

You might mean the full regression testing of entire application, including all the features, negative paths, edge cases etc. And you are right - you can’t possibly do all these things every iteration, maybe except for some very small products.

But the thing is: you shouldn’t do this kind of comprehensive testing in every iteration, you shouldn’t feel the need to do it, and you shouldn’t be asked to do it.

The iteration is the concept from the agile software development world - in the model where software is developed for extended period of time and delivered once, at the end, the word “iteration” has no meaning. The whole point of iterating over software during the development is to introduce small, but valuable change and deliver it to the users. Since the change is small, most of the product is the same as it was and therefore does not need to undergo comprehensive verification again. It is assumed that it was already tested in the past.

But maybe changes have broader scope than it looks like, or they introduce unintended consequences in other parts of the program. These are fair concerns. However, most changes introduced by developers don’t impact the entire system and you should not assume by default that they might. It definitely pays off to identify code with potentially large impact and make informed decision about scope of testing needed during the iteration.

It’s worth adding that if the team routinely finds itself in the situation that changes ripple through the entire system and comprehensive testing is necessary, then that’s a sign of deeper issues to tackle. It might mean that the architecture of the system is wrong and the team should seriously consider paying off the technical debt. Or maybe the software is more of a prototype, proof of concept, than the solution. In that case, perhaps comprehensive testing is not needed until requirements are better understood by the team and stakeholders can green-light one approach.

Many testers struggle with working in this kind of environment. They see their role as a last line of defense between a possibly buggy software and its users. The idea that users might get a software that was not comprehensively tested and verified make them feel uneasy.

If you are one of them, you need to understand that this kind of risk is calculated and intrinsic to software development model that values short iteration and frequent delivery. These ideas reinforce each other. The product is not comprehensively tested, which means there might be bugs, mistakes and problems that won’t be caught before users encounter them. But because the product is not comprehensively tested, it can be released faster. When the team learns about problems in the software, they can be fixed and a new release may be made.

If this argument does not convince you, I would urge to just give it a try. Acknowledge that you personally are not sold to this approach, but don’t block it if this is what the team wants to do. Keep your reservations to yourself for few releases and see what happens. You might learn that the product is not, in fact, significantly lower quality than before. And if it turns out the quality has lowered, then you will have hard empirical data to show to the team. Most people will agree these results are undesirable and some kind of change is needed.

Finally, you should not be asked to do comprehensive testing during each iteration. Personally, I don’t have much experience in that regard - I have never been in a team that would attempt that. While I can’t say what things might work in such situation, I guess the first thing to do is understanding the position such request is coming from. Perhaps stakeholders need to be taught the tradeoffs of iterations; or perhaps the team needs to adopt to more cautious software development models at the cost of speed.

I have not mentioned automation yet, which contributes to all of the points so far. Your work as a tester should be supported by automation that runs at least once a day and delivers up-to-date and reliable information about the state of the product. While automation has limits and can’t check everything, it can check some things. You should understand what automation can and can’t do in your project, so you can make an informed decisions about which features and paths you don’t need to spend much time on. You should feel confident in automation results. And when someone asks you to repeatedly perform specific tests, you might use that as opportunity to discuss spending some time on improving automation instead.

At the beginning of this article I said that the answer depends on the meaning of “all the stuff”. It might mean comprehensive testing, which I focused on so far. But it might also mean all the things that have changed during the iteration.

It does happen there’s so much work done by developers that testers in the team are swamped. I was in that situation not too long ago, and what did work for us was pushing for whole team ownership. Most of development tasks don’t need to be tested or verified by a dedicated tester - they only need to be tested by someone who did not write the code. In our case, all we had to do was signalling the problem during standup meeting, and rest of the team quickly came up with a solution - that some of developers should stop taking new work and instead focus on testing things that are already done. Having clearly spelled out definition of done and acceptance criteria for each ticket definitely helps.

However, the best way to deal with that problem is to actively avoid it. In general, as a tester, try to test changes as soon as possible. Don’t wait until the end of the iteration. If the team practices PR-based workflows, you can try testing the development build created before the code is merged. It helps to have the CI system that builds artifacts for each PR, but alternatively you can set up the build environment on your own machine. For exceptionally large features, give feedback to developer before code is written, and encourage early sharing of code and ideas. Daily standup, if there is one, is a good opportunity to show desire of this kind of conversations. You can also consider reaching out to developer and discuss things in private.

It might happen that despite using all the suggestions above, testers are still routinely swamped with work at the end of the sprint. In my experience, this is more likely to happen in teams that try to strictly follow rigid scrum structure, with iterations taking a week or two. Since work is assigned at the beginning of the sprint, there is a period of time when developers are focused on their work, and testers don’t have many things to do. That dynamic changes towards the end of the sprint. In that case, I would highly encourage to bring this up during the retrospective meeting. Most of the people will agree that this is a systematic problem that is not sustainable in the long term. As usual with self-organizing teams, it’s up to the team to come up with a solution. Just remember to monitor the situation and raise this topic again if there is no visible improvement.

Association for Software Testing Board of Directors report - April 2024

2024-05-02T23:56:19+02:00

In my Association for Software Testing 2023 elections proceedings I advocated for transparency. In the spirit of practising what I preach, this blog post summarizes my work for AST in April 2024.

Found a way to send messages to BlueSky and Mastodon from Zapier.
Met with Chris and Gwen to discuss remaining items from Phone.com migration (see January report). I hope to finally close that in May.
Removed old, no longer used passwords from shared LastPass folders.

Integration of Zapier with BlueSky and Mastodon deserve a bit longer explanation.

There are two sources of our automatic messages - we periodically re-publish posts and videos from our archives, and we forward blog posts from our members. For a longest time, we published that on Facebook, LinkedIn and Twitter. Last year Twitter announced they will require paid subscription to access API, which in turn rendered most existing automations useless. We had to turn off ours.

BlueSky and Mastodon quickly rose to prominence as possible Twitter replacements, so we created new accounts there. However, Zapier did not support them natively. In fact, it still does not.

But after a bit of focused searching, I was able to learn that Zapier also supports arbitrary code execution jobs, at least as long as code is JavaScript or Python, and follows certain rules. What’s more, I was able to find scripts created by other people. After adjusting them a bit, they ran perfectly.

Integrating Zapier with BlueSky and Mastodon was one of the things I wanted to tackle since I joined a Board, and it makes me really happy that I was able to finally complete this.

If you are interested in becoming a better tester, advancing the practice of software testing or just want to support my work, consider subscribing to AST newsletter and becoming a member. If you have any questions, ideas or would like to work together, you can email me at mirekdlugosz %at% associationforsoftwaretesting.org or get in touch through other channels.

Rust binaries stability

2024-04-27T22:21:27+02:00

Recently, I’ve been thinking about Rust stability. Most web search results are discussing stability of the language itself, especially in the context of taking old Rust code and compiling it with recent toolchain. This is obviously important, but not the kind of stability that I had in mind. What I wanted to know is: “can I take my old Rust executable and use it on recent Linux?”

When I publish this post, Rust toolchain is at version 1.77.2. Rust itself was created in 2006, made publicly available in 2010 and saw first stable release (1.0) in 2015. Obviously I am not interested in anything built before 2015 - you can’t take bleeding-edge programming language and expect stability. Also, it’s not like I would be using toolchain that old today.

Skimming through the documentation, it’s easy to notice nightly APIs that you have to explicitly opt-in to use. While I heard some of them remain in unchanged form for years, it’s clear to me that you can’t expect stability when you use API marked as not stable.

The typical Rust program will compile to mostly statically linked binary, with GNU libc linked dynamically. This is not the full story, as you can use musl instead, which also supports static linking, and apparently it is possible to avoid any C library if you try really hard. I am not really interested in these cases, especially since they seem to target somewhat special needs, including embedded systems.

For a typical Rust binary, the long-term stability will depend on two components that are not part of Rust at all: kernel and glibc. Linux takes long-term stability very seriously and is well known for not breaking the user space, so it’s not really a concern.

I don’t know much about GNU libc, but I found the article discussing techniques employed to ensure backwards compatibility. This gives me some confidence that they take long-term stability similarly to Linux. When I think about this, about the only case of glibc breaking backwards compatibility I heard of is related to anti-cheat software used by games, and as far as I understand, this was dubious from the very beginning.

It all indicates that Rust executables should be pretty stable. But as a tester, I know experience beats the theory every time. And I already have a modern Linux, so if only I could find some old Rust executables, I could see for myself if they still work or not.

This is easy enough to do with GitHub search. I searched for projects written in Rust that were created before 2018. There’s nothing special about 2018 - it’s just a label for “old”. Then for every project it found I checked if that’s an application (and not a library or documentation), if it does releases through GitHub, and what is the oldest release with binary files attached. I repeated that until I decided I have enough. I’m not trying to be thorough, I want to find an answer in less than an hour.

I don’t have any reason to assume these applications are malicious, but since these are random executables from the Internet, I wanted to separate them from the rest of the system. Podman helped me with that. I have used fedora:38 image, only because my host is Fedora 38, and I assumed this way I should minimize potential compatibility problems, if any.

I don’t really know how to use these programs, and it was not my goal to learn. I just tried to run each executable with --help and --version flags, hoping they are recognized. I expected to see something if program still works, and get obvious error message if it does not.

Out of 9 projects tested, 8 appear to still work. The oldest one was from September 2016. The table below shows detailed results. Name of the application is link to the specific version that I tested.

Application	Version	Date	Works
ripgrep	0.0.2	2016-09-06	✅ Yes
watchexec	1.0.0	2016-10-15	✅Yes
rip	0.8.3	2016-10-18	✅Yes
just	v0.2.23	2016-11-24	✅Yes
wstunnel	1.0	2017-12-07	✅Yes
Firecracker	v.0.1.0	2018-03-05	✅ / ❌It’s complicated
RustScan	1.0.1	2020-07-19	✅ / ❌It’s complicated
cicada	v0.9.29	2022-02-24	✅Yes
swc	v1.2.24	2020-09-08	❌No
swc	v1.3.105	2024-01-21	✅Yes

I count Firecracker and RustScan as working, but they deserve a bit of explanation. They both showed errors on Fedora, but appear to work on Debian. Errors were “failed to initialize syslog: InvalidFd” (Firecracker) and “./rustscan: cannot execute: required file not found” (RustScan). For me they sound like missing runtime dependencies, and I consider them fundamentally the same as forgetting to specify a mandatory positional argument. But they also show something I did not consider initially - that long term stability might depend on Linux distribution as well.

On the other hand, I count swc as the only one that is not working. That’s because the oldest version, v1.2.24, crashed with a segmentation fault. That was also a case for three later versions I have tried. The first version that appears to work is v1.3.105, released earlier this year. I don’t know what this application is doing, but it seems to be something unusual.

To sum up: if you search the web trying to learn if Rust executables are stable in the long term, then the answer seems to be “yes”.

Why don’t we replace the testers with AI? Postscript

2024-04-16T20:05:21+02:00

When researching my previous article, I stumbled upon an essay by Dave Karpf titled “Two Failure Modes of Emerging Technologies”.

The first failure mode is when the technology is actually successful, finds itself used outside of planned context and that causes a whole swath of unintended consequences. Second - technology fails to deliver what it promised, but still finds itself widely used and incorporated into critical systems. Everyone learns to works around problems, or worse - stops to see there are problems to begin with.

Most of AI critique falls closer to the first mode of failure. My article is no different - my main argument is that AI will not get as good as it needs to be to replace testers anytime soon.

But if I were taking bets, my money would be on AI following the second mode of failure - it will not live up to it’s hype, and still get widely deployed.

It’s unfortunate I did not spend more time reflecting on this.

Why don’t we replace the testers with AI?

2024-04-16T20:04:12+02:00

The Association for Software Testing is crowd-sourcing a book, Navigating the World as a Context-Driven Tester. The book is edited by Lee Hawkins, who posts questions on Mastodon, LinkedIn, BlueSky, Twitter, Slack, and the AST mailing list. You don’t have to be a member to give your answer to the most recent prompt, “Why don’t we replace the testers with AI?”. Mine is below.

We can’t replace testers with AI because, as of April 2024, there’s no AI capable of doing all the things that testers are doing.

Now, that’s the answer that misses the point of the question. Obviously, the intent is: assuming AI develops rapidly and may get to the point where it is capable of performing some / most / all of the things that testers do, are there any reasons to not replace testers with AI?

That “assuming” is doing some heavy lifting here. Unfortunately, AI discourse is ridden with hype, daydreamers and straight-up con artists. I think it’s important to try to tip the scales by highlighting the problems that need to be solved. Let’s dive into what exactly “assuming” here means.

We assume that AI of the future will not hallucinate. “Hallucination” in the context of AI is a marketing term used for situations where AI system outputs statements that are not true. This is one of the most widely known problems, as it takes only a few minutes of interacting with AI bot for it to say things that are demonstrably false. This is also a heavily researched topic. There is some research showing that “hallucinations” might be inevitable in the current breed of AIs. For all we know, no company has been able to solve that yet - OpenAI, Microsoft, Google and everyone else just add an increasing number of layers around their models, hoping that users encounter canned response earlier than a problematic one.

We assume that we have not yet reached the limit of what current AI technology can do. OpenAI takes a position that all the problems in AI space right now are resource-bound. What they say is that we just need a little larger models and little faster computers; what they imply is that we already have all the theory needed to build any AI we might need. If you go to any computer science or psychology department in the world, I’m not sure if you will find a single person that shares that belief.

We assume that if there are any problems that are fundamentally qualitative in their nature (i.e. won’t be solved by a little faster computer), we will solve them. The tricky thing about unknown unknowns is that you don’t know about them. And the thing about breakthroughs is that you don’t know when they are going to happen, if at all. Hilbert’s problems are a good exemplification - some of the problems are still unresolved after decades of work, while others were solved soon after their posting. When Hilbert announced them, it was impossible to predict which are going to be which.

We assume there are sustainable business models that can support AI. All current AI products are developed and run at the loss. This is not a necessarily a bad thing - every research takes resources before it turns into something that can generate profit, and most of companies have to lose money before they can earn enough to sustain themselves (this is what “investment” means). However, right now we still don’t know what kind of problems AI is going to be good at solving, and many of AI leaders seem to be fundamentally disinterested in engaging with this question.

We assume that AI natural resource usage can be lowered and kept at manageable level. AI models are very resource-intensive, especially during the training. Data centers are well-known sponges for electricity and water. As I’m writing these words, we are in the streak of 10 record warm months, and judging by the first half of April, it’s going to be at least 11 record warm months. The climate change is the biggest challenge that humanity faces right now. The nations are finally starting to create and enforce laws that attempt to limit the natural resources usage, many companies pledge to achieve net-zero emissions within a next decade (including Microsoft and Google), and customers are actually taking environmental costs into consideration in their purchasing decisions.

We assume that AI can be trained on legal material, and AI output will not create additional legal risks for users. Creative workers were first to be hit by AI work, and many were not happy to learn that these systems are trained on their pieces without explicit consent. New York Times is in legal battle against OpenAI on the ground of copyright infringement. This, and many more lawsuits in the future will establish the limits, legal rules and total costs of AI usage and training. Smarter companies are holding off wide deployments of AI tools until someone else fights these legal battles and establishes precedent. Actual creators of these tools count themselves among smarter companies.

We assume that general quality of the work AI tools produce is going to be acceptable. Every time I see anyone pitching an AI tool, I have a single question - if the tool is so great, why don’t you use it yourself? You saw Microsoft talking a lot about purported time-savings coming from using GitHub Copilot, but you never saw Microsoft talking about using Copilot for Windows, Office or Visual Studio development. Makes you wonder why.

All of above make me think that “AI taking testers jobs” might not be a problem that we are going to have anytime soon.

30 Days of AI in Testing experience report

2024-04-02T21:49:26+02:00

In March, Ministry of Testing held 30 Days of AI in Testing challenge. I participated, and post my general impressions below.

If you have never heard of “$x days of $something” challenges, the idea is simple - there is some kind of prompt posted every day, and your goal is to write a short piece in response. You publish the work on a blog, dedicated forum or on social media. Prompts might include a task to complete, and these tasks will vary from very simple to somewhat complex. Some days majority of your time will be spent on the task itself, not writing.

I have heard about challenges like that years ago, but this was my first time attempting it.

One thing that I was definitely not prepared for is the pace. To be honest, it was absolutely exhausting. I started on a high note, fully motivated to write something every single day. At 9th day there was a technical glitch that caused a daily prompt to be published much later than usual, and I skipped that day due to scheduling problems. Only then I realized how badly I needed the break, and that 30 days straight are not going to be sustainable for me. I ended up skipping 5 days.

Finding the right rhythm and schedule was somewhat challenging, too. Prompts were published at the forum at 1 AM my local time, and arrived by email at around 11 AM. I usually checked the forum before going to sleep, to give myself some time to think about the reply. Then I would try to kick off the day by writing a reply as one of the first things in the morning. I found it much harder to fit writing in my afternoon / evening schedule. On the other hand, composing a reply could easily get out of control, creeping into my working hours. The solution I applied is not revolutionary - after few days I started to timebox myself much more aggressively. This resulted in omitting some points I thought about raising, but couldn’t carry out in satisfactory manner in reasonable time.

I participated primarily to see how others are using AI in their day-to-day work, or hope to use it in the future. The challenge definitely did push me into taking a closer look at various tools and getting some hands-on experience with them. I was able to try them in tasks that I wouldn’t come up with on my own. That gave me some idea what these tools might be good at, and where they definitely need some more work.

At the same time, I wouldn’t say it changed my overall approach to AI. I haven’t seen anything that would change my mind about the problems pullulating in the entire space, including output quality, copyright, privacy, data security and natural resource usage.

Few of the tasks involved using chatbots to handle some generic testing tasks - coming up with test ideas, generating test data, creating test code or explaining existing code. I have used Microsoft Bing Copilot for these. Every time I made a point to give it a problem that I have already faced in the past, so I can compare the output with some baseline reference material. There were some surprises, when Copilot did things I didn’t think about myself. But almost without a fail, Copilot would also omit something that I did cover. For me, that tips the balance against using LLMs for most of the tasks - I might be slower, but I produce higher quality work, gain experience and acquire some new domain-specific knowledge. These are things that are going to pay off manifold in the future. My current opinion is that by using AI systems you might be faster, but you miss opportunities to learn and upskill yourself.

One of the reasons to take part was to build connections and interact with other members. I don’t think it worked for me. Doing a task and writing a response was challenging and time consuming already, I just wasn’t able to put conversations with others on top of that. These kinds of interactions were not facilitated by the forum software, which displays a long list of messages one under another. It was especially problematic in the early days where each thread saw hundreds of responses. To be fair, I did saw more people responding to other folks later on, when activity has toned down a bit.

Unexpected benefit of taking part in the challenge was that it forced me to write. A lot. In total, I have written a bit over 8500 words (51 000 characters). For comparison, all the blog posts published so far on this website total to 42 500 words and a bit over 280 000 characters. To put things in perspective, in these 30 days I have written about 20% of what I’ve published here in 8 and a half years.

Especially in early days, the challenge kind of unlocked something in me, and I became much more prolific writer than ever before. Apart from forum posts, I published two articles on this blog, wrote notes, multiple emails and became much more active on LinkedIn or Mastodon. I even have two blog posts written that are not published yet - something that I haven’t experienced since 2012 or so. My only worry is this change might not be permanent and will fade as quickly as it appeared; that after short period of high productivity I will succumb to levels even lower than before.

The final observation I’m going to share is that Ministry of Testing forums were rather unusable in the first half of the month. While the goal of challenge is to do a small task every day, it’s only natural that not everyone can do that. It’s encouraged to write a reply to previous prompt at later date - better to do things late than to drop out. But as a result, all threads from recent days constantly received new replies. The default view of the forum, where threads are listed by time of last reply, was constantly swamped by challenge threads. Things improved after two or three weeks, when initial excitement faded and only most persistent people remained. Most of them responded daily, so previous days saw little activity and unrelated threads were easier to notice.

Honestly, I don’t know if I will ever participate in similar challenge in the future. I think it will require a topic that is highly interesting for me. If I do participate, I will definitely not go in with the goal of writing something every single day. I imagine I will try to respond to third, maybe half of all prompts. I will also give myself permission to slack and skip days from the get-go. I will definitely timebox my daily participation aggressively from the very first day. Perhaps I will focus exclusively on hands-on experience and completing the tasks, while skipping the writing part. Another approach I might try is treating that more as a networking event, focus on reading what others are publishing, ask clarifying or supplementary questions and encourage them to keep going.

I thought I’ll finish with some numbers. I have participated in 25 days, which means I have skipped 5 days. In total, I have written 51 562 characters, which gives a bit over 2000 characters per post. For reference, average article on this blog is about 6000 characters, and this article is about 7700 characters.

The chart below shows the number of replies to each day prompt, as of April 1st. Since one person could write multiple posts in each thread, this is at best a proxy variable for much more interesting survival analysis. Nonetheless, the pattern is pretty clear.

Don’t blindly serve WebP format

2024-03-12T17:57:46+01:00

If you have done any webdev work in last few years, you must have heard about WebP. It’s an image format that promises up to 34% smaller file sizes without noticeable quality downgrade. It’s pretty much universally supported since late 2020.

With smaller file sizes and widespread support, you might think it’s a good idea to just serve all your images in WebP. Or, if you want to be extra backward-compatible - serve WebP to all browsers that claim to support it, and original image to remaining few.

I also thought it’s a good idea, and made a switch on this very website. I’m creating WebP with a compression factor set to 80. And then I noticed that one file is actually larger after conversion.

Following this thread, I compared size of all WebP images with their original counterparts. Turned out WebP produced larger files in 3% (4 out of 117). In worst case, 20 kB PNG file turned into 122 kB WebP - over sixfold increase in size!

Since then, when I generate WebP, I compare file size to original and keep it only if new format produces smaller file. This way browsers will always receive the smallest file I can produce, regardless of the format.

I guess the main takeaway here is ages old “measure before optimizing”.

What’s the best format for a test plan?

2024-03-04T10:15:57+01:00

The Association for Software Testing is crowd-sourcing a book, Navigating the World as a Context-Driven Tester. The book is edited by Lee Hawkins, who posts questions on Twitter, LinkedIn, Mastodon, Slack, and the AST mailing list. You don’t have to be a member to give your answer to the most recent prompt, “What’s the best format for a test plan?”. Mine is below.

It should come as no surprise that there is no single “best” format.

Each document that we create should be for someone and should respond to some real or perceived need. Otherwise, why bother writing it?

I imagine perspective on planning changes as you move up in corporate hierarchy. For directors and executives it’s all about expected resource utilization and budgeting. I think they want to know how many people to hire, how many devices they need to buy, how much their cloud costs are going to go up; and what they can skip or cut out while still meeting the strategic objectives. I wouldn’t know, I have never had this kind of job.

As an individual contributor, my perspective is different. When I am asked to create a test plan, my first thought is “who am I writing this for?” I might not need it, and I definitely don’t need it in this very heavy form exemplified by templates floating around the web. Other testers in my team often don’t need it either - they are just glad they didn’t have to write it. Developers in my team don’t care. Other testers in adjacent teams are usually busy with their own things. My manager might be happy to see it, and he might or might not forward it up the chain, but I never heard back from as close as skip-level manager.

Usually after test plan is written, there’s a request for review and comments sent to a number of people. In the best case I will get few questions pointing out parts that could be clarified, or some suggestions for things that I have missed. I appreciate them. This helps me to be more thorough and allows others to better understand what is happening in other parts of the project.

However, I doubt if test plan is the most effective way of achieving these goals. While everything that test plans usually cover is important, I feel whatever I put in document should be the conclusion of conversation, not the beginning of it.

Let’s say I think we will need a team of three people to complete all testing activities in given time, while there’s only me and one colleague. Putting that somewhere deep in test plan will not make a new person appear out from nowhere. In fact, I am much better off talking with my manager about this - because either we need to arrange for new person right now, or we need to highlight to other stakeholders that certain things are at risk because there are not enough people.

The same applies to any hardware and software that might be needed for testing. Most companies have processes for obtaining these, which I would have to follow independently of any test plan document. If I have any doubts on whether my request is going to be accepted, I’d better talk with my manager instead of hoping he will notice this crucial detail somewhere in a test plan that I send him.

This is also true for test ideas that I might put in test plan. I am happy to share them with developers - the best thing that can happen is that I bring their attention to a thing they have not thought about, and they will fix the bugs before any code is subjected to testing. But I haven’t seen any evidence that a long, mostly irrelevant document is the best way to start that conversation.

In fact, I consider thinking about test ideas to be the most valuable part of test planning. Usually I would try to create a list of things I think would be worth testing, based on any resources I have available at the time. Often while writing things down I will realize I’m not exactly sure what is supposed to happen - which means that either I didn’t fully understood what’s going on, or I found a problem with specification. Sometimes I will also write down questions I will want to find answers for.

That list of test ideas is going to be my starting point for actual testing. However, not all test ideas need to be performed repeatedly, and not all of them are worth automating - which leaves open the question if they should be included in any kind of formal test plan.

And even if I did include the whole list in a test plan, considering it to be complete would be a grave mistake. I assume I will add more items later. Some ideas might come to me while doing other things, some might be prompted by conversations that are going to happen as project evolves, and some I will create in response to the actual behavior of developed software. When I test, there are always things that pique my interest, and more often than not they are worth following - no matter if they are included in test plan or not.

I think all the above points, and then some, are now widely understood in the industry. That’s why test plans are not so ubiquitous as they once were. Personally, I can count on one hand the number of times I’ve been asked to create a test plan in a last few years. The software I work on is in continuous development and sees many relatively small releases. Large things, if they happen, take months to develop, not years. We practice backlog refinements, three amigos meetings and collectively write acceptance criteria, which often are a lightweight form of test plan. Along with the other testers we are in the same team as developers, closely watch progress on new features and get a chance to interact with early versions of the software. I participate in code reviews and suggest improvements to unit tests, which allows me to share test ideas.

We get many of the same benefits that test plan could provide, but in less formal way and without spending much time writing it. However, if I were asked to write a test plan, I would start by reading “What Should A Test Plan Contain?” by Michael Bolton.

Association for Software Testing Board of Directors report - February 2024

2024-03-01T21:38:26+01:00

In my Association for Software Testing 2023 elections proceedings I advocated for transparency. In the spirit of practising what I preach, this blog post summarizes my work for AST in February 2024.

Fixed a problem where email sent to Group is not distributed to all members of that Group. Google support was not particularly helpful, although I must note that there was actual human involved, and she did try. A fix involved changing the way our Wordpress site is sending emails. Documented everything internally.
Pushed Phone.com migration (see previous report). Wrote down all cost saving opportunities and action items, assigned these to other Board members. Not all items are resolved yet.
Fixed a BBST Graduates list page. There were some strange things in HTML that caused Wordpress to parse it badly, and in turn it displayed all wrong.
Fixed a problem of periodic posts on LinkedIn displaying black box instead of YouTube video preview.
Updated internal dictionary / documentation of shared email accounts
Monthly Board of Directors meeting.

Association for Software Testing Board of Directors report - January 2024

2024-02-01T22:08:38+01:00

In my Association for Software Testing 2023 elections proceedings I advocated for transparency. In the spirit of practising what I preach, this blog post summarizes my work for AST in January 2024.

Implemented DNS changes required for VeryConnect (software running our members portal)
Did about 75% of Phone.com migration. We want to stop using phone number from them and someone has to log in to all our accounts, check what phone number is set in profile and change it when necessary. While doing that I have identified some action items for other Board members and cost saving opportunities, which I need to write down.
Created runbook for accessing shared email accounts despite Google security mechanisms. As usual, this is internal only.
Started a discussion with Google Customer service for a problem where email sent to Group is not distributed to all members of that Group.
Monthly Board of Directors meeting.

You might have noticed there was no report for December. That’s because we have kicked off a month with quarterly Board of Directors meeting, which is much longer and much more demanding that monthly ones. With pretty busy end of the year at work and festive season, unfortunately I did not contribute anything to AST that month.

When the build is green, the product is of sufficient quality to release

2024-01-25T23:08:31+01:00

The Association for Software Testing is crowd-sourcing a book, Navigating the World as a Context-Driven Tester. The book is edited by Lee Hawkins, who posts questions on Twitter, LinkedIn, Mastodon, Slack, and the AST mailing list. You don’t have to be a member to give your answer to the most recent prompt, “When the build is green, the product is of sufficient quality to release”. Mine is below.

No, not really. At least not in any general sense. But you are free to choose to release based on single metric. If this is what you want, I would advise you to work on creating an environment that allows for this choice to succeed. I would also assume that you are aware of risks that this option entails, and you have considered other options as well.

If you want to release a software, you need a release strategy. You don’t have to call it like that and you don’t have to write it down, but it makes sense to spend few hours thinking about it. Here are some questions you should be able to answer:

How many releases do you want to make? When you work under contract, there might be set number of releases that you will be able to make. Some specialized software is being used only once - think about space rocket controls, or software used during sports events like Olympics.
How often do you want to release? Assuming there are going to be multiple releases, is there a specific frequency of releases that you want to maintain? Or are you going to release when you feel you are ready?
Are you able to fix discovered problems after the release? Especially if you don’t plan multiple releases, would you be able to release fixes for critical problems discovered afterwards?
What are the risks of releasing software with unfixed problems? This greatly depends on the nature of these problems, but think also about your reputation and competition.
What are the risks of holding a release to fix problems? This is mostly about all the things accompanying and related to release, e.g. marketing campaign that started well before the planned release date. Can you move things around? Can you afford to extend it?
What does it take to actually make a release? All the technical aspects of getting the software to customers, including distribution logistics for physical copies, if there are any.
What external factors might force you to make a release? These might be technical, like new version of operating system, or related to changed laws and regulations.
How much in advance will you know about these external factors? Changes in laws are usually announced months or years in advance, but you might have only weeks to respond to changes in operating system.
How does a release align with other things that company is doing? Your software is probably only a part of the company offering. Are you independent of other projects? Even when your software is the only thing that company provides, there are probably other branches in company that support it and they all have to work with common goal in mind.
Who decides to make a release? Is that a development team, a product manager, a branch director, sales people? Software can’t release itself, so who decides that this specific version is going to be made available to customers?

You can decide to empower the team to make release decisions whenever they see fit, disregarding most of outside factors and concerns. That’s a perfectly valid choice!

The common risk of every single software release is that there might be important problems that you don’t know about, and that you underestimated problems that you do know about - that they turn out to be more serious than you thought. By making a choice to release a new version based only on build results, you decide that these and some other risks are acceptable for you.

First set of risks revolves around quality of your automated checks suite that is run before the release. Common problems with such suites are lack of data variability and coverage gaps in areas that are hard to test, especially interactions with real external systems. If you don’t maintain the discipline of only merging code that is covered by tests, you might also have insufficient coverage of newer features. Automated checks suite must also maintain a balance between thoroughness and execution time. It is common to keep separate pipelines for slow performance and security testing, and there’s a risk of forgetting to run them before making a release.

If you make it easy to make a release, and you make a habit of frequent releases, there’s a risk of losing vigilance. A Big Important Release acts as external pressure that keeps everyone cautious. When releases become a common thing, you might underestimate the importance of found issues, working under the assumption that you can always fix them later and make a new release. That assumption is true and valid for many software issues, but it will become a problem when it starts to be an excuse for sloppiness.

When release decision is made based on build results, releases are usually more frequent and smaller in scope (contain a limited number of changes). The argument goes, smaller releases are easier to test and inherently less risky, so it’s better to release more frequently. On the other hand, smaller changes carry a risk of losing the forest for the trees. As your team is focused on small tasks, there might be nobody to consider the impact of these small changes on the product as a whole. Things that work well in isolation sometimes do not work well together.

Finally, there’s a risk of that choice being wrong for your actual case. Many developers appreciate easy, small and frequent releases. But most of your customers just want to get the task done - they don’t want to wait for update or be interrupted by it. They might have slow Internet connection or pay for downloaded data. They might be averse to change - people tend to be conservative and bear with arduous way of doing things, just because they have always done it like that.

If you want to release based on build results, you want to ensure that you have other ways to minimize these risks, and you have a safety net that minimizes the impact of them when they inevitably occur.

First, make it technically easy to release a new version of software. That usually means that release pipeline is fully automated. Whenever you decide to make a release, you should be able to just push a button and be confident that new version is available to customers.

Second, prepare a plan for when broken version is released, or release process itself fails. That might mean a simple way to yank and roll back a release, or empowering a team to quickly fix an issue and make another release. You might also schedule fire drills when you pretend that working release is broken, just to prepare for a real situation.

Third, if your delivery process allows it, consider rolling releases, blue/green deployments and post-release monitoring. The general idea is that you release new version to some small subset of your customers, and gradually make it available to others. That should be paired with monitoring, so if you notice that there are some problems with new version, you pause the release. This way most of customers never experience the broken behavior.

Finally, you need the ability to deliver new versions frequently. This is a given when you work on a software as a service, where you fully control who uses which version. But this dynamic changes as you move to software that is installed on customer devices, especially devices that are not always online.

To sum up, the quality of software, as signaled by build status, is only a small part of release equation. There’s nothing wrong in deciding to focus on it, disregarding all the other factors. But you need to ensure that this is the right choice in your general business environment, and you should work on safety net that minimises the risk that this choice entails.

Playwright - accessing page object in event handler

2024-01-03T18:37:03+01:00

Playwright exposes a number of browser events and provides a mechanism to respond to them. Since many of these events signal errors and problems, most of the time you want to log them, halt program execution, or ignore and move on. Logging is also shown in Playwright documentation about network, which I will use as a base for examples in this article.

Problem statement

Documentation shows event handlers created with lambda expressions, but lambda poses significant problems once you leave the territory of toy examples:

they should fit in single line of code
you can’t share them across modules
you can’t unit test them in isolation

Usually you want to define event handlers as normal functions. But when you attempt that, you might run into another problem - Playwright invokes event handler with some event-related data, that data does not contain any reference back to page object, and page object might contain some important contextual information.

In other words, we would like to do something similar to code below. Note that this example does not work - if you run it, you will get NameError: name 'page' is not defined.

from playwright.sync_api import sync_playwright
from playwright.sync_api import Playwright


def request_handler(request):
    print(f"{page.url} issued request: {request.method} {request.url}")


def response_handler(response):
    print(f"{page.url} received response: {response.status} {response.url}")


def run_test(playwright: Playwright):
    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("https://mirekdlugosz.com")

    page.on("request", request_handler)
    page.on("response", response_handler)

    page.goto("https://httpbin.org/status/404")

    browser.close()


with sync_playwright() as playwright:
    run_test(playwright)

I can think of three ways of solving that: by defining a function inside a function, with functools.partialand with a factory function. Let’s take a look at all of them.

Defining a function inside a function

Most Python users are so used to defining functions at the top level of module or inside a class (we call these “methods”) that they might consider function definitions to be somewhat special. In fact, some other programming languages do encumber where functions can be defined. But in Python you can define them anywhere, including inside other functions.

from playwright.sync_api import sync_playwright
from playwright.sync_api import Playwright


def run_test(playwright: Playwright):
    def request_handler(request):
        print(f"{page.url} issued request: {request.method} {request.url}")


    def response_handler(response):
        print(f"{page.url} received response: {response.status} {response.url}")


    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("https://mirekdlugosz.com")

    page.on("request", request_handler)
    page.on("response", response_handler)

    page.goto("https://httpbin.org/status/404")

    browser.close()


with sync_playwright() as playwright:
    run_test(playwright)

This works because function body is not evaluated until function is called and functions have access to names defined in their encompassing scope. So Python will look up page only when event handler is invoked by Playwright; since it’s not defined in function itself, Python will look for it in the function where event handler was defined (and then next function, if there is one, then module and eventually builtins).

I think this solution solves the most important part of the problem - it allows to write event handlers that span multiple lines. Technically it is also possible to share these handlers across modules, but you won’t see that often. They can’t be unit tested in isolation, as they depend on their parent function.

`functools.partial`

functools.partial documentation may be confusing, as prose sounds exactly like a description of standard function, code equivalent assumes pretty good understanding of Python internals, and provided example seems completely unnecessary.

I think about partial this way: it creates a function that has some of the arguments already filled in.

To be fair, partial is rarely needed. It allows to write shorter code, as you don’t have to repeat the same arguments over and over again. It may also allow you to provide saner library API - you can define single generic and flexible function with a lot of arguments, and few helper functions intended for external use, each with a small number of arguments.

But it’s invaluable when you have to provide your own function, but you don’t have control over arguments it will receive. Which is exactly the problem we are facing.

from functools import partial

from playwright.sync_api import sync_playwright
from playwright.sync_api import Playwright


def request_handler(request, page=None):
    print(f"{page.url} issued request: {request.method} {request.url}")


def response_handler(response, page=None):
    print(f"{page.url} received response: {response.status} {response.url}")


def run_test(playwright: Playwright):
    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("https://mirekdlugosz.com")

    local_request_handler = partial(request_handler, page=page)
    local_response_handler = partial(response_handler, page=page)

    page.on("request", local_request_handler)
    page.on("response", local_response_handler)

    page.goto("https://httpbin.org/status/404")

    browser.close()


with sync_playwright() as playwright:
    run_test(playwright)

Notice that our function takes the same arguments as Playwright event handler, and then some. When it’s time to assign event handlers, we use partial to create a new function, one that only needs argument that we will receive from Playwright - the other one is already filled in. But when function is executed, it will receive both arguments.

Factory function

Functions in Python may not only define other functions in their bodies, but also return functions. They are called “higher-order functions” and aren’t used often, with one notable exception of decorators.

from playwright.sync_api import sync_playwright
from playwright.sync_api import Playwright


def request_handler_factory(page):
    def inner(request):
        print(f"{page.url} issued request: {request.method} {request.url}")
    return inner


def response_handler_factory(page):
    def inner(response):
        print(f"{page.url} received response: {response.status} {response.url}")
    return inner


def run_test(playwright: Playwright):
    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("https://mirekdlugosz.com")

    page.on("request", request_handler_factory(page))
    page.on("response", response_handler_factory(page))

    page.goto("https://httpbin.org/status/404")

    browser.close()


with sync_playwright() as playwright:
    run_test(playwright)

The key here is that inner function has access to all of enclosing scope, including values passed as arguments to outer function. This allows us to pass specific values that are only available in the place where outer function is called.

Summary

The first solution is a little different than other two, because it does not solve all of the problems set forth. On the other hand, I think it’s the easiest to understand - even beginner Python programmers should intuitively grasp what is happening and why.

In my experience higher-order functions takes some getting used to, while partial is not well-known and may be confusing at first. But they do solve our problem completely.