Akatama's Slice of the Internet

Cleaner Tests, Smarter Workflows: Lessons from Improving Calibre-Web for IIAB

By Jimmy Lindsey

When I first started improving Calibre-Web's tests for IIAB, I thought it would be a quick update. Add a few tests, tweak a workflow, and then move on. It didn't turn out that way.

IIAB is an open-source offline server platform that brings digital libraries, educational resources, and web applications to schools and communities without reliable internet access. One of those applications is Calibre-Web, which lets users browse and manage eBooks through a web interface.

What began as a small change to add a few missing tests turned into a much larger effort. I restructured our pytest fixtures, refined our use of Pytest-BDD in our tests, and refactored our integration tests workflow so that it was cleaner and more reusable.

In the end, the code changes were small, but the impact was significant: better organization, faster workflows, and a much clearer foundation for future contributors. Here's how that evolution unfolded, and what I learned in the process.

Documentation Changes

When I was working on the tests that involved downloading videos, I realized that my current setup on openSUSE Tumbleweed wasn't working properly. Now, IIAB only officially supports the most recent releases of Ubuntu, Debian and Raspberry Pi OS. However, this turned out to be more than just a distro preference issue. We have an integration test workflow where we purposefully do not install Calibre-Web through IIAB, and it would have the same problem.

After talking with several people that worked on the video downloading functionality¹, I eventually figured out what configuration was required and wrote out documentation in the README. Officially, we are not supporting IIAB-less installs of Calibre-Web. If we’re going to run tests in this scenario, we need to document it clearly. That ensures the workflow stays maintainable and future contributors can run tests on their systems. You can see the current version of this documentation here.

General Test Improvements

Up to this point, we only had two tests in one file. As I added more tests, we ended up with several small utility functions and fixtures that made sense to move into their own file. So I created conftest.py. If you aren't familiar with pytest, conftest.py is a special file that defines fixtures and other shared code. You can even define multiple conftest.py files across directories if needed. Currently, we only need one of these, but if we start moving our tests into their separate directories, it's something to consider.

Most of the below code was in the original test file, test_basic_behavior.py, and was moved here. It looks like this:

import pytest

from urllib.parse import urljoin
from selenium import webdriver
import time


# overrides default Chrome options to always call Chrome with --no-sandbox
@pytest.fixture(scope="session")
def splinter_driver_kwargs(splinter_webdriver):
    """Override Chrome WebDriver options"""
    if splinter_webdriver == "chrome":
        chrome_options = webdriver.ChromeOptions()

        # List of Chromium Command Line Switches
        # https://peter.sh/experiments/chromium-command-line-switches/
        chrome_options.add_argument("--no-sandbox")

        return {"options": chrome_options}

    elif splinter_webdriver == "firefox":
        firefox_options = webdriver.FirefoxOptions()
        return {"options": firefox_options}
    else:
        raise ValueError(
            "Invalid browser passed to --splinter_Wewbdriver. Only Chrome and Firefox are allowed"
        )

# how to visit Calibre-Web, all in one place
def visit_website(browser, step_context):
    step_context["ip_address"] = "localhost:8083"
    url = urljoin("".join(["http://", str(step_context["ip_address"])]), "/")
    browser.visit(url)

You can see the rest of this file in the repo.

Upload Book Tests

The first set of tests I wrote were for uploading two books (one for each test). This required creating a directory for test files, which can be found in /tests/functional/files. Right now, it only contains the two ebooks used in this test, but I’m sure more will be added in the future. These books were picked because they are relatively small and in the public domain. You can see the feature file for this test or see the full test itself.

"""Book upload feature tests."""
# ... other imports are here
# our helper functions from conftest
from tests.functional.conftest import login_if_not_logged_in, visit_website


@scenario("book_upload.feature", "Upload book 1")
def test_upload_book_1():
    """Upload book 1"""


# note that since this given is the same for both test_upload_book_1 and test_upload_book_2,
# it is called by both. Also, this is where we are using our helper functions defined in conftest.py
@given("Calibre-Web is running and I am logged in as admin")
def _(browser, step_context):
    """Calibre-Web is running and I am logged in as admin"""
    visit_website(browser, step_context)
    login_if_not_logged_in(browser, "Admin", "changeme")


# this uses OS utilities to pass the correct path to Calibre-Web's upload function
@when("I click on upload and upload the first book")
def _(browser, step_context):
    """I click on upload and upload the first book"""
    filename = "tests/functional/files/The_King_In_Yellow.epub"
    file = os.path.join(os.getcwd(), filename)
    browser.fill("btn-upload", file)


@then("I should see book 1")
def _(browser):
    """I should see book 1"""
    assert browser.find_by_id("title").value == "The King in Yellow", (
        "Expected to see first book title present"
    )

    assert browser.find_by_id("authors").value == "Robert W. Chambers", (
        "Expected to see first book author present"
    )
# ... The other test is down here, 

Download Video Tests

This test is pretty much the same as the upload book tests above, but instead we are testing the ability to download videos from websites such as Youtube. Unfortunately, I cannot link to the full test or the feature file, as the test does not work on our GitHub Actions workflows. YouTube blocks any yt-dlp requests from unauthenticated users, and we don’t currently have a reliable way to sign in during the workflow. I can verify that the test works great on my personal computer, so hopefully soon I can find a solution to allow us to run this test with the workflow. For now, I’m exploring other sites that could serve as reliable test sources.

Create User Tests

For this test, I decided to take a deeper look at how Pytest-BDD works. In the Upload Book Tests section, I didn’t include all the tests. That is because both of the tests are pretty much the same, just with different data. I knew there had to be a cleaner way to parameterize the tests so Pytest-BDD would recognize them. Here is how I wrote the feature file to make it data-driven:

Feature: Create User
  Testing create User


  Scenario Outline: Create users
      Given Calibre-Web is running and I am logged in as admin
      When I click on Admin button and create user with <username>, <password>, and <email>
      Then I should see that <username> is created

      Examples:
      | username | password  | email         |
      | chloe    | Chloe123! | chloe@iiab.io |
      | ella     | Ella123!  | ella@iiab.io  |

Pytest-BDD refers to this as a "Scenario Outline". It will use the values in the table under "Example" to fill in any variable between the angle brackets, which is useful for reporting when there is a test failure. You do need to do something special in the test code itself so that these values are passed into the test:

# ... other imports here
# note the parsers import
from pytest_bdd import given, scenario, then, when, parsers



@scenario("create_users.feature", "Create users")
def test_create_user_1():
    """Create users"""


@given("Calibre-Web is running and I am logged in as admin")
def _(browser, step_context):
    """Calibre-Web is running and I am logged in as admin"""
    visit_website(browser, step_context)
    login_if_not_logged_in(browser, "Admin", "changeme")


# parsers.parse takes the When step and associates it with each row of data from the Examples table
@when(
    parsers.parse(
        "I click on Admin button and create user with {username}, {password}, and {email}"
    )
)
def _(browser, username, password, email):
    admin_button = browser.find_by_id("top_admin")
    admin_button.click()

    add_new_user_button = browser.find_by_id("admin_new_user")
    add_new_user_button.click()

    browser.fill("name", username)
    browser.fill("email", email)
    browser.fill("password", password)

    save_button = browser.find_by_id("user_submit")
    save_button.click()


# parsers.parse takes the Then step and associates it with each row of data from the Examples table
@then(parsers.parse("I should see that {username} is created"))
def _(browser, username):
    time.sleep(0.5)
    assert browser.find_by_id("name").value == username, (
        f"Expected to see user {username} is present"
    )

In the end, this approach feels far more maintainable and readable than my previous setup. I do wish the test data was a little closer to the tests themselves (like what I see when I write tests in C#). Overall, this makes the tests easier to maintain, since any changes to the flow would only need to be made in one place. Also, if we decided we wanted to create more than two users, we would only have to update the table in the feature file to add another row.

Refactoring the Integration Test Workflow

The integration-test.yml workflow used to run the tests on Google Chrome and Firefox sequentially. However, the create user tests broke this, as you cannot create the same user multiple times. As a result, I decided to split these test runs into their own jobs that would run in parallel. I realized that in doing this, a large portion of the workflow would be duplicated. As such, I looked into composite actions. Basically, they are a set of reusable steps that can be called in any workflow.

There are a few things to note about composite actions:

• They are called by your workflow, so you need to make sure the repo that contains them is cloned first.
• Since an operating system is not specified in the composite action, you need to specify a shell for all steps that use the run key.
• The name of the composite action is the name of the containing folder. In my case, calibre-web-setup.
  • The name of the file that defines the composite action itself is either action.yml, action.yaml or Dockerfile.

All we are doing in this composite action is the required setup for Calibre-Web. Note that we aren't installing any browsers, since that's unique to the job itself. You can see the full composite action here.

Now integration-test.yml looks like this:

name: Integration tests
run-name: Integration tests
on: [push]
jobs:
  integration-test-chrome:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository code
        uses: actions/checkout@v4
      - uses: ./.github/workflows/calibre-web-setup # calling the composite action
      - name: Install Chrome
        uses: browser-actions/setup-chrome@v1
      - run: chrome --version
      - name: Execute Integration Tests for Chrome
        run: pytest -s --splinter-webdriver chrome --splinter-headless
  integration-test-firefox:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository code
        uses: actions/checkout@v4
      - uses: ./.github/workflows/calibre-web-setup # calling the composite action
      - name: Install Firefox
        uses: browser-actions/setup-firefox@v1
      - run: firefox --version
      - name: Execute Integration Tests for Firefox
        run: pytest -s --splinter-webdriver firefox --splinter-headless

You can see that since I moved so many of the steps out to the composite action, it makes the workflow really clean and easy to read. Now all we are doing here is cloning the repo, setting up Calibre-Web with the composite action, installing the required browser and running the tests on that browser. In addition, if the setup for Calibre-Web changes, which has already happened recently, now it only needs to be updated in the composite action. I call that a win!

Next Steps

While the testing framework feels solid, there’s still room to expand coverage and explore new scenarios. Here are a few ideas for what's next:

1. Update the upload book tests to use a Session Outline.
2. Get the video download tests working on the workflow
3. Two tests where each user creates a shelf and assigns a book to it
4. Two tests where we verify that users can only see their own shelves

Conclusion

What started as “just a few new tests” turned into a complete rethink of how we structure and automate tests for Calibre-Web. Along the way, I got to dig deeper into Pytest-BDD, improve our workflows with composite actions, and clean up technical debt that had been quietly slowing us down.

The end result isn’t flashy. A few better tests, faster runs, and a cleaner YAML file, but it makes a huge difference in how confidently we can develop and contribute. That’s the part I enjoy most about work like this: small improvements that make the whole system a little more dependable.