Automated commit and PR-level performance testing based on CI (taking the deployment in Vercel as an example)

✨ Introduction and Summary

✨

This article explains how to implement automated commit and PR-level performance testing using CI/CD pipelines and how to achieve this goal through GitHub Actions and Lighthouse CI tools. The article provides a detailed explanation of Lighthouse's evaluation metrics and their role in web performance optimization, as well as how to integrate Lighthouse testing into CI/CD pipelines and automatically generate reports. Additionally, it explores how to ensure continuous website performance optimization through threshold detection and regular monitoring.

📝 Main content

Introduction

Website performance has a deep and far-reaching influence on user experience and SEO. Lighthouse is a powerful tool for front-end performance testing, and manual testing is not only time-consuming but also prone to missing some key issues. So, how can we automate the performance testing in the CI/CD pipeline during code commits or pull requests?

GitHub Actions makes this possible by running Lighthouse audits automatically on each commit or pull request, helping to ensure that performance always meets the requirement.

What is Lighthouse？

As an open-source automated testing tool, Lighthouse helps developers analyze website performance and provide recommendations for optimization. Lighthouse performs in-depth audits on any webpage, whether it is public or requiring authentication, ensuring it meets high standards across all aspects.

Lighthouse mainly evaluates a webpage in the following areas:

✅ Performance: analyze the webpage’s loading speed, optimize resource requests, reduce render-blocking resources, and improve user interaction experience.

✅ Accessibility: ensure the website is highly accessible to all users, including those with visual or hearing impairments.

✅ PWA: check whether it meets the PWA standard, such as offline access and responsive design.

✅ Best Practices: check if it complies with modern web development best practices, such as HTTPS usage and resource loading optimization.

✅ SEO: evaluate whether it meets SEO guidelines to improve the search engine ranking.

What is Lighthouse CI？

Lighthouse CI is a suite of tools designed to simplify and enhance the automated execution, storage, data retrieval, and report validation of Lighthouse. It helps developers to seamlessly integrate into the CI/CD pipeline, continuously monitor performance metrics, and manage audit results with ease, making website optimization more intelligent, consistent, and traceable.

How to use it in CI/CD?

We can utilize GitHub Actions to set up an automated workflow that runs Lighthouse tests and generates reports every time code is committed or a Pull Request Is Created.

Configure GitHub Actions

⚠️

The example is based on deployment on Vercel, but the approach is similar across different platforms

The key idea is waiting for the completion of the new deployment (CD), and GitHub Actions can detect its completion, then trigger the Lighthouse audit.

Approaches that the author is aware of for detecting the completion of a new deployment (CD)

Cloud platform：

Deploy via the cloud platform API in GitHub Actions. The deployment is completed when the execution is completed.
Poll periodically to check if the deployment is completed in GitHub Actions.

To enable Lighthouse checks for both pull requests and code commits, create separate workflow files in the .github/workflows directory.

For example:

pr-lighthouse.yml for pull requests

code


name: pr-deploy-and-audit
on: [pull_request]
jobs:
  deploy_and_audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Vercel Action
        id: vercel_action
        uses: amondnet/vercel-action@v25.2.0
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          vercel-org-id: ${{ secrets.VERCEL_ORG_ID}}
          vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID}}
          scope: ${{ secrets.VERCEL_SCOPE }}
      - name: Add comment to PR
        id: loading_lighthouse_comment_to_pr
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          number: ${{ github.event.pull_request.number }}
          header: lighthouse
          message: |
            🚦 Running Lighthouse audit...
      - name: Add comment to PR
        id: loading_budget_comment_to_pr
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          number: ${{ github.event.pull_request.number }}
          header: budget
          message: |
            ⏱ Running budget checks...
      - name: Audit URLs using Lighthouse
        id: lighthouse_audit
        uses: treosh/lighthouse-ci-action@v12
        with:
          # add as many urls here as you wish to audit
          # prefixed by the preview-url
          urls: |
            ${{ steps.vercel_action.outputs.preview-url }}
          runs: 3
          # depend on the specific lighthouse configuration
          configPath: './lighthouserc.js'
          serverBaseUrl: ${{ secrets.LHCI_SERVER_URL }}
          serverToken: ${{ secrets.LHCI_SERVER_TOKEN }}
          basicAuthUsername: ${{ secrets.LHCI_BASIC_AUTH_USERNAME }}
          basicAuthPassword: ${{ secrets.LHCI_BASIC_AUTH_PASSWORD }}
      - name: Format lighthouse result
        id: lighthouse_result
        if: ${{ always() }}
        uses: actions/github-script@v7
        with:
          github-token: ${{secrets.GITHUB_TOKEN}}
          script: |
            const links = ${{ steps.lighthouse_audit.outputs.links }}
            const results = (${{ steps.lighthouse_audit.outputs.assertionResults }}).filter(result => result.isRepresentativeRun);
            
            const score = res => res >= 90 ? '🟢' : res >= 50 ? '🟠' : '🔴';
            
            const comment = results.map((result) => {
              const summary = result.summary;
              const url = result.url;
            
              return `
              🌎 [${url}](${url})
              ⚡️ [Lighthouse report](${links[url]})
            
              | Category | Score |
              | --- | --- |
              ${Object.keys(summary).map((key) => {
                const percentage = Math.round(summary[key] * 100);
                return `| ${score(percentage)} ${key} | ${percentage} |`;
              }).join('\n')}
            
              `;
            }).join('---');
            
            core.setOutput("comment", comment);
      - name: Format budget result
        id: budget_result
        if: ${{ always() }}
        uses: actions/github-script@v7
        with:
          github-token: ${{secrets.GITHUB_TOKEN}}
          script: |
            const assertions = ${{ steps.lighthouse_audit.outputs.assertionResults }};
            if (!assertions.length) {
              core.setOutput("comment", '✅ Budget met, nothing to see here');
            } else {
              const comment = assertions.map((result) => {
                return `
            ❌ **${result.auditProperty || ''}.${result.auditId}** failure on [${result.url}](${result.url})
            
            *${result.auditTitle}* - [docs](${result.auditDocumentationLink})
            
            | Actual | Expected |
            | --- | --- |
            | ${result.actual} | ${result.operator} ${result.expected} |
            `;
              }).join('---');
            
              core.setOutput("comment", comment);
            }
      - name: Add Lighthouse comment to PR
        id: lighthouse_comment_to_pr
        if: ${{ always() }}
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          number: ${{ github.event.pull_request.number }}
          header: lighthouse
          message: |
            ${{ steps.lighthouse_result.outputs.result }}
      - name: Add Budget comment to PR
        id: budget_comment_to_pr
        if: ${{ always() }}
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          number: ${{ github.event.pull_request.number }}
          header: budget
          message: |
            ${{ steps.budget_result.outputs.result }}

commit-lighthouse.yml for code commits

code


name: CI
on: [push]
jobs:
  lhci:
    name: Lighthouse
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
      - name: Vercel Action
        id: vercel_action
        uses: amondnet/vercel-action@v25.2.0
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          vercel-org-id: ${{ secrets.VERCEL_ORG_ID}}
          vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID}}
          scope: ${{ secrets.VERCEL_SCOPE }}
      - name: Audit Vercel preview URL using Lighthouse
        uses: treosh/lighthouse-ci-action@v12
        with:
          urls: |
            ${{ steps.urls.outputs.urls }}
          runs: 3
          # depend on the specific lighthouse configuration
          configPath: './lighthouserc.js'
          serverBaseUrl: ${{ secrets.LHCI_SERVER_URL }}
          serverToken: ${{ secrets.LHCI_SERVER_TOKEN }}
          basicAuthUsername: ${{ secrets.LHCI_BASIC_AUTH_USERNAME }}
          basicAuthPassword: ${{ secrets.LHCI_BASIC_AUTH_PASSWORD }}

Configuration of required environment variables

vercel-action

Vercel Action - GitHub Marketplace

This action make a deployment with github actions instead of Vercel builder

https://github.com/marketplace/actions/vercel-action#how-to-use

lighthouse part

Lighthouse CI Action - GitHub Marketplace

Audit URLs using Lighthouse and test performance with Lighthouse CI

https://github.com/marketplace/actions/lighthouse-ci-action

Advanced Configuration

Continuous Tracking for a Specific Branch (Vercel Only)

Existing problems known to the author

Because Vercel-Action by default setting returns the URL in hash format of Vercel, which changes across different deployments, not the URL format that uses the branch name as the prefix, every audit will use a different URL for the same content. As a result, it is impossible to compare the data over time when using the Lighthouse server.

⚠️

Not only that, but there is also no branch format URL in Vercel

Solution

To solve this, add the alias-domains configuration to vercel-action, which instructs Vercel to generate the same branch URL that matches Vercel’s default format. The same URL will then be used in the outputs.


- name: Vercel Action
  id: vercel_action
  uses: amondnet/vercel-action@v25.2.0
  with:
    vercel-token: ${{ secrets.VERCEL_TOKEN }}
    github-token: ${{ secrets.GITHUB_TOKEN }}
    vercel-org-id: ${{ secrets.VERCEL_ORG_ID}}
    # Add the alias-domains configuration, for manually generating a git branch in the same format as the one automatically generated by Vercel
    alias-domains: ${{ secrets.PROJECT_NAME}}-git-{{ BRANCH }}-${{ secrets.TEAM_NAME}}.vercel.app
    vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID}}
    scope: ${{ secrets.VERCEL_SCOPE }}

Hosting your own Lighthouse Server It allows for storing historical records, generating more comprehensive statistics, and performing long-term data comparisons.

lighthouse-ci/docs/server.md at main · GoogleChrome/lighthouse-ci

Automate running Lighthouse for every commit, viewing the changes, and preventing regressions - GoogleChrome/lighthouse-ci

https://github.com/GoogleChrome/lighthouse-ci/blob/main/docs/server.md

How to deploy your own Lighthouse

Coming soon 🚧

Advanced optimization suggestions

Set Performance Thresholds

You can add Lighthouse threshold detection in CI, causing CI tasks to fail when performance drops below a certain threshold, reminding developers to optimize their code.

Conduct Regular Lighthouse Tests

In addition to PR-triggered tests, you can also schedule regular Lighthouse monitoring to prevent long-term accumulated performance issues.

Integrate Web Vitals Monitoring

You can combine it with Web Vitals monitoring tools to analyze real-time performance data from user visits.

📖 Extended reading

A Practical Note on Optimizing Core Web Performance Indicators Using Next.js and Cloudflare

Webpage Core Performance Optimization Guide and Practical Implementation Records

🤗 Summarize and summarize

Not at the moment

📎 Reference article

Main Reference Not available

💡

acceptable become a member Telegram's Little Warehouse of Seven Rows | Internet Memories | Blogs cap Seven lines Technical Exchange Group Find more tips oh 🥰 and also discuss various issues in the chat group ❓

Welcome to the Bottom comment section. Share your thoughts and experiences with Let's discuss and improve together!