Cypress Tips June 2025

RAG for Cypress questions 🤖

Retrieval-augmented generation

Imagine yourself coding a Cypress test. In your code editor you prepare a test; you enter its title and type the comments planning the test steps.

it('changes the text after clicking the button', () => {
  cy.visit('index.html')
  // there is an element with id "message" that contains some text
  // after clicking on the button with id "button"
  // the text changes to something else
})

You can see the initial test running in Cypress

The starting Cypress test

Let’s ask AI to write the test for us. I will use GPT-4o model included with GitHub Copilot to generate the solution

GPT-4o generated answer

Hmm, that is not a good answer. Like at all! Using older models like “Gemini 2.0 Flash” and “cursor-small” produces similar results.

Let’s switch the model to Claude Sonnet 3.7 and try again.

Claude Sonnet 3.7 generated answer

Trust but verify

Ok, this is slightly better answer with one subtle bug. The initial message text is stored and loaded from an alias “originalText”. Cypress re-evaluates the aliases when you retrieve them - thus the original text and the updated text after clicking the button … will always be the same!

The generated solution does not work at all

We can fix the solution by adding the “type: static” to the alias

cy.get('#message')
  .invoke('text')
  .as('originalText', { type: 'static' })

Saving the initial text as static value fixes the test

More and more data

Let’s try using Cursor. It “vacuums” all the data in the local project, reading the index.html file and grabbing the actual HTML text before and after the click.

Cursor answer misinterprets the task

The above answer fundamentally misunderstands the task. I am not looking for specific text assertions, since in most cases they would not be known. The task was to check if the unknown text changes!

Ok, why is AI suggesting wrong answers? Older / smaller models generate simpler incorrect answers, while newer larger models throw everything at the problem and get confused. Can we do better?

Aside: I strongly believe the rush to produce MCP servers that extend AI chat sessions with more skills is misguided, since it leads AI on wild goose chases.

Give AI a relevant example

Here is a better way to answer Cypress-related question. If you were a human familiar with this newsletter, before answering the “implement the test following the comments” prompt, you would open your browser, go to https://cypress.tips/search and search for an existing Cypress test example similar to the task at hand:

Using cypress.tips/search to search for an existing answer

There is already a similar example from at glebbahmutov.com/cypress-examples!

Cypress examples “Text changes” recipe

Nice. Let’s grab the Markdown of this found recipe and copy to clipboard

The found recipe's raw Markdown text

Go back to the VSCode Copilot or Cursor chat and include the raw Markdown with your original query:

Cursor prompt with included recipe Markdown

Can an older / smaller LLM generate a good answer now? Yes, it can!

Good answer generated by the small LLM model

Beautiful. Including relevant Cypress example with the prompt lets even smaller / faster / cheaper LLM produce good tests. Even accounting for more tokens, the larger prompt can still come out cheaper because we can use a cheaper model to run it, not even counting the human time spent on reviewing / fixing the answer.

The newest LLMs scrape a lot of data, but most data on the Internet is low quality. Using tested, verified examples from my blog posts, cypress-examples site, online lessons is THE best way to generate high-quality answers in your specific situations.

Stay tuned for more, as I show how to automate the manual “search” step using text vector embeddings. I am working on blog posts and videos to show how Retrieval-augmented generation (RAG) system could work for Cypress test writing.

🤖🎉