I am using python and selenium to read specific s on a page into an array. The page refreshes its data every 10 seconds or so automatically. I have the following code in a loop and it loops 5-200 times before getting the error.

The error I get is

Exception has occurred: StaleElementReferenceException
Message: stale element reference: stale element not found in the current frame"

which is thrown on the line

text_array.append(div.get_attribute('innerHTML'))

I have tried a bunch of solutions I found in this group, including the try except section, but they have not prevented the error.

Thanks for any help that can be given.

text_array = []
wait = WebDriverWait(driver, 10)  # Wait for up to 10 seconds
try:
    div_id = "tool-table-row"
    div_elements = wait.until(EC.presence_of_all_elements_located((By.ID, div_id)))
except NoSuchElementException or StaleElementReferenceException:
    time.sleep(10)
    continue
 
for div in div_elements:
    text_array.append(div.get_attribute('innerHTML'))

I am using python and selenium to read specific s on a page into an array. The page refreshes its data every 10 seconds or so automatically. I have the following code in a loop and it loops 5-200 times before getting the error.

The error I get is

Exception has occurred: StaleElementReferenceException
Message: stale element reference: stale element not found in the current frame"

which is thrown on the line

text_array.append(div.get_attribute('innerHTML'))

I have tried a bunch of solutions I found in this group, including the try except section, but they have not prevented the error.

Thanks for any help that can be given.

text_array = []
wait = WebDriverWait(driver, 10)  # Wait for up to 10 seconds
try:
    div_id = "tool-table-row"
    div_elements = wait.until(EC.presence_of_all_elements_located((By.ID, div_id)))
except NoSuchElementException or StaleElementReferenceException:
    time.sleep(10)
    continue
 
for div in div_elements:
    text_array.append(div.get_attribute('innerHTML'))

• python
• selenium-webdriver

2 Answers 2

In order to understand the problem and attempt to fix it, you need to understand what a stale element is.

What is a stale element?

A stale element is an element that you locate, store in a variable, and then the page updates in some way losing that element reference. A StaleElementReferenceException is caused when that variable is accessed.

A simple example:

e = driver.find_element(...)
# page updates
e.click() # `StaleElementReferenceException` is thrown

The example I most often see is when people grab a list of elements from a page and then loop through that list clicking on each element. The problem occurs when that click either refreshes or navigates away from the page so that the second time you enter the loop, the list of elements is stale and the action throws the StaleElementReferenceException.

links = driver.find_elements(...)
for link in links
    link.click() # `StaleElementReferenceException` is thrown on 2nd entry into loop
    # navigates to new page
    # do stuff
    driver.back()

Once you have a stale element, there's no way to "fix" that reference other than fetching it again, e.g.

e = driver.find_element(...)
# page updates
e = driver.find_element(...)
e.click()

This doesn't "fix" e, it just replaces the element stored with a fresh reference.

Now that we understand what a stale element is, on to fixing your code....

There are a few things I would change.

1. Your try-except is in the wrong place. The code that throws the stale element exception is div.get_attribute(). That's because once the page updates, all the elements in div_elements are stale. You come to the top of the loop and it grabs an element from div_elements and puts it into div. It next calls div.get_attribute() and that's where it throws. As I described above, to fix this we need to always have a fresh reference for each loop.

2. EC.presence_of_all_elements_located() is waiting for presence of elements. Presence just means that the element exists in the DOM, not that it's visible or ready to be interacted with, e.g. .get_attribute(). You need to wait for visible instead using EC.visibility_of_all_elements_located().

3. To prevent the stale element, we need to change the loop so that each loop gets a fresh copy of the element.

The updated code would look something like

from selenium import webdriver
from seleniummon.exceptions import StaleElementReferenceException
from selenium.webdrivermon.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()

url = "https://www.practicesoftwaretesting/"
driver.get(url)

text_array = []
wait = WebDriverWait(driver, 10)
div_elements = wait.until(EC.visibility_of_all_elements_located((By.ID, "tool-table-row")))
i = 0
while i < len(div_elements):
    try:
        div_elements = wait.until(EC.visibility_of_all_elements_located((By.ID, "tool-table-row")))
        text_array.append(div_elements[i].get_attribute('innerHTML'))
        i = i + 1
    except StaleElementReferenceException:
        # do nothing and loop again at the same i

The chance of a stale reference showing up with this code is slim but it could happen between the time the element is fetched and .get_attribute() is called so I put it in there just to be safe.

from selenium import webdriver
from selenium.webdrivermon.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from seleniummon.exceptions import StaleElementReferenceException, NoSuchElementException
import time

# Initialize the driver
driver = webdriver.Chrome()  # Change this if using a different browser driver
driver.get(url)

text_array = []  # Array to store extracted text
div_id = "tool-table-row"  # ID of the elements to be extracted

while True:
    wait = WebDriverWait(driver, 10)  # Wait up to 10 seconds for elements to appear
    try:
        # Re-find elements in every loop iteration to avoid stale references
        div_elements = wait.until(EC.presence_of_all_elements_located((By.ID, div_id)))

        for div in div_elements:
            try:
                # Get the inner HTML of each element
                text_array.append(div.get_attribute('innerHTML'))
            except StaleElementReferenceException:
                # If the element becomes stale, continue the loop and find it again
                print("Stale element encountered, retrying...")
                continue  

    except (NoSuchElementException, StaleElementReferenceException):
        # If elements are not found or become stale, wait and retry
        print("Element not found or stale, retrying...")
        time.sleep(2)  # Short delay before retrying
        continue  # Restart the loop

    time.sleep(10)  # Wait before the next iteration, as the page refreshes every 10 seconds

driver.quit()