I'm using Selenium with Python to perform web scraping in headless mode on an Ubuntu system. My script configures logging and uses a context manager to handle the WebDriver. However, the script halts after the log message:

Driver [/home/shane/.wdm/drivers/chromedriver/linux64/132.0.6834.83/chromedriver-linux64/chromedriver] found in cache

It doesn't navigate to the target URL or proceed with any further actions. Here’s the relevant code snippet:

from datetime import datetime
import logging
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from contextlib import contextmanager

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    handlers=[
        logging.StreamHandler(),
        logging.FileHandler('app.log', mode='a')
    ]
)

@contextmanager
def create_driver():
    driver = None
    try:
        options = webdriver.ChromeOptions()
        options.binary_location = '/usr/bin/chromium-browser'  # Update as needed
        options.add_argument('--headless')
        options.add_argument('--no-sandbox')
        options.add_argument('--disable-dev-shm-usage')
        options.add_argument('--disable-gpu')
        options.add_argument('--disable-blink-features=AutomationControlled')
        options.add_argument('--disable-extensions')
        options.add_argument('--disable-setuid-sandbox')
        options.add_argument('--window-size=1920,1080')

        service = Service(ChromeDriverManager().install())
        driver = webdriver.Chrome(service=service, options=options)
        logging.info("WebDriver created successfully.")
        yield driver
    except Exception as e:
        logging.error(f"Error creating WebDriver: {e}")
        if driver:
            driver.quit()
        raise
    finally:
        if driver:
            driver.quit()
            logging.info("WebDriver closed successfully.")

def search_web(query):
    try:
        with create_driver() as driver:
            url = '/'
            logging.info(f"Navigating to search page: {url}")
            driver.get(url)
            return "complete"
    except Exception as e:
        logging.error(f"Error during web search: {e}")
        return f"Error performing search: {str(e)}"

if __name__ == '__main__':
    result = search_web("Hello")
    print(result.strip())

Logs: Here’s the output log where the script gets stuck:

2025-01-18 12:28:52,663 - INFO - ====== WebDriver manager ======
2025-01-18 12:28:52,741 - INFO - Get LATEST chromedriver version for google-chrome
2025-01-18 12:28:52,814 - INFO - Get LATEST chromedriver version for google-chrome
2025-01-18 12:28:52,882 - INFO - Driver [/home/shane/.wdm/drivers/chromedriver/linux64/132.0.6834.83/chromedriver-linux64/chromedriver] found in cache
2025-01-18 12:29:53,123 - ERROR - Error creating WebDriver: Message: session not created: DevToolsActivePort file doesn't exist
Stacktrace:
#0 0x6515224747ca <unknown>
#1 0x651521f6c2f0 <unknown>
#2 0x651521fa8f34 <unknown>
#3 0x651521fa3999 <unknown>
#4 0x651521f9f8c6 <unknown>
#5 0x651521fecbc9 <unknown>
#6 0x651521fec216 <unknown>
#7 0x651521fe0753 <unknown>
#8 0x651521fadbaa <unknown>
#9 0x651521faedfe <unknown>
#10 0x65152243f38b <unknown>
#11 0x651522443307 <unknown>
#12 0x65152242be7c <unknown>
#13 0x651522443ec7 <unknown>
#14 0x65152241024f <unknown>
#15 0x6515224632f8 <unknown>
#16 0x6515224634c0 <unknown>
#17 0x651522473646 <unknown>
#18 0x71479c294ac3 <unknown>

2025-01-18 12:29:53,124 - ERROR - Error during web search: Message: session not created: DevToolsActivePort file doesn't exist
Stacktrace:
#0 0x6515224747ca <unknown>
#1 0x651521f6c2f0 <unknown>
#2 0x651521fa8f34 <unknown>
#3 0x651521fa3999 <unknown>
#4 0x651521f9f8c6 <unknown>
#5 0x651521fecbc9 <unknown>
#6 0x651521fec216 <unknown>
#7 0x651521fe0753 <unknown>
#8 0x651521fadbaa <unknown>
#9 0x651521faedfe <unknown>
#10 0x65152243f38b <unknown>
#11 0x651522443307 <unknown>
#12 0x65152242be7c <unknown>
#13 0x651522443ec7 <unknown>
#14 0x65152241024f <unknown>
#15 0x6515224632f8 <unknown>
#16 0x6515224634c0 <unknown>
#17 0x651522473646 <unknown>
#18 0x71479c294ac3 <unknown>

Error performing search: Message: session not created: DevToolsActivePort file doesn't exist
Stacktrace:
#0 0x6515224747ca <unknown>
#1 0x651521f6c2f0 <unknown>
#2 0x651521fa8f34 <unknown>
#3 0x651521fa3999 <unknown>
#4 0x651521f9f8c6 <unknown>
#5 0x651521fecbc9 <unknown>
#6 0x651521fec216 <unknown>
#7 0x651521fe0753 <unknown>
#8 0x651521fadbaa <unknown>
#9 0x651521faedfe <unknown>
#10 0x65152243f38b <unknown>
#11 0x651522443307 <unknown>
#12 0x65152242be7c <unknown>
#13 0x651522443ec7 <unknown>
#14 0x65152241024f <unknown>
#15 0x6515224632f8 <unknown>
#16 0x6515224634c0 <unknown>
#17 0x651522473646 <unknown>
#18 0x71479c294ac3 <unknown>


Environment Details:

OS: Ubuntu 22.04
Selenium: 4.x
WebDriver Manager: Latest version
Chromium Browser: Installed at /usr/bin/chromium-browser

Verified that Chromium is correctly installed and its version matches the ChromeDriver version.

he WebDriver should navigate to the target URL and complete the search operation.

Why is the WebDriver halting after finding the driver in the cache, and how can I fix this issue?

I'm using Selenium with Python to perform web scraping in headless mode on an Ubuntu system. My script configures logging and uses a context manager to handle the WebDriver. However, the script halts after the log message:

Driver [/home/shane/.wdm/drivers/chromedriver/linux64/132.0.6834.83/chromedriver-linux64/chromedriver] found in cache

It doesn't navigate to the target URL or proceed with any further actions. Here’s the relevant code snippet:

from datetime import datetime
import logging
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from contextlib import contextmanager

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(levelname)s - %(message)s',
    handlers=[
        logging.StreamHandler(),
        logging.FileHandler('app.log', mode='a')
    ]
)

@contextmanager
def create_driver():
    driver = None
    try:
        options = webdriver.ChromeOptions()
        options.binary_location = '/usr/bin/chromium-browser'  # Update as needed
        options.add_argument('--headless')
        options.add_argument('--no-sandbox')
        options.add_argument('--disable-dev-shm-usage')
        options.add_argument('--disable-gpu')
        options.add_argument('--disable-blink-features=AutomationControlled')
        options.add_argument('--disable-extensions')
        options.add_argument('--disable-setuid-sandbox')
        options.add_argument('--window-size=1920,1080')

        service = Service(ChromeDriverManager().install())
        driver = webdriver.Chrome(service=service, options=options)
        logging.info("WebDriver created successfully.")
        yield driver
    except Exception as e:
        logging.error(f"Error creating WebDriver: {e}")
        if driver:
            driver.quit()
        raise
    finally:
        if driver:
            driver.quit()
            logging.info("WebDriver closed successfully.")

def search_web(query):
    try:
        with create_driver() as driver:
            url = 'https://www.example/'
            logging.info(f"Navigating to search page: {url}")
            driver.get(url)
            return "complete"
    except Exception as e:
        logging.error(f"Error during web search: {e}")
        return f"Error performing search: {str(e)}"

if __name__ == '__main__':
    result = search_web("Hello")
    print(result.strip())

Logs: Here’s the output log where the script gets stuck:

2025-01-18 12:28:52,663 - INFO - ====== WebDriver manager ======
2025-01-18 12:28:52,741 - INFO - Get LATEST chromedriver version for google-chrome
2025-01-18 12:28:52,814 - INFO - Get LATEST chromedriver version for google-chrome
2025-01-18 12:28:52,882 - INFO - Driver [/home/shane/.wdm/drivers/chromedriver/linux64/132.0.6834.83/chromedriver-linux64/chromedriver] found in cache
2025-01-18 12:29:53,123 - ERROR - Error creating WebDriver: Message: session not created: DevToolsActivePort file doesn't exist
Stacktrace:
#0 0x6515224747ca <unknown>
#1 0x651521f6c2f0 <unknown>
#2 0x651521fa8f34 <unknown>
#3 0x651521fa3999 <unknown>
#4 0x651521f9f8c6 <unknown>
#5 0x651521fecbc9 <unknown>
#6 0x651521fec216 <unknown>
#7 0x651521fe0753 <unknown>
#8 0x651521fadbaa <unknown>
#9 0x651521faedfe <unknown>
#10 0x65152243f38b <unknown>
#11 0x651522443307 <unknown>
#12 0x65152242be7c <unknown>
#13 0x651522443ec7 <unknown>
#14 0x65152241024f <unknown>
#15 0x6515224632f8 <unknown>
#16 0x6515224634c0 <unknown>
#17 0x651522473646 <unknown>
#18 0x71479c294ac3 <unknown>

2025-01-18 12:29:53,124 - ERROR - Error during web search: Message: session not created: DevToolsActivePort file doesn't exist
Stacktrace:
#0 0x6515224747ca <unknown>
#1 0x651521f6c2f0 <unknown>
#2 0x651521fa8f34 <unknown>
#3 0x651521fa3999 <unknown>
#4 0x651521f9f8c6 <unknown>
#5 0x651521fecbc9 <unknown>
#6 0x651521fec216 <unknown>
#7 0x651521fe0753 <unknown>
#8 0x651521fadbaa <unknown>
#9 0x651521faedfe <unknown>
#10 0x65152243f38b <unknown>
#11 0x651522443307 <unknown>
#12 0x65152242be7c <unknown>
#13 0x651522443ec7 <unknown>
#14 0x65152241024f <unknown>
#15 0x6515224632f8 <unknown>
#16 0x6515224634c0 <unknown>
#17 0x651522473646 <unknown>
#18 0x71479c294ac3 <unknown>

Error performing search: Message: session not created: DevToolsActivePort file doesn't exist
Stacktrace:
#0 0x6515224747ca <unknown>
#1 0x651521f6c2f0 <unknown>
#2 0x651521fa8f34 <unknown>
#3 0x651521fa3999 <unknown>
#4 0x651521f9f8c6 <unknown>
#5 0x651521fecbc9 <unknown>
#6 0x651521fec216 <unknown>
#7 0x651521fe0753 <unknown>
#8 0x651521fadbaa <unknown>
#9 0x651521faedfe <unknown>
#10 0x65152243f38b <unknown>
#11 0x651522443307 <unknown>
#12 0x65152242be7c <unknown>
#13 0x651522443ec7 <unknown>
#14 0x65152241024f <unknown>
#15 0x6515224632f8 <unknown>
#16 0x6515224634c0 <unknown>
#17 0x651522473646 <unknown>
#18 0x71479c294ac3 <unknown>


Environment Details:

OS: Ubuntu 22.04
Selenium: 4.x
WebDriver Manager: Latest version
Chromium Browser: Installed at /usr/bin/chromium-browser

Verified that Chromium is correctly installed and its version matches the ChromeDriver version.

he WebDriver should navigate to the target URL and complete the search operation.

Why is the WebDriver halting after finding the driver in the cache, and how can I fix this issue?

• python
• selenium-webdriver
• web-scraping
• selenium-chromedriver
• headless-browser

• This question is similar to: DevToolsActivePort file doesn't exist error message. Selenium in Python running on MacOS. If you believe it’s different, please edit the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem. – S A Commented Jan 18 at 15:05

1 Answer 1

Adding options.add_argument('--remote-debugging-pipe') fixed the issue.