I am trying to access a website but the selenium will not load the page, when printing the page source it is empty.
I've used selenium several times before and it all worked fine so this is very odd to me. I would really appreciate some advice on revising the code. My chromedriver is very recent: .0.6834.83/mac-arm64/chromedriver-mac-arm64.zip
from selenium.webdrivermon.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("--start-maximized")
options.add_argument(
"user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
)
# driver_path = 'pathtomydriverhere'
driver = webdriver.Chrome(service=Service(driver_path), options=options)
try:
# Open the URL
url = ";
driver.get(url)
# Wait for the body tag to load
WebDriverWait(driver, 60).until(lambda d: d.execute_script("return document.readyState") == "complete")
# Debug: Print the page source
print(driver.page_source)
except Exception as e:
print(f"Error occurred: {e}")
finally:
# Close the browser
driver.quit()
Ran selenium to load a page but the page will not load. Tried WebDriverWait and time.sleep() but both do not work, the result of print(driver.page_source) is
<html><head></head><body></body></html>
I am trying to access a website but the selenium will not load the page, when printing the page source it is empty.
I've used selenium several times before and it all worked fine so this is very odd to me. I would really appreciate some advice on revising the code. My chromedriver is very recent: https://storage.googleapis.com/chrome-for-testing-public/132.0.6834.83/mac-arm64/chromedriver-mac-arm64.zip
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("--start-maximized")
options.add_argument(
"user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
)
# driver_path = 'pathtomydriverhere'
driver = webdriver.Chrome(service=Service(driver_path), options=options)
try:
# Open the URL
url = "http://qikan.cqvip.com/Qikan/Article/Detail?id=35677542"
driver.get(url)
# Wait for the body tag to load
WebDriverWait(driver, 60).until(lambda d: d.execute_script("return document.readyState") == "complete")
# Debug: Print the page source
print(driver.page_source)
except Exception as e:
print(f"Error occurred: {e}")
finally:
# Close the browser
driver.quit()
Ran selenium to load a page but the page will not load. Tried WebDriverWait and time.sleep() but both do not work, the result of print(driver.page_source) is
<html><head></head><body></body></html>
- 2 It looks like it blocks page loading for Selenium. Try undetected-chromedriver. – S A Commented Jan 20 at 12:25
- @SA Yes adding undetected-chromedriver worked, thank you! – es07889 Commented Jan 20 at 15:02
1 Answer
Reset to default 0undetected-chromedriver worked; added the following to the header and to call ChromeOptions and the driver.
import undetected_chromedriver as uc
options = uc.ChromeOptions()
driver = uc.Chrome(options=options)