I am trying to create a script using python and selenium to automate the checkout process at bestbuy.ca.

I get all the way to the final stage where you click to review the final order, but get the following 403 forbidden message (as seen in the network response) when I try to click through to the final step.

Is there something server side that has detected that I am using selenium and preventing me to proceed?

How can I hide the fact that it is selenium being used?

These are the options I am using for selenium:

options = Options()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("start-maximized")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)

I currently have 10 second delays after each action(ie open page, wait, click add to cart, wait, click checkout, wait)

I have implemented a random useragent to be used on each run:

import fake_useragent
ua = UserAgent()
userAgent = ua.random
options.add_argument(f'user-agent={userAgent}')

I have also modified my chromedriver binary as per the comments in THIS THREAD

Error seen when proceeding to order review page:

I am trying to create a script using python and selenium to automate the checkout process at bestbuy.ca.

I get all the way to the final stage where you click to review the final order, but get the following 403 forbidden message (as seen in the network response) when I try to click through to the final step.

Is there something server side that has detected that I am using selenium and preventing me to proceed?

How can I hide the fact that it is selenium being used?

These are the options I am using for selenium:

options = Options()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument("start-maximized")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)

I currently have 10 second delays after each action(ie open page, wait, click add to cart, wait, click checkout, wait)

I have implemented a random useragent to be used on each run:

import fake_useragent
ua = UserAgent()
userAgent = ua.random
options.add_argument(f'user-agent={userAgent}')

I have also modified my chromedriver binary as per the comments in THIS THREAD

Error seen when proceeding to order review page:

• javascript
• python
• html
• selenium
• selenium-chromedriver

• 1 Have you tried with user-agent? – KunduK Commented Apr 7, 2021 at 16:09
• 2 Best guess is this actually due to timing issues - sending in selenium actions faster than the page can react to them. Try inserting some delays. – David K. Hess Commented Apr 7, 2021 at 16:09
• 1 Does the same happen in debug mode too? – Lia Commented Apr 7, 2021 at 17:04
• 1 @DavidK.Hess I currently have 10 second delays after each action (ie open page, wait, click add to cart, wait, click checkout, wait) – Bajan Commented Apr 7, 2021 at 17:18
• 1 Are you logged-in to bestbuy.ca or processing the order as a guest? – Life is complex Commented Apr 12, 2021 at 3:06

3 Answers 3

After much testing the last few days, here are the options that have allowed me to bypass the restrictions I was facing.

1. Modified cdc_ string in my chromedriver

2. Chromedriver options:

    options.add_argument('--disable-blink-features=AutomationControlled')
    options.add_argument("--disable-extensions")
    options.add_experimental_option('useAutomationExtension', False)
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    chrome_driver = webdriver.Chrome(options=options)
   

3. Change the property value of the navigator for webdriver to undefined:

chrome_driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")

After all three of these were implemented I no longer faced any 403 error when navigating the site and the cart/checkout process.

In my case, either using code to control the browser, or simply starting Chrome through python and manually using the browser always leads to the 403 error, even just adding a product to the cart.

As you said, I think that this site someway knows that the user is using Selenium or some sort of automation tool and the server is blocking API requests.

Searching in stackoverflow I found this https://stackoverflow.com/a/52108199/3228768 but editing the chromedriver results anyway in a failure.

The only way I completed the flow is settings this options:

u = 'https://www.bestbuy.ca/en-ca/category/appliances/26517'
# relevant part start here
options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features")
options.add_argument("--disable-blink-features=AutomationControlled")
# relevant part ends here 
driver = webdriver.Chrome(executable_path=r"chromedriver.exe", options=options)
driver.maximize_window()
driver.get(u)

In this way I managed to add a product to the cart. I think you could use it to proceed the flow until checkout. Let me know.

Try this one: https://github.com/ultrafunkamsterdam/undetected-chromedriver

It avoids the selenium detection quite well, I've been having good result with it so far. Headless is not guaranteed though.

import undetected_chromedriver as uc
driver = uc.Chrome()
driver.get('https://www.bestbuy.ca/')