I'm trying to use scrapy with selenium to be able to interact with javascript and still have the powerful scraping framework that scrapy offers. I've written a script that visits , enters "Amsterdam" in the search bar and then clicks on the search button succesfully. After clicking on the search button I want scrapy to retreive an element from the newly rendered page. Unfortunately scrapy doesn't return any values.

This is what my code looks like:

from selenium import webdriver
from scrapy.loader import ItemLoader
from scrapy import Request
from scrapy.crawler import CrawlerProcess
from properties import PropertiesItem
import scrapy


class BasicSpider(scrapy.Spider):
    name = "basic"
    allowed_domains = ["web"]
    # Start on a property page
    start_urls = ['']

    def __init__(self):
        chrome_path = '/Users/username/Documents/chromedriver'
        self.driver = webdriver.Chrome(chrome_path)

    def parse(self, response):
        self.driver.get(response.url)
        text_box = self.driver.find_element_by_xpath('//*[@id="searchText"]')
        submit_button = self.driver.find_element_by_xpath('//*[@id="button_search"]')
        text_box.send_keys("Amsterdam")
        submit_button.click()

        l = ItemLoader(item=PropertiesItem(), response=response)
        l.add_xpath('description', '//*[@id="results"]/ul/li[1]/div[2]/h3/a/')

        return l.load_item()


process = CrawlerProcess()
process.crawl(BasicSpider)
process.start()

"properties" is another script that looks like this:

from scrapy.item import Item, Field

class PropertiesItem(Item):
    # Primary fields
    description = Field()

Q: How do I succesfully make scrapy find the element I call "description" by its xpath on the page selenium reached and return it as output?

Thanks in advance!

I'm trying to use scrapy with selenium to be able to interact with javascript and still have the powerful scraping framework that scrapy offers. I've written a script that visits http://www.iens.nl, enters "Amsterdam" in the search bar and then clicks on the search button succesfully. After clicking on the search button I want scrapy to retreive an element from the newly rendered page. Unfortunately scrapy doesn't return any values.

This is what my code looks like:

from selenium import webdriver
from scrapy.loader import ItemLoader
from scrapy import Request
from scrapy.crawler import CrawlerProcess
from properties import PropertiesItem
import scrapy


class BasicSpider(scrapy.Spider):
    name = "basic"
    allowed_domains = ["web"]
    # Start on a property page
    start_urls = ['http://www.iens.nl']

    def __init__(self):
        chrome_path = '/Users/username/Documents/chromedriver'
        self.driver = webdriver.Chrome(chrome_path)

    def parse(self, response):
        self.driver.get(response.url)
        text_box = self.driver.find_element_by_xpath('//*[@id="searchText"]')
        submit_button = self.driver.find_element_by_xpath('//*[@id="button_search"]')
        text_box.send_keys("Amsterdam")
        submit_button.click()

        l = ItemLoader(item=PropertiesItem(), response=response)
        l.add_xpath('description', '//*[@id="results"]/ul/li[1]/div[2]/h3/a/')

        return l.load_item()


process = CrawlerProcess()
process.crawl(BasicSpider)
process.start()

"properties" is another script that looks like this:

from scrapy.item import Item, Field

class PropertiesItem(Item):
    # Primary fields
    description = Field()

Q: How do I succesfully make scrapy find the element I call "description" by its xpath on the page selenium reached and return it as output?

Thanks in advance!

• javascript
• python
• selenium
• scrapy

• @eLRuLL it did reach parse, otherwise selenium wouldn't have moved to the next page right? – titusAdam Commented Jan 10, 2017 at 14:50
• You may want to have a look at this, for other ways to couple Scrapy with Selenium: stackoverflow./a/36085533/1204332 – Ivan Chaer Commented Jan 10, 2017 at 15:02

1 Answer 1

the response object you are assigning to your ItemLoader is the scrapy response, not Selenium's.

I would remend creating a new Selector with the page source returned by selenium:

from scrapy import Selector
...

selenium_response_text = driver.page_source

new_selector = Selector(text=selenium_response_text)
l = ItemLoader(item=PropertiesItem(), selector=new_selector)
...

that way the add_xpath will get information from that response structure instead of scrapy (that you don't actually need).