I am trying to Scrape Screener.in website to extract some information related to stocks. However while trying to extract Quarterly Results section there are some field which is hidden and when click on + button it show additional information related to parent header. I need to have this information

I am using below python code which is giving me a dataframe but without additional information

url = f'/'
print(url)
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
page = urlopen(req).read()
soup = BeautifulSoup(page, 'html.parser')
table = soup.find_all("table", {"class": "data-table responsive-text-nowrap"})[0]
df = pd.read_html(StringIO(str(table)))[0]
df

Above code is working fine however I am not able to pull additional information

Can somebody help me with this?

I am trying to Scrape Screener.in website to extract some information related to stocks. However while trying to extract Quarterly Results section there are some field which is hidden and when click on + button it show additional information related to parent header. I need to have this information

I am using below python code which is giving me a dataframe but without additional information

url = f'https://www.screener.in/company/TATAPOWER/consolidated/'
print(url)
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
page = urlopen(req).read()
soup = BeautifulSoup(page, 'html.parser')
table = soup.find_all("table", {"class": "data-table responsive-text-nowrap"})[0]
df = pd.read_html(StringIO(str(table)))[0]
df

Above code is working fine however I am not able to pull additional information

Can somebody help me with this?

• python
• pandas
• web-scraping
• beautifulsoup
• python-requests

• 1 Many sites are driven by JS and the html is not generated until the JS events are triggered to add elements to the DOM. You can't parse HTML that isn't there, so in such cases you may need a different library more like a web scraper or web automation library (there are many but as an example when I was taking a short course on web development we were introduced to Splinter, a python framework I believe based on Selenium). Although you may use your automation code to "click" the right button ... and still use BS to parse the new html. – topsail Commented Feb 15 at 18:07

1 Answer 1

As already commented, the content is reloaded on demand, but it is precisely these requests that can be replicated in order to obtain the content as well.

To do this, you have to iterate over the rows of the table and make the request if necessary.

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = f'https://www.screener.in/company/TATAPOWER/consolidated/'
soup = BeautifulSoup(requests.get(url, headers={'User-Agent': 'Mozilla/5.0'}).text)

keys = ['Item'] + list(soup.select_one('#quarters thead tr').stripped_strings)

data = []

for row in soup.select('#quarters tbody tr')[:-1]:
    if row.td.button:
        data.append(dict(zip(keys,[c.text for c in row.select('td')])))
        d = requests.get(f'https://www.screener.in/api/company/3371/schedules/?parent={row.td.button.text.strip(" +")}&section=quarters&consolidated=', headers={'User-Agent': 'Mozilla/5.0'}).json()
        first_key = next(iter(d))
        data.append({"Item": first_key, **d[first_key]})     
    else:
        data.append(dict(zip(keys,row.stripped_strings)))

pd.DataFrame(data)

Result: