Create web scraper for extracting data

[aashutosh-samagra]

1. Illustrated Technical Book Hindi Webpage with PDFs Link

2. Rabi Crop Details English Webpages with Text Link

[rachitavya]

Hello @ChakshuGautam @aashutosh-samagra 👋,

As much as I can get from the issue mentioned:

1. We want to download all the pdfs on the link.
2. We want data of every rabi crop from each crop's link. This scrapped data will be stored in JSON format maybe.

Correct me if I am wrong.

I can do the task using the bs4 in python by iterating to every crop's link from the page source.

[ChakshuGautam]

@rachitavya your assumptions are correct. We are looking at a single file python script with minimal dependencies to do this. bs4 should work.

[rachitavya]

I can do it by end of the day tomorrow. Kindly assign me. @ChakshuGautam

[ChakshuGautam]

Hey @rachitavya how is it going?

[rachitavya]

Hello @ChakshuGautam

• PDF task is completed.
• Regarding rabi crops website, the scrapper is able to scrap all the data from all the links into a JSON except the tables. Tables are the one left as of now. On it.
  Will be posting updates here.

[rachitavya]

Hey @ChakshuGautam,
There are multiple types of tables present in the web pages (having different dimensions in no particular pattern). I am a bit confused how to write one generic code for such variety of tables that we have.

[ChakshuGautam]

Got it. If there is no pattern, don't need to generalize. Let's feed that to an LLM and ask it to parse it.

[rachitavya]

Understood. This will be done by tomorrow. 👍

[rachitavya]

Hello @ChakshuGautam 👋🏻

The tasks are done and I want to mention a couple of constraints:

• I've tried many LLMs for table parsing and ChatGPT turns out to be the best. There are API key constraint regarding it.
• This fetching of table from API is taking so much time slowing down the scrapping.

On the other hand, I tried coding approach also, but it works only on those tables who are in proper (non complex) structures.

[ChakshuGautam]

Can you share the repo here. Let me Also get into this.

[rachitavya]

Here it is: https://github.com/rachitavya/.github

[singhalkarun]

Hey @rachitavya,

1. Can we please move the code to a separate repository, you should not use .github repository for this.
2. I see you have mentioned there is some constraint associated with API Key for ChatGPT. Can you please describe? You can find steps here to generate and use API Key.
3. Can you please detail on how many different types of tables are there? We don't need to write generic code. If the ChatGPT fails to help us, we can write different codes for different structures instead of trying to generalise them, our prime goal is to get data, and it's a one time thing for now.
4. Also, we are expecting output to be in form of pdf files.