In this video, I'll be showing you how to build an AI web scraper using Python. The application itself is super cool as it scrapes the site based on the URL you give it, grabs the DOM content, and allows us to then pass a prompt to the AI that will be able to grab information from the website that we want. In addition to this quick demo of how everything works, I will also take a deep dive into how to build the AI web scraper utilizing tools such Selenium, BeatifulSoup, LangChain, & more!
Get a free credit when you sign up for BrightData with a new account: https://brdta.com/techwithtim_ai
If you want to land a developer job check out my program with CourseCareers: https://techwithtim.net/dev
? Video Resources ?
Code In This Video: https://github.com/techwithtim/AI-Web-Scraper
ChromeDriver Download: https://googlechromelabs.github.io/chrome-for-testing/#stable
Selenum Video Tutorial: https://www.youtube.com/watch?v=NB8OceGZGjA
Ollama Download Link: https://ollama.com/
Ollama Github: https://github.com/ollama/ollama
⏳ Timestamps ⏳
00:00 | Overview
00:15 | Install/Setup
04:30 | Starting the Streamlit UI
07:34 | Web Scraping with Selenium
15:59 | Unblocking Websites/Captchas/IP Bans
22:30 | Cleaning DOM Content
31:10 | Parsing Content with Ollama (LLM)
39:45 | Ollama Installation
42:45 | Testing the Scraper
Hashtags
#python #ai #webscraping