$\"How$
\u00a0 <\/p>\n
Internet scraping is the method of amassing data from web sites routinely. A traditional scraper normally extracts uncooked textual content, HTML parts, or the complete web page content material. However when you’re constructing AI brokers or massive language mannequin (LLM) purposes, sending the whole webpage to the mannequin will not be all the time the very best strategy.<\/p>\n
A greater means is to first clear the web page, convert it into Markdown, after which use an LLM to grasp the content material and return solely the reply the consumer wants. This makes the output cleaner, simpler to learn, and simpler to make use of in one other workflow.<\/p>\n
It additionally helps cut back token utilization. As an alternative of passing a messy webpage stuffed with navigation hyperlinks, buttons, scripts, footers, and repeated content material, we solely ship the helpful web page content material to the mannequin. The LLM then returns a centered reply in Markdown as an alternative of dumping the entire web page again to the consumer.<\/p>\n
On this information, we are going to construct a easy AI internet scraper in Python utilizing Jupyter Pocket book. It is going to fetch a webpage, clear the HTML, convert it into Markdown, settle for a consumer question, and return a transparent Markdown reply primarily based on the web page content material.<\/p>\n
\u00a0<\/p>\n
#\u00a0<\/span>Setting Up<\/h2>\n
\u00a0
We are going to use Jupyter Pocket book for this challenge. It makes it simpler to check every step first earlier than turning the scraper into a correct software programming interface (API) or software.<\/p>\n
Begin by putting in the required Python packages:<\/p>\n
!pip set up requests beautifulsoup4 markdownify openai ftfy python-dotenv<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
We are going to use:<\/p>\n
Within the subsequent cell, import the required libraries:<\/p>\n
\nimport os\nimport re\nimport requests\n\nfrom bs4 import BeautifulSoup, Remark\nfrom ftfy import fix_text\nfrom markdownify import markdownify as markdownify_html\nfrom openai import OpenAI\nfrom dotenv import load_dotenv\nfrom IPython.show import Markdown, show<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
Subsequent, make sure that your OpenAI API key’s accessible as an atmosphere variable. The safer means is to create a .env<\/code> file in the identical folder as your pocket book and add your key there:<\/p>\n
\nOPENAI_API_KEY=your_api_key_here<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
Then load it contained in the pocket book:<\/p>\n
\nload_dotenv()\n\nshopper = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
You may also examine that the important thing was loaded accurately:<\/p>\n
\nif not os.getenv(\"OPENAI_API_KEY\"):\n    elevate ValueError(\"OPENAI_API_KEY is lacking. Add it to your .env file first.\")<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
Additionally make sure that your OpenAI platform account has billing arrange. For brand new API accounts, it’s possible you’ll want so as to add pay as you go credit earlier than you’ll be able to run API calls. If a mannequin will not be accessible in your account, use one other mannequin out of your OpenAI dashboard.<\/p>\n
Now outline the mannequin identify:<\/p>\n
\nMODEL_NAME = \"gpt-5.4-nano\"<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
We’re utilizing a smaller mannequin right here as a result of this job doesn’t want a big reasoning mannequin. The purpose is straightforward: learn the cleaned webpage content material, perceive the consumer question, and return a centered Markdown reply.<\/p>\n
\u00a0<\/p>\n
#\u00a0<\/span>Fetching the Webpage<\/h2>\n\u00a0
Now we are going to create the primary operate. This operate will fetch the webpage utilizing the requests<\/code> package deal and return the uncooked HTML.<\/p>\n
\ndef fetch_page(url: str) -> str:\n    \"\"\"\n    Obtain the HTML content material from a webpage.\n    \"\"\"\n    headers = {\n        \"Consumer-Agent\": \"SimpleAIScraper\/1.0\"\n    }\n\n    response = requests.get(url, headers=headers, timeout=15)\n    response.raise_for_status()\n\n    return response.textual content<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
The Consumer-Agent<\/code> header tells the web site that the request is coming from our scraper. Some web sites block requests that don’t embrace a consumer agent, so including one makes the request a bit extra dependable.<\/p>\n
We additionally use timeout<\/code> to keep away from ready indefinitely if the web site doesn’t reply. The raise_for_status()<\/code> name will cease the code if the request fails \u2014 for instance, if the web page returns a 404<\/code> or 500<\/code> error.<\/p>\n
Now let’s take a look at the operate with an actual web site:<\/p>\n
\nuncooked = fetch_page(\"https:\/\/www.olostep.com\/\")\nprint(uncooked[:500])<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
This can obtain the uncooked HTML from the webpage and print the primary 500 characters.<\/p>\n
\u00a0<\/p>\n

Uncooked HTML output | Picture by Writer<\/span><\/center>
\n\u00a0<\/p>\nAt this stage, the output will nonetheless look messy as a result of it incorporates the complete web page HTML, together with tags, scripts, format parts, and different content material we don’t want.<\/p>\n
\u00a0<\/p>\n
#\u00a0<\/span>Cleansing the HTML<\/h2>\n\u00a0
The uncooked HTML from a webpage normally incorporates loads of content material we don’t want. It will possibly embrace scripts, styling, navigation menus, buttons, kinds, headers, footers, popups, and different format parts.<\/p>\n
Earlier than sending the web page content material to the LLM, we have to clear the HTML. This helps cut back noise and makes the ultimate Markdown a lot simpler for the mannequin to grasp.<\/p>\n
We are going to use BeautifulSoup to parse the HTML and take away pointless parts.<\/p>\n
\ndef clean_html(html):\n    html = fix_text(html)\n\n    soup = BeautifulSoup(html, \"html.parser\")\n\n    # Take away apparent noisy tags\n    for tag in soup([\n        \"script\", \"style\", \"noscript\", \"svg\", \"img\", \"iframe\",\n        \"nav\", \"header\", \"footer\", \"aside\", \"form\", \"button\"\n    ]):\n        tag.decompose()\n\n    noise_words = [\n        \"cursor\",\n        \"modal\",\n        \"popup\",\n        \"floating\",\n        \"signup\",\n        \"login\",\n        \"cookie\",\n        \"banner\",\n        \"navbar\",\n        \"menu\",\n        \"footer\",\n        \"header\",\n        \"subscribe\",\n        \"newsletter\",\n        \"loading\",\n        \"wait\",\n        \"success\",\n        \"auth\",\n        \"w-nav\",\n        \"w-form\"\n    ]\n\n    # First accumulate noisy tags\n    tags_to_remove = []\n\n    for tag in soup.find_all(True):\n        if tag.attrs is None:\n            proceed\n\n        class_value = tag.get(\"class\", [])\n        id_value = tag.get(\"id\", \"\")\n\n        if isinstance(class_value, record):\n            class_text = \" \".be part of(class_value).decrease()\n        else:\n            class_text = str(class_value).decrease()\n\n        id_text = str(id_value).decrease()\n\n        if any(phrase in class_text or phrase in id_text for phrase in noise_words):\n            tags_to_remove.append(tag)\n\n    # Then take away them safely\n    for tag in tags_to_remove:\n        tag.decompose()\n\n    physique = soup.physique if soup.physique else soup\n\n    return str(physique)<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
First, we use fix_text()<\/code> to scrub any damaged or unusual textual content encoding points. Then BeautifulSoup parses the HTML so we are able to take away the elements we don’t want.<\/p>\n
We take away apparent noisy tags like script<\/code>, type<\/code>, nav<\/code>, header<\/code>, footer<\/code>, type<\/code>, and button<\/code>. These sections normally don’t assist reply the consumer question and might waste tokens.<\/p>\n
After that, we search for noisy class names and IDs. Many web sites use phrases like popup<\/code>, cookie<\/code>, navbar<\/code>, publication<\/code>, or modal<\/code> inside their HTML. If a tag incorporates these phrases, we accumulate it and take away it safely.<\/p>\n
Now let’s run the operate on the uncooked HTML:<\/p>\n
\nclear = clean_html(uncooked)\nprint(clear[:500])<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
As you’ll be able to see, the webpage is now a lot cleaner. It nonetheless incorporates helpful HTML tags and textual content, however a lot of the noisy format, scripts, navigation, and popups have been eliminated.<\/p>\n
\u00a0<\/p>\n

Cleaned HTML output | Picture by Writer<\/span><\/center>
\n\u00a0<\/p>\n#\u00a0<\/span>Changing HTML to Markdown<\/h2>\n\u00a0
Now we are going to convert the cleaned HTML into Markdown. Markdown is simpler to learn, simpler to avoid wasting, and simpler for the LLM to grasp in comparison with uncooked HTML.<\/p>\n
This step additionally helps cut back enter tokens as a result of we take away pointless formatting, photographs, clean strains, and repeated textual content. For the conversion, we are going to use markdownify.<\/p>\n
\ndef html_to_markdown(html):\n    markdown_text = markdownify_html(\n        html,\n        heading_style=\"ATX\",\n        bullets=\"-\"\n    )\n\n    markdown_text = fix_text(markdown_text)\n\n    # Take away picture markdown\n    markdown_text = re.sub(r\"![.*?](.*?)\", \"\", markdown_text)\n\n    # Take away additional areas and clean strains\n    markdown_text = re.sub(r\"[ t]+\", \" \", markdown_text)\n    markdown_text = re.sub(r\"n{3,}\", \"nn\", markdown_text)\n\n    strains = []\n\n    skip_lines = [\n        \"click to try\",\n        \"wait...\",\n        \"you've successfully reserved your spot.\",\n        \"thank you! your submission has been received!\",\n        \"oops! something went wrong while submitting the form.\",\n        \"product\",\n        \"resources\",\n        \"company\"\n    ]\n\n    for line in markdown_text.splitlines():\n        line = line.strip()\n\n        if not line:\n            proceed\n\n        if line.decrease() in skip_lines:\n            proceed\n\n        strains.append(line)\n\n    return \"n\".be part of(strains)<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
First, we use markdownify to transform the cleaned HTML into Markdown. We set the heading type to ATX<\/code>, which implies headings will use customary Markdown syntax with #<\/code>, ##<\/code>, and ###<\/code>.<\/p>\n
Then we run fix_text()<\/code> once more to scrub any remaining encoding points. After that, we take away picture Markdown as a result of picture hyperlinks are normally not helpful for answering text-based questions.<\/p>\n
We additionally take away additional areas and clean strains so the ultimate content material is compact. This makes the web page simpler to examine and helps cut back the variety of tokens despatched to the mannequin.<\/p>\n
The skip_lines<\/code> record removes repeated web site textual content equivalent to type messages, navigation labels, and small call-to-action textual content. You possibly can replace this record primarily based on the web site you might be scraping.<\/p>\n
Now let’s run the operate:<\/p>\n
\nmd = html_to_markdown(clear)\nprint(md[:500])<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
As you’ll be able to see, the textual content is now a lot cleaner and nearer to the format we would like. As an alternative of uncooked HTML, we now have readable Markdown with helpful headings, paragraphs, and bullet factors.<\/p>\n
\u00a0<\/p>\n

Markdown output | Picture by Writer<\/span><\/center>
\n\u00a0<\/p>\n#\u00a0<\/span>Asking a Consumer Question In opposition to the Web page<\/h2>\n\u00a0
Now we are going to create the operate that sends the cleaned Markdown content material to the LLM. This operate takes two inputs: the webpage content material in Markdown and the consumer question.<\/p>\n
As an alternative of asking the mannequin to summarize the entire web page, we ask it to reply a particular query utilizing solely the web page content material. This makes the response extra centered and helpful.<\/p>\n
\ndef answer_query_from_page(markdown_text, user_query):\n    immediate = f\"\"\"\nYou might be an AI internet scraping assistant.\n\nYou'll obtain Markdown extracted from a webpage.\n\nYour job is to reply the consumer's question utilizing solely the helpful web page content material.\n\nConsumer question:\n{user_query}\n\nWebpage Markdown:\n{markdown_text}\n\nDirections:\n- Return solely clear Markdown.\n- Use solely data from the webpage Markdown.\n- Don't invent lacking particulars.\n- Ignore navigation hyperlinks, buttons, CTAs, popups, ornamental labels, picture captions, and repeated advertising and marketing fragments.\n- Ignore strains like \"Begin at no cost\", \"Contact Gross sales\", \"Your AI Agent\", and ornamental workflow examples except they immediately reply the question.\n- Concentrate on headings, paragraphs, product descriptions, function sections, pricing particulars, documentation textual content, and factual claims.\n- If the web page doesn't include the reply, say: \"The web page doesn't include this data.\"\n- Maintain the reply brief, clear, and centered.\n\"\"\"\n\n    response = shopper.responses.create(\n        mannequin=MODEL_NAME,\n        enter=immediate\n    )\n\n    return response.output_text<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
The immediate is a very powerful a part of this step. It tells the mannequin what function it ought to play, what content material it could possibly use, and how much reply it ought to return.<\/p>\n
We additionally inform the mannequin to make use of solely the supplied Markdown. That is necessary as a result of we don’t want the mannequin to guess or add data that’s not current on the webpage.<\/p>\n
The instruction to return solely clear Markdown makes the output simpler to show in a pocket book, save to a file, or cross into one other AI workflow.<\/p>\n
This operate is the place the AI internet scraper turns into genuinely helpful. We’re not simply extracting web page textual content \u2014 we’re asking the LLM to grasp the cleaned web page and return the precise reply the consumer is searching for.<\/p>\n
\u00a0<\/p>\n
#\u00a0<\/span>Creating the Full AI Internet Scraper<\/h2>\n\u00a0
Now we are going to create the ultimate operate that connects the whole lot collectively.<\/p>\n
This operate will take the URL and the consumer question as inputs. It is going to then fetch the webpage, clear the HTML, convert the content material into Markdown, and return the reply utilizing the gpt-5.4-nano<\/code> mannequin.<\/p>\n
\ndef ai_web_scraper(url, user_query):\n    raw_html = fetch_page(url)\n    cleaned_html = clean_html(raw_html)\n    markdown_text = html_to_markdown(cleaned_html)\n    reply = answer_query_from_page(markdown_text, user_query)\n\n    return reply<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
That is our full AI internet scraper pipeline. As an alternative of manually working every step one after the other, we are able to now name a single operate and get a clear Markdown reply from any webpage.<\/p>\n
The move is straightforward:<\/p>\n
\nFetch the webpage.\n<\/li>\n
Clear the HTML.\n<\/li>\n
Convert it into Markdown.\n<\/li>\n
Ask the LLM a query.\n<\/li>\n
Return the ultimate reply.\n<\/li>\n<\/ul>\nThis retains the code easy and straightforward to reuse later in an API, chatbot, or agent workflow.<\/p>\n
\u00a0<\/p>\n
#\u00a0<\/span>Testing the AI Internet Scraper<\/h2>\n\u00a0
Now let’s take a look at our AI internet scraper. We are going to present it with an internet site URL and ask what the corporate does.<\/p>\n
\nurl = \"https:\/\/www.olostep.com\/\"\nuser_query = \"What does this firm do?\"\noutcome = ai_web_scraper(url, user_query)\n\nshow(Markdown(outcome))<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
In return, we get a correct Markdown response concerning the firm and its product. That is significantly better than returning the complete webpage content material as a result of the reply is concentrated, readable, and immediately associated to the consumer question.<\/p>\n
\u00a0<\/p>\n

Scraper output for a corporation overview question | Picture by Writer<\/span><\/center>
\n\u00a0<\/p>\nNow let’s strive a unique web page and ask about pricing.<\/p>\n
\nurl = \"https:\/\/www.olostep.com\/pricing\"\nuser_query = \"Assist me perceive the pricing\"\noutcome = ai_web_scraper(url, user_query)\n\nshow(Markdown(outcome))<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
In just a few seconds, we get a clear response that’s straightforward to grasp. As an alternative of manually visiting the pricing web page and looking for the related data, the scraper extracts the web page, cleans it, and asks the LLM to elucidate solely what issues.<\/p>\n
\u00a0<\/p>\n

Scraper output for a pricing question | Picture by Writer<\/span><\/center>
\n\u00a0<\/p>\nWe are able to additionally save the ultimate response as a Markdown file.<\/p>\n
\nwith open(\"ai_scraper_result.md\", \"w\", encoding=\"utf-8\") as file:\n    file.write(outcome)\n\nprint(\"Markdown saved to ai_scraper_result.md\")<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
Output:<\/p>\n
\nMarkdown saved to ai_scraper_result.md<\/code><\/pre>\n<\/div>\n\u00a0<\/p>\n
Now the result’s saved as a Markdown file, which you’ll be able to open, edit, share, or use in one other workflow.<\/p>\n
\u00a0<\/p>\n
#\u00a0<\/span>Remaining Ideas<\/h2>\n\u00a0
Constructing your personal AI instruments is way simpler now. With just a few strains of Python and an LLM, we turned a traditional webpage right into a easy question-answering engine that may learn the web page, perceive the consumer question, and return a clear Markdown reply.<\/p>\n
That is highly effective as a result of you don’t all the time want a fancy system to unravel a particular drawback. Typically, a small specialised answer is sufficient.<\/p>\n
However it’s also necessary to keep in mind that the whole lot has a price. Operating the app on a server prices cash. Calling an LLM prices cash. Sustaining the scraper, fixing damaged pages, dealing with errors, and bettering the system over time additionally prices money and time.<\/p>\n
So earlier than constructing your personal customized answer, it’s price  current instruments like Olostep<\/a><\/strong>, Firecrawl<\/a><\/strong>, or Exa<\/a><\/strong>. In some instances, paying for a ready-made scraping or internet intelligence API might make extra sense. In different instances \u2014 particularly if the duty is small, native, or very particular \u2014 constructing your personal light-weight answer could be the higher choice.
\u00a0
\u00a0<\/p>\n
Abid Ali Awan<\/a><\/strong><\/b><\/a> (@1abidaliawan<\/a>) is a licensed information scientist skilled who loves constructing machine studying fashions. Presently, he’s specializing in content material creation and writing technical blogs on machine studying and information science applied sciences. Abid holds a Grasp’s diploma in know-how administration and a bachelor’s diploma in telecommunication engineering. His imaginative and prescient is to construct an AI product utilizing a graph neural community for college kids battling psychological sickness.<\/p>\n<\/p><\/div>\n\n","protected":false},"excerpt":{"rendered":"
\u00a0 Internet scraping is the method of amassing data from web sites routinely. A traditional scraper normally extracts uncooked textual content, HTML parts, or the complete web page content material. However when you’re constructing AI brokers or massive language mannequin (LLM) purposes, sending the whole webpage to the mannequin will not be all the time […]<\/p>\n","protected":false},"author":2,"featured_media":17773,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[73,1258,10178,4127,505],"class_list":["post-17771","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-build","tag-python","tag-scraper","tag-simple","tag-web"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/17771","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=17771"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/17771\/revisions"}],"predecessor-version":[{"id":17772,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/17771\/revisions\/17772"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/17773"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=17771"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=17771"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=17771"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}