How To Use IndexNow API With Python For Bulk Indexing

IndexNow is a protocol developed by Microsoft Bing and adopted by Yandex that permits site owners and search engine marketing professionals to simply notify engines like google when a webpage has been up to date by way of an API.

And at this time, Microsoft introduced that it’s making the protocol simpler to implement by guaranteeing that submitted URLs are shared between engines like google.

Given its optimistic implications and the promise of a quicker indexing expertise for publishers, the IndexNow API must be on each search engine marketing skilled’s radar.

Utilizing Python for automating URL submission to the IndexNow API or making an API request to the IndexNow API for bulk URL indexing could make managing IndexNow extra environment friendly for you.

On this tutorial, you’ll discover ways to do exactly that, with step-by-step directions for utilizing the IndexNow API to submit URLs to Microsoft Bing in bulk with Python.

Notice: The IndexNow API is much like Google’s Indexing API with just one distinction: the Google Indexing API is just for job ads or broadcasting net pages that include a video object inside it.

Google introduced that they will take a look at the IndexNow API however hasn’t up to date us since.

Bulk Indexing Utilizing IndexNow API with Python: Getting Began

Under are the requirements to grasp and implement the IndexNow API tutorial.

Under are the Python packages and libraries that will probably be used for the Python IndexNow API tutorial.

• Advertools (should).
• Pandas (should).
• Requests (should).
• Time (elective).
• JSON (elective).

Earlier than getting began, studying the fundamentals may help you to grasp this IndexNow API and Python tutorial higher. We will probably be utilizing an API Key and a .txt file to offer authentication together with particular HTTP Headers.

IndexNow API Utilization Steps with Python.

1. Import The Python Libraries

To make use of the mandatory Python libraries, we’ll use the “import” command.

• Advertools will probably be used for sitemap URL extraction.
• Requests will probably be used for making the GET and POST requests.
• Pandas will probably be used for taking the URLs within the sitemap into a listing object.
• The “time” module is to forestall a “An excessive amount of request” error with the “sleep()” technique.
• JSON is for probably modifying the POST JSON object if wanted.

Under, you can find the entire crucial import traces for the IndexNow API tutorial.

import advertools as adv
import pandas as pd
import requests
import json
import time

2. Extracting The Sitemap URLs With Python

To extract the URLs from a sitemap file, totally different net scraping strategies and libraries can be utilized akin to Requests or Scrapy.

However to maintain issues easy and environment friendly, I’ll use my favourite Python search engine marketing bundle – Advertools.

With solely a single line of code, the entire URLs inside a sitemap could be extracted.

sitemap_urls = adv.sitemap_to_df("https://www.instance.com/sitemap_index.xml")

The “sitemap_to_df” technique of the Advertools can extract all of the URLs and different sitemap-related tags akin to “lastmod” or “precedence.”

Under, you’ll be able to see the output of the “adv.sitemap_to_df” command.

Sitemap URL Extraction could be carried out by way of Advertools’ “sitemap_to_df” technique.

All the URLs and dates are specified throughout the “sitemap_urls” variable.

Since sitemaps are helpful sources for engines like google and SEOs, Advertools’ sitemap_to_df technique can be utilized for a lot of totally different duties together with a Sitemap Python Audit.

However that’s a subject for an additional time.

3. Take The URLs Into A Listing Object With “to_list()”

Python’s Pandas library has a way for taking a knowledge body column (knowledge collection) into a listing object, to_list().

Under is an instance utilization:

sitemap_urls["loc"].to_list()

Under, you’ll be able to see the outcome:

Pandas’ “to_list” technique can be utilized with Advertools for itemizing the URLs.

All URLs throughout the sitemap are in a Python listing object.

4. Perceive The URL Syntax Of IndexNow API Of Microsoft Bing

Let’s check out the URL syntax of the IndexNow API.

Right here’s an instance:

https://<searchengine>/indexnow?url=url-changed&key=your-key

The URL syntax represents the variables and their relations to one another throughout the RFC 3986 requirements.

• The <searchengine> represents the search engine identify that you’ll use the IndexNow API for.
• “?url=” parameter is to find out the URL that will probably be submitted to the search engine by way of IndexNow API.
• “&key=” is the API Key that will probably be used throughout the IndexNow API.
• “&keyLocation=” is to offer an authenticity that reveals that you’re the proprietor of the web site that IndexNow API will probably be used for.

The “&keyLocation” will deliver us to the API Key and its “.txt” model.

5. Collect The API Key For IndexNow And Add It To The Root

You’ll want a legitimate key to make use of the IndexNow API.

Use this hyperlink to generate the Microsoft Bing IndexNow API Key.

 There isn’t a restrict for producing the IndexNow API Key.

Clicking the “Generate” button creates an IndexNow API Key.

Whenever you click on on the obtain button, it can obtain the “.txt” model of the IndexNow API Key.

IndexNow API Key could be generated by Microsoft Bing’s said deal with.

Downloaded IndexNow API Key as txt file.

The TXT model of the API key would be the file identify and in addition to throughout the textual content file.

IndexNow API Key in TXT File must be the identical with the identify of the file, and the precise API Key worth.

The subsequent step is importing this TXT file to the basis of the web site’s server.

Since I exploit FileZilla for my FTP, I’ve uploaded it simply to my net server’s root.

By placing the .txt file into the net server’s root folder, the IndexNow API setup could be accomplished.

The subsequent step is performing a easy for a loop instance for submitting the entire URLs throughout the sitemap.

6. Submit The URLs Inside The Sitemap With Python To IndexNow API

To submit a single URL to the IndexNow, you should use a single “requests.get()” occasion. However to make it extra helpful, we’ll use a for a loop.

To submit URLs in bulk to the IndexNow API with Python, comply with the steps under:

1. Create a key variable with the IndexNow API Key worth.
2. Substitute the <searchengine> part with the search engine that you just wish to submit URLs (Microsoft Bing, or Yandex, for now).
3. Assign the entire URLs from the sitemap inside a listing to a variable.
4. Use the “txt” file throughout the root of the net server with its URL worth.
5. Place the URL, key, and key location URL throughout the string manipulation worth.
6. Begin your for a loop, and use the “requests.get()” for the entire URLs throughout the sitemap.

Under, you’ll be able to see the implementation:

key = "22bc7c564b334f38b0b1ed90eec8f2c5"
url = sitemap_urls["loc"].to_list()

for i in url:
          endpoint = f"https://bing.com/indexnow?url={i}&key={key}&keyLocation={location}"
          response = requests.get(endpoint)
          print(i)
          print(endpoint)
          print(response.status_code, response.content material)
          #time.sleep(5)

In the event you’re involved about sending too many requests to the IndexNow API, you should use the Python time module to make the script wait between each request.

Right here you’ll be able to see the output of the script:

The empty string because the request’s response physique represents the success of the IndexNow API request based on Microsoft Bing’s IndexNow documentation.

The 200 Standing Code signifies that the request was profitable.

With the for a loop, I’ve submitted 194 URLs to Microsoft Bing.

Based on the IndexNow Documentation, the HTTP 200 Response Code alerts that the search engine is conscious of the change within the content material or the brand new content material. However it doesn’t essentially assure indexing.

For example, I’ve used the identical script for an additional web site. After 120 seconds, Microsoft Bing says that 31 outcomes are discovered. And conveniently, it reveals 4 pages.

The one downside is that on the primary web page there are solely two outcomes, and it says that the URLs are blocked by Robots.txt even when the blocking was eliminated earlier than submission.

This could occur if the robots.txt was modified to take away some URLs earlier than utilizing the IndexNow API as a result of plainly Bing doesn’t verify the Robots.txt once more.

Thus, for those who beforehand blocked them, they attempt to index your web site however nonetheless use the earlier model of the robots.txt file.

It reveals what is going to occur for those who use IndexNow API by blocking Bingbot by way of Robots.txt.

On the second web page, there is just one outcome:

Microsoft Bing may use a special indexation and pagination technique than Google. The second web page reveals just one among the many 31 outcomes.

On the third web page, there is no such thing as a outcome, and it reveals the Microsoft Bing Translate for translating the string throughout the search bar.

It reveals typically, Microsoft Bing infers the “web site” search operator as part of the question.

After I checked Google Analytics, it reveals that Bing nonetheless hadn’t crawled the web site or listed it. I do know that is true as I additionally checked the log information.

Under, you will note the Bing Webmaster Device’s report for the instance web site:

It says that I submitted 38 URLs.

The subsequent step will contain the majority request with the POST Technique and a JSON object.

7. Carry out An HTTP Publish Request To The IndexNow API

To carry out an HTTP put up request to the IndexNow API for a set of URLs, a JSON object must be used with particular properties.

• Host property represents the search engine hostname.
• Key represents the API Key.
• Key represents the situation of the API Key’s txt file throughout the net server.
• urlList represents the URL set that will probably be submitted to the IndexNow API.
• Headers signify the POST Request Headers that will probably be used that are “Content material-type” and “charset.”

Since this can be a POST request, the “requests.put up” will probably be used as a substitute of the “requests.get().”

Under, you can find an instance of a set of URLs submitted to Microsoft Bing’s IndexNow API.

knowledge = {
  "host": "www.bing.com",
  "key": "22bc7c564b334f38b0b1ed90eec8f2c5",
  "keyLocation": "https://www.instance.com/22bc7c564b334f38b0b1ed90eec8f2c5.txt",
  "urlList": [
    'https://www.example.com/technical-seo/http-header/',
    'https://www.example.com/python-seo/nltk/lemmatize',
    'https://www.example.com/pagespeed/broser-hints/preload',
    'https://www.example.com/python-seo/nltk/stemming',
    'https://www.example.com/python-seo/categorize-queries/',
    'https://www.example.com/python-seo/nltk/tokenization',
    'https://www.example.com/review/oncrawl/',
    'https://www.example.com/technical-seo/hreflang/',
    'https://www.example.com/technical-seo/multilingual-seo/'
      ]
}
headers = {"Content material-type":"utility/json", "charset":"utf-8"}
r = requests.put up("https://bing.com/", knowledge=knowledge, headers=headers)
r.status_code, r.content material

Within the instance above, now we have carried out a POST Request to index a set of URLs.

We’ve got used the “knowledge” object for the “knowledge parameter of requests.put up,” and the headers object for the “headers” parameter.

Since we POST a JSON object, the request ought to have the “content-type: utility/json” key and worth with the “charset:utf-8.”

After I make the POST request, 135 seconds later, my dwell logfile evaluation dashboard began to point out the fast hits from the Bingbot.

8. Create Customized Operate For IndexNow API To Make Time

Making a customized perform for IndexNow API is beneficial to lower the time that will probably be spent on the code preparation.

Thus, I’ve created two totally different customized Python features to make use of the IndexNow API for bulk requests and particular person requests.

Under, you can find an instance for less than the majority requests to the IndexNow API.

The customized perform for bulk requests is known as “submit_url_set.”

Even for those who simply fill within the parameters, nonetheless it is possible for you to to make use of it correctly.

def submit_url_set(set_:listing, key, location, host="https://www.bing.com", headers={"Content material-type":"utility/json", "charset":"utf-8"}):
     key = "22bc7c564b334f38b0b1ed90eec8f2c5"
     set_ = sitemap_urls["loc"].to_list()
     knowledge = {
     "host": "www.bing.com",
     "key": key,
     "keyLocation": "https://www.instance.com/22bc7c564b334f38b0b1ed90eec8f2c5.txt",
     "urlList": set_
     }
     r = requests.put up(host, knowledge=knowledge, headers=headers)
     return r.status_code

A proof of this tradition perform:

• The “Set_” parameter is to offer a listing of URLs.
• “Key” parameter is to offer an IndexNow API Key.
• “Location” parameter is to offer the situation of the IndexNow API Key’s txt file throughout the net server.
• “Host” is to offer the search engine host deal with.
• “Headers” is to offer the headers which can be crucial for the IndexNow API.

I’ve outlined a few of the parameters with default values akin to “host” for Microsoft Bing. If you wish to use it for Yandex, you have to to state it whereas calling the perform.

Under is an instance utilization:

submit_url_set(set_=sitemap_urls["loc"].to_list(), key="22bc7c564b334f38b0b1ed90eec8f2c5", location="https://www.instance.com/22bc7c564b334f38b0b1ed90eec8f2c5.txt")

If you wish to extract sitemap URLs with a special technique, or if you wish to use the IndexNow API for a special URL set, you have to to alter “set_” parameter worth.

Under, you will note an instance of the Customized Python perform for the IndexNow API for less than particular person requests.

def submit_url(url, location, key = "22bc7c564b334f38b0b1ed90eec8f2c5"):
     key = "22bc7c564b334f38b0b1ed90eec8f2c5"
     url = sitemap_urls["loc"].to_list()
     for i in url:
          endpoint = f"https://bing.com/indexnow?url={i}&key={key}&keyLocation={location}"
          response = requests.get(endpoint)
          print(i)
          print(endpoint)
          print(response.status_code, response.content material)
          #time.sleep(5)

Since that is for a loop, you’ll be able to submit extra URLs one after the other. The search engine can prioritize these kind of requests in another way.

Among the bulk requests will embody non-important URLs, the person requests is likely to be seen as extra affordable.

If you wish to embody the sitemap URL extraction throughout the perform, you need to embody Advertools naturally into the features themselves.

Ideas For Utilizing The IndexNow API With Python

An Overview of How The IndexNow API Works, Capabilities & Makes use of

• The IndexNow API doesn’t assure that your web site or the URLs that you just submitted will probably be listed.
• You need to solely submit URLs which can be new or for which the content material has modified.
• The IndexNow API impacts the crawl price range.
• Microsoft Bing has a threshold for the URL Content material High quality and Calculation of the Crawl Want for a URL. If the submitted URL isn’t adequate, they could not crawl it.
• You’ll be able to submit as much as 10,000 URLs.
• The IndexNow API suggests submitting URLs even when the web site is small.
• Submitting the identical pages many occasions inside a day can block the IndexNow API from crawling the redundant URLs or the supply.
• The IndexNow API is beneficial for websites the place the content material adjustments steadily, like each 10 minutes.
• IndexNow API is beneficial for pages which can be gone and are returning a 404 response code. It lets the search engine know that the URLs are gone.
• IndexNow API can be utilized for notifying of latest 301 or 302 redirects.
• The 200 Standing Response Code signifies that the search engine is conscious of the submitted URL.
• The 429 Standing Code signifies that you made too many requests to the IndexNow API.
• In the event you put a “txt” file that incorporates the IndexNow API Key right into a subfolder, the IndexNow API can be utilized just for that subfolder.
• If in case you have two totally different CMS, you should use two totally different IndexNow API Keys for 2 totally different web site sections
• Subdomains want to make use of a special IndexNow API key.
• Even for those who already use a sitemap, utilizing IndexNow API is beneficial as a result of it effectively tells the various search engines of web site adjustments and reduces pointless bot crawling.
• All engines like google that undertake the IndexNow API (Microsoft Bing and Yandex) share the URLs which can be submitted between one another.

IndexNow API Documentation and utilization suggestions could be discovered above.

On this IndexNow API tutorial and guideline with Python, now we have examined a brand new search engine expertise.

As a substitute of ready to be crawled, publishers can notify the various search engines to crawl when there’s a want.

IndexNow reduces using search engine knowledge middle sources, and now you understand how to make use of Python to make the method extra environment friendly, too.

Extra sources:

An Introduction To Python & Machine Studying For Technical search engine marketing

The best way to Use Python to Monitor & Measure Web site Efficiency

Superior Technical search engine marketing: A Full Information

Featured Picture: metamorworks/Shutterstock