8 Helpful Python Libraries for search engine optimization & How To Use Them

Editor’s notice: As 2021 winds down, we’re celebrating with a 12 Days of Christmas Countdown of the most well-liked, useful skilled articles on Search Engine Journal this yr.

This assortment was curated by our editorial group primarily based on every article’s efficiency, utility, high quality, and the worth created for you, our readers.

Every day till December twenty fourth, we’ll repost the most effective columns of the yr, beginning at No. 12 and counting all the way down to No. 1. Our countdown begins in the present day with our No. 3 column, which was initially revealed on March 18, 2021.

Ruth Everett’s article on using Python libraries for automating and undertaking search engine optimization duties makes a marketer’s work a lot simpler. It’s very straightforward to learn and ideal for novices and much more skilled search engine optimization professionals that need to use Python extra.  

Nice work on this, Ruth, and we actually admire your contributions to Search Engine Journal.

Take pleasure in!   

Python libraries are a enjoyable and accessible approach to get began with studying and utilizing Python for search engine optimization.

A Python library is a group of helpful capabilities and code that permit you to full a lot of duties without having to jot down the code from scratch.

There are over 100,000 libraries obtainable to make use of in Python, which can be utilized for capabilities from knowledge evaluation to creating video video games.

On this article, you’ll discover a number of totally different libraries I’ve used for finishing search engine optimization tasks and duties. All of them are beginner-friendly and also you’ll discover loads of documentation and sources that can assist you get began.

Why Are Python Libraries Helpful For search engine optimization?

Every Python library accommodates capabilities and variables of all kinds (arrays, dictionaries, objects, and many others.) which can be utilized to carry out totally different duties.

For search engine optimization, for instance, they can be utilized to automate sure issues, predict outcomes, and supply clever insights.

It’s doable to work with simply vanilla Python, however libraries might be used to make duties a lot simpler and faster to jot down and full.

Python Libraries For search engine optimization Duties

There are a variety of helpful Python libraries for search engine optimization duties together with knowledge evaluation, net scraping, and visualizing insights.

This isn’t an exhaustive record, however these are the libraries I discover myself utilizing probably the most for search engine optimization functions.

Pandas

Pandas is a Python library used for working with desk knowledge. It permits for high-level knowledge manipulation the place the important thing knowledge construction is a DataFrame.

DataFrames are just like Excel spreadsheets, nonetheless, they aren’t restricted to row and byte limits and are additionally a lot sooner and extra environment friendly.

One of the best ways to get began with Pandas is to take a easy CSV of knowledge (a crawl of your web site, for instance) and save this inside Python as a DataFrame.

After getting this saved in Python, you possibly can carry out a lot of totally different evaluation duties together with aggregating, pivoting, and cleansing knowledge.

For instance, if I’ve an entire crawl of my web site and need to extract solely these pages which are indexable, I’ll use a built-in Pandas operate to incorporate solely these URLs in my DataFrame.

import pandas as pd 
df = pd.read_csv('/Customers/rutheverett/Paperwork/Folder/file_name.csv')
df.head
indexable = df[(df.indexable == True)]
indexable

Requests

The following library is named Requests and is used to make HTTP requests in Python.

Requests makes use of totally different request strategies comparable to GET and POST to make a request, with the outcomes being saved in Python.

One instance of this in motion is a straightforward GET request of URL, this may print out the standing code of a web page:

import requests
response = requests.get('https://www.deepcrawl.com') print(response)

You may then use this outcome to create a decision-making operate, the place a 200 standing code means the web page is out there however a 404 means the web page isn’t discovered.

if response.status_code == 200:
    print('Success!')
elif response.status_code == 404:
    print('Not Discovered.')

It’s also possible to use totally different requests comparable to headers, which show helpful details about the web page just like the content material kind or how lengthy it took to cache the response.

headers = response.headers
print(headers)

response.headers['Content-Type']

There may be additionally the power to simulate a selected person agent, comparable to Googlebot, in an effort to extract the response this particular bot will see when crawling the web page.

headers = {'Person-Agent': 'Mozilla/5.0 (appropriate; Googlebot/2.1; +http://www.google.com/bot.html)'} ua_response = requests.get('https://www.deepcrawl.com/', headers=headers) print(ua_response)

Lovely Soup

Lovely Soup is a library used to extract knowledge from HTML and XML recordsdata.

Enjoyable truth: The BeautifulSoup library was really named after the poem from Alice’s Adventures in Wonderland by Lewis Carroll.

As a library, BeautifulSoup is used to make sense of net recordsdata and is most frequently used for net scraping, as it could remodel an HTML doc into totally different Python objects.

For instance, you possibly can take a URL and use Lovely Soup along with the Requests library to extract the title of the web page.

from bs4 import BeautifulSoup 
import requests
url="https://www.deepcrawl.com" 
req = requests.get(url) 
soup = BeautifulSoup(req.textual content, "html.parser")
title = soup.title print(title)

Moreover, utilizing the find_all methodology, BeautifulSoup allows you to extract sure components from a web page, comparable to all a href hyperlinks on the web page:

url="https://www.deepcrawl.com/information/technical-seo-library/" 
req = requests.get(url) 
soup = BeautifulSoup(req.textual content, "html.parser")

for hyperlink in soup.find_all('a'): 
    print(hyperlink.get('href'))

Placing Them Collectively

These three libraries may also be used collectively, with Requests used to make the HTTP request to the web page we wish to use BeautifulSoup to extract data from.

We are able to then remodel that uncooked knowledge right into a Pandas DataFrame to carry out additional evaluation.

URL = 'https://www.deepcrawl.com/weblog/'
req = requests.get(url)
soup = BeautifulSoup(req.textual content, "html.parser")

hyperlinks = soup.find_all('a')

df = pd.DataFrame({'hyperlinks':hyperlinks})
df

Matplotlib And Seaborn

Matplotlib and Seaborn are two Python libraries used for creating visualizations.

Matplotlib means that you can create a lot of totally different knowledge visualizations comparable to bar charts, line graphs, histograms, and even heatmaps.

For instance, if I needed to take some Google Developments knowledge to show the queries with probably the most recognition over a interval of 30 days, I may create a bar chart in Matplotlib to visualise all of those.

Seaborn, which is constructed upon Matplotlib, gives much more visualization patterns comparable to scatterplots, field plots, and violin plots along with line and bar graphs.

It differs barely from Matplotlib because it makes use of fewer syntax and has built-in default themes.

A technique I’ve used Seaborn is to create line graphs in an effort to visualize log file hits to sure segments of an internet site over time.

sns.lineplot(x = "month", y = "log_requests_total", hue="class", knowledge=pivot_status)
plt.present()

This specific instance takes knowledge from a pivot desk, which I used to be in a position to create in Python utilizing the Pandas library, and is one other manner these libraries work collectively to create an easy-to-understand image from the information.

Advertools

Advertools is a library created by Elias Dabbas that can be utilized to assist handle, perceive, and make choices primarily based on the information we’ve as search engine optimization professionals and digital entrepreneurs.

Sitemap Evaluation

This library means that you can carry out a lot of totally different duties comparable to downloading, parsing, and analyzing XML Sitemaps to extract patterns or analyze how typically content material is added or modified.

Robots.txt Evaluation

One other fascinating factor you are able to do with this library is to make use of a operate to extract an internet site’s robots.txt right into a DataFrame, in an effort to simply perceive and analyze the foundations set.

It’s also possible to run a take a look at throughout the library in an effort to examine whether or not a specific user-agent is ready to fetch sure URLs or folder paths.

URL Evaluation

Advertools additionally allows you to parse and analyze URLs in an effort to extract data and higher perceive analytics, SERP, and crawl knowledge for sure units of URLs.

It’s also possible to break up URLs utilizing the library to find out issues such because the HTTP scheme getting used, the principle path, further parameters, and question strings.

Selenium

Selenium is a Python library that’s typically used for automation functions. The commonest use case is testing net purposes.

One in style instance of Selenium automating a movement is a script that opens a browser and performs a lot of totally different steps in an outlined sequence comparable to filling in varieties or clicking sure buttons.

Selenium employs the identical precept as is used within the Requests library that we coated earlier.

Nonetheless, it is not going to solely ship the request and look forward to the response but additionally render the webpage that’s being requested.

To get began with Selenium, you will have a WebDriver in an effort to make the interactions with the browser.

Every browser has its personal WebDriver; Chrome has ChromeDriver and Firefox has GeckoDriver, for instance.

These are straightforward to obtain and arrange together with your Python code. Here’s a helpful article explaining the setup course of, with an instance challenge.

Scrapy

The ultimate library I needed to cowl on this article is Scrapy.

Whereas we are able to use the Requests module to crawl and extract inner knowledge from a webpage, in an effort to cross that knowledge and extract helpful insights we additionally want to mix it with BeautifulSoup.

Scrapy basically means that you can do each of those in a single library.

Scrapy can be significantly sooner and extra highly effective, completes requests to crawl, extracts and parses knowledge in a set sequence, and means that you can protect the information.

Inside Scrapy, you possibly can outline a lot of directions such because the title of the area you wish to crawl, the beginning URL, and sure web page folders the spider is allowed or not allowed to crawl.

Scrapy can be utilized to extract the entire hyperlinks on a sure web page and retailer them in an output file, for instance.

class SuperSpider(CrawlSpider):
   title="extractor"
   allowed_domains = ['www.deepcrawl.com']
   start_urls = ['https://www.deepcrawl.com/knowledge/technical-seo-library/']
   base_url="https://www.deepcrawl.com"
   def parse(self, response):
       for hyperlink in response.xpath('//div/p/a'):
           yield {
               "hyperlink": self.base_url + hyperlink.xpath('.//@href').get()
           }

You may take this one step additional and observe the hyperlinks discovered on a webpage to extract data from all of the pages that are being linked to from the beginning URL, form of like a small-scale replication of Google discovering and following hyperlinks on a web page.

from scrapy.spiders import CrawlSpider, Rule
 
 
class SuperSpider(CrawlSpider):
    title="follower"
    allowed_domains = ['en.wikipedia.org']
    start_urls = ['https://en.wikipedia.org/wiki/Web_scraping']
    base_url="https://en.wikipedia.org"
 
    custom_settings = {
        'DEPTH_LIMIT': 1
    }
 
    def parse(self, response):
        for next_page in response.xpath('.//div/p/a'):
            yield response.observe(next_page, self.parse)
 
        for quote in response.xpath('.//h1/textual content()'):
            yield {'quote': quote.extract() }

Be taught extra about these tasks, amongst different instance tasks, right here.

Remaining Ideas

As Hamlet Batista all the time mentioned, “the easiest way to be taught is by doing.”

I hope that discovering a few of the libraries obtainable has impressed you to get began with studying Python, or to deepen your information.

Python Contributions From The search engine optimization Trade

Hamlet additionally liked sharing sources and tasks from these within the Python search engine optimization group. To honor his ardour for encouraging others, I needed to share a few of the wonderful issues I’ve seen from the group.

As a beautiful tribute to Hamlet and the search engine optimization Python group he helped to domesticate, Charly Wargnier has created search engine optimization Pythonistas to gather contributions of the wonderful Python tasks these within the search engine optimization group have created.

Hamlet’s priceless contributions to the search engine optimization Neighborhood are featured.

Moshe Ma-yafit created a brilliant cool script for log file evaluation, and on this put up explains how the script works. The visualizations it is ready to show together with Google Bot Hits By Machine, Each day Hits by Response Code, Response Code % Whole, and extra.

Koray Tuğberk GÜBÜR is presently engaged on a Sitemap Well being Checker. He additionally hosted a RankSense webinar with Elias Dabbas the place he shared a script that data SERPs and Analyses Algorithms.

It basically data SERPs with common time variations, and you’ll crawl all of the touchdown pages, mix knowledge and create some correlations.

John McAlpin wrote an article detailing how you should use Python and Knowledge Studio to spy in your rivals.

JC Chouinard wrote a full information to utilizing the Reddit API. With this, you possibly can carry out issues comparable to extracting knowledge from Reddit and posting to a Subreddit.

Rob Might is engaged on a brand new GSC evaluation instrument and constructing a number of new area/actual websites in Wix to measure in opposition to its higher-end WordPress competitor whereas documenting it.

Masaki Okazawa additionally shared a script that analyzes Google Search Console Knowledge with Python.

🎉 Pleased #RSTwittorial Thursday with @saksters 🥳

Analyzing Google Search Console Knowledge with #Python 🐍🔥

Right here’s the output 👇 pic.twitter.com/9l5Xc6UsmT

— RankSense (@RankSense) February 25, 2021

2021 SEJ Christmas Countdown:

Featured picture: jakkaje879/Shutterstock