[ad_1]
Google up to date the record of their official crawlers by including the title and knowledge for a comparatively unknown crawler that publishers have been seeing at times however no documentation for it beforehand existed.
Though Google added official documentation for this crawler the data offered appears to encourage extra clarification.
Particular Case Crawlers
Google has a number of sorts of crawlers (also referred to as bots and spiders).
The totally different types of crawlers are:
- Frequent Crawlers
These bots are principally used for indexing totally different sorts of content material. However some frequent crawlers are additionally for search testing instruments, inner Google product group use, and crawling associated to AI. - Consumer-triggered fetchers
These are bots which can be triggered by customers. This consists of makes use of equivalent to for fetching feeds or website verification. - Particular-case crawlers
These are for particular instances like for cellular advertisements webpage high quality checks or for push notification messages through Google APIs. These bots don’t obey the worldwide consumer agent directives in robots.txt which can be signaled with the asterisk (*).
The brand new crawler documentation is for the Google-Security consumer agent. The crawler is just not new however the documentation is new.
Google-Security Crawler
The documentation within the Particular-case Crawlers Google-Security crawler is one which’s utilized by Google’s processes for locating malware.
Distinctive amongst Particular-case crawlers, the Google-Security Crawler utterly ignores all robots.txt directives.
The brand new documentation for the Google-Security Crawler:
“The Google-Security consumer agent handles abuse-specific crawling, equivalent to malware discovery for publicly posted hyperlinks on Google properties.
This consumer agent ignores robots.txt guidelines.”
The complete agent string for the crawler:
"Google-Security"
Learn the brand new documentation for the Google-Security consumer agent on the Google Search Central web page for crawlers within the part dedicated to Particular-case crawlers.
Overview of Google crawlers and fetchers (consumer brokers) – Particular-case crawlers
[ad_2]
