[ad_1]
Monday, September 21, 2020
Final 12 months we launched the
robots.txt parser and matcher that we use in
our manufacturing techniques to the open supply world. Since then, we have seen individuals construct new instruments
with it,
contribute to the
open supply library (successfully bettering our manufacturing systems- thanks!), and launch new
language variations like golang and
rust, which make it simpler for
builders to construct new instruments.
With the intern season ending right here at Google, we wished to spotlight two new releases associated to
robots.txt that have been made potential by two interns engaged on the Search Open Sourcing group,
Andreea Dutulescu and
Ian Dolzhanskii.
Robots.txt Specification Check
First, we’re releasing a
testing framework for robots.txt
parser builders, created by Andreea. The undertaking gives a testing instrument that may validate
whether or not a robots.txt parser follows the Robots Exclusion Protocol, or to what extent. Presently
there is no such thing as a official and thorough technique to assess the correctness of a parser, so Andreea constructed a
instrument that can be utilized to create robots.txt parsers which can be following the protocol.
Java robots.txt parser and matcher
Second, we’re releasing an official
Java port of the C++ robots.txt parser,
created by Ian. Java is the
third hottest programming language
on GitHub and it is extensively used at Google as properly, so no marvel it has been probably the most requested
language port. The parser is a 1-to-1 translation of the C++ parser by way of capabilities and
habits, and it has been completely examined for parity towards a big corpora of robots.txt
guidelines. Groups are already planning to make use of the Java robots.txt parser in Google manufacturing
techniques, and we hope that you will discover it helpful, too.
As normal, we welcome your contributions to those tasks. In case you constructed one thing with the
C++ robots.txt parser or with these new
releases, tell us so we will probably aid you unfold the phrase! In case you discovered a bug, assist
us repair it by opening a problem on GitHub or instantly contributing with a pull request. In case you
have questions or feedback about these tasks, catch us on
Twitter!
It was our real pleasure to host Andreea and Ian, and we’re unhappy that their internship is
ending. Their contributions assist make the Web a greater place and we hope that we will
welcome them again to Google sooner or later.
[ad_2]
