Google pushes for an official web crawler standard 1

While the Robots Exclusion Protocol has been around for a quarter of a century, it was only an unofficial standard — and that has created problems with teams interpreting the format differently. One might handle an edge case differently than another. Google’s initiative, which includes submitting its approach to the Internet Engineering Task Force, would “better define” how crawlers are supposed to handle robots.txt and create fewer rude surprises.

The draft isn’t fully available, but it would work with more than just websites, include a minimum file size, set a max one-day cache time and give sites a break if there are server problems.

There’s no guarantee this will become a standard, at least as-is. If it does, though, it could help web visitors as much as it does creators. You might see more consistent web search results that respect sites’ wishes. If nothing else, this shows that Google isn’t completely averse to opening important assets if it thinks they’ll advance both its technology and the industry at large.

About the author

E-Crypto News was developed to assist all cryptocurrency investors in developing profitable cryptocurrency portfolios through the provision of timely and much-needed information. Investments in cryptocurrency require a level of detail, sensitivity, and accuracy that isn’t required in any other market and as such, we’ve developed our databases to help fill in information gaps.

Related Posts

E-Crypto News Executive Interviews

Automated trading with HaasBot Crypto Trading Bots

Blockchain/Cryptocurrency Questions and Answers

What Are E-stablecoins And How Do They Operate?
What Are E-Stablecoins And How Do They Operate?
August 11, 2022
How to Choose a Legit Crypto Casino?
August 5, 2022
Spend Crypto
5 Ways to Spend Crypto
August 2, 2022
What Is A DAO LLC?
What Is A DAO LLC?
August 2, 2022
Can Running A Lightning Node Earn You Passive Income?
Can Running A Lightning Node Earn You Passive Income?
July 5, 2022


CryptoCurrencyUSDChange 1hChange 24hChange 7d
Bitcoin24,401 0.31 % 0.52 % 6.16 %
Ethereum1,954.1 0.75 % 1.93 % 15.40 %
Tether1.001 0.02 % 0.01 % 0.04 %
USD Coin1.000 0.44 % 0.25 % 0.18 %
BNB320.63 0.32 % 1.85 % 1.62 %
Cardano0.5739 0.54 % 1.50 % 11.90 %
XRP0.3823 0.36 % 0.50 % 2.66 %
Binance USD1.003 0.27 % 0.17 % 0.31 %
Solana42.12 0.56 % 2.22 % 3.81 %
Dogecoin0.08130 0.69 % 10.07 % 18.13 %

bitcoin
Bitcoin (BTC) $ 24,349.00
ethereum
Ethereum (ETH) $ 1,943.32
tether
Tether (USDT) $ 1.00
usd-coin
USD Coin (USDC) $ 1.00
bnb
BNB (BNB) $ 319.79
cardano
Cardano (ADA) $ 0.57137
xrp
XRP (XRP) $ 0.381248
binance-usd
Binance USD (BUSD) $ 1.00
solana
Solana (SOL) $ 45.35
dogecoin
Dogecoin (DOGE) $ 0.080974