HOW TO Unblocked resources with Google Webmaster Tools

For a several weeks before. The Google Webmaster reports has been forgotten. So I wasn’t encountered the blocked resources from GoogleBot which was increased everyday causing of Google Panda 4. With its ‘Fetch as Google’ feature within Google Webmaster Tools (GWT), it shows how these blocked resources matter.
The blocked resources follows the robots.txt directives for all files that Googlebot fetches, include JavaScript, CSS and image files. If your site’s robots.txt file disallows crawling of these assets, it directly harms how well their algorithms render and index your content.
In this post, I will explain how can I diagnosing blocked Googlebot issues and solve them with Google Webmaster Tools.

Here is mine:

How to check what resources are blocked

Fortunately, Google Webmaster Tools can help you find the problem that may be blocked GoogleBot accessing your site. From Search Console Dashboard, navigate to Google Index > Blocked Resources.
At this section, you will find how many pages with blocked resources which listed below. Choose the main domain to open a list of blocked resources. At here, you can review them and find why it’s blocked.
In my case, the file style.css denied GoogleBot accessing, so the page won’t render correctly. Continue click on it, I find a list of pages has partial rendered. I choose the first item, it open a windows with information and suggested actions to help me solve this problem:

Finish 3 steps like figure above then check and fix another blocked as Google recommended.
Solve this problem is easy but you need a few weeks to allow Google re-index and update your website status.
There are steps that I’ve done to prevent it happens again in future:

1. Handling resources blocked by robots.txt: Allowing the bots to index anything and everything in your site, can actually harm your sites rankings. If you using WordPress, I unblocked wp-includes folder where has default scripts & files which used by many themes and plugins. The next position is wp-content with the same reason. Here is my new robot.txt:

   User-agent: *
   Disallow: /wp-admin/
   Disallow: /trackback/
   Disallow: /xmlrpc.php
   Disallow: /go/
   

   As you see, I blocked wp-admin folder, trackback and xmlrpc.php for prevent spam, /go/ for our affiliate links.
   Remember, If Google crawls that these resources are blocked, then Googlebot can’t use them when it renders those pages for search.

2. Make sure that Googlebot can crawl your JavaScript, CSS and image files by using the Fetch as Google feature in Google Webmaster Tools.
   

   

   
   Google Webmaster Tools shows screenshots rendered both as Googlebot and as a typical user, if the problem has fixed, you will got the result similar the image below:

   

   
   After solving the mobile usability issues, you can Submit to index, because the changes you made can influence your rankings in search results. Submit to index will make Google crawl your new changes and index your page faster. This button only appeared beside the Status result, after the render processed.
   You have two choices:
   1. Crawl only this URL: You are allowed to submit only 500 URLs per month.
   2. This URL and its direct links: You are allowed 10 submission per month for this option.
3. Be sure your website pass the Mobile-Friendly Test. In Mobilegeddon, it’s very important because Google announced that mobile-friendliness will become a ranking factor in the mobile search results on April 21, 2015. With this update, GoogleBot now has a smartphone, it’s using to visit and indexing your website, haha.
4. Guarantee of technical quality for running your website flawlessly, some common problems causes GoogleBot is indexing your website hardly like DNS, Firewall, Over-Optimize for Server Configuration…

TIPS: Don’t include your website’s XML sitemap in robot.txt because Google doesn’t interested with spider’s arguments that available in its.

NOTES: If you using MaxCDN service, be sure their custom robot.txt wasn’t activated or its content has the same with your original file on your server.
To verify it, login to MaxCDN, navigate to Contact Zones then Manage, find Robot.txt section in the SEO tab, if it was enabled, review and update its content. Wait a few minutes before Fetch and Render again.