How to Remove sitemap.xml & robots.txt from SERPs?

One of my professional friends asked me a question. The question was “how to prevent/remove the sitemap.xml and
robot.txt from the Google SERP results”

Well lets discuss…

While my friend was checking indexed pages on the Google with this search operator “site:www.YourDomaiName.com“, Google was showing the results with sitemap.xml and robot.txt links. Actually sitemap.xml doesn’t bother us, if it is showing on SERP but robot.txt should not be disclosed on Search Engines, it hampers the privacy.

So if you want to remove sitemap.xml and robot.txt from the SERP results. Just adding this to your Apache config or
your .htaccess file to solve that:

For: robot.txt
<FilesMatch “robots\.txt”>
Header set X-Robots-Tag “noindex”
</FilesMatch>

For: xml.txt
<FilesMatch “sitemap\.xml”>
Header set X-Robots-Tag “noindex”
</FilesMatch>

April 16, 2013 at 9:19 pm 3 comments

Fix the Canonical Issue for Your Domain

First check whether it is in the server or not [.htaccess is a typically hidden file so use commands to display it or access it form file manager in the control panel of your  website ], if you already have an .htaccess file, download it and edit it with Notepad or Dreamweaver. Otherwise you have to create a new text document in a plain text editor and save it with the name .htaccess.

Options +FollowSymlinks
RewriteEngine on
RewriteBase /

### non-www to www re-direct ###
RewriteCond %{HTTP_HOST} ^example.com\.com$ [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [L,R=301]

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /index\.php\ HTTP/
RewriteRule ^index\.php$ http://www.example.com/ [R=301,L]

### re-direct index.html to root / ###
RewriteCond %{THE_REQUEST} ^.*\/index\.html\ HTTP/
RewriteRule ^(.*)index\.html$ /$1 [R=301,L]

### re-direct index.htm to root / ###
RewriteCond %{THE_REQUEST} ^.*\/index\.htm\ HTTP/
RewriteRule ^(.*)index\.htm$ /$1 [R=301,L]

### re-direct index.php to root / ###
RewriteCond %{THE_REQUEST} ^.*\/index\.php\ HTTP/
RewriteRule ^(.*)index\.php$ /$1 [R=301,L]

### re-direct default.html to root / ###
RewriteCond %{THE_REQUEST} ^.*\/default\.html\ HTTP/
RewriteRule ^(.*)default\.html$ /$1 [R=301,L]

### re-direct home.html to root / ###
RewriteCond %{THE_REQUEST} ^.*\/home\.html\ HTTP/
RewriteRule ^(.*)home\.html$ /$1 [R=301,L]

### re-direct IP address to www
RewriteCond %{http_host} !^www.example.com$ [nc]
RewriteRule ^(.*)$ http://www.example.com/$1 [r=301,nc,L]

You can also use rel canonical tags if you need…
<link rel=”canonical” href=”http://www.example.com/&#8221; />

March 29, 2013 at 8:31 am Leave a comment

SEO/SMO/ORM Software & Tools

Paid SEO/SMO Software:

Free SEO/SMO Software:

  • http://www.opensiteexplorer.org: Try Open Site Explorer, the new Search Engine for Links from SEOmoz. Perform competitive website research. Explore backlinks, anchor text and more for free.
  • http://www.majesticseo.com: Majestic SEO – Link intelligence tools for SEO and Internet PR and Marketing. Site Explorer shows inbound link and site summary data.
  • chrispederick.com/work/web-developer: Firefox in itself is the most flexible browser out there. But when you pair it with the Web Developer extension Firefox becomes a useful SEO tool. With it you can easily remove the cascading styles (inline, external or both), you can take a fast glance at all the external links in a page, or you can remove all the javascript components at once just to name a few of its features. The nicest part of the extension is that you can add your own SEO tools to the menu (online or desktop).
  • tools.seobook.com: Free link analysis and keyword research tools brought to you by SEO expert Aaron Wall.
  • http://www.ranks.nl/tools/spider.html: Page Analyzer. Keyword placement, keyword density, prominence, link popularity, Google positions and much more.
  • http://www.ahrefs.com: The largest and the freshest index of live links. Index is updated every 15 min. Ahrefs Site Explorer is free to try.
  • http://www.linkdiagnosis.com:
  • Easily see backlinks from your competitors. LinkDiagnosis tool can show you all the important information about your competitor’s links. The report includes pagerank, anchor texts, no-follow information and more.
  • http://www.builtwith.com:
  • Web technology information profiler tool. Find out what a website is built with.
  • tools.pingdom.com/fpt/: Pingdom offers a free online tool that analyzes a URL’s load time. You simply enter a URL, and the tool displays a summary of the page’s load time, size, and number of corresponding HTTP requests.
  • developers.google.com/speed/pagespeed/insights: PageSpeed is a free online tool (plugins are also available) that tests a URL’s load performance. You enter a URL, and then, the tool presents a list of performance-based suggestions (e.g., optimize images, minify JavaScript, etc.), which are categorized into high, medium, and low priority items.
  • urivalet.com: Check Server Headers – Test Page Load Times – URI Valet
  • http://www.sitereportcard.com: SiteReportCard provides free website optimization, promotion and analysis tools to help you improve your website. Thetools help you find broken links, validate HTML, improve meta tags, check spelling and much more.
  • http://www.spyonweb.com: SpyOnWeb identifies multiple sites that probably belong to the same owner. On the backend, the service clusters sites that share an IP address, Google AdSense ID, Google Analytics ID, Yahoo! Publisher Network ID, or Yandex. Direct ID. Then, when you query a website’s URL, the tool presents a list of sites that have been clustered together with that URL using the previously listed information.
  • home.snafu.de/tilman/xenulink.html: A free multi-threaded link checking software to analyze web sites to find broken links. Runs on Windows 95, 98, NT, 2000, XP, Vista and 7.
  • validator.w3.org: checklink Link Checker is a free online tool that checks the validity of links for a given URL. You simply enter a URL, and then, the tool recursively checks the URL’s links for URI scheme errors, redirects, and error status codes.
  • http://www.backlinkwatch.com : backlink Watch greatly helps you to know every bit about your inbound links. It includes anchor text, pagerank, quality of backlinks by finding total outbound links any restriction of nofollow tag.
  • http://www.masspagerank.com: PageRank is a numeric value that represents how important a page is on the web. Google™ figures that when one page links to another page, it is effectively casting a vote for the other page.
  • validator.w3.org: HTML, HyperText Markup Language, Validation, W3C Markup Validation Service
  • http://www.seo-browser.com: SEO Browser is a fast, efficient, and comprehensive SEO analysis tool that can help you examine and understand the component parts of a Web page.
  • http://www.copyscape.com: Cityscape is a free plagiarism checker. The software lets you detect duplicate content and check if your articles are original.
  • adwords.google.com/select/KeywordToolExternal: The AdWords Keyword Tool is one of the most popular free keyword tools available. The tool gives you three options for querying keywords: (1) input a list of keywords, (2) input a URL, which the tool scrapes to identify keywords, or (3) select from a predefined list of categories.
  • freekeywords.wordtracker.com: Wordtracker is one of the oldest keyword research tools, and it uses the fermium model. With the free version, you enter a keyword phrase, and then, the tool presents up to 100 related keywords, along with estimated search totals.
  • ubersuggest.org: Ubersuggest is a free online keyword suggestion tool that uses Google Suggest (and other suggest services) to generate a list of related keywords. You simply enter a keyword phrase. Then, the tool adds a letter or digit in front of your phrase and displays the corresponding suggestions.
  • http://www.virante.com/seo-tools/lda-content-optimizer: Use the tool, you enter a URL (or a block of content) and a targeted keyword. The tool computes the LDA relevancy score for that URL and keyword, and then, it identifies collocated keywords from top ranking pages that are missing from the given URL (and would improve its relevancy).
  • http://www.spyfu.com: SpyFu is a competitive keyword analysis tool that uses the freemium model. When you enter a domain, the tool displays summary information, which includes the domain’s daily PPC ad budget, the domain’s daily organic SEO traffic value, and a graph of the domain’s historical budget. The tool’s domain report also shows the domain’s top 10 paid keywords, ad competitors, organic keywords, and organic competitors.
  • sharedcount.com: Shared Count is a free online service (and accompanying API) that presents the most popular social metrics for a given URL. Currently, this list of social metrics is derived from Facebook (likes, shares, and comments), Twitter (tweets), Google+ (+1s), Digg (diggs), Pinterest (pins), LinkedIn (shares), Delicious (bookmarks), and StumbleUpon (stumbles).
  • socialmention.com: SocialMention is a free social media search engine that aggregates user generated content from 100+ social media sources, including Twitter, Facebook, FriendFeed, YouTube, Digg, and Google. You simply search for a topic, and the search engine will return the most recent results from the social Web, along with descriptive statistics for those results (e.g., sentiment breakdown, top keywords, top users, etc.).
  • followerwonk.com: Follower Wonk is a Twitter analysis tool that uses the freemium model. The tool is great for identifying influencers, building out personas, or finding outreach targets for guest posts. As a free user, you are able to accomplish three tasks: (1) search Twitter bios for specific keywords, (2) compare the social graphs of up to three accounts, and (3) analyze the social graph characteristics of a single account (e.g., influence of the account, influence of the account’s followings, etc.).
  • http://www.helpareporter.com: HARO is a free online service that allows reporters to connect directly with expert sources. As a reporter, you simply submit a query on HARO’s website, and then, that query will appear with others in an email, which is delivered to the sources. As a source, you simply monitor the stream of daily emails from HARO, and if you see a query that is relevant to your expertise, you can respond to the querying reporter directly.
  • http://www.bloggerlinkup.com: Blogger LinkUp (BLU) is a completely free service that allows anyone to request guest posts, sources for interviews, and product reviews from a community of users. If you have a request, you submit it on BLU’s website. Then, that request is aggregated with other requests and sent to the community in an email. On the flip side, if you join the free community, you will receive an email every Monday, Wednesday, and Friday that contains the most recent requests. If you see a relevant request, you respond directly to the requester.
  • myblogguest.com: MyBlogGuest is a free community dedicated to helping bloggers write and publish guest posts. Once you sign up for a free account, you receive access to various forums that allow you to introduce yourself to community, search for sites looking for guest posts, and search for bloggers looking to write guest posts. The community also offers a searching mechanism that allows you to search for specific categories, tags, or users.
  • http://www.google.com/analytics: Google Analytics is one of the most popular free analytics packages available. Once you create an account, you are asked to insert a few lines of JavaScript on your site. After that code is in place, you’ll be able to start monitoring your site’s traffic through the Google Analytics interface.
  • http://www.google.com/webmasters: Google is the most popular search engine in the world, and as a result, Google Webmaster Tools (GWT) is one of the most useful free SEO tools available because it gives you direct access to some of Google’s information about your site.
  • http://www.bing.com/toolbox/webmaster: Bing isn’t as popular as Google, but Bing Webmaster Tools (BWT) is still an extremely valuable tool because it allows you to view your site through the eyes of yet another search engine.
  • http://www.webgnomes.org/free-seo-analysis: The tool on the list is a shameless plug for free SEO analysis. You simply enter a URL, and then, the tool sends you a report that covers 20 important on-page SEO best practices, which are divided into 5 categories: Accessibility, Content, HTML Markup (<head>), HTML Markup (<body>), and URL.
  • http://www.coolsocial.net: Analyze your Social Media Impact. Take a look at your competitors.
  • http://www.bulkseotools.com/bulk-check-alexa-ranking.php: Bulk alexa rank checker. Bulk check alexa rank.
  • http://www.authoritydomains.com/bulk-ip-checker.php: Use the free bulk class c IP checker to find duplicate IP addresses as well as duplicate class c blocks. You can view results online or download to a CSV for excel.
  • http://www.ipligence.com/freetools: Information about IP addresses: Owner or internet provider, exact latitude and longitude, time zone, city, region, country and continent of the visitor connection.
  • http://www.xml-sitemaps.com: Create your Sitemap XML, HTML, ROR and more…
  • http://www.geo-tag.de/generator/en.html: Use this HTML Geo-Tag-Generator to create html geo tags for your WEB-Site and show your location to the community

Free/Paid ORM Software & Tools:

  • http://www.trackur.com: Accurate Social Media Monitoring Tools & Measurement Software
  • http://www.naymz.com: Naymz was established to help you create, define, and manage who you are on the Web.
  • http://www.brandseye.com: BrandsEye is an ORM tool that keeps you informed about how your brand is perceived online and allows you to manage and shape your online reputation.
  • http://www.brandwatch.com: Social media monitoring and analytics tool, chosen by pioneering brands and agencies all over the world.
  • rankur.com: Rankur is an affordable tool for social media monitoring, online reputation management, and brand protection. Every 24 hours
  • http://www.whostalkin.com: WhosTalkin helps search for conversations, surrounding the topics that we care about most. Join us, in conversation today.
  • tweetbeep.com: Get an email when they do talk about you!
  • http://www.mywot.com: WOT is a free website reputation, rating and review tool that helps find trustworthy websites and protects against online scams, untrustworthy links, and unreliable web stores. Free add-on for Firefox, IE, Chrome, Opera and Safari browsers.
  • http://www.google.com/alerts: Google Alerts are email updates of the latest relevant Google results (web, news, etc.) based on your queries.

February 22, 2013 at 8:44 am 3 comments

2012 in review

The WordPress.com stats helper monkeys prepared a 2012 annual report for this blog.

Here’s an excerpt:

600 people reached the top of Mt. Everest in 2012. This blog got about 3,000 views in 2012. If every person who reached the top of Mt. Everest viewed this blog, it would have taken 5 years to get that many views.

Click here to see the complete report.

December 31, 2012 at 5:47 am Leave a comment

Google Webmaster Tools Sent Messages 90% Black Hat Issues Related

Matt Cutts said that 90% of the messages sent out via Google Webmaster Tools are related to black hat issues.

Watch the video and if have a question? visit Google Webmaster Help Forum


Source: GoogleWebmasterHelp

December 4, 2012 at 5:23 am Leave a comment

Older Posts

SEO Expert › About Maloy Gupta

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 10 other subscribers

Search Engine Optimizer - Blog Stats

  • 19,171 Hit Counter Is Delayed By Up To 60 Seconds.

My Calendar

May 2024
M T W T F S S
 12345
6789101112
13141516171819
20212223242526
2728293031  

Search Posts Monthly Wise

Become a Follower @ Facebook


Maloy Gupta has received rating of 5 out of 5 based on 5 ratings.See more user reviews.

Prepared by Maloy Gupta
© - 2012 All Rights Received.