1. Scott Hamilton Robert E Ron Paul Lee

    Scott Hamilton Robert E Ron Paul Lee Straight Awesome

    Joined:
    Jan 5, 2008
    Messages:
    29,016
    Location:
    TN
    Ratings:
    +14,152
  2. Starguard

    Starguard Fresh Meat

    Joined:
    Nov 21, 2006
    Messages:
    7,402
    Location:
    Midwest
    Ratings:
    +766
    User-Agent: *Disallow: /francais/newsDisallow: /english/newsDisallow: /arabic/newsDisallow: /espanol/newsDisallow: /portugues/newsDisallow: /deutsch/news
  3. Bailey

    Bailey It's always Christmas Eve Super Moderator

    Joined:
    Apr 1, 2004
    Messages:
    27,146
    Location:
    Adelaide, South Australia
    Ratings:
    +39,745
  4. Kyle

    Kyle You will regret this!

    Joined:
    Mar 29, 2004
    Messages:
    9,150
    Location:
    California?!?!
    Ratings:
    +2,814
    AFP is suing Google over its indexing of their news stories, which they sell to other agencies. Unfortunately for the AFP, Google is indexing them from the sites of the other agencies.

    There is also some question as to whether the Google News indexing properly follows the robots.txt guidelines, but it's unlikely anything is going to come of this case.

    By the way, all of what I just said?

    That should be in the FIRST POST IN THE THREAD.
    • Agree Agree x 1
  5. Scott Hamilton Robert E Ron Paul Lee

    Scott Hamilton Robert E Ron Paul Lee Straight Awesome

    Joined:
    Jan 5, 2008
    Messages:
    29,016
    Location:
    TN
    Ratings:
    +14,152
    When AFP laucned the suit, they didn't have a robot text at all!
  6. Kyle

    Kyle You will regret this!

    Joined:
    Mar 29, 2004
    Messages:
    9,150
    Location:
    California?!?!
    Ratings:
    +2,814
    Well, thank you for keeping us updated on a site's spiders.txt file.

    :rolleyes:
  7. Scott Hamilton Robert E Ron Paul Lee

    Scott Hamilton Robert E Ron Paul Lee Straight Awesome

    Joined:
    Jan 5, 2008
    Messages:
    29,016
    Location:
    TN
    Ratings:
    +14,152
    I just find it interesting, as I'm spending about 50% of my time on SEO now.