flat assembler
Message board for the users of flat assembler.

Index > Feedback > why the hell you block googlebot?

Author
Thread Post new topic Reply to topic
asmrox



Joined: 19 Jan 2008
Posts: 160
asmrox
i tried to search some info, and my old topics but...
http://board.flatassembler.net/robots.txt
you blocked google. Why?

http://www.google.com/search?q=site:board.flatassembler.net
Post 05 Mar 2008, 17:53
View user's profile Send private message Reply with quote
Ehtyar



Joined: 26 Sep 2006
Posts: 51
Ehtyar
I have to second this, it's bugged me for a while, phpbb search just can't match google.

Ehtyar.
Post 05 Mar 2008, 21:24
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17665
Location: In your JS exploiting you and your system
revolution
I think that the robots create too much traffic. I have heard that if you try to unblock just googlebot then other bots also come along and index your site.

flatassembler.net is already very slow (Tomasz, it's time to find a new host), and with robots going through and indexing this can only make it slower.
Post 06 Mar 2008, 02:50
View user's profile Send private message Visit poster's website Reply with quote
edfed



Joined: 20 Feb 2006
Posts: 4242
Location: 2018
edfed
and if flat board is very slow, we cannot access it in less than 3 seconds...
then, yes, changing the host, leave this bot blocked (make this board elitist cause bad referenced by google) and everything will be ok.
Post 06 Mar 2008, 10:21
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17665
Location: In your JS exploiting you and your system
revolution
If a new host can cope with the traffic then I see no reason to continue to block the bots. Unless there is anther reason for blocking the bots that has not been revealed?
Post 06 Mar 2008, 10:27
View user's profile Send private message Visit poster's website Reply with quote
edfed



Joined: 20 Feb 2006
Posts: 4242
Location: 2018
edfed
bots are bad. they pollute the screen with random posts.
Post 06 Mar 2008, 11:14
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17665
Location: In your JS exploiting you and your system
revolution
edfed: The bots being discussed are not "chat bots"
Post 06 Mar 2008, 11:22
View user's profile Send private message Visit poster's website Reply with quote
edfed



Joined: 20 Feb 2006
Posts: 4242
Location: 2018
edfed
ok! then i hope it will never be a forum bot, like ads by ggogle...
there are a lot of forums with a post from google or else company. if it appens on fasm board, i'll be sad.
Post 06 Mar 2008, 11:44
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17665
Location: In your JS exploiting you and your system
revolution
edfed: read up on "search bots", they index the forum so that you can search for things on sites like google. They never make posts.
Post 06 Mar 2008, 11:47
View user's profile Send private message Visit poster's website Reply with quote
asmrox



Joined: 19 Jan 2008
Posts: 160
asmrox
Post 07 Mar 2008, 12:53
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 7796
Location: Kraków, Poland
Tomasz Grysztar
google bot was much more agressive than the other ones.
Post 07 Mar 2008, 13:10
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17665
Location: In your JS exploiting you and your system
revolution
I thought this:
http://board.flatassembler.net/robots.txt wrote:
User-agent: *
Disallow: /
would block all robots?
Post 07 Mar 2008, 13:45
View user's profile Send private message Visit poster's website Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
revolution wrote:
I thought this:
http://board.flatassembler.net/robots.txt wrote:
User-agent: *
Disallow: /
would block all robots?

Only bots that actually respect robots.txt? Smile - and some of them might be looking for hardcoded strings, not accepting wildcard...

_________________
Image - carpe noctem
Post 07 Mar 2008, 13:50
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17665
Location: In your JS exploiting you and your system
revolution
f0dder wrote:
Only bots that actually respect robots.txt? Smile - and some of them might be looking for hardcoded strings, not accepting wildcard...
So it seems that yahoo, live and altavista all couldn't care less about the wildcard thing. Naughty, naughty.
Post 07 Mar 2008, 13:54
View user's profile Send private message Visit poster's website Reply with quote
itsnobody



Joined: 01 Feb 2008
Posts: 93
Location: Silver Spring, MD
itsnobody
if the search bots were unblocked the traffic would increase by like 10 times
Post 08 Mar 2008, 12:37
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on GitHub, YouTube, Twitter.

Website powered by rwasa.