Comments on: Robots.txt is not a security measure http://diovo.com/2008/09/robotstxt-is-not-a-security-measure/ Startups, Programming and stuff... Mon, 07 May 2012 14:22:46 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Niyaz http://diovo.com/2008/09/robotstxt-is-not-a-security-measure/comment-page-1/#comment-21929 Niyaz Mon, 26 Mar 2012 07:22:37 +0000 http://www.diovo.com/?p=264#comment-21929 You must be trolling. In that case, welcome! You must be trolling. In that case, welcome!

]]> By: Bypasser http://diovo.com/2008/09/robotstxt-is-not-a-security-measure/comment-page-1/#comment-21927 Bypasser Mon, 26 Mar 2012 07:16:19 +0000 http://www.diovo.com/?p=264#comment-21927 Why are those who do not know how, such cowards? Do not penalize those who know what you DO NOT. Newbstats Why are those who do not know how, such cowards?

Do not penalize those who know what you DO NOT.

Newbstats

]]> By: Anthony http://diovo.com/2008/09/robotstxt-is-not-a-security-measure/comment-page-1/#comment-4379 Anthony Fri, 01 Jul 2011 22:13:59 +0000 http://www.diovo.com/?p=264#comment-4379 I think lawyers should look into this! They could have another pay day with this neglect. Internet law allows for copyright simply by stating a site or software is copyrighted. I was told by an attorney to just put copyright on my software that I wrote and it would be copyrighted. I imagine the same goes for web pages. If an author does not want a work released then it should not be released! Same should go for the robots.txt file. Copyrighted material that is not wanted to be released should not be released. I understand it would be wise to have security configured. That is another issue. That is not the same thing (There are laws for that the FBI takes care of such things, ask Kevin Mitnick). Copyrighted material should be able to be allowed or disallowed by the author. It should be law that if they don't obey the author can claim damages. I feel this could currently hold up in court. As yet no individual has went to court over it. It is possible. I think lawyers should look into this! They could have another pay day with this neglect.
Internet law allows for copyright simply by stating a site or software is copyrighted.

I was told by an attorney to just put copyright on my software that I wrote and it would be copyrighted. I imagine the same goes for web pages.

If an author does not want a work released then it should not be released!

Same should go for the robots.txt file.

Copyrighted material that is not wanted to be released should not be released.

I understand it would be wise to have security configured. That is another issue.
That is not the same thing (There are laws for that the FBI takes care of such things, ask Kevin Mitnick). Copyrighted material should be able to be allowed or disallowed by the author. It should be law that if they don’t obey the author can claim damages.

I feel this could currently hold up in court. As yet no individual has went to court over it.
It is possible.

]]> By: Brian Drake http://diovo.com/2008/09/robotstxt-is-not-a-security-measure/comment-page-1/#comment-2499 Brian Drake Wed, 13 Apr 2011 14:39:58 +0000 http://www.diovo.com/?p=264#comment-2499 Anything that requests pages from a website – be it a web browser, a legitimate search engine or a malicious crawler – does it in the exactly the same way. They all look exactly the same to any other computer on the Internet, including the computers running that website. (That is unless they choose to distinguish themselves, but why would a malicious crawler do that?) Robots.txt only works on good crawlers that take it upon themselves to enforce the rules it contains. Seriously, how did you think the Web worked? Anything that requests pages from a website – be it a web browser, a legitimate search engine or a malicious crawler – does it in the exactly the same way. They all look exactly the same to any other computer on the Internet, including the computers running that website. (That is unless they choose to distinguish themselves, but why would a malicious crawler do that?)

Robots.txt only works on good crawlers that take it upon themselves to enforce the rules it contains. Seriously, how did you think the Web worked?

]]> By: The robots.txt file - Web Hosting http://diovo.com/2008/09/robotstxt-is-not-a-security-measure/comment-page-1/#comment-1134 The robots.txt file - Web Hosting Wed, 08 Sep 2010 11:58:52 +0000 http://www.diovo.com/?p=264#comment-1134 [...] [...] [...] [...]

]]> By: How can a crawler bypass robots.txt? | Diovo http://diovo.com/2008/09/robotstxt-is-not-a-security-measure/comment-page-1/#comment-1133 How can a crawler bypass robots.txt? | Diovo Mon, 16 Mar 2009 09:04:45 +0000 http://www.diovo.com/?p=264#comment-1133 [...] I wrote that robots.txt will not prevent bad crawlers from accessing your private data, a reader wondered how a crawler can bypass [...] [...] I wrote that robots.txt will not prevent bad crawlers from accessing your private data, a reader wondered how a crawler can bypass [...]

]]> By: Yogesh http://diovo.com/2008/09/robotstxt-is-not-a-security-measure/comment-page-1/#comment-1132 Yogesh Fri, 13 Mar 2009 12:08:44 +0000 http://www.diovo.com/?p=264#comment-1132 How? How a crawler can bypass robots.txt How?
How a crawler can bypass robots.txt

]]>