Detecting Spiders

I know that I can get a visitor's IP Address with the following code:


But I need to be able to determine whether or not this return values is a spider or a bot, etc.  What is the best way to go about doing something like this?


5/4/2009 10:10:42 PM 91979 articles. 2 followers. Follow

3 Replies

Similar Articles

[PageSpeed] 47

Here is an article that might help you

- William

Please mark the most helpful reply/replies as "Answer".

Give some of my PWSK modules a try.
5/4/2009 10:57:53 PM

If I understand the article, I would need to update a .config file on my machine.  However, I need to identify crawlers so I can maintain statistics for my sight so I can't expect users' machines to be updated with these .config files.  I really need some sort of service where my page can "dynamically" check whether or not the visitor is a robot.

5/5/2009 1:27:40 AM

The only config file that needs to be updated is the one on the web server where your site is.  The only other way to detect crawlers is by the User Agent String.

This article does it by looking at the agent string for patterns

- William

Please mark the most helpful reply/replies as "Answer".

Give some of my PWSK modules a try.
5/5/2009 2:02:11 AM

Similar Artilces:

Getting Start With .net
hi i'm intersted to learn (vb or delphi) . i programming with delphi6 but i'd like to tp know something about programming in internet specially with .net . i hear about with delphi 7 that i can programming with delphi in .net .... is it right ? what 's our opinian about it ? and better to learn Vb.Net or continu with delphi(or thanks. Ali Darabian I think there is some support for Delphi in .Net. However, you will find much more support from the user community if you choose a more popular language such as VB.Net or C#. Since your al...

how to get start with .NET
the net help people installed  to my desktop 1-microsoft visual studio .NET enterprise developer 2003 2-microsoft .NET framework 1.1 what else do i need ...???  and how can i try with some simple code in VS.Net and run it to see wat happen...????? please help If you want to develope web applications, then you'll also need IIS on your PC.  That'll require XP-Professional or Windows 2000/2003, but XP-Home edition won't allow IIS installation. As for how to get started, you can search the web for some tutorials or buy a good book on VS.  The topic is way too vast to...

Where to start and how did you get started?
Hi, I have had a feeling for a long time to jump into the adds market for some time due to my low income (shooting for the stars) and I don't have anything better to do these days. But I find it really difficult not to give up due to lack of guides (what I can find atleast) and all the hard words which only makes me confused. But I won't give up. Never. Not sure if this is the right place to post this, but I deffinitly would be really greatfull for help getting started. How did you get started? +If there are any teacher that wanna show me the ropes private, I can give payment with ...

Getting started .net website
Hi folks,Does anyone here know of a good website to just sit back and read for learning .net? Thanks in advance.Baldeep Right here... Click the Get Started and Learn sections here. Quite a few out there - - this one, has a lot of good videos -  also, but then, I might be a little prejudiced David WierMCP/ - where to look first!Please Vote for ASPNet101 - 'Best Community Resource'!Control Grouper - easily control properties for multiple controls with one control!Calendar Express - The Best HTML Calendar Generator on the web!(P...

Getting started with JScript.NET
I'm a Javascript and Actionscript developer, and the thought of being able to do server-sides in something I know was too good to resist. Does anyone know some good tutorials for developing ASP.NET pages in JScript?  I can find lots of console code, but simple things are hard: like I can't even figure out how to print something out to the web page (just a string). What I'm really trying to do is access a database, verify a login, and print JSON out to the response.  What should I do to start?  Is there a better way to do this? (IE: using some JSON library already?). I'm commun...

How do I get started with AJAX.Net?
Hi All, I'm a seasoned developer but have not yet ventured into atlas et al.  I prefer books, but will take whatever I can get if it's good and will get me going. Any recommendations? Thx in advance. Jeff try here first then learn from sample and docJames Wu (MIB426).NET is only way to goMCP, MCSE, MCDBA, MCSD, MCAD I find the materials online display features and implementations but are lacking architectural depth, and am seeking more comprehensive resources. By architectural depth, I mean that I cannot watch ...

Getting Started with .net and a few questions
Hello I've done some asp and php programming along with various other languages in the past and slowing getting my head around .net.I have a small project of creating a Holiday Booking system on our Intranet and just trying to get the first part to work properly will hopefully get me sorted for the whole project. The Intranet is running on .net 2 and I prefer to use the VB code rather than C#. This is connecting to an SQL server.My initial page that I am trying to create is just for an admin user to view which dates in a table have been specified as Bank Holidays and be able to add ...

Getting Started Without VS.NET
Hi all, does anyone know how any online resources that can show me how to create web apps that include code-behind, and uer controls without using VS.NET? I have VS.NET in work, but would also like to do work at home, but cannot install VS.NET on my home laptop because I simply dont' have the space or processor power. Any online articles would be welcome, as will any open-source apps - I currently have SharpDevelop, but I find it unstable as it crashes my laptop alot of the time. Thanks Tryst Did you try Web Martix? It is a FREE tool from Microsoft.Regards, Narayana ...

Newbie Question
I have a new project which has just started. One of the requirements is to generate XML files. Do I have to create schema files? Does anyone have a tutorial or sample code to help me get started. I have some background using XML, but not in .NET. Thx Mark How to create a new xml file with XmlDocument? More examples hereSonu Kapoor [MVP] Feel free to ask me any .NET question | Ajax Data Controls Forum Thanks, this is very helpful. What do I need to do extra to setup the schema for this xml file? Is the schema really necessary as well? Thx Mark You dont need to define any sc...

Getting started with mySQL Connector/Net 5.1
i have a vs2005 project which someone else built.  I need to connect to mySQL in this project. it is currently using the OleDbConnection to mySQL and I cannot make the connection, so I was told by the developer to try the the mySQL Connector/Net 5.1.  I have read the very encouraging documentation.  I am in a file called SqlDataProvider.vbI want to replace Dim objConnection As OleDbConnectionwithDim objConnection As MySqlConnection I get the blue squiggly  "MySqlConncection is undefined." I added this import to the top. Imports MySql.Da...

Getting started with .net mobile/wireless application programming
Hi friends, I am very much amateur to the world of mobile/wireless application programming that's why m is having thousands of doubts and confusion regarding the technology. My first few concerns about the terms i.e. mobile web applications vs. windows mobile applications first thing are these 2 things are same if not what are the main differences. I hope you will also show me some light how to move further ahead in this field as i m having small exp approx. 1.5 yrs in application. Any help in the form of links, URL, code will be highly appreciated. Thanks in advanced...

getting start with and crystal repots(web form)
I use Crystal Report9 in Visual Studio.NET 2003..... Example at Northwind database I want a application that.....a form include a combobox that contain CustomerID.....If I choose one CustomerID and click OK button....a Report will display information about this Customer......I think that must use to do that ???? (How do I start to coding --please step by step or sample) please help me !!! Thank you so much!!! very good I have Dropdownlist when I choose the 'month'--> the all data of this field will show I use this Private Sub ddlMonth_Sel...

Getting 3.1 to start in debug mode in 2003.
  Hi All, I have a run time version of dnn installed and working at D:\Inetpub\wwwroot\dotnetnuke on my laptop. I want to be able to learn how to write DNN modules etc and so I have installed the source code in D:\Inetpub\wwwroot\dotnetnuke3100\. I set the debug flag in web.config to yes..... as in the lines.. <!-- set debugmode to false for running application --> <compilation debug="true" /> I updated the solution file D:\Inetpub\wwwroot\dotnetnuke3100\Solutions\DotNetNuke.All to look at localhost/dotnetnuke3100 which is my source code version. I have created the...

Free .Net and Sql Server Training Videos for Developers just getting started
I figured since this is the Getting started forum I would mention a great .Net training video site with some free videos.http://www.TechnicalVideos.netThanksMike James...

Starting an Application in .net
i am new to could any one give me simple ajax,net example so that i start building on my ajax skills please.........  I started with the todolist video on the main page, and its all you need to know to get started. Don't forget to click "Mark as Answer" on the post that helped you.This credits that member, earns you a point and marks your thread as Resolved so we will all know you have been helped.    ...

Web resources about - Detecting Spiders -

Methods of detecting exoplanets - Wikipedia, the free encyclopedia
Any planet is an extremely faint light source compared to its parent star . In addition to the intrinsic difficulty of detecting such a faint ...

Detecting unauthorized cross-app resource access on OS X
Last week, Luyi Xing, a Ph.D. student from Indiana University Bloomington and his colleagues released a research paper on several vulnerabilities ...


Gold Coast Suns detecting 'a trickle of respect'
... produce a quarter "as good as we've played" in its first win at Etihad Stadium at the weekend Gold Coast chief executive Travis Auld is detecting ...

Detecting APT Activity with Network Traffic Analysis
Computerworld Australia is the leading source of technology news, analysis and tools for IT decision makers, managers and professionals.

Malware-detecting 'sandboxing' technology no silver bullet - Palo Alto, mcafee, trend micro, security ...
The security technology called "sandboxing" aims at detecting malware code by subjecting it to run in a computer-based system of one type of ...

Researchers close to developing a blood test for detecting Alzheimer’s
A team of scientists say they are getting closer to detecting Alzheimer’s antibodies at the preclinical stage, giving patients an opportunity ...

Threat-detecting surveillence cameras among first start-up projects to benefit from local Microsoft Innovation ...
South Australia's start-up support initiatives give funding to system that foils armed hold-ups by locking the shop doors to suspicious people. ...

Detecting APT Activity with Network Traffic Analysis
Defining Technology for Australia's IT Leaders.

Shrimp's eyes inspire new cancer-detecting camera
Shrimps' eyes have been revealed as the latest weapon against cancer, with their unique vision set to be adapted to diagnostic cameras.

Resources last updated: 12/6/2015 11:52:34 PM