Search engines...

[rammy]

Hi,
Is it possible to submit a URL to a search engine and obtain the results? I want to be able to do this via code and use this to evaluate a site.

Thanx a lot.

[JoshT]

Yes, each major search engine puts the search in the querystring - look at the major one to see how they encode your search terms.

[rammy]

but how do i use this in either PHP or Perl to obtain the results via code? Any ideas??

Thanx a lot.

[JoshT]

With Perl, use LWP::Simple - there's decent examples in the Perl documentation.

[rammy]

thanx JoshT, will look it up.

[rammy]

Hi,
Thanx for the info on LWP::Simple...but Im stuck when I try to send a search engine's URL. for Eg: if I try to retrieve the page content from www.yahoo.com it works fine. But if I try to access


http://www.google.com/search?hl=en&q=php


the code does not work. Is this because the url is put together dynamically? Any ideas on how i can get this to work?

Here is my code:

use LWP::Simple;

my $content = get("http://www.google.com/search?hl=en&q=php");
if (defined $content)
{
#$content will contain the html for with the url mentioned.
print $content;
}
else
{
#If an error occurs then $content will not be defined.
print "Error: Get failed\n";
}



Thanx a ton again.

[JoshT]

This works for me (note I wrote it to be ran at the command prompt, not as a web page).

#usage:
# webget.pl www.google.com
# webget.pl www.google.com google.txt

use strict;
use warnings;

# Create a user agent object
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent("Internet-Exploder/1.0999" );

my $website = $ARGV[0];
my $filename = $ARGV[1];
my $out; #handle to output

die "No website specified.\n\n" unless $website;

if (not ($website =~ /^http:\/\//)) {$website = "http://" . $website;}

# Create a request
my $req = HTTP::Request->new(GET => $website);

# Pass request to the user agent and get a response back
my $res = $ua->request($req);

# Check the outcome of the response
if ($res->is_success) {
if ($filename) {
open OUTFILE, "> $filename" or die "Couldn't open $filename: $!\n\n";
$out = *OUTFILE;
} else {
$out = *STDOUT;
}
print $out $res->content;

close($out);
} else {
print "Couldn't get $website.\n\n";
}

[rammy]

JoshT, that worked perfectly. thanx a whole lot. :)