Checking if a website is up doesn't work correctly

Hi there,
Somewhere I found an example how to check if a website is up.

Here my sample:

#! /usr/bin/perl

use strict;

use LWP::Simple;
my $url="https://notabug.org";
if (! head($url)) {
    die "$url is DOWN!!!!" 
}

Running above code I get
  https://notabug.org is DOWN!!!! at ./check_url.pl line 8.


However, firefox shows the site works ok.


What am I doing wrong?


-- 
Thanks,
Manfred
0
ml_news
2/13/2018 11:25:31 AM
perl.beginners 29312 articles. 3 followers. Follow

6 Replies
78 Views

Similar Articles

[PageSpeed] 18

Hi Manfred!

On Tue, 13 Feb 2018 12:25:31 +0100
Manfred Lotz <ml_news@posteo.de> wrote:

> Hi there,
> Somewhere I found an example how to check if a website is up.
>=20
> Here my sample:
>=20
> #! /usr/bin/perl
>=20
> use strict;
>=20
> use LWP::Simple;
> my $url=3D"https://notabug.org";
> if (! head($url)) {
>     die "$url is DOWN!!!!"=20
> }
>=20
> Running above code I get
>   https://notabug.org is DOWN!!!! at ./check_url.pl line 8.
>=20

This code seems to work fine here:

#!/usr/bin/perl

use strict;
use warnings;

use LWP::Simple;
  my $url=3D"https://shlomif.github.io/";
# my $url=3D"https://notabug.org/";
if (! head($url)) {
    die "$url is DOWN!!!!";
}

seems like notabug blocks libwww-perl.


>=20
> However, firefox shows the site works ok.
>

same here.

Regards,

	Shlomi
=20
>=20
> What am I doing wrong?
>=20
>=20



--=20
-----------------------------------------------------------------
Shlomi Fish       http://www.shlomifish.org/
Chuck Norris/etc. Facts - http://www.shlomifish.org/humour/bits/facts/

<talexb> =E2=80=9CHey, I have a flat tire. Can you help me change it with a=
 can opener
and a pound of sesame seeds?=E2=80=9D
    =E2=80=94 talexb on parsing HTML or XML with regular expressions.

Please reply to list if it's a mailing list post - http://shlom.in/reply .
0
shlomif
2/13/2018 11:53:40 AM
--001a1147a42ae0279005651ac0bf
Content-Type: text/plain; charset="UTF-8"

The site doesn't like 'head' requests? get works
#!/usr/bin/perl

use strict;
use warnings;

use LWP::Simple;
#  my $url="https://shlomif.github.io/";
my $url="http://www.notabug.org/";
print "$url is ", (
                (! get($url)) ?  "DOWN!!!!"
                                : "up"
                ), "\n";

$ is_it_up.pl
http://www.notabug.org/ is up


On Tue, Feb 13, 2018 at 5:25 AM, Manfred Lotz <ml_news@posteo.de> wrote:

> Hi there,
> Somewhere I found an example how to check if a website is up.
>
> Here my sample:
>
> #! /usr/bin/perl
>
> use strict;
>
> use LWP::Simple;
> my $url="https://notabug.org";
> if (! head($url)) {
>     die "$url is DOWN!!!!"
> }
>
> Running above code I get
>   https://notabug.org is DOWN!!!! at ./check_url.pl line 8.
>
>
> However, firefox shows the site works ok.
>
>
> What am I doing wrong?
>
>
> --
> Thanks,
> Manfred
>
> --
> To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> For additional commands, e-mail: beginners-help@perl.org
> http://learn.perl.org/
>
>
>


-- 

a

Andy Bach,
afbach@gmail.com
608 658-1890 cell
608 261-5738 wk

--001a1147a42ae0279005651ac0bf
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">The site doesn&#39;t like &#39;head&#39; requests? get wor=
ks<br>#!/usr/bin/perl<br><br>use strict;<br>use warnings;<br><br>use LWP::S=
imple;<br>#=C2=A0 my $url=3D&quot;<a href=3D"https://shlomif.github.io/">ht=
tps://shlomif.github.io/</a>&quot;;<br>my $url=3D&quot;<a href=3D"http://ww=
w.notabug.org/">http://www.notabug.org/</a>&quot;;<br>print &quot;$url is &=
quot;, (<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 (! get($url)) ?=C2=A0 &quot;DOWN!!!!&quot;<br>=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 : &quot;up&quot;<br>=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ),=
 &quot;\n&quot;;<br><br>$ <a href=3D"http://is_it_up.pl">is_it_up.pl</a><br=
><a href=3D"http://www.notabug.org/">http://www.notabug.org/</a> is up<br><=
br></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, =
Feb 13, 2018 at 5:25 AM, Manfred Lotz <span dir=3D"ltr">&lt;<a href=3D"mail=
to:ml_news@posteo.de" target=3D"_blank">ml_news@posteo.de</a>&gt;</span> wr=
ote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border=
-left:1px #ccc solid;padding-left:1ex">Hi there,<br>
Somewhere I found an example how to check if a website is up.<br>
<br>
Here my sample:<br>
<br>
#! /usr/bin/perl<br>
<br>
use strict;<br>
<br>
use LWP::Simple;<br>
my $url=3D&quot;<a href=3D"https://notabug.org" rel=3D"noreferrer" target=
=3D"_blank">https://notabug.org</a>&quot;;<br>
if (! head($url)) {<br>
=C2=A0 =C2=A0 die &quot;$url is DOWN!!!!&quot;<br>
}<br>
<br>
Running above code I get<br>
=C2=A0 <a href=3D"https://notabug.org" rel=3D"noreferrer" target=3D"_blank"=
>https://notabug.org</a> is DOWN!!!! at ./<a href=3D"http://check_url.pl" r=
el=3D"noreferrer" target=3D"_blank">check_url.pl</a> line 8.<br>
<br>
<br>
However, firefox shows the site works ok.<br>
<br>
<br>
What am I doing wrong?<br>
<span class=3D"HOEnZb"><font color=3D"#888888"><br>
<br>
--<br>
Thanks,<br>
Manfred<br>
<br>
--<br>
To unsubscribe, e-mail: <a href=3D"mailto:beginners-unsubscribe@perl.org">b=
eginners-unsubscribe@perl.org</a><br>
For additional commands, e-mail: <a href=3D"mailto:beginners-help@perl.org"=
>beginners-help@perl.org</a><br>
<a href=3D"http://learn.perl.org/" rel=3D"noreferrer" target=3D"_blank">htt=
p://learn.perl.org/</a><br>
<br>
<br>
</font></span></blockquote></div><br><br clear=3D"all"><br>-- <br><div clas=
s=3D"gmail_signature" data-smartmail=3D"gmail_signature"><br>a<br><br>Andy =
Bach,<br><a href=3D"mailto:afbach@gmail.com" target=3D"_blank">afbach@gmail=
..com</a><br>608 658-1890 cell<br>608 261-5738 wk</div>
</div>

--001a1147a42ae0279005651ac0bf--
0
afbach
2/13/2018 4:47:42 PM
On Tue, 13 Feb 2018 10:47:42 -0600
Andy Bach <afbach@gmail.com> wrote:

> The site doesn't like 'head' requests? get works
> #!/usr/bin/perl
> 
> use strict;
> use warnings;
> 
> use LWP::Simple;
> #  my $url="https://shlomif.github.io/";
> my $url="http://www.notabug.org/";
> print "$url is ", (
>                 (! get($url)) ?  "DOWN!!!!"
>                                 : "up"
>                 ), "\n";
> 
> $ is_it_up.pl
> http://www.notabug.org/ is up
> 

You are right. 

But am afraid this is not all of it. If I test
http://scripts.sil.org/OFL then I get an error but it is fine in
firefox.

Very strange.

-- 
Manfred



> 
> On Tue, Feb 13, 2018 at 5:25 AM, Manfred Lotz <ml_news@posteo.de>
> wrote:
> 
> > Hi there,
> > Somewhere I found an example how to check if a website is up.
> >
> > Here my sample:
> >
> > #! /usr/bin/perl
> >
> > use strict;
> >
> > use LWP::Simple;
> > my $url="https://notabug.org";
> > if (! head($url)) {
> >     die "$url is DOWN!!!!"
> > }
> >
> > Running above code I get
> >   https://notabug.org is DOWN!!!! at ./check_url.pl line 8.
> >
> >
> > However, firefox shows the site works ok.
> >
> >
> > What am I doing wrong?
> >
> >
> > --
> > Thanks,
> > Manfred
> >
> > --
> > To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> > For additional commands, e-mail: beginners-help@perl.org
> > http://learn.perl.org/
> >
> >
> >  
> 
> 
0
ml_news
2/13/2018 5:33:12 PM
--001a113fe1b41a97b405651d50ad
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

$ wget http://scripts.sil.org/OFL
--2018-02-13 13:42:50--  http://scripts.sil.org/OFL
Resolving scripts.sil.org (scripts.sil.org)... 209.12.63.143
Connecting to scripts.sil.org (scripts.sil.org)|209.12.63.143|:80...
connected.
HTTP request sent, awaiting response... 302 Found
Location: http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DO=
FL
[following]
--2018-02-13 13:42:52--
http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL
Reusing existing connection to scripts.sil.org:80.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: /cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL&_sc=3D1 [following]
--2018-02-13 13:42:52--
http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL&_sc=3D1
Reusing existing connection to scripts.sil.org:80.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: /cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL [following]
--2018-02-13 13:42:53--
http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL
Reusing existing connection to scripts.sil.org:80.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: =E2=80=98OFL=E2=80=99

    [
<=3D>
] 37,439      59.6KB/s   in 0.6s

2018-02-13 13:42:55 (59.6 KB/s) - =E2=80=98OFL=E2=80=99 saved [37439]

so it may not be following the 302s. I'm not sure you're using the correct
tool here.  A little more straight forward

andy@wiwmb-md-afb-mint:~/spam$ wget http://scripts.sil.org/robots.txt
--2018-02-13 13:47:27--  http://scripts.sil.org/robots.txt
Resolving scripts.sil.org (scripts.sil.org)... 209.12.63.143
Connecting to scripts.sil.org (scripts.sil.org)|209.12.63.143|:80...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 36 [text/plain]
Saving to: =E2=80=98robots.txt=E2=80=99

100%[=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>]
36          --.-K/s   in 0s

2018-02-13 13:47:27 (2.99 MB/s) - =E2=80=98robots.txt=E2=80=99 saved [36/36=
]

but
$ is_it_up.pl
http://scripts.sil.org/robots.txt is DOWN!!!!

You might look at more LWP tools:
$ lwp-dump https://www.sil.org
HTTP/1.1 403 Forbidden
Cache-Control: max-age=3D10
Connection: keep-alive
Date: Tue, 13 Feb 2018 19:49:47 GMT
Server: cloudflare
Content-Type: text/html; charset=3DUTF-8
Expires: Tue, 13 Feb 2018 19:49:57 GMT
CF-RAY: 3eca501a5d569895-LAX
Expect-CT: max-age=3D604800, report-uri=3D"
https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Set-Cookie: __cfduid=3Ddd8038f4f2c995fa4b4c7fa8beb2b42f31518551387;
expires=3DWed, 13-Feb-19 19:49:47 GMT; path=3D/; domain=3D.sil.org; HttpOnl=
y
X-Frame-Options: SAMEORIGIN

<!DOCTYPE html>
<!--[if lt IE 7]> <html class=3D"no-js ie6 oldie" lang=3D"en-US"> <![endif]=
-->
<!--[if IE 7]>    <html class=3D"no-js ie7 oldie" lang=3D"en-US"> <![endif]=
-->
<!--[if IE 8]>    <html class=3D"no-js ie8 oldie" lang=3D"en-US"> <![endif]=
-->
<!--[if gt IE 8]><!--> <html class=3D"no-js" lang=3D"en-US"> <!--<![endif]-=
->
<head>
<title>Access denied | www.sil.org used Cloudflare to restrict
access</title>
<meta charset=3D"UTF-8" />
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-8" />
<meta http-equiv=3D...
(+ 2770 more bytes not shown)

so it's up, but "forbidden" probably as the user agent isn't set or some
other policy reason.


On Tue, Feb 13, 2018 at 11:33 AM, Manfred Lotz <ml_news@posteo.de> wrote:

> On Tue, 13 Feb 2018 10:47:42 -0600
> Andy Bach <afbach@gmail.com> wrote:
>
> > The site doesn't like 'head' requests? get works
> > #!/usr/bin/perl
> >
> > use strict;
> > use warnings;
> >
> > use LWP::Simple;
> > #  my $url=3D"https://shlomif.github.io/";
> > my $url=3D"http://www.notabug.org/";
> > print "$url is ", (
> >                 (! get($url)) ?  "DOWN!!!!"
> >                                 : "up"
> >                 ), "\n";
> >
> > $ is_it_up.pl
> > http://www.notabug.org/ is up
> >
>
> You are right.
>
> But am afraid this is not all of it. If I test
> http://scripts.sil.org/OFL then I get an error but it is fine in
> firefox.
>
> Very strange.
>
> --
> Manfred
>
>
>
> >
> > On Tue, Feb 13, 2018 at 5:25 AM, Manfred Lotz <ml_news@posteo.de>
> > wrote:
> >
> > > Hi there,
> > > Somewhere I found an example how to check if a website is up.
> > >
> > > Here my sample:
> > >
> > > #! /usr/bin/perl
> > >
> > > use strict;
> > >
> > > use LWP::Simple;
> > > my $url=3D"https://notabug.org";
> > > if (! head($url)) {
> > >     die "$url is DOWN!!!!"
> > > }
> > >
> > > Running above code I get
> > >   https://notabug.org is DOWN!!!! at ./check_url.pl line 8.
> > >
> > >
> > > However, firefox shows the site works ok.
> > >
> > >
> > > What am I doing wrong?
> > >
> > >
> > > --
> > > Thanks,
> > > Manfred
> > >
> > > --
> > > To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> > > For additional commands, e-mail: beginners-help@perl.org
> > > http://learn.perl.org/
> > >
> > >
> > >
> >
> >
>
> --
> To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> For additional commands, e-mail: beginners-help@perl.org
> http://learn.perl.org/
>
>
>


--=20

a

Andy Bach,
afbach@gmail.com
608 658-1890 cell
608 261-5738 wk

--001a113fe1b41a97b405651d50ad
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>$ wget <a href=3D"http://scripts.sil.org/OFL">ht=
tp://scripts.sil.org/OFL</a><br>--2018-02-13 13:42:50--=C2=A0 <a href=3D"ht=
tp://scripts.sil.org/OFL">http://scripts.sil.org/OFL</a><br>Resolving <a hr=
ef=3D"http://scripts.sil.org">scripts.sil.org</a> (<a href=3D"http://script=
s.sil.org">scripts.sil.org</a>)... 209.12.63.143<br>Connecting to <a href=
=3D"http://scripts.sil.org">scripts.sil.org</a> (<a href=3D"http://scripts.=
sil.org">scripts.sil.org</a>)|209.12.63.143|:80... connected.<br>HTTP reque=
st sent, awaiting response... 302 Found<br>Location: <a href=3D"http://scri=
pts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&amp;id=3DOFL">http://script=
s.sil.org/cms/scripts/page.php?site_id=3Dnrsi&amp;id=3DOFL</a> [following]<=
br>--2018-02-13 13:42:52--=C2=A0 <a href=3D"http://scripts.sil.org/cms/scri=
pts/page.php?site_id=3Dnrsi&amp;id=3DOFL">http://scripts.sil.org/cms/script=
s/page.php?site_id=3Dnrsi&amp;id=3DOFL</a><br>Reusing existing connection t=
o <a href=3D"http://scripts.sil.org:80">scripts.sil.org:80</a>.<br>HTTP req=
uest sent, awaiting response... 302 Moved Temporarily<br>Location: /cms/scr=
ipts/page.php?site_id=3Dnrsi&amp;id=3DOFL&amp;_sc=3D1 [following]<br>--2018=
-02-13 13:42:52--=C2=A0 <a href=3D"http://scripts.sil.org/cms/scripts/page.=
php?site_id=3Dnrsi&amp;id=3DOFL&amp;_sc=3D1">http://scripts.sil.org/cms/scr=
ipts/page.php?site_id=3Dnrsi&amp;id=3DOFL&amp;_sc=3D1</a><br>Reusing existi=
ng connection to <a href=3D"http://scripts.sil.org:80">scripts.sil.org:80</=
a>.<br>HTTP request sent, awaiting response... 302 Moved Temporarily<br>Loc=
ation: /cms/scripts/page.php?site_id=3Dnrsi&amp;id=3DOFL [following]<br>--2=
018-02-13 13:42:53--=C2=A0 <a href=3D"http://scripts.sil.org/cms/scripts/pa=
ge.php?site_id=3Dnrsi&amp;id=3DOFL">http://scripts.sil.org/cms/scripts/page=
..php?site_id=3Dnrsi&amp;id=3DOFL</a><br>Reusing existing connection to <a h=
ref=3D"http://scripts.sil.org:80">scripts.sil.org:80</a>.<br>HTTP request s=
ent, awaiting response... 200 OK<br>Length: unspecified [text/html]<br>Savi=
ng to: =E2=80=98OFL=E2=80=99<br><br>=C2=A0=C2=A0=C2=A0 [=C2=A0=C2=A0 &lt;=
=3D&gt;=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ] 37,439=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 59.6KB/s=C2=A0=C2=A0 in 0.6s=C2=A0=C2=A0 <br><br>2018-02-13=
 13:42:55 (59.6 KB/s) - =E2=80=98OFL=E2=80=99 saved [37439]<br><br></div>so=
 it may not be following the 302s. I&#39;m not sure you&#39;re using the co=
rrect tool here.=C2=A0 A little more straight forward<br><br>andy@wiwmb-md-=
afb-mint:~/spam$ wget <a href=3D"http://scripts.sil.org/robots.txt">http://=
scripts.sil.org/robots.txt</a> <br>--2018-02-13 13:47:27--=C2=A0 <a href=3D=
"http://scripts.sil.org/robots.txt">http://scripts.sil.org/robots.txt</a><b=
r>Resolving <a href=3D"http://scripts.sil.org">scripts.sil.org</a> (<a href=
=3D"http://scripts.sil.org">scripts.sil.org</a>)... 209.12.63.143<br>Connec=
ting to <a href=3D"http://scripts.sil.org">scripts.sil.org</a> (<a href=3D"=
http://scripts.sil.org">scripts.sil.org</a>)|209.12.63.143|:80... connected=
..<br>HTTP request sent, awaiting response... 200 OK<br>Length: 36 [text/pla=
in]<br>Saving to: =E2=80=98robots.txt=E2=80=99<br><br>100%[=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D&gt;] 36=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 --.-K/s=C2=A0=C2=A0 in 0s=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <br><b=
r>2018-02-13 13:47:27 (2.99 MB/s) - =E2=80=98robots.txt=E2=80=99 saved [36/=
36]<br><br></div>but<br>$ <a href=3D"http://is_it_up.pl">is_it_up.pl</a><br=
><a href=3D"http://scripts.sil.org/robots.txt">http://scripts.sil.org/robot=
s.txt</a> is DOWN!!!!<br><div><br></div><div>You might look at more LWP too=
ls:<br>$ lwp-dump <a href=3D"https://www.sil.org">https://www.sil.org</a><b=
r>HTTP/1.1 403 Forbidden<br>Cache-Control: max-age=3D10<br>Connection: keep=
-alive<br>Date: Tue, 13 Feb 2018 19:49:47 GMT<br>Server: cloudflare<br>Cont=
ent-Type: text/html; charset=3DUTF-8<br>Expires: Tue, 13 Feb 2018 19:49:57 =
GMT<br>CF-RAY: 3eca501a5d569895-LAX<br>Expect-CT: max-age=3D604800, report-=
uri=3D&quot;<a href=3D"https://report-uri.cloudflare.com/cdn-cgi/beacon/exp=
ect-ct">https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct</a>&quot=
;<br>Set-Cookie: __cfduid=3Ddd8038f4f2c995fa4b4c7fa8beb2b42f31518551387; ex=
pires=3DWed, 13-Feb-19 19:49:47 GMT; path=3D/; domain=3D.<a href=3D"http://=
sil.org">sil.org</a>; HttpOnly<br>X-Frame-Options: SAMEORIGIN<br><br>&lt;!D=
OCTYPE html&gt;<br>&lt;!--[if lt IE 7]&gt; &lt;html class=3D&quot;no-js ie6=
 oldie&quot; lang=3D&quot;en-US&quot;&gt; &lt;![endif]--&gt;<br>&lt;!--[if =
IE 7]&gt;=C2=A0=C2=A0=C2=A0 &lt;html class=3D&quot;no-js ie7 oldie&quot; la=
ng=3D&quot;en-US&quot;&gt; &lt;![endif]--&gt;<br>&lt;!--[if IE 8]&gt;=C2=A0=
=C2=A0=C2=A0 &lt;html class=3D&quot;no-js ie8 oldie&quot; lang=3D&quot;en-U=
S&quot;&gt; &lt;![endif]--&gt;<br>&lt;!--[if gt IE 8]&gt;&lt;!--&gt; &lt;ht=
ml class=3D&quot;no-js&quot; lang=3D&quot;en-US&quot;&gt; &lt;!--&lt;![endi=
f]--&gt;<br>&lt;head&gt;<br>&lt;title&gt;Access denied | <a href=3D"http://=
www.sil.org">www.sil.org</a> used Cloudflare to restrict access&lt;/title&g=
t;<br>&lt;meta charset=3D&quot;UTF-8&quot; /&gt;<br>&lt;meta http-equiv=3D&=
quot;Content-Type&quot; content=3D&quot;text/html; charset=3DUTF-8&quot; /&=
gt;<br>&lt;meta http-equiv=3D...<br>(+ 2770 more bytes not shown)<br><br></=
div><div>so it&#39;s up, but &quot;forbidden&quot; probably as the user age=
nt isn&#39;t set or some other policy reason.<br></div><div><br></div></div=
><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Feb 13, =
2018 at 11:33 AM, Manfred Lotz <span dir=3D"ltr">&lt;<a href=3D"mailto:ml_n=
ews@posteo.de" target=3D"_blank">ml_news@posteo.de</a>&gt;</span> wrote:<br=
><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1=
px #ccc solid;padding-left:1ex"><span class=3D"">On Tue, 13 Feb 2018 10:47:=
42 -0600<br>
Andy Bach &lt;<a href=3D"mailto:afbach@gmail.com">afbach@gmail.com</a>&gt; =
wrote:<br>
<br>
&gt; The site doesn&#39;t like &#39;head&#39; requests? get works<br>
&gt; #!/usr/bin/perl<br>
&gt;<br>
&gt; use strict;<br>
&gt; use warnings;<br>
&gt;<br>
&gt; use LWP::Simple;<br>
&gt; #=C2=A0 my $url=3D&quot;<a href=3D"https://shlomif.github.io/" rel=3D"=
noreferrer" target=3D"_blank">https://shlomif.github.<wbr>io/</a>&quot;;<br=
>
&gt; my $url=3D&quot;<a href=3D"http://www.notabug.org/" rel=3D"noreferrer"=
 target=3D"_blank">http://www.notabug.org/</a>&quot;<wbr>;<br>
&gt; print &quot;$url is &quot;, (<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(! get($u=
rl)) ?=C2=A0 &quot;DOWN!!!!&quot;<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: &quot;up&quot;<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0), &quot;=
\n&quot;;<br>
&gt;<br>
&gt; $ <a href=3D"http://is_it_up.pl" rel=3D"noreferrer" target=3D"_blank">=
is_it_up.pl</a><br>
&gt; <a href=3D"http://www.notabug.org/" rel=3D"noreferrer" target=3D"_blan=
k">http://www.notabug.org/</a> is up<br>
&gt;<br>
<br>
</span>You are right.<br>
<br>
But am afraid this is not all of it. If I test<br>
<a href=3D"http://scripts.sil.org/OFL" rel=3D"noreferrer" target=3D"_blank"=
>http://scripts.sil.org/OFL</a> then I get an error but it is fine in<br>
firefox.<br>
<br>
Very strange.<br>
<span class=3D"HOEnZb"><font color=3D"#888888"><br>
--<br>
Manfred<br>
</font></span><div class=3D"HOEnZb"><div class=3D"h5"><br>
<br>
<br>
&gt;<br>
&gt; On Tue, Feb 13, 2018 at 5:25 AM, Manfred Lotz &lt;<a href=3D"mailto:ml=
_news@posteo.de">ml_news@posteo.de</a>&gt;<br>
&gt; wrote:<br>
&gt;<br>
&gt; &gt; Hi there,<br>
&gt; &gt; Somewhere I found an example how to check if a website is up.<br>
&gt; &gt;<br>
&gt; &gt; Here my sample:<br>
&gt; &gt;<br>
&gt; &gt; #! /usr/bin/perl<br>
&gt; &gt;<br>
&gt; &gt; use strict;<br>
&gt; &gt;<br>
&gt; &gt; use LWP::Simple;<br>
&gt; &gt; my $url=3D&quot;<a href=3D"https://notabug.org" rel=3D"noreferrer=
" target=3D"_blank">https://notabug.org</a>&quot;;<br>
&gt; &gt; if (! head($url)) {<br>
&gt; &gt;=C2=A0 =C2=A0 =C2=A0die &quot;$url is DOWN!!!!&quot;<br>
&gt; &gt; }<br>
&gt; &gt;<br>
&gt; &gt; Running above code I get<br>
&gt; &gt;=C2=A0 =C2=A0<a href=3D"https://notabug.org" rel=3D"noreferrer" ta=
rget=3D"_blank">https://notabug.org</a> is DOWN!!!! at ./<a href=3D"http://=
check_url.pl" rel=3D"noreferrer" target=3D"_blank">check_url.pl</a> line 8.=
<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; However, firefox shows the site works ok.<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; What am I doing wrong?<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; --<br>
&gt; &gt; Thanks,<br>
&gt; &gt; Manfred<br>
&gt; &gt;<br>
&gt; &gt; --<br>
&gt; &gt; To unsubscribe, e-mail: <a href=3D"mailto:beginners-unsubscribe@p=
erl.org">beginners-unsubscribe@perl.org</a><br>
&gt; &gt; For additional commands, e-mail: <a href=3D"mailto:beginners-help=
@perl.org">beginners-help@perl.org</a><br>
&gt; &gt; <a href=3D"http://learn.perl.org/" rel=3D"noreferrer" target=3D"_=
blank">http://learn.perl.org/</a><br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt;<br>
&gt;<br>
<br>
--<br>
To unsubscribe, e-mail: <a href=3D"mailto:beginners-unsubscribe@perl.org">b=
eginners-unsubscribe@perl.org</a><br>
For additional commands, e-mail: <a href=3D"mailto:beginners-help@perl.org"=
>beginners-help@perl.org</a><br>
<a href=3D"http://learn.perl.org/" rel=3D"noreferrer" target=3D"_blank">htt=
p://learn.perl.org/</a><br>
<br>
<br>
</div></div></blockquote></div><br><br clear=3D"all"><br>-- <br><div class=
=3D"gmail_signature" data-smartmail=3D"gmail_signature"><br>a<br><br>Andy B=
ach,<br><a href=3D"mailto:afbach@gmail.com" target=3D"_blank">afbach@gmail.=
com</a><br>608 658-1890 cell<br>608 261-5738 wk</div>
</div>

--001a113fe1b41a97b405651d50ad--
0
afbach
2/13/2018 7:50:55 PM
On Tue, 13 Feb 2018 13:50:55 -0600
Andy Bach <afbach@gmail.com> wrote:

> $ wget http://scripts.sil.org/OFL
> --2018-02-13 13:42:50--  http://scripts.sil.org/OFL
> Resolving scripts.sil.org (scripts.sil.org)... 209.12.63.143
> Connecting to scripts.sil.org (scripts.sil.org)|209.12.63.143|:80...
> connected.
> HTTP request sent, awaiting response... 302 Found
> Location:
> http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL
> [following] --2018-02-13 13:42:52--
> http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL
> Reusing existing connection to scripts.sil.org:80.
> HTTP request sent, awaiting response... 302 Moved Temporarily
> Location: /cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL&_sc=3D1 [followin=
g]
> --2018-02-13 13:42:52--
> http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL&_sc=
=3D1
> Reusing existing connection to scripts.sil.org:80.
> HTTP request sent, awaiting response... 302 Moved Temporarily
> Location: /cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL [following]
> --2018-02-13 13:42:53--
> http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL
> Reusing existing connection to scripts.sil.org:80.
> HTTP request sent, awaiting response... 200 OK
> Length: unspecified [text/html]
> Saving to: =E2=80=98OFL=E2=80=99
>=20
>     [
> <=3D>
> ] 37,439      59.6KB/s   in 0.6s
>=20
> 2018-02-13 13:42:55 (59.6 KB/s) - =E2=80=98OFL=E2=80=99 saved [37439]
>=20
> so it may not be following the 302s. I'm not sure you're using the
> correct tool here.  A little more straight forward
>=20
> andy@wiwmb-md-afb-mint:~/spam$ wget http://scripts.sil.org/robots.txt
> --2018-02-13 13:47:27--  http://scripts.sil.org/robots.txt
> Resolving scripts.sil.org (scripts.sil.org)... 209.12.63.143
> Connecting to scripts.sil.org (scripts.sil.org)|209.12.63.143|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 36 [text/plain]
> Saving to: =E2=80=98robots.txt=E2=80=99
>=20
> 100%[=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>] =20
> 36          --.-K/s   in 0s
>=20
> 2018-02-13 13:47:27 (2.99 MB/s) - =E2=80=98robots.txt=E2=80=99 saved [36/=
36]
>=20
> but
> $ is_it_up.pl
> http://scripts.sil.org/robots.txt is DOWN!!!!
>=20
> You might look at more LWP tools:
> $ lwp-dump https://www.sil.org
> HTTP/1.1 403 Forbidden
> Cache-Control: max-age=3D10
> Connection: keep-alive
> Date: Tue, 13 Feb 2018 19:49:47 GMT
> Server: cloudflare
> Content-Type: text/html; charset=3DUTF-8
> Expires: Tue, 13 Feb 2018 19:49:57 GMT
> CF-RAY: 3eca501a5d569895-LAX
> Expect-CT: max-age=3D604800, report-uri=3D"
> https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
> Set-Cookie: __cfduid=3Ddd8038f4f2c995fa4b4c7fa8beb2b42f31518551387;
> expires=3DWed, 13-Feb-19 19:49:47 GMT; path=3D/; domain=3D.sil.org; HttpO=
nly
> X-Frame-Options: SAMEORIGIN
>=20
> <!DOCTYPE html>
> <!--[if lt IE 7]> <html class=3D"no-js ie6 oldie" lang=3D"en-US">
> <![endif]--> <!--[if IE 7]>    <html class=3D"no-js ie7 oldie"
> lang=3D"en-US"> <![endif]--> <!--[if IE 8]>    <html class=3D"no-js ie8
> oldie" lang=3D"en-US"> <![endif]--> <!--[if gt IE 8]><!--> <html
> class=3D"no-js" lang=3D"en-US"> <!--<![endif]--> <head>
> <title>Access denied | www.sil.org used Cloudflare to restrict
> access</title>
> <meta charset=3D"UTF-8" />
> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-8" =
/>
> <meta http-equiv=3D...
> (+ 2770 more bytes not shown)
>=20
> so it's up, but "forbidden" probably as the user agent isn't set or
> some other policy reason.
>=20
>=20

I tried WWW::Mechanize, and (of course) got also 403.

Really strange.=20

Is there another tool I could use for checking? I mean some tool in the
Perl universe?

--=20
Manfred



> On Tue, Feb 13, 2018 at 11:33 AM, Manfred Lotz <ml_news@posteo.de>
> wrote:
>=20
> > On Tue, 13 Feb 2018 10:47:42 -0600
> > Andy Bach <afbach@gmail.com> wrote:
> > =20
> > > The site doesn't like 'head' requests? get works
> > > #!/usr/bin/perl
> > >
> > > use strict;
> > > use warnings;
> > >
> > > use LWP::Simple;
> > > #  my $url=3D"https://shlomif.github.io/";
> > > my $url=3D"http://www.notabug.org/";
> > > print "$url is ", (
> > >                 (! get($url)) ?  "DOWN!!!!"
> > >                                 : "up"
> > >                 ), "\n";
> > >
> > > $ is_it_up.pl
> > > http://www.notabug.org/ is up
> > > =20
> >
> > You are right.
> >
> > But am afraid this is not all of it. If I test
> > http://scripts.sil.org/OFL then I get an error but it is fine in
> > firefox.
> >
> > Very strange.
> >
> > --
> > Manfred
> >
> >
> > =20
> > >
> > > On Tue, Feb 13, 2018 at 5:25 AM, Manfred Lotz <ml_news@posteo.de>
> > > wrote:
> > > =20
> > > > Hi there,
> > > > Somewhere I found an example how to check if a website is up.
> > > >
> > > > Here my sample:
> > > >
> > > > #! /usr/bin/perl
> > > >
> > > > use strict;
> > > >
> > > > use LWP::Simple;
> > > > my $url=3D"https://notabug.org";
> > > > if (! head($url)) {
> > > >     die "$url is DOWN!!!!"
> > > > }
> > > >
> > > > Running above code I get
> > > >   https://notabug.org is DOWN!!!! at ./check_url.pl line 8.
> > > >
> > > >
> > > > However, firefox shows the site works ok.
> > > >
> > > >
> > > > What am I doing wrong?
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Manfred
> > > >
> > > > --
> > > > To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> > > > For additional commands, e-mail: beginners-help@perl.org
> > > > http://learn.perl.org/
> > > >
> > > >
> > > > =20
> > >
> > > =20
> >
> > --
> > To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> > For additional commands, e-mail: beginners-help@perl.org
> > http://learn.perl.org/
> >
> >
> > =20
>=20
>=20
0
ml_news
2/13/2018 8:34:09 PM
--001a114f4f9e2173ea05651e9543
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

> Is there another tool I could use for checking? I mean some tool in the
Perl universe?

Well, lwp-dump is a perl util - comes w/ LWP I believe. The sil.org, for
one, just returns forbidden/403 for their own policy reasons, but as far as
your "is it up?" question, that should be answer enough.  It uses
LWP::UserAgent.  To play fair (though it doesn't help with sil.org) you
should be looking for /robots.txt as you're creating a robot.

Pretty sure there's a libcurl interface (Net::Curl and WWW::Curl for two)
which might have better luck impersonating a proper user to get around the
policy.  But your urls so far have shown some odd repsonse using wget so
you may want to check them out first before your script has at them.

On Tue, Feb 13, 2018 at 2:34 PM, Manfred Lotz <ml_news@posteo.de> wrote:

> On Tue, 13 Feb 2018 13:50:55 -0600
> Andy Bach <afbach@gmail.com> wrote:
>
> > $ wget http://scripts.sil.org/OFL
> > --2018-02-13 13:42:50--  http://scripts.sil.org/OFL
> > Resolving scripts.sil.org (scripts.sil.org)... 209.12.63.143
> > Connecting to scripts.sil.org (scripts.sil.org)|209.12.63.143|:80...
> > connected.
> > HTTP request sent, awaiting response... 302 Found
> > Location:
> > http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL
> > [following] --2018-02-13 13:42:52--
> > http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL
> > Reusing existing connection to scripts.sil.org:80.
> > HTTP request sent, awaiting response... 302 Moved Temporarily
> > Location: /cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL&_sc=3D1 [follow=
ing]
> > --2018-02-13 13:42:52--
> > http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL&_sc=
=3D1
> > Reusing existing connection to scripts.sil.org:80.
> > HTTP request sent, awaiting response... 302 Moved Temporarily
> > Location: /cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL [following]
> > --2018-02-13 13:42:53--
> > http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&id=3DOFL
> > Reusing existing connection to scripts.sil.org:80.
> > HTTP request sent, awaiting response... 200 OK
> > Length: unspecified [text/html]
> > Saving to: =E2=80=98OFL=E2=80=99
> >
> >     [
> > <=3D>
> > ] 37,439      59.6KB/s   in 0.6s
> >
> > 2018-02-13 13:42:55 (59.6 KB/s) - =E2=80=98OFL=E2=80=99 saved [37439]
> >
> > so it may not be following the 302s. I'm not sure you're using the
> > correct tool here.  A little more straight forward
> >
> > andy@wiwmb-md-afb-mint:~/spam$ wget http://scripts.sil.org/robots.txt
> > --2018-02-13 13:47:27--  http://scripts.sil.org/robots.txt
> > Resolving scripts.sil.org (scripts.sil.org)... 209.12.63.143
> > Connecting to scripts.sil.org (scripts.sil.org)|209.12.63.143|:80...
> > connected.
> > HTTP request sent, awaiting response... 200 OK
> > Length: 36 [text/plain]
> > Saving to: =E2=80=98robots.txt=E2=80=99
> >
> > 100%[=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D>]
> > 36          --.-K/s   in 0s
> >
> > 2018-02-13 13:47:27 (2.99 MB/s) - =E2=80=98robots.txt=E2=80=99 saved [3=
6/36]
> >
> > but
> > $ is_it_up.pl
> > http://scripts.sil.org/robots.txt is DOWN!!!!
> >
> > You might look at more LWP tools:
> > $ lwp-dump https://www.sil.org
> > HTTP/1.1 403 Forbidden
> > Cache-Control: max-age=3D10
> > Connection: keep-alive
> > Date: Tue, 13 Feb 2018 19:49:47 GMT
> > Server: cloudflare
> > Content-Type: text/html; charset=3DUTF-8
> > Expires: Tue, 13 Feb 2018 19:49:57 GMT
> > CF-RAY: 3eca501a5d569895-LAX
> > Expect-CT: max-age=3D604800, report-uri=3D"
> > https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
> > Set-Cookie: __cfduid=3Ddd8038f4f2c995fa4b4c7fa8beb2b42f31518551387;
> > expires=3DWed, 13-Feb-19 19:49:47 GMT; path=3D/; domain=3D.sil.org; Htt=
pOnly
> > X-Frame-Options: SAMEORIGIN
> >
> > <!DOCTYPE html>
> > <!--[if lt IE 7]> <html class=3D"no-js ie6 oldie" lang=3D"en-US">
> > <![endif]--> <!--[if IE 7]>    <html class=3D"no-js ie7 oldie"
> > lang=3D"en-US"> <![endif]--> <!--[if IE 8]>    <html class=3D"no-js ie8
> > oldie" lang=3D"en-US"> <![endif]--> <!--[if gt IE 8]><!--> <html
> > class=3D"no-js" lang=3D"en-US"> <!--<![endif]--> <head>
> > <title>Access denied | www.sil.org used Cloudflare to restrict
> > access</title>
> > <meta charset=3D"UTF-8" />
> > <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-8=
" />
> > <meta http-equiv=3D...
> > (+ 2770 more bytes not shown)
> >
> > so it's up, but "forbidden" probably as the user agent isn't set or
> > some other policy reason.
> >
> >
>
> I tried WWW::Mechanize, and (of course) got also 403.
>
> Really strange.
>
> Is there another tool I could use for checking? I mean some tool in the
> Perl universe?
>
> --
> Manfred
>
>
>
> > On Tue, Feb 13, 2018 at 11:33 AM, Manfred Lotz <ml_news@posteo.de>
> > wrote:
> >
> > > On Tue, 13 Feb 2018 10:47:42 -0600
> > > Andy Bach <afbach@gmail.com> wrote:
> > >
> > > > The site doesn't like 'head' requests? get works
> > > > #!/usr/bin/perl
> > > >
> > > > use strict;
> > > > use warnings;
> > > >
> > > > use LWP::Simple;
> > > > #  my $url=3D"https://shlomif.github.io/";
> > > > my $url=3D"http://www.notabug.org/";
> > > > print "$url is ", (
> > > >                 (! get($url)) ?  "DOWN!!!!"
> > > >                                 : "up"
> > > >                 ), "\n";
> > > >
> > > > $ is_it_up.pl
> > > > http://www.notabug.org/ is up
> > > >
> > >
> > > You are right.
> > >
> > > But am afraid this is not all of it. If I test
> > > http://scripts.sil.org/OFL then I get an error but it is fine in
> > > firefox.
> > >
> > > Very strange.
> > >
> > > --
> > > Manfred
> > >
> > >
> > >
> > > >
> > > > On Tue, Feb 13, 2018 at 5:25 AM, Manfred Lotz <ml_news@posteo.de>
> > > > wrote:
> > > >
> > > > > Hi there,
> > > > > Somewhere I found an example how to check if a website is up.
> > > > >
> > > > > Here my sample:
> > > > >
> > > > > #! /usr/bin/perl
> > > > >
> > > > > use strict;
> > > > >
> > > > > use LWP::Simple;
> > > > > my $url=3D"https://notabug.org";
> > > > > if (! head($url)) {
> > > > >     die "$url is DOWN!!!!"
> > > > > }
> > > > >
> > > > > Running above code I get
> > > > >   https://notabug.org is DOWN!!!! at ./check_url.pl line 8.
> > > > >
> > > > >
> > > > > However, firefox shows the site works ok.
> > > > >
> > > > >
> > > > > What am I doing wrong?
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Manfred
> > > > >
> > > > > --
> > > > > To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> > > > > For additional commands, e-mail: beginners-help@perl.org
> > > > > http://learn.perl.org/
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> > > For additional commands, e-mail: beginners-help@perl.org
> > > http://learn.perl.org/
> > >
> > >
> > >
> >
> >
>
> --
> To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> For additional commands, e-mail: beginners-help@perl.org
> http://learn.perl.org/
>
>
>


--=20

a

Andy Bach,
afbach@gmail.com
608 658-1890 cell
608 261-5738 wk

--001a114f4f9e2173ea05651e9543
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>&gt; Is there another tool I could use for check=
ing? I mean some tool in the Perl universe?<br><br></div>Well, lwp-dump is =
a perl util - comes w/ LWP I believe. The <a href=3D"http://sil.org">sil.or=
g</a>, for one, just returns forbidden/403 for their own policy reasons, bu=
t as far as your &quot;is it up?&quot; question, that should be answer enou=
gh.=C2=A0 It uses LWP::UserAgent.=C2=A0 To play fair (though it doesn&#39;t=
 help with <a href=3D"http://sil.org">sil.org</a>) you should be looking fo=
r /robots.txt as you&#39;re creating a robot.<br><br></div>Pretty sure ther=
e&#39;s a libcurl interface (Net::Curl and WWW::Curl for two) which might h=
ave better luck impersonating a proper user to get around the policy.=C2=A0=
 But your urls so far have shown some odd repsonse using wget so you may wa=
nt to check them out first before your script has at them.<br></div><div cl=
ass=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Feb 13, 2018 at =
2:34 PM, Manfred Lotz <span dir=3D"ltr">&lt;<a href=3D"mailto:ml_news@poste=
o.de" target=3D"_blank">ml_news@posteo.de</a>&gt;</span> wrote:<br><blockqu=
ote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc s=
olid;padding-left:1ex"><div class=3D"HOEnZb"><div class=3D"h5">On Tue, 13 F=
eb 2018 13:50:55 -0600<br>
Andy Bach &lt;<a href=3D"mailto:afbach@gmail.com">afbach@gmail.com</a>&gt; =
wrote:<br>
<br>
&gt; $ wget <a href=3D"http://scripts.sil.org/OFL" rel=3D"noreferrer" targe=
t=3D"_blank">http://scripts.sil.org/OFL</a><br>
&gt; --2018-02-13 13:42:50--=C2=A0 <a href=3D"http://scripts.sil.org/OFL" r=
el=3D"noreferrer" target=3D"_blank">http://scripts.sil.org/OFL</a><br>
&gt; Resolving <a href=3D"http://scripts.sil.org" rel=3D"noreferrer" target=
=3D"_blank">scripts.sil.org</a> (<a href=3D"http://scripts.sil.org" rel=3D"=
noreferrer" target=3D"_blank">scripts.sil.org</a>)... 209.12.63.143<br>
&gt; Connecting to <a href=3D"http://scripts.sil.org" rel=3D"noreferrer" ta=
rget=3D"_blank">scripts.sil.org</a> (<a href=3D"http://scripts.sil.org" rel=
=3D"noreferrer" target=3D"_blank">scripts.sil.org</a>)|209.12.63.<wbr>143|:=
80...<br>
&gt; connected.<br>
&gt; HTTP request sent, awaiting response... 302 Found<br>
&gt; Location:<br>
&gt; <a href=3D"http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&=
amp;id=3DOFL" rel=3D"noreferrer" target=3D"_blank">http://scripts.sil.org/c=
ms/<wbr>scripts/page.php?site_id=3Dnrsi&amp;<wbr>id=3DOFL</a><br>
&gt; [following] --2018-02-13 13:42:52--<br>
&gt; <a href=3D"http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&=
amp;id=3DOFL" rel=3D"noreferrer" target=3D"_blank">http://scripts.sil.org/c=
ms/<wbr>scripts/page.php?site_id=3Dnrsi&amp;<wbr>id=3DOFL</a><br>
&gt; Reusing existing connection to <a href=3D"http://scripts.sil.org:80" r=
el=3D"noreferrer" target=3D"_blank">scripts.sil.org:80</a>.<br>
&gt; HTTP request sent, awaiting response... 302 Moved Temporarily<br>
&gt; Location: /cms/scripts/page.php?site_id=3D<wbr>nrsi&amp;id=3DOFL&amp;_=
sc=3D1 [following]<br>
&gt; --2018-02-13 13:42:52--<br>
&gt; <a href=3D"http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&=
amp;id=3DOFL&amp;_sc=3D1" rel=3D"noreferrer" target=3D"_blank">http://scrip=
ts.sil.org/cms/<wbr>scripts/page.php?site_id=3Dnrsi&amp;<wbr>id=3DOFL&amp;_=
sc=3D1</a><br>
&gt; Reusing existing connection to <a href=3D"http://scripts.sil.org:80" r=
el=3D"noreferrer" target=3D"_blank">scripts.sil.org:80</a>.<br>
&gt; HTTP request sent, awaiting response... 302 Moved Temporarily<br>
&gt; Location: /cms/scripts/page.php?site_id=3D<wbr>nrsi&amp;id=3DOFL [foll=
owing]<br>
&gt; --2018-02-13 13:42:53--<br>
&gt; <a href=3D"http://scripts.sil.org/cms/scripts/page.php?site_id=3Dnrsi&=
amp;id=3DOFL" rel=3D"noreferrer" target=3D"_blank">http://scripts.sil.org/c=
ms/<wbr>scripts/page.php?site_id=3Dnrsi&amp;<wbr>id=3DOFL</a><br>
&gt; Reusing existing connection to <a href=3D"http://scripts.sil.org:80" r=
el=3D"noreferrer" target=3D"_blank">scripts.sil.org:80</a>.<br>
&gt; HTTP request sent, awaiting response... 200 OK<br>
&gt; Length: unspecified [text/html]<br>
&gt; Saving to: =E2=80=98OFL=E2=80=99<br>
&gt;<br>
&gt;=C2=A0 =C2=A0 =C2=A0[<br>
&gt; &lt;=3D&gt;<br>
&gt; ] 37,439=C2=A0 =C2=A0 =C2=A0 59.6KB/s=C2=A0 =C2=A0in 0.6s<br>
&gt;<br>
&gt; 2018-02-13 13:42:55 (59.6 KB/s) - =E2=80=98OFL=E2=80=99 saved [37439]<=
br>
&gt;<br>
&gt; so it may not be following the 302s. I&#39;m not sure you&#39;re using=
 the<br>
&gt; correct tool here.=C2=A0 A little more straight forward<br>
&gt;<br>
&gt; andy@wiwmb-md-afb-mint:~/spam$ wget <a href=3D"http://scripts.sil.org/=
robots.txt" rel=3D"noreferrer" target=3D"_blank">http://scripts.sil.org/rob=
ots.<wbr>txt</a><br>
&gt; --2018-02-13 13:47:27--=C2=A0 <a href=3D"http://scripts.sil.org/robots=
..txt" rel=3D"noreferrer" target=3D"_blank">http://scripts.sil.org/robots.<w=
br>txt</a><br>
&gt; Resolving <a href=3D"http://scripts.sil.org" rel=3D"noreferrer" target=
=3D"_blank">scripts.sil.org</a> (<a href=3D"http://scripts.sil.org" rel=3D"=
noreferrer" target=3D"_blank">scripts.sil.org</a>)... 209.12.63.143<br>
&gt; Connecting to <a href=3D"http://scripts.sil.org" rel=3D"noreferrer" ta=
rget=3D"_blank">scripts.sil.org</a> (<a href=3D"http://scripts.sil.org" rel=
=3D"noreferrer" target=3D"_blank">scripts.sil.org</a>)|209.12.63.<wbr>143|:=
80...<br>
&gt; connected.<br>
&gt; HTTP request sent, awaiting response... 200 OK<br>
&gt; Length: 36 [text/plain]<br>
&gt; Saving to: =E2=80=98robots.txt=E2=80=99<br>
&gt;<br>
&gt; 100%[=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D<wbr>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<wbr>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<wbr>=3D=3D=3D&gt;]<b=
r>
&gt; 36=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 --.-K/s=C2=A0 =C2=A0in 0s<br>
&gt;<br>
&gt; 2018-02-13 13:47:27 (2.99 MB/s) - =E2=80=98robots.txt=E2=80=99 saved [=
36/36]<br>
&gt;<br>
&gt; but<br>
&gt; $ <a href=3D"http://is_it_up.pl" rel=3D"noreferrer" target=3D"_blank">=
is_it_up.pl</a><br>
&gt; <a href=3D"http://scripts.sil.org/robots.txt" rel=3D"noreferrer" targe=
t=3D"_blank">http://scripts.sil.org/robots.<wbr>txt</a> is DOWN!!!!<br>
&gt;<br>
&gt; You might look at more LWP tools:<br>
&gt; $ lwp-dump <a href=3D"https://www.sil.org" rel=3D"noreferrer" target=
=3D"_blank">https://www.sil.org</a><br>
&gt; HTTP/1.1 403 Forbidden<br>
&gt; Cache-Control: max-age=3D10<br>
&gt; Connection: keep-alive<br>
&gt; Date: Tue, 13 Feb 2018 19:49:47 GMT<br>
&gt; Server: cloudflare<br>
&gt; Content-Type: text/html; charset=3DUTF-8<br>
&gt; Expires: Tue, 13 Feb 2018 19:49:57 GMT<br>
&gt; CF-RAY: 3eca501a5d569895-LAX<br>
&gt; Expect-CT: max-age=3D604800, report-uri=3D&quot;<br>
&gt; <a href=3D"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"=
 rel=3D"noreferrer" target=3D"_blank">https://report-uri.cloudflare.<wbr>co=
m/cdn-cgi/beacon/expect-ct</a>&quot;<br>
&gt; Set-Cookie: __cfduid=3D<wbr>dd8038f4f2c995fa4b4c7fa8beb2b4<wbr>2f31518=
551387;<br>
&gt; expires=3DWed, 13-Feb-19 19:49:47 GMT; path=3D/; domain=3D.<a href=3D"=
http://sil.org" rel=3D"noreferrer" target=3D"_blank">sil.org</a>; HttpOnly<=
br>
&gt; X-Frame-Options: SAMEORIGIN<br>
&gt;<br>
&gt; &lt;!DOCTYPE html&gt;<br>
&gt; &lt;!--[if lt IE 7]&gt; &lt;html class=3D&quot;no-js ie6 oldie&quot; l=
ang=3D&quot;en-US&quot;&gt;<br>
&gt; &lt;![endif]--&gt; &lt;!--[if IE 7]&gt;=C2=A0 =C2=A0 &lt;html class=3D=
&quot;no-js ie7 oldie&quot;<br>
&gt; lang=3D&quot;en-US&quot;&gt; &lt;![endif]--&gt; &lt;!--[if IE 8]&gt;=
=C2=A0 =C2=A0 &lt;html class=3D&quot;no-js ie8<br>
&gt; oldie&quot; lang=3D&quot;en-US&quot;&gt; &lt;![endif]--&gt; &lt;!--[if=
 gt IE 8]&gt;&lt;!--&gt; &lt;html<br>
&gt; class=3D&quot;no-js&quot; lang=3D&quot;en-US&quot;&gt; &lt;!--&lt;![en=
dif]--&gt; &lt;head&gt;<br>
&gt; &lt;title&gt;Access denied | <a href=3D"http://www.sil.org" rel=3D"nor=
eferrer" target=3D"_blank">www.sil.org</a> used Cloudflare to restrict<br>
&gt; access&lt;/title&gt;<br>
&gt; &lt;meta charset=3D&quot;UTF-8&quot; /&gt;<br>
&gt; &lt;meta http-equiv=3D&quot;Content-Type&quot; content=3D&quot;text/ht=
ml; charset=3DUTF-8&quot; /&gt;<br>
&gt; &lt;meta http-equiv=3D...<br>
&gt; (+ 2770 more bytes not shown)<br>
&gt;<br>
&gt; so it&#39;s up, but &quot;forbidden&quot; probably as the user agent i=
sn&#39;t set or<br>
&gt; some other policy reason.<br>
&gt;<br>
&gt;<br>
<br>
</div></div>I tried WWW::Mechanize, and (of course) got also 403.<br>
<br>
Really strange.<br>
<br>
Is there another tool I could use for checking? I mean some tool in the<br>
Perl universe?<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
--<br>
Manfred<br>
<br>
<br>
<br>
&gt; On Tue, Feb 13, 2018 at 11:33 AM, Manfred Lotz &lt;<a href=3D"mailto:m=
l_news@posteo.de">ml_news@posteo.de</a>&gt;<br>
&gt; wrote:<br>
&gt;<br>
&gt; &gt; On Tue, 13 Feb 2018 10:47:42 -0600<br>
&gt; &gt; Andy Bach &lt;<a href=3D"mailto:afbach@gmail.com">afbach@gmail.co=
m</a>&gt; wrote:<br>
&gt; &gt;<br>
&gt; &gt; &gt; The site doesn&#39;t like &#39;head&#39; requests? get works=
<br>
&gt; &gt; &gt; #!/usr/bin/perl<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; use strict;<br>
&gt; &gt; &gt; use warnings;<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; use LWP::Simple;<br>
&gt; &gt; &gt; #=C2=A0 my $url=3D&quot;<a href=3D"https://shlomif.github.io=
/" rel=3D"noreferrer" target=3D"_blank">https://shlomif.github.<wbr>io/</a>=
&quot;;<br>
&gt; &gt; &gt; my $url=3D&quot;<a href=3D"http://www.notabug.org/" rel=3D"n=
oreferrer" target=3D"_blank">http://www.notabug.org/</a>&quot;<wbr>;<br>
&gt; &gt; &gt; print &quot;$url is &quot;, (<br>
&gt; &gt; &gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0(! get($url)) ?=C2=A0 &quot;DOWN!!!!&quot;<br>
&gt; &gt; &gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: &quot;up&quot;=
<br>
&gt; &gt; &gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0), &quot;\n&quot;;<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; $ <a href=3D"http://is_it_up.pl" rel=3D"noreferrer" target=
=3D"_blank">is_it_up.pl</a><br>
&gt; &gt; &gt; <a href=3D"http://www.notabug.org/" rel=3D"noreferrer" targe=
t=3D"_blank">http://www.notabug.org/</a> is up<br>
&gt; &gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; You are right.<br>
&gt; &gt;<br>
&gt; &gt; But am afraid this is not all of it. If I test<br>
&gt; &gt; <a href=3D"http://scripts.sil.org/OFL" rel=3D"noreferrer" target=
=3D"_blank">http://scripts.sil.org/OFL</a> then I get an error but it is fi=
ne in<br>
&gt; &gt; firefox.<br>
&gt; &gt;<br>
&gt; &gt; Very strange.<br>
&gt; &gt;<br>
&gt; &gt; --<br>
&gt; &gt; Manfred<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; On Tue, Feb 13, 2018 at 5:25 AM, Manfred Lotz &lt;<a href=3D=
"mailto:ml_news@posteo.de">ml_news@posteo.de</a>&gt;<br>
&gt; &gt; &gt; wrote:<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; Hi there,<br>
&gt; &gt; &gt; &gt; Somewhere I found an example how to check if a website =
is up.<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; Here my sample:<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; #! /usr/bin/perl<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; use strict;<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; use LWP::Simple;<br>
&gt; &gt; &gt; &gt; my $url=3D&quot;<a href=3D"https://notabug.org" rel=3D"=
noreferrer" target=3D"_blank">https://notabug.org</a>&quot;;<br>
&gt; &gt; &gt; &gt; if (! head($url)) {<br>
&gt; &gt; &gt; &gt;=C2=A0 =C2=A0 =C2=A0die &quot;$url is DOWN!!!!&quot;<br>
&gt; &gt; &gt; &gt; }<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; Running above code I get<br>
&gt; &gt; &gt; &gt;=C2=A0 =C2=A0<a href=3D"https://notabug.org" rel=3D"nore=
ferrer" target=3D"_blank">https://notabug.org</a> is DOWN!!!! at ./<a href=
=3D"http://check_url.pl" rel=3D"noreferrer" target=3D"_blank">check_url.pl<=
/a> line 8.<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; However, firefox shows the site works ok.<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; What am I doing wrong?<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; --<br>
&gt; &gt; &gt; &gt; Thanks,<br>
&gt; &gt; &gt; &gt; Manfred<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt; --<br>
&gt; &gt; &gt; &gt; To unsubscribe, e-mail: <a href=3D"mailto:beginners-uns=
ubscribe@perl.org">beginners-unsubscribe@perl.org</a><br>
&gt; &gt; &gt; &gt; For additional commands, e-mail: <a href=3D"mailto:begi=
nners-help@perl.org">beginners-help@perl.org</a><br>
&gt; &gt; &gt; &gt; <a href=3D"http://learn.perl.org/" rel=3D"noreferrer" t=
arget=3D"_blank">http://learn.perl.org/</a><br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt;<br>
&gt; &gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt; --<br>
&gt; &gt; To unsubscribe, e-mail: <a href=3D"mailto:beginners-unsubscribe@p=
erl.org">beginners-unsubscribe@perl.org</a><br>
&gt; &gt; For additional commands, e-mail: <a href=3D"mailto:beginners-help=
@perl.org">beginners-help@perl.org</a><br>
&gt; &gt; <a href=3D"http://learn.perl.org/" rel=3D"noreferrer" target=3D"_=
blank">http://learn.perl.org/</a><br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt; &gt;<br>
&gt;<br>
&gt;<br>
<br>
--<br>
To unsubscribe, e-mail: <a href=3D"mailto:beginners-unsubscribe@perl.org">b=
eginners-unsubscribe@perl.org</a><br>
For additional commands, e-mail: <a href=3D"mailto:beginners-help@perl.org"=
>beginners-help@perl.org</a><br>
<a href=3D"http://learn.perl.org/" rel=3D"noreferrer" target=3D"_blank">htt=
p://learn.perl.org/</a><br>
<br>
<br>
</div></div></blockquote></div><br><br clear=3D"all"><br>-- <br><div class=
=3D"gmail_signature" data-smartmail=3D"gmail_signature"><br>a<br><br>Andy B=
ach,<br><a href=3D"mailto:afbach@gmail.com" target=3D"_blank">afbach@gmail.=
com</a><br>608 658-1890 cell<br>608 261-5738 wk</div>
</div>

--001a114f4f9e2173ea05651e9543--
0
afbach
2/13/2018 9:21:48 PM
Reply: