Subject: lynx - not all sites readable

From: Hans <[email-masked]>
Date: Wed, 14 Sep 2016 22:18:59 +0200

Hi list,

can someone explain, why lynx sometimes forbid websites or supresses websites?

I needed a driver from Nvidia. As I had no X available, I tried download using 
lynx. 
However, I could open www.nvidia.com, but could not jump to the submenu alled 
"Treiber".  Then I tried to go to www.nvidia.de, but lynx told me, "This site 
is forbidden".

I found no logic for these issues. Tried http 1.0 and http 1.1, but there was 
no change. Does someone know more? IMO lynx is very usefull, when there is no 
X available!

Best regards

Hans

---

From: jeremy bentham <[email-masked]>
Date: Thu, 15 Sep 2016 19:38:48 -0700

> I needed a driver from Nvidia. As I had no X available, I tried download using 

Many sites block lynx, because it can be used as a crawler.
Sometimes it helps to obscure the user-agent:  I stick hyphens in
in the word "lynx" there and it helps.  Not really lying...just
not telling the truth {-; .

-- 
 Dave Williams         [email-masked]

---

From: emetib <[email-masked]>
Date: Wed, 14 Sep 2016 16:10:58 -0700 (PDT)

personally i prefer links

just seems to work better.  especially if you install gpm also.  make a great combo for terminal only browsing.

henning ->
because it would be fun.

i've had to browse forums before using a text only browser trying to fix my X.  it will make you a happy camper to have a graphic browser if you ever have to spend a couple of hours trying to find something on the web with a text browser.

em

---

From: Jude DaShiell <[email-masked]>
Date: Wed, 14 Sep 2016 17:03:22 -0400 (EDT)

lynx isn't suppressing or forbidding anything!  The sites you visit are 
doing all of that blocking and forbidding.

On Wed, 14 Sep 2016, Hans wrote:

> Date: Wed, 14 Sep 2016 16:18:59
> From: Hans <[email-masked]>
> To: [email-masked]
> Subject: lynx - not all sites readable
> Resent-Date: Wed, 14 Sep 2016 20:19:31 +0000 (UTC)
> Resent-From: [email-masked]
> 
> Hi list,
>

--

---

From: Jude DaShiell <[email-masked]>
Date: Wed, 14 Sep 2016 17:06:29 -0400 (EDT)

One of these days, I'd like to put a website up and permit only text 
browsers like links and lynx and edbrowse full access and either block 
all of the graphical browsers or simply cause all of them to crash.

On Wed, 14 Sep 2016, Jude DaShiell wrote:

> Date: Wed, 14 Sep 2016 17:03:22
> From: Jude DaShiell <[email-masked]>
> To: Hans <[email-masked]>, [email-masked]
> Subject: Re: lynx - not all sites readable
> Resent-Date: Wed, 14 Sep 2016 21:03:51 +0000 (UTC)
> lynx isn't suppressing or forbidding anything!  The sites you visit are doing 
> all of that blocking and forbidding.
>

--

---

From: Curt <[email-masked]>
Date: Thu, 15 Sep 2016 12:13:16 +0000 (UTC)

> all of the graphical browsers or simply cause all of them to crash.

That's the spirit. An eye for an eye is what I say (with a little
malicious crashing as whipped cream on the ocular globe).

-- 
“Whatever is rejected from the self, appears in the world as an event.” 
C.G. Jung

---

From: Karen Lewellen <[email-masked]>
Date: Thu, 15 Sep 2016 10:03:10 -0400 (EDT)

> malicious crashing as whipped cream on the ocular globe).

Oh how
  funny!  Honestly I would simply put them through the frustration 
encountered when  the low graphics browsers do not work...then explain how 
to create the site properly in the first place.
That, and  raise funds on kickstarter to pay  someone to create a browser 
that  embodies the best of both.
The ignorance is the greater issue.
I can no longer access my linked in account because the person working on 
the site thinks the enter key works, on their script buttons...and it does 
not.
Kare

---

From: Henning Follmann <[email-masked]>
Date: Wed, 14 Sep 2016 17:46:31 -0400

On Wed, Sep 14, 2016 at 05:06:29PM -0400, Jude DaShiell wrote:
> browsers like links and lynx and edbrowse full access and either
> block all of the graphical browsers or simply cause all of them to
> crash.
> 
That is stupid, petty and pretty much a waste of time.

-H

-- 
Henning Follmann           | [email-masked]

---

From: Andre Majorel <[email-masked]>
Date: Thu, 15 Sep 2016 10:47:55 +0200

> That is stupid, petty and pretty much a waste of time.

Yes, isn't it wonderful ?

To the OP : both sites (and the links under "Drivers") work with
Elinks as far as I can see.

I must say, it baffles me that Lynx continues to get more
mentions than Elinks, Links and W3m together even though they
render HTML at least as well and usually better that Lynx. Or
have things changed in recent years ?

-- 
André Majorel <http://www.teaser.fr/~amajorel/>
bugs.debian.org, a spambot's best friend.

---

From: Karen Lewellen <[email-masked]>
Date: Thu, 15 Sep 2016 09:47:14 -0400 (EDT)

> have things changed in recent years ?

Actually not as well or better, depending on the task.
Lynx is updated regularly, which  may be part of why you hear about it with 
greater frequency.
Plug in aspects like html5  in some distributions, the ability to manage 
script buttons sometimes too if the button is well created
.
Links is no longer updated, does not even support cookies.
e-links has not been updated for a while, while lynx was updated a few 
months  ago.

Karen

---

From: Dutch Ingraham <[email-masked]>
Date: Thu, 15 Sep 2016 10:24:42 -0500

On Thu, Sep 15, 2016 at 09:47:14AM -0400, Karen Lewellen wrote:
> Links is no longer updated, 

Where did you get this information?  As far as I can see, links was updated on
July 1, 2016.[1][2]  Does that qualify as "no longer updated"?  

[1] https://en.wikipedia.org/wiki/Links_(web_browser)
[2] https://www.archlinux.org/packages/core/x86_64/links/

---

From: Karen Lewellen <[email-masked]>
Date: Thu, 15 Sep 2016 16:47:01 -0400 (EDT)

Hi Dutch,

Actually I got that information from a Google staffer working with 
accessibility.  Such was why, at the time, gmail stopped working with 
links.
they have changed that factor though, and links often produces more 
detailed search results than lynx.
I appreciate your providing this correction.
Still, that brings up two questions of my own regarding links.
first, does the browser still not accept cookies?
and second, what is the comparative solution to Hans' forbidden  site 
problem?
I do not know if I can alter the send user agent  command in links.
I would welcome an idea though, because there are times when using links 
is the wiser option, but that barrier presents a problem.
Thanks,
Karen

---

From: Dutch Ingraham <[email-masked]>
Date: Thu, 15 Sep 2016 17:08:14 -0500

> first, does the browser still not accept cookies?

I'm not sure of your definition of "accept," but in the general sense links does
not store and set cookies.  This post [1] seems to make a distinction, though,
between persistent cookies and volatile cookies.

> and second, what is the comparative solution to Hans' forbidden  site

I had no problem visiting nvidia.de/Treiber with links.

> I do not know if I can alter the send user agent  command in links.

links -> setup -> Network options -> HTTP options -> Header options

[1] https://superuser.com/questions/478633/links-cookie-location

---

From: Karen Lewellen <[email-masked]>
Date: Thu, 15 Sep 2016 18:55:25 -0400 (EDT)

Hi Dutch,
The forbidden site issue I encounter  is not the one Hans referenced, but 
others.
take
masterworkshealing.com
for example,
So the idea would be an option  that I could change, as I directed Hans 
to  make a  short term change to the send user agent header one in lynx.
I realize naturally there are times when the  user agent header needs to 
be sent.  Likely why Lynx does not make this change perminent.  what about 
in links though?
As for storing cookies,  indeed,  I am wondering what to do if I try 
visiting a site and am not allowed to continue because the browser will not 
accept cookies...as in literary says as much.
for example,
www.walmart.ca
will not even move forward in links for this reason.
I am not sure if the reference you place below will explain the user agent 
header question, or the cookies one?
Thanks,
Karen

---

From: Dutch Ingraham <[email-masked]>
Date: Thu, 15 Sep 2016 18:32:27 -0500

> The forbidden site issue I encounter  is not the one Hans referenced, but

I can visit that site with links.

> for example,
> So the idea would be an option  that I could change, as I directed Hans to
> make a  short term change to the send user agent header one in lynx.
> I realize naturally there are times when the  user agent header needs to be
> sent.  Likely why Lynx does not make this change perminent.  what about in
> links though?

See http://links.twibright.com/user_en.html#subch-network-options  

> will not even move forward in links for this reason.

On my machine, that site is not viewable in the standard links config due to 
lack of javascript, not cookies.  Interestingly, walmart.com doesn't have that
issue.

---

From: Karen Lewellen <[email-masked]>
Date: Thu, 15 Sep 2016 19:57:27 -0400 (EDT)

Hi Dutch,

Something told me to try my example again  before I shared it, try this in 
links.

www.healingwiththemasters.com
I just tried and got the 403 forbidden error.
As for Walmart.com allowing the links browser, I am afraid that matters 
not as   I am in Canada and my reference is to the Canadian site.
My speech  states that the issue is a cookies one,  providing the offer of 
viewing the flyer without cookies.
I will check your resource for the user agent question.
Thanks,
Karen

---

From: Dutch Ingraham <[email-masked]>
Date: Thu, 15 Sep 2016 19:39:24 -0500

> I just tried and got the 403 forbidden error.

Yes, I get the same thing.  Note that if you turn off javascript on your
full-graphical browser (Firefox, etc), that same site will not
error-out, but will likely not display anything, either.

> As for Walmart.com allowing the links browser, I am afraid that matters not
> as   I am in Canada and my reference is to the Canadian site.

---

From: Karen Lewellen <[email-masked]>
Date: Thu, 15 Sep 2016 22:43:02 -0400 (EDT)

Hi Dutch,
I am not sure I agree with the java script focus.  after all that site 
will work in lynx, when I turn off the send user agent header option, just 
like I suggested to Hans.
why would  it work in a non graphical environment with lynx,  if I take the 
above 
referenced step if Java script is not  involved under those circumstances?
I cannot say as to the graphical ones, I do not have them.

---

From: Dutch Ingraham <[email-masked]>
Date: Thu, 15 Sep 2016 22:07:37 -0500

On Thu, Sep 15, 2016 at 10:43:02PM -0400, Karen Lewellen wrote:
> Hi Dutch,
> I am not sure I agree with the java script focus.  after all that site will
> work in lynx, when I turn off the send user agent header option, just like I
> suggested to Hans.
> above referenced step if Java script is not  involved under those
> circumstances?

No, it is likely the user agent string.  If I set links' user agent
string to fake firefox, the site will load.

---

From: Johann Spies <[email-masked]>
Date: Thu, 22 Sep 2016 09:08:33 +0200

I can open http://www.healingwiththemasters.com/ with elinks but not with
lynx or links.

Regards
Johann.

---

From: Vincent Lefevre <[email-masked]>
Date: Thu, 15 Sep 2016 12:29:50 +0200

> have things changed in recent years ?

Links does not support UTF-8. The other ones do.

-- 
Vincent Lefèvre <[email-masked]> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

---

From: Dutch Ingraham <[email-masked]>
Date: Thu, 15 Sep 2016 15:34:10 -0500

>On 2016-09-15 12:29:50 +0200, Vincent Lefevre wrote:

>Links does not support UTF-8. The other ones do.

Where is the support for that conclusion?  It appears to be incorrect.[1]

[1] https://wiki.gentoo.org/wiki/UTF-8#links_and_elinks

---

From: Vincent Lefevre <[email-masked]>
Date: Fri, 16 Sep 2016 12:26:55 +0200

> [1] https://wiki.gentoo.org/wiki/UTF-8#links_and_elinks

And concerning elinks, it supports UTF-8 by default (I don't have
any config file for elinks), so no need to set a special config as
mentioned on this page.

---

From: Vincent Lefevre <[email-masked]>
Date: Fri, 16 Sep 2016 11:27:43 +0200

> [1] https://wiki.gentoo.org/wiki/UTF-8#links_and_elinks

OK, so if I understand correctly, it does not support UTF-8
by default, and one needs to explicitly enable it. That's
still a buggy way to support UTF-8. This should be automatic
based on locales, just like with other applications that run
in a terminal.

---

From: Jude DaShiell <[email-masked]>
Date: Thu, 15 Sep 2016 05:59:21 -0400 (EDT)

Links can be made to work accessibly; elinks cannot.  Last version of 
elinks I used I couldn't find any setting in it to number links.
w3m runs stuff a bit differently but can be blocked by javascript on 
sites.  The creator of the web is frustrated with the way it turned out 
and specifically for accessibility reasons.  He wasn't blind from birth 
either.

On Thu, 15 Sep 2016, Andre Majorel wrote:

> Date: Thu, 15 Sep 2016 04:47:55
> From: Andre Majorel <[email-masked]>
> Resent-Date: Thu, 15 Sep 2016 08:48:30 +0000 (UTC)

--

---

From: Hans <[email-masked]>
Date: Thu, 15 Sep 2016 11:08:25 +0200

Hi Andre,
> Elinks as far as I can see.
yes, elinks is working well and let me download the drivers. But that did not 
answer my question, why lynx ignored the links. 

> have things changed in recent years ?

Due to your suggestion, I tried elinks. For my personal taste I like the 
colour setting of lynx more. However, I see elinks is also great with another 
GUI philosophie. It reminds me of old packet radio programs in DOS like turbo 
packet, top, stop or  similar written in turbo pascal.

In the past I also tried links, which I also was not so emphazised of, as its 
very simple interface. However, this is a personal meaning, and I believe, 
that other people might love, what I dislike. :)

Thanks for the hint and happy hacking!

Hans

---

From: David Wright <[email-masked]>
Date: Thu, 15 Sep 2016 11:43:41 -0500

> answer my question, why lynx ignored the links. 

Imagine a company puts a clickable map of the UK on their webpage and
asks you to click on the county you live in. They're labelled
Northamptonshire, Rutland, Norfolk etc.

A graphical browser might show you the links at the bottom of the
window as you move your mouse over the counties on the map. It might
say things like http://..../item=58 http://..../item=32 etc.
Lynx will list those addresses but it hasn't a clue how to
display the names of the counties, which might be in the text
of the page or even as bits in the image of the map.
And the names of the counties might be irrelevant anyway: those
links could correspond with regions, like East Midlands and
East Anglia, that aren't labelled on the page at all.

Lynx will list them separately (last) or merged with the rest
as requested ( -hiddenlinks=[option] ).

Cheers,
David.

---

From: Andre Majorel <[email-masked]>
Date: Thu, 15 Sep 2016 12:16:39 +0200

On 2016-09-15 11:08 +0200, Hans wrote:

> Due to your suggestion, I tried elinks. For my personal taste
> I like the colour setting of lynx more.

My Elinks have colour disabled so use the xterm's colours (in my
case, black text on white-ish background). I often end up doing
this as most terminal applications that use colour have a
positive contrast colour scheme (light on dark) which I find
tiring.

-- 
André Majorel <http://www.teaser.fr/~amajorel/>
lists.debian.org, an essential online resource for spammers.

---

From: David Wright <[email-masked]>
Date: Wed, 14 Sep 2016 19:01:48 -0500

> That is stupid, petty and pretty much a waste of time.

Is there any reason to try to make replies to your post get CC'd to the OP
with Mail-Followup-To: Jude ..., Hans ..., [email-masked] ?

Cheers,
David.

---

From: Hans <[email-masked]>
Date: Wed, 14 Sep 2016 22:26:08 +0200

Correction: 
> However, I could open www.nvidia.com, but could not jump to the submenu
> alled "Treiber".  Then I tried to go to www.nvidia.de, but lynx told me,
> "This site is forbidden".
> 
Must be: 
www.nvidia.de could be opened
www.nvidia.com was forbidden.

Sorry for that.

Hans

---

From: Brian <[email-masked]>
Date: Thu, 15 Sep 2016 00:11:06 +0100

> www.nvidia.com was forbidden.

What do you get with

lynx -useragent "Mozilla/5.0 (X11; Linux i686; rv:38.0) Gecko/20100101 Firefox/38.0 Iceweasel/38.8.0" www.nvidia.com ?

---

From: Karen Lewellen <[email-masked]>
Date: Wed, 14 Sep 2016 20:06:40 -0400 (EDT)

Hi,
Forgive this not being in context.
If you are using the most current development edition of Lynx from April 
25  of this year and you are reaching sites that present a message like 
the following,
403 forbidden,
I suggest you try the following.
open the options menu and arrow down to the item that says
   "send user agent," with an exclamation mark because the change  cannot 
be saved  permanently.
unchecked this option do the temporary  save options  and you will get 
confirmation that the value has been accepted.
if you are still on the site in question, things may improve right 
away.  if not try the site again, with positive results..or such is 
my experience   almost all of the time when I reach the forbidden message.
I hope this is helpful,
Karen

---

From: Hans <[email-masked]>
Date: Thu, 15 Sep 2016 08:36:51 +0200

Am Mittwoch, 14. September 2016, 20:06:40 CEST schrieb Karen Lewellen:
Hi Karen,

thanks for your hint! This is working. Deactivating sending the agent, let me 
read the site. However, still there is the problem, that I cannot step to 
submenus on the site. If one wants to see, what I mean:

go to www.nvidia.com, then try to go to the "Drivers" link. The colour here is 
purple of the link, and cannot be reached.

Hans

---

From: Karen Lewellen <[email-masked]>
Date: Thu, 15 Sep 2016 10:31:03 -0400 (EDT)

> purple of the link, and cannot be reached.

Hi Hans,
Okay,  here is what I just did to reach that area.
Indeed you are correct, especially if you have the links are numbered 
option turned on, there is no clear way to reach the drivers area.
However lynx will display  and allow you to reach hidden links on a page.
Once you go through the steps providing the content on this site, choose 
the l key as in the letter l.
You are presented in order the links on the page, starting  with the 
visible ones.
Now,  if you use the search function, /
and enter the word hidden,  you are taken to the list of hidden links  on 
the page, including one for the drivers.
with links are numbered turned on, at least for me, it is number 114.
activate  the hidden link option for the drivers and you are taken to the 
page where they are  listed.
Does this make sense?
Kare

---

From: Hans <[email-masked]>
Date: Thu, 15 Sep 2016 16:42:19 +0200

Hi Karen,

I tried as you described. However, this showed me the word "Treiber" but your 
better hint was the one with the letter "l". That way I can see, these links 
are hidden and could navigate to it (it is letter 107 here).
Gowever, it does not explain, why lynx suppresses these. Ok, these are hidden, 
but accessible, so lynx should be able to acccess them.

Maybe it is still configurationable, I will try once more.

Thanks for your help, it makes things clearer.

Best

Hans

---

From: [email-masked]
Date: Fri, 16 Sep 2016 00:01:19 +0000 (UTC)

On Thu, 15 Sep 2016, Hans wrote:

> Hi Karen,
>
> I tried as you described. However, this showed me the word "Treiber"
> but your better hint was the one with the letter "l". That way I can
> see, these links are hidden and could navigate to it (it is letter
> 107 here).

In my lynx environment one must use shift + l (ie, capital 'L'), to
invoke this command, whose name is LIST.

And, no matter what the keymap might be, one can call it by typing ':'
followed by 'LIST'. (Just ':list', in lowercase, seems to work, too).

Also useful, in case you would like to search for a particular url, is
the command ADDRLIST, which is just like LIST except that it names all
links with their url, (thereby enabling searches for urls).

I have ADDRLIST mapped to capital 'A'. Enter ':keymap' to find the
shortcuts in your environment.

> Gowever, it does not explain, why lynx suppresses these. Ok, these

It looks to me like you have concisely summed up an explanation
yourself.

> but accessible, so lynx should be able to acccess them.

And lynx can, when you tell it to.

> Maybe it is still configurationable, I will try once more.

If I understand you correctly, you would like hidden links to appear
on the rendered page.

While I hope you find a satisfactory solution, and I look forward to
reading about any you find, the question arises: Where in the rendered
page ought lynx to place these no-longer-hidden links?

As Karen points out, we cannot sensibly look to the intentions of the
author of the web page to give lynx any guidance here, since the web
page's author has explicitly told lynx to hide them!

Kind regards.

-- 
Wrote David Woolley, long ago:

> One final point to remember, is that most web site are not about
> communicating information; they are about selling, and these days
> the two are almost incompatible. (It's an interesting point that, in
> spite of the accessibility lobby's objections to PDF, PDF is where
> you will find the real information on most commercial sites, and the
> HTML on the sites is used where PDF would have been a better
> vehicle, because those parts have all form and no content!)

https://lists.nongnu.org/archive/html/lynx-dev/1999-08/msg00694.html

---

From: Karen Lewellen <[email-masked]>
Date: Thu, 15 Sep 2016 10:52:29 -0400 (EDT)

Hi Hans,
lynx no more suppresses site content, than a monitor  refuses to show 
material.
All lynx can do is give you what the site designer  has used in their 
code.
If the person drafting the site decides to use  techniques that do not 
respond to the keyboard, do not create active links  try  to do something 
loopy like a sort of dynamic element, it is not the fault of  lynx.
In fact you may wish to contact the people behind this site and inform 
them.  Especially since lynx is often used by those in their market.
Does this make sense?
Remember the references to the Internet as a super highway.
Lynx is just a car, if someone leaves pot holes  in the road, one does not 
blame the car for this   yes?
Happy to help,
Kare