vp_web getURLContentAsText

Discussions related to Visual Prolog
User avatar
Tonton Luc
VIP Member
Posts: 814
Joined: 16 Oct 2001 23:01

vp_web getURLContentAsText

Unread post by Tonton Luc » 6 Oct 2009 8:41

Hi,
The following code gives exactly the same results (TxtHTML_1 = TxtHTML_2) and it's not normal.

Code: Select all

    trap(TxtHTML_1 = getURLContentAsText(URL),_,fail()),     sleep(1000),     trap(TxtHTML_2 = getURLContentAsText(URL),_,fail()),
It seems to me that the buffer (or something else) is not reinitialized ...maybe in this clause (in vp_web.pro) :

Code: Select all

clauses /* new version*/     getURLContentAsBin(URL) = Res :-           IT   = vp_web::internetOpenws(),           FIT = vp_web::internetOpenUrlWs(IT,URL),           binres := binary::createAtomic(0),           sz := 0,           programControl::repeat,               Temp = binary::createAtomic(bufferSize),               Remain = vp_web::internetReadFile(FIT, Temp, bufferSize),               if Remain >= 0 then                     sz:=sz+Remain,                     Temp2 = binary::reSizeAtomic(Temp,Remain),                     binres := binary::concatAtomic(binres, Temp2)               end if,          Remain = 0,          vp_web::internetClosehandle(FIT),          vp_web::internetClosehandle(IT),          Res = binary::reSizeAtomic(binres,sz),     !.%  

User avatar
Thomas Linder Puls
VIP Member
Posts: 2448
Joined: 28 Feb 2000 0:01

Unread post by Thomas Linder Puls » 6 Oct 2009 9:54

It would help if you told:
  • Where this code comes from
  • Why it is not "normal"
Regards Thomas Linder Puls
PDC

User avatar
Tonton Luc
VIP Member
Posts: 814
Joined: 16 Oct 2001 23:01

Unread post by Tonton Luc » 6 Oct 2009 10:16

Yes : I was found it in Gildas Menier site : http://www.arsaniit.com/

User avatar
Thomas Linder Puls
VIP Member
Posts: 2448
Joined: 28 Feb 2000 0:01

Unread post by Thomas Linder Puls » 6 Oct 2009 12:50

Looking at the code you have shown it seems that:
  • The variable "Remain" actually contains the number of bytes read. (Minor issue but it confuses a bit).
  • The resize into "Temp2" is very strange, I fail to see what purpose it has.
If I should write the code I would have collected the result in a binary stream instead. Something like this (I have used integer where I dont know the exact domain:

Code: Select all

clauses     getURLContentAsBin(URL) = Res :-         Stream = outputStream_binary::new(),         Buffer = binary::createAtomic(bufferSize),         IT   = vp_web::internetOpenws(),         FIT = vp_web::internetOpenUrlWs(IT,URL),         try             getURLContentAsBin2(Stream, FIT, Buffer, bufferSize)         finally             vp_web::internetClosehandle(FIT),             vp_web::internetClosehandle(IT)         end try,         Res = Stream:getBinary().   class predicates     getURLContentAsBin2 : (outputStream Stream, integer FIT, binary Buffer, integer BufferSize). clauses     getURLContentAsBin2(Stream, FIT, Buffer, BufferSize) :-         Read = vp_web::internetReadFile(FIT, Buffer, BufferSize),         Stream:writeBytes(convert(pointer, Buffer), Read),         if Read > 0 then             getURLContentAsBin2(Stream, FIT, Buffer, BufferSize)         end if.
Regards Thomas Linder Puls
PDC

User avatar
Tonton Luc
VIP Member
Posts: 814
Joined: 16 Oct 2001 23:01

Unread post by Tonton Luc » 6 Oct 2009 14:33

Hi,
Your code works fine but the problem is not resolved.
Maybe something staying in the temporary internet files (cache).
:cry:

User avatar
Tonton Luc
VIP Member
Posts: 814
Joined: 16 Oct 2001 23:01

Unread post by Tonton Luc » 7 Oct 2009 7:50

I've found a solution : just need to use INTERNET_FLAG_NO_CACHE_WRITE flag with internetOpenUrl, and each request return differents results.

User avatar
Thomas Linder Puls
VIP Member
Posts: 2448
Joined: 28 Feb 2000 0:01

Unread post by Thomas Linder Puls » 7 Oct 2009 7:54

:P
Regards Thomas Linder Puls
PDC

User avatar
Thomas Linder Puls
VIP Member
Posts: 2448
Joined: 28 Feb 2000 0:01

Unread post by Thomas Linder Puls » 7 Oct 2009 7:57

By the way, I have realized that the purpose of the resize into Temp2 is to make the binary smaller in case it wasn't filled completely. (I still prefer my own version :wink: .)
Regards Thomas Linder Puls
PDC

User avatar
Tonton Luc
VIP Member
Posts: 814
Joined: 16 Oct 2001 23:01

Unread post by Tonton Luc » 8 Oct 2009 8:42

Hi,
Other question : when I use

Code: Select all

send_response_header("200 OK", "Content-Type: text/html; charset=utf-8\r\n\r\n", false)
in an Isapi dLL (in a web server) and

Code: Select all

trap(TxtHTML = getURLContentAsText(URL),_,fail())

Code: Select all

predicates     getURLContentAsText: (string URL) -> string procedure (i). clauses     getURLContentAsText(URL)= Res :-       Bin2 = getURLContentAsBin(URL),       Text = uncheckedConvert(string8,Bin2),       Res  = string8::mapToString(Text)       .

Post Reply