HttpRequest - What am I doing wrong!?!?! Help

Hi,

I'm trying to log into a website directly from my application and then save it for
later. After hours of trying, I think I'm pretty spent and maybe I'm not seeing
something here.

I'm using Firefox's Live Http Headers to view what information needs to be
passed to this particular website. This is the content that is being sent to
the site:

(from live HTTP headers)
POST /control/selectclinic IsFirstTime=true&AccountNumber=XXXXX&MainZip=90025&selectclinicbutton.x=81&selectclinicbutton.y=5

So, here's my code:

bPageRet is boolean 
// Run a request on a secure server
bPageRet = HTTPRequest("[zoasis.com];,...
		   "IsFirstTime=true"+CR+"AccountNumber=XXXXX"+CR+"MainZip=90025"+CR+...
		   "selectclinicbutton.x=38"+CR+"selectclinicbutton.y=3"+CR+"submit")

// If an error occurs
IF bPageRet = False THEN
	// evaluate the error
	SWITCH ErrorInfo(errCode)
		// Invalid certificate name
		CASE HTTPErrorInvalidCertificateName : 
			//automatically ignore cert.
			HTTP.IgnoreError = HTTPIgnoreInvalidCertificateName
			// Re-run the request
			HTTPRequest("[zoasis.com];,...
			"IsFirstTime=true"+CR+"AccountNumber=XXXXX"+CR+"MainZip=90025"+CR+...
			"selectclinicbutton.x=38"+CR+"selectclinicbutton.y=3"+CR+"submit")

			EDT_Edit1 = HTTPGetResult()

		// Invalid or expired cert.	
              CASE HTTPErrorExpiredCertificate : 
		HTTP.IgnoreError = HTTPIgnoreExpiredCertificate
		      // Re-run the request 
			HTTPRequest("[zoasis.com];,...
			"IsFirstTime=true"+CR+"AccountNumber=XXXXX"+CR+"MainZip=90025"+CR+...
			"selectclinicbutton.x=38"+CR+"selectclinicbutton.y=3"+CR+"submit")
	END
END

Am i not executing the httpRequest command correctly? I'm pretty certain that
all fields have been convered.

TIA

Hi Georgio,

I think you need to "urlencode" the string you send.

PROCEDURE UrlEncode(_S)

sRes is string

//Possible : 32 45-46 48-57 65-90 95 97-122
bOk is boolean
N is int
C is int

FOR C = 1 TO Length(_S)
	
	N = Asc(_S[[C]])
	
	bOk = True
	SWITCH True  
		CASE (N>=45) AND (N<=46)
		CASE (N>=48) AND (N<=57)
		CASE (N>=65) AND (N<=90)
		CASE (N=95)
		CASE (N>=97) AND (N<=122)
		OTHER CASE
			bOk = False
	END 
	
	IF bOk THEN
		sRes += _S[[C]]
	ELSE
		IF (N=32) THEN
			sRes += "+"
		ELSE
			sRes += "%"+NumToString(N,"02X")
		END
	END
	
END

RESULT sRes

The function filters out spaces and encodes unsafe ascii characters.

HTH, regards,
Piet

p.s. I don't think you can use CR in a http request. Perhaps do several subsequent requests?




Edited 1 time(s). Last edit at 09/26/2008 11:32AM by Piet van Zanten.

Hi Piet,

MyURL is string = URLEncode("[my] web site/my page")

Already exists as a W-Language procedure and does good job (as far as I had to use it).

As for the CR+...+"submit" this is the documented syntax in the help to send form data.

Best regards.

Hi Georgio,

I looked at the web site, and I see no https but only http protocol. This will lead to permanent failure.

Also, selectclinicbutton is the sumbit button. You should not send information about that input field. The only field you must input are:
- IsFirstTime
- AccountNumber
- MainZip

Then submit that information.

Something like:

bPageRet = HTTPRequest("[zoasis.com]
"AccountNumber=XXXXX"+CR+"MainZip=90025"+CR+"submit")

Best regards.

Hi Alexandre,

Thanks for the tip. I used the above example exactly as you have layed out.

No go.

The website is giving me the error:

Missing required field(s): Hospital Account Number, Hospital Zip Code

The above fields are definetly in the command statement. Could this possibly
be a windev bug?

Thanks again!

Georgio

Hi Georgio,

I don't know because I have no data to test. But it could be. (The encoding can also be a problem.) This is charset=UTF-8 and in such a format the data should be sent back to the server with good encoding in the header.

You might want to check that. (There is a bug in SOAP with encoding... so try specifying the encoding in the HTTP header.)

Regards.

Hi Alexandre,

I'd be really appreciative of your help. The information I have is for
a demo account that the company gives to developers for test.

May I email it to you?

Thanks!
Georgio

Hi Georgio,

I also had quite a lot of trouble to get things working using HTTPRequest(). It is like working in the dark, because you don't get to see the actual message being sent by Windev.

I solved this with a simple little program called tcptrace from [www.pocketsoap.com]. You may want to have a look at it. It redirects all HTTP-traffic to some other port, and reports whatever comes along. Sort of like a proxy.

When I saw the actual message being sent by Windev, the solution to my problem was obvious: I forgot to append a CR/LF to the additional headers I was sending. So simple if you can actually SEE whatever Windev is sending.

The next step was to SEE what my browser is sending when I connect to the site interactively. There are many programs that can do that, some of them freeware (like tcptrace). Comparing the messages that I was sending to the ones sent by my web-browser, eventually resolved the remaining issues. After all, if my message is exactly the same as the message sent by my browser, it must be correct.

/sohan

Sohan,

So simple but so clever. I already have a screen-scrapper utility installed
and I'll try that now.

Thanks!
Georgio

Ok,

Now, I'm upset. I've run this every which way I could have. I have a screen
scrapper that shows me that NONE of my post data is being sent.

So, just to test, I tried this with 4 other websites. I even wrote a simple
cgi on my website with 1 field to post ... nothing.

Has anyone ever successfully used the HttpRequest with POST in
Windev 12?

Sorry, I've been at this for 3 days and nothing. Nothing from PC soft
either.

Georgio

Hi Georgio,

I'm afraid I don't understand what you mean by "screen scrapper". Data sent by HTTPRequest has nothing to do with the 'screen'. Did you try tcptrace? And yes, posting works fine in WD12.

/sohan

Hi Georgio,

When I use your HTTPRequest, this is what Windev sends:

POST / HTTP/1.1
Content-Length: 60
Content-Type: application/x-www-form-urlencoded
Accept:  */*
Host: localhost
User-Agent: 
Connection: close

IsFirstTime=true
AccountNumber=XXXXX
MainZip=90025
submit

(Ignore the "localhost", it is because I am using tcptrace). As you can see, the word "submit" is simply copied into the body of the posted data (below the empty line). I would be surprised if this is correct. The documentation to HTTPRequest states that all input fields should be on a seperate line, and appended with "submit". I wonder if this is correct, since the word "submit" does not seem to be interpreted by Windev.

But you can simply avoid this game of trial and error by looking at what your browser is sending. When I type [zoasis.com] in my browser, enter "xxxxx" as the ID Number and "90025" as Hospital Zip Code, this is what my browser sends:

POST /control/selectclinic HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/msword, application/x-silverlight, */*
Referer: [zoasis.com]
Accept-Language: nl
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
Host: zoasis.com
Content-Length: 98
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: JSESSIONID=LppZ1sDqzwyCGHP0JJ2f898db8295Y6nL4g1xPmqyx13q7ftFQJy!1632793846

IsFirstTime=false&AccountNumber=xxxxx&MainZip=90025&selectclinicbutton.x=37&selectclinicbutton.y=3

First thing to notice is that the posted data is on a single line and does not contain the word "submit" (surprise surprise). Next thing is the cookie. Your browser has picked up a cookie somewhere and sends it along with every request. Your HTTPRequest() is failing because you did not supply it. Don't let the HTML page that comes along with it confuse you. It says: "Missing required field(s): Hospital Account Number, Hospital Zip Code" etc, but that's BS. It means that the cookie is missing. The browser received the cookie on first entry of the site/page. Keep in mind that if you approach the site through a browser, the POST request is the second request, not the first one. The first request is a simple GET of the main url, in this case "[zoasis.com];. This is where the cookie is set. You will have to focus on the header received, not the (html) result. The first header received is:

HTTP/1.1 200 OK
Server: Sun-ONE-Web-Server/6.1
Date: Sun, 28 Sep 2008 01:32:09 GMT
Content-Type: text/html; charset=UTF8
Set-Cookie: JSESSIONID=LppZ1sDqzwyCGHP0JJ2f898db8295Y6nL4g1xPmqyx13q7ftFQJy!1632793846; path=/
Transfer-encoding: chunked

There is the cookie. As soon as you receive it, you must supply it with every request that you send. Next you can issue the POST request. Use the third parameter of HTTPRequest to send the cookie along with the request. Don't forget to append a CR to every 'additional HTTP header' that you send. When I do this I get the following response:

HTTP/1.1 200 OK
Server: Sun-ONE-Web-Server/6.1
Date: Sun, 28 Sep 2008 01:41:52 GMT
Content-Type: text/html; charset=UTF8
Set-Cookie: clinicId=null; expires=Thursday, 01-Jan-1970 01:00:00 GMT
Set-Cookie: clinicZip=null; expires=Thursday, 01-Jan-1970 01:00:00 GMT
Transfer-encoding: chunked

The resulting HTML is the same as I see in my browser, so the request has succeeded. This is as far as I can go with it, since I do not have a valid account-id.

You have to keep in mind that HTTPRequest is just an atomic command. It sends some data to the server (either a GET or a POST) and that's about it. Your browser does much more than that - receiving and sending cookies is one of these things To give you an idea: the browser issued 8 requests in total to service your POST request. Another common thing that your browser handles is redirection. The server responds with code 302 which means that the page requested is to be found elsewhere. This means that you will have to issue another request with the given location. In short: to do anything non-trivial, you will have to build a rudimentary http-client around HTTPRequest.

As I mentioned before, things become a lot more clear if you can actually see the HTTP-messages going to and from the server, especially the headers. To see what's going on inside my browser I purchased HTTPWatch. It's a plugin for your browser and reports exactly what's going on real-time. I mention this only to point out that they have an excellent http tutorial on their site: [www.httpwatch.com]. Most of my http knowledge comes from these pages.

Cheers,
/sohan

HI Sohan,

A screen scrapper extracts data from the display output of another program ... in most cases, a website. From there I can parse the data for my own use. The screen scrappe program I use has a proxy server attached so I can see how data is being sent to a program or website (even while using windev).

A good "real-world" example of a screen scrapping program would be a piece of software called "scroogle". This is actually a SS proxy that allows google searches without the google ads. With scroogle, I can see exactly what data is being sent to google's search field and parse the resulting data.

Thanks for your last example, this should help me greatly.

Georgio

Sohan,

Excellent post!!! I'm finally gaining some headway.

This thread should be in a "best of"; the help pages for the HttpRequest command
is somewhat elusive.

Thank you, Thank you!
Georgio

HTTPRequest("[zoasis.com]

Hi all,

I know this is a little late, however, google, download and install "WireShark" (aka Ethereal). This is the best thing since sliced bread when it comes to debugging network traffic.

Cheers!
ken

Hi Ken,

Yes, I L-O-V-E wireshark, however, I've never used it for windows ... only in Linux or Unix platforms/flavors.

Georgio