如何在安全的网站中使用webclient？-Howtousewebclientinasecuresite?

9  

One way would be through automating a browser -- you mentioned WebClient, so I'm guessing you might be referring to WebClient in .NET.

一种方法是通过自动化浏览器 - 你提到了WebClient,所以我猜你可能指的是.NET中的WebClient。

Two main points:

两个要点:

• There's nothing special about https related to WebClient - it just works

与WebClient相关的https没有什么特别之处 - 它只是起作用

• COOKIEs are typically used to carry authentication -- you'll need to capture and replay them

COOKIE通常用于进行身份验证 - 您需要捕获并重放它们

Here's the steps I'd follow:

这是我要遵循的步骤:

1. GET the login form, capture the the COOKIE in the response.

获取登录表单,捕获响应中的COOKIE。

2. Using Xpath and HtmlAgilityPack, find the "input type=hidden" field names and values.

使用Xpath和HtmlAgilityPack,找到“input type = hidden”字段名称和值。

3. POST to login form's action with user name, password, and hidden field values in the request body. Include the COOKIE in the request headers. Again, capture the COOKIE in the response.

使用用户名,密码和请求正文中的隐藏字段值POST登录表单的操作。在请求标头中包含COOKIE。再次,在响应中捕获COOKIE。

4. GET the pages you want, again, with the COOKIE in the request headers.

再次使用请求标头中的COOKIE获取所需的页面。

On step 2, I mention a somewhat complicated method for automating the login. Usually, you can post with username and password directly to the known login form action without getting the initial form or relaying the hidden fields. Some sites have form validation (different from field validation) on their forms which makes this method not work.

在第2步,我提到了一种有点复杂的自动登录方法。通常,您可以使用用户名和密码直接发布到已知的登录表单操作,而无需获取初始表单或中继隐藏字段。某些网站在其表单上进行了表单验证(与字段验证不同),这使得此方法无效。

HtmlAgilityPack is a .NET library that allows you to turn ill-formed html into an XmlDocument so you can XPath over it. Quite useful.

HtmlAgilityPack是一个.NET库,允许您将格式错误的HTML转换为XmlDocument,以便对其进行XPath。非常有用。

Finally, you may run into a situation where the form relies on client script to alter the form values before submitting. You may need to simulate this behavior.

最后,您可能会遇到这样一种情况,即表单依赖客户端脚本在提交之前更改表单值。您可能需要模拟此行为。

Using a tool to view the http traffic for this type of work is extremely helpful - I recommend ieHttpHeaders, Fiddler, or FireBug (net tab).

使用工具查看此类工作的http流量非常有用 - 我建议使用ieHttpHeaders,Fiddler或FireBug(网络标签)。

1  

You can easily simulate user input. You can submit form on the web page from you program by sending post\get request to a website.
Typical login form looks like:

您可以轻松模拟用户输入。您可以通过向网站发送post \ get请求从您的程序在网页上提交表单。典型的登录表单如下:

You can send a post request to the website providing values for Username & Password fields. What happens after you send your request is largely depends on a website, usually you will be redirected to some page. You authorization info will be stored in the sessions\COOKIE. So if you scrape client can maintain web session\understands COOKIEs you will be able to access protected pages.

您可以向网站发送帖子请求,提供用户名和密码字段的值。发送请求后会发生什么情况在很大程度上取决于网站,通常您会被重定向到某个页面。您的授权信息将存储在sessions \ COOKIE中。因此,如果你刮客户端可以维护网络会话\了解COOKIE,你将能够访问受保护的页面。

It's not clear from your question what language\framework you're going to use. For example there is a framework for screen scraping (including login functionality) written in perl - WWW::Mechanize

从您的问题中不清楚您将使用哪种语言\框架。例如,有一个用perl编写的屏幕抓取框架(包括登录功能) - WWW :: Mechanize

Note, that you can face some problems if site you're trying to login to uses java scripts or some kind of CAPTCHA.

请注意,如果您尝试登录的网站使用的是Java脚本或某种CAPTCHA,则可能会遇到一些问题。

0  

Can you please clarify? Is the WebClient class you speak of the one in HTTPUnit/Java?

你能澄清一下吗? WebClient类是否是HTTPUnit / Java中的一个?

If so, your session should be saved automatically.

如果是这样,您的会话应自动保存。

0  

It isn't clear from your question which WebClient class (or language) you are referring to.

从您的问题中不清楚您指的是哪个WebClient类(或语言)。

If have a Java Runtime you can use the Apache HttpClient class; here's an example I wrote using Groovy that accesses the delicious API over SSL:

如果有Java Runtime,您可以使用Apache HttpClient类;这是我使用Groovy编写的一个示例,它通过SSL访问美味的API:

   def client = new HttpClient()

   def credentials = new UsernamePasswordCredentials( "username", "password" )
   def authScope = new AuthScope("api.del.icio.us", 443, AuthScope.ANY_REALM)
   client.getState().setCredentials( authScope, credentials )

   def url = "https://api.del.icio.us/v1/posts/get"

   def method = new PostMethod( url )
   method.addParameter( "tag", tag )
   client.executeMethod( method )