HttpWebRequest connecting to tivo webserver

[Chris Becker]

I am trying to write a little app that will parse my Tivo's Now Playing
xml doc using ASP.NET (or winforms, i just want something to work at
this point).

Below is the code that I am using. What happens is that I get 401:
Unauthorized when I call req.GetResponse(). But using the same url,
username, password and accepting the cert manually in IE or firefox, I
am able to access the url.

Any insight as to what is happening would be greatly appreciated. I am
thinking that this is a username/password problem because of the 401:
Unauthorized response, but maybe it is a certificate problem? I
haven't found too many examples out there on handling ssl
request/responses using HttpWebRequest or WebClient.

Imports System.xml
Imports System.Net
Imports System.Security.Cryptography.X509Certificates
Public Class WebForm1
Inherits System.Web.UI.Page

Private Sub Page_Load(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles MyBase.Load

'Supposedly, the following should accept all SSL certificates.
'Might not be safe and secure, but I wanted to get a working copy
first
ServicePointManager.CertificatePolicy = New
AcceptAllCertificatePolicy()

Const IP As String = "192.168.1.100"
Dim url As String = "https://" + IP +
"/TiVoConnect?Command=QueryContainer&Container=%2FNowPlaying&Recurse=Yes"

Dim creds As New CredentialCache
'BEGIN QUESTION: Is the following URI correct?
creds.Add(New Uri("https://" + IP + "/"), "Basic", New
NetworkCredential("tivo", "MyMAKNumberRemoved"))
'END QUESTION

Dim req As HttpWebRequest
req = HttpWebRequest.Create(url)
req.ConnectionGroupName = Guid.NewGuid.ToString 'Added this
just to make sure that my various tests weren't getting cached
req.KeepAlive = False

'BEGIN QUESTION: I have tried the following together and both without
the other
req.Headers.Add("Authorization", "Basic " +
Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes("tivo:MyMAKNumberRemoved")))
req.Credentials = creds
'END QUESTION

Dim answer As HttpWebResponse
'c Here's where I get the 401:Unauthorized error
answer = req.GetResponse()

Dim reader As New XmlTextReader(url)
Dim xmltext As New System.Text.StringBuilder
Dim doc As New XmlDocument
doc.Load(reader)
Response.Clear()

Response.Write(doc.InnerText)
End Sub
End Class

'Accept all SSL certificates.
Friend Class AcceptAllCertificatePolicy
Implements ICertificatePolicy

Public Function CheckValidationResult(ByVal srvPoint As
ServicePoint, _
ByVal cert As X509Certificate, ByVal request As WebRequest, ByVal
problem As Integer) _
As Boolean Implements ICertificatePolicy.CheckValidationResult

Return True
End Function
End Class

 

[Joerg Jooss]

I am trying to write a little app that will parse my Tivo's Now
Playing xml doc using ASP.NET (or winforms, i just want something to
work at this point).

Below is the code that I am using. What happens is that I get 401:
Unauthorized when I call req.GetResponse(). But using the same url,
username, password and accepting the cert manually in IE or firefox, I
am able to access the url.

Any insight as to what is happening would be greatly appreciated. I
am thinking that this is a username/password problem because of the
401: Unauthorized response, but maybe it is a certificate problem? I
haven't found too many examples out there on handling ssl
request/responses using HttpWebRequest or WebClient.

[...]

Note that you shouldn't be implementing HTTP authentication by setting
headers yourself -- use a NetworkCredentials object and the
HttpWebRequest.Credentials property.

The question is how the TiVo web app implements authentication. Does
the browser challenge you to enter a username and password, or do you
enter the information in a web form?

Cheers,

 

[danieljroot]

No Tivo, so I can't test it, but your code looks like it should work.
A good tool to watch HTTP traffic on your pc is Fiddler. Run it, then
launch IE (I've had problems with FireFox), connect to the Tivo. Then
run your app and compare the requests...

 

[danieljroot]

http://www.fiddlertool.com/fiddler/

 

[Chris Becker]

Tivo implements authentication with a challenge... my browser prompts
me. It does not use a web form.

I have tried both ways of specifying credentials. You can see in the
code the following:

creds.Add(New Uri("https://" + IP + "/"), "Basic", New
NetworkCredential("tivo", "MyMAKNumberRemoved"))
....
req.Credentials = creds

I did try this scenario without also adding the header manually and it
still did not work. Maybe my uri in the creds.Add() is not correct:
https://IP/? What should that be?

I will also try the fiddler debugging route again, but I wasn't able to
glean any info from that last night (I never saw my creds go across the
wire).

 

[Chris Becker]

So here is what fiddler captures from the the initial request from IE:

CONNECT 192.168.1.102:443 HTTP/1.0
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;
..NET CLR 1.0.3705; .NET CLR 1.1.4322)
Host: 192.168.1.102
Content-Length: 0
Proxy-Connection: Keep-Alive
Pragma: no-cache

The data sent represents a SSL ClientHello Message. For your
convenience, the data is extracted below.

Major Version: 3
Minor Version: 0
Random: 6B E3 E7 A9 45 61 61 77 9D 61 06 1D 41 2C 6A BE
SessionID: empty
Ciphers:
<Bunch of stuff cut out here>


I have modified my app to TRY to mimic some of the values, but none of
it gets set:

Dim req As HttpWebRequest
req = HttpWebRequest.Create(url)
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
'req.ContentLength = 0 'Exception: ContentLength cannot be set
for a non-write operation
req.Method = "GET"
req.ConnectionGroupName = Guid.NewGuid.ToString
req.KeepAlive = False

req.Credentials = creds

Here is what fiddler captures from my new modified app:

CONNECT 192.168.1.102:443 HTTP/1.1
Proxy-Connection: Keep-Alive
Host: 192.168.1.102:443

The data sent represents a SSL ClientHello Message. For your
convenience, the data is extracted below.

Major Version: 3
Minor Version: 0
Random: 42 DE AA AB 21 10 ED 58 A3 A4 31 7E 17 98 76 78 82 74 04 87 BD
D9 DE BE 96 A5 F2 91 0F 24 ED 13
SessionID: empty
Ciphers:
<Bunch of stuff cut out here>

Now what? Why aren't things like UserAgent, KeepAlive getting sent
across with my new code? I did notice that HTTP/1.0 is sent using IE
and HTTP/1.1 is sent using my app. How do I change that in light of
the problem with the properties that I've already set not being sent?

 

[Chris Becker]

I finally figured it out: Tivo uses Digest, not Basic, for the
credentials. So I changed this line:

creds.Add(New Uri("https://" + IP + "/"), "Basic", New
NetworkCredential("tivo", "MyMAKNumberRemoved"))

to:

creds.Add(New Uri("https://" + IP + "/"), "Digest", New
NetworkCredential("tivo", "MyMAKNumberRemoved"))

Now my question, for anyone who is experienced in Fiddler, is how could
I have determined this from my Fiddler outputs that I posted
previously? Or is it just guessing game?

 

[Joerg Jooss]

I finally figured it out: Tivo uses Digest, not Basic, for the
credentials. So I changed this line:

creds.Add(New Uri("https://" + IP + "/"), "Basic", New
NetworkCredential("tivo", "MyMAKNumberRemoved"))

to:

creds.Add(New Uri("https://" + IP + "/"), "Digest", New
NetworkCredential("tivo", "MyMAKNumberRemoved"))

Now my question, for anyone who is experienced in Fiddler, is how
could I have determined this from my Fiddler outputs that I posted
previously? Or is it just guessing game?

No. Normally you can see it in the WWW-Authenticate header that is sent
by the server to challenge the client for credentials.

You weren't able to see that because you've been using SSL. When using
SSL, proxies will just tunnel the HTTP traffic (that's what the CONNECT
method does).

Cheers,