I would like to read streaming binary data from a web site (using C#)
as soon as it is available for reading. I do not want to wait till all
the data has arrived. Can someone help me with the commands needed to
achieve this?
Can't plain WebClient and HttpWebRequest/HttpWebResponse do that?
Arne
There is a callback function in BeginGetResponse, but it seemed that
it doesn't get invoked when the website is sending data continuously.
It waits for the data to be retrieved completely. There is a property
in SilverLight version, but there is probably a way to do this without
using SilverLight.
And if you do it sync and get the response stream object and
start reading from that?
Arne
You mean GetResponse()? I think I tried that (I can try again to be
sure). It just gets stuck. Doesn't the website need to finish sending
all the data before GetResponse returns the value? What if the website
never stops sending? That is the scenario I am referring to.
I just tested with:
WebClient wc = new WebClient();
StreamReader sr = new
StreamReader(wc.OpenRead("http://arnepc3/loop"));
string line;
while((line = sr.ReadLine()) != null)
{
Console.WriteLine(line);
}
And it gets a line per second (loop outputs and flushes
a line per second).
Arne
[...]
I am new to these technologies. Your approach seems interesting. Do
you think with your approach there is a way to post the data and then
read the binary data?
AFAIK, WebClient does not support POST requests directly. It is intended
for simpler types of queries.
However, you can use the WebRequest class directly. It is not much more
complicated, and does allow for POST requests. The data is still
available as soon as it's received by the client; contrary to the
assumption you alluded to in a previous post, it is not true that all of
the data must have arrived before any of it is available in the stream.
See below for a concise-but-complete code example that illustrates the
use of both WebClient and WebRequest for the purpose of receiving
streamed data. (Note that the code is for illustration purposes only,
containing no real error handling and recovery, nor other niceties one
might find in a real program).
I think it's possible that you could sub-class WebClient, overriding the
protected GetWebRequest() method so that you can convert the request to
the necessary POST request you want. But IMHO it's easier to just use
WebRequest directly at that point, rather than to try to get WebClient
to do what you want.
Pete
using System;
using System.Diagnostics;
using System.IO;
using System.Net;
namespace TestWebStreaming
{
class Program
{
static void Main(string[] args)
{
using (LoopingServer server = new LoopingServer())
{
WebRequest request =
WebRequest.Create("http://localhost:8888/");
using (WebResponse response = request.GetResponse())
{
_ReadResponseStream(response.GetResponseStream());
}
using (WebClient client = new WebClient())
{
_ReadResponseStream(client.OpenRead("http://localhost:8888/"));
}
}
}
private static void _ReadResponseStream(Stream stream)
{
using (StreamReader reader = new StreamReader(stream))
{
Debug.WriteLine("connected to server");
string strLine;
while ((strLine = reader.ReadLine())!= null)
{
Console.WriteLine(strLine);
}
}
}
}
class LoopingServer : IDisposable
{
private HttpListener _listener;
public LoopingServer()
{
_listener = new HttpListener();
_listener.Prefixes.Add("http://localhost:8888/");
_listener.Start();
_listener.BeginGetContext(_Connected, _listener);
}
private void _Connected(IAsyncResult ar)
{
try
{
HttpListener listener = (HttpListener)ar.AsyncState;
HttpListenerContext context = listener.EndGetContext(ar);
StreamWriter writer = new
StreamWriter(context.Response.OutputStream);
Debug.WriteLine("connected client");
System.Timers.Timer timer = new System.Timers.Timer(1000);
int clines = 0;
timer.Elapsed += (sender, e) =>
{
if (clines++ < 10)
{
string strLine= DateTime.Now.ToString();
Debug.WriteLine("sending line \"{0}\"",
(object)strLine);
writer.WriteLine(strLine);
writer.Flush();
}
else
{
Debug.WriteLine("closing client");
timer.Stop();
writer.Dispose();
}
};
timer.Start();
listener.BeginGetContext(_Connected, listener);
}
catch (Exception exc)
{
Debug.WriteLine("ERROR: {0}: \"{1}\"",
exc.GetType().Name, exc.Message);
}
}
public void Dispose()
{
if (_listener != null)
{
_listener.Close();
_listener = null;
}
}
}
}
On 3/1/11 12:05 PM, eagle123 wrote:[...]
I am new to these technologies. Your approach seems interesting. Do
you think with your approach there is a way to post the data and then
read the binary data?AFAIK, WebClient does not support POST requests directly. It is intended
for simpler types of queries.However, you can use the WebRequest class directly. It is not much more
complicated, and does allow for POST requests. The data is still
available as soon as it's received by the client; contrary to the
assumption you alluded to in a previous post, it is not true that all of
the data must have arrived before any of it is available in the stream.See below for a concise-but-complete code example that illustrates the
use of both WebClient and WebRequest for the purpose of receiving
streamed data. (Note that the code is for illustration purposes only,
containing no real error handling and recovery, nor other niceties one
might find in a real program).I think it's possible that you could sub-class WebClient, overriding the
protected GetWebRequest() method so that you can convert the request to
the necessary POST request you want. But IMHO it's easier to just use
WebRequest directly at that point, rather than to try to get WebClient
to do what you want.
using System;
using System.Diagnostics;
using System.IO;
using System.Net;namespace TestWebStreaming
{
class Program
{
static void Main(string[] args)
{
using (LoopingServer server = new LoopingServer())
{
WebRequest request =
WebRequest.Create("http://localhost:8888/");
using (WebResponse response = request.GetResponse())
{
_ReadResponseStream(response..GetResponseStream());
}using (WebClient client = new WebClient())
{private static void _ReadResponseStream(Stream stream)
{
using (StreamReader reader = new StreamReader(stream))
{
Debug.WriteLine("connected to server");string strLine;while ((strLine = reader.ReadLine()) != null)
{
Console.WriteLine(strLine);
}
}
}
}class LoopingServer : IDisposable
{
private HttpListener _listener;public LoopingServer()
{
_listener = new HttpListener();
_listener.Prefixes.Add("http://localhost:8888/");
_listener.Start();
_listener.BeginGetContext(_Connected, _listener);
}private void _Connected(IAsyncResult ar)
{
try
{
HttpListener listener = (HttpListener)ar.AsyncState;
HttpListenerContext context = listener.EndGetContext(ar);
StreamWriter writer = new
StreamWriter(context.Response.OutputStream);Debug.WriteLine("connected client");System.Timers.Timer timer = new System.Timers.Timer(1000);
int clines = 0;timer.Elapsed += (sender, e) =>
{
if (clines++ < 10)
{
string strLine = DateTime.Now.ToString();Debug.WriteLine("sending line \"{0}\"",
(object)strLine);
writer.WriteLine(strLine);
writer.Flush();
}
else
{
Debug.WriteLine("closing client");
timer.Stop();
writer.Dispose();
}
};
timer.Start();listener.BeginGetContext(_Connected,listener);
}
catch (Exception exc)
{
Debug.WriteLine("ERROR: {0}: \"{1}\"",
exc.GetType().Name, exc.Message);
}
}public void Dispose()
{
if (_listener != null)
{
_listener.Close();
_listener = null;
}
}
}
Thanks Pete for taking time to provide the explanation and the code.
Much appreciated.
I will experiment with it and will provide feedback.
On 3/1/11 12:05 PM, eagle123 wrote:
[...]
I am new to these technologies. Your approach seems interesting. Do
you think with your approach there is a way to post the data and then
read the binary data?AFAIK, WebClient does not support POST requests directly. It is intended
for simpler types of queries.However, you can use the WebRequest class directly. It is not much more
complicated, and does allow for POST requests. The data is still
available as soon as it's received by the client; contrary to the
assumption you alluded to in a previous post, it is not true that all of
the data must have arrived before any of it is available in the stream.See below for a concise-but-complete code example that illustrates the
use of both WebClient and WebRequest for the purpose of receiving
streamed data. (Note that the code is for illustration purposes only,
containing no real error handling and recovery, nor other niceties one
might find in a real program).....I think it's possible that you could sub-class WebClient, overriding the
protected GetWebRequest() method so that you can convert the request to
the necessary POST request you want. But IMHO it's easier to just use
WebRequest directly at that point, rather than to try to get WebClient
to do what you want.
Thanks Pete for taking time to provide the explanation and the code.
Much appreciated.
I will experiment with it and will provide feedback.
In my case, GetResponse times out. It times out at the following line.
HttpWebResponse response2 =
(HttpWebResponse)myHttpWebRequest2.GetResponse();
Exception details are copied below.
System.Net.WebException was unhandled
Message=The operation has timed out
Source=System
StackTrace:
at System.Net.HttpWebRequest.GetResponse()
at Test.Main(String[] args) in d:\personal\10003328\visual
studio 2010\Projects\Sync4\Sync4\Program.cs:line 147
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly,
String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile,
Evidence assemblySecurity, String[] args)
at
Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object
state) ....
Does it have to do anything with the fact that the data I am supposed
to getting is binary?
No.
What makes GetResponse() to return?
I noticed that in your code, you are using Writeline. It would put a
new line character. What if the data I am supposed to receive has no
new line characters?
What if a large volume of binary data is being sent that has no new
line characters?
On 3/1/11 12:05 PM, eagle123 wrote:
[...]
I am new to these technologies. Your approach seems interesting. Do
you think with your approach there is a way to post the data and then
read the binary data?
AFAIK, WebClient does not support POST requests directly. It is intended
for simpler types of queries.
However, you can use the WebRequest class directly. It is not muchmore
complicated, and does allow for POST requests. The data is still
available as soon as it's received by the client; contrary to the
assumption you alluded to in a previous post, it is not true that allof
the data must have arrived before any of it is available in the stream.
See below for a concise-but-complete code example that illustrates the
use of both WebClient and WebRequest for the purpose of receiving
streamed data. (Note that the code is for illustration purposes only,
containing no real error handling and recovery, nor other niceties one
might find in a real program).
I think it's possible that you could sub-class WebClient, overriding the
protected GetWebRequest() method so that you can convert the request to
the necessary POST request you want. But IMHO it's easier to just use
WebRequest directly at that point, rather than to try to get WebClient
to do what you want. ...
Thanks Pete for taking time to provide the explanation and the code.
Much appreciated.
I will experiment with it and will provide feedback.In my case, GetResponse times out. It times out at the following line.HttpWebResponse response2 =
(HttpWebResponse)myHttpWebRequest2.GetResponse();Exception details are copied below.System.Net.WebException was unhandled
Message=The operation has timed out
Source=System
StackTrace:
at System.Net.HttpWebRequest.GetResponse()
at Test.Main(String[] args) in d:\personal\10003328\visual
studio 2010\Projects\Sync4\Sync4\Program.cs:line 147
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly,
String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile,
Evidence assemblySecurity, String[] args)
at
Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object
state) ...
Does it have to do anything with the fact that the data I am supposed
to getting is binary?
No.
What makes GetResponse() to return?
It starts receiving HTTP response body.
I noticed that in your code, you are using Writeline. It would put a
new line character. What if the data I am supposed to receive has no
new line characters?
Should not change the logic (besides the obvious that you need to
read binary instead of lines).
What if a large volume of binary data is being sent that has no new
line characters?
It should not be a problem.
If (and this is the important part) the server actually start
sending the data before the data are completed.
It is very common for web servers and various server side
scripting to wait start sending data until all data are
produced.
(this allows setting Content-Length header and similar)
So are you sure that the server actually starts
sending data?
Arne
On 3/1/11 12:05 PM, eagle123 wrote:
[...]
I am new to these technologies. Your approach seems interesting. Do
you think with your approach there is a way to post the data and then
read the binary data?AFAIK, WebClient does not support POST requests directly. It is intended
for simpler types of queries.However, you can use the WebRequest class directly. It is not much more
complicated, and does allow for POST requests. The data is still
available as soon as it's received by the client; contrary to the
assumption you alluded to in a previous post, it is not true that all of
the data must have arrived before any of it is available in the stream.See below for a concise-but-complete code example that illustrates the
use of both WebClient and WebRequest for the purpose of receiving
streamed data. (Note that the code is for illustration purposes only,
containing no real error handling and recovery, nor other niceties one
might find in a real program).I think it's possible that you could sub-class WebClient, overriding the
protected GetWebRequest() method so that you can convert the request to
the necessary POST request you want. But IMHO it's easier to just use
WebRequest directly at that point, rather than to try to get WebClient
to do what you want. ...
Thanks Pete for taking time to provide the explanation and the code.
Much appreciated.
I will experiment with it and will provide feedback.In my case, GetResponse times out. It times out at the following line.HttpWebResponse response2 =
(HttpWebResponse)myHttpWebRequest2.GetResponse();Exception details are copied below.System.Net.WebException was unhandled
Message=The operation has timed out
Source=System
StackTrace:
at System.Net.HttpWebRequest.GetResponse()
at Test.Main(String[] args) in d:\personal\10003328\visual
studio 2010\Projects\Sync4\Sync4\Program.cs:line 147
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly,
String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile,
Evidence assemblySecurity, String[] args)
at
Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object
state) ...
Does it have to do anything with the fact that the data I am supposed
to getting is binary?
No.
What makes GetResponse() to return?
It starts receiving HTTP response body.
I noticed that in your code, you are using Writeline. It would put a
new line character. What if the data I am supposed to receive has no
new line characters?
Should not change the logic (besides the obvious that you need to
read binary instead of lines).
What if a large volume of binary data is being sent that has no new
line characters?
It should not be a problem.
If (and this is the important part) the server actually start
sending the data before the data are completed.
It is very common for web servers and various server side
scripting to wait start sending data until all data are
produced.
(this allows setting Content-Length header and similar)
So are you sure that the server actually starts
sending data?
Thanks again Arne for the quick response. You guys are really good.
There are two modes. I am getting the response back in ‘One-Time’
mode. I am getting time-out error in ‘Continuous’ mode. I increased
the timeout to 10 minutes and still I get time-out error. As the data
is supposed to be sent continuously, web server can not be waiting
till all data is completed. I should be getting the response back in
seconds.
You have a very good point though. May be for some reason, web server
is just not sending the response back in ‘Continuous’ mode.
I am going to check with the web site owner to see if they have an
explanation.
On 02-03-2011 14:41, eagle123 wrote:
On 3/1/11 12:05 PM, eagle123 wrote:
[...]
I am new to these technologies. Your approach seems interesting. Do
you think with your approach there is a way to post the data and then
read the binary data?
AFAIK, WebClient does not support POST requests directly. It is intended
for simpler types of queries.
However, you can use the WebRequest class directly. It is not much more
complicated, and does allow for POST requests. The data is still
available as soon as it's received by the client; contrary to the
assumption you alluded to in a previous post, it is not true that all of
the data must have arrived before any of it is available in the stream.
See below for a concise-but-complete code example that illustrates the
use of both WebClient and WebRequest for the purpose of receiving
streamed data. (Note that the code is for illustration purposes only,
containing no real error handling and recovery, nor other niceties one
might find in a real program).
I think it's possible that you could sub-class WebClient, overriding the
protected GetWebRequest() method so that you can convert the request to
the necessary POST request you want. But IMHO it's easier to just use
WebRequest directly at that point, rather than to try to get WebClient
to do what you want.
...
Thanks Pete for taking time to provide the explanation and the code.
Much appreciated.
I will experiment with it and will provide feedback.
In my case, GetResponse times out. It times out at the following line..
HttpWebResponse response2 =
(HttpWebResponse)myHttpWebRequest2.GetResponse();
Exception details are copied below.
System.Net.WebException was unhandled
Message=The operation has timed out
Source=System
StackTrace:
at System.Net.HttpWebRequest.GetResponse()
at Test.Main(String[] args) in d:\personal\10003328\visual
studio 2010\Projects\Sync4\Sync4\Program.cs:line 147
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly,
String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile,
Evidence assemblySecurity, String[] args)
at
Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object
state)
...
Does it have to do anything with the fact that the data I am supposed
to getting is binary?
No.
What makes GetResponse() to return?
It starts receiving HTTP response body.
I noticed that in your code, you are using Writeline. It would put a
new line character. What if the data I am supposed to receive has no
new line characters?
Should not change the logic (besides the obvious that you need to
read binary instead of lines).
What if a large volume of binary data is being sent that has no new
line characters?
It should not be a problem.
If (and this is the important part) the server actually start
sending the data before the data are completed.
It is very common for web servers and various server side
scripting to wait start sending data until all data are
produced.
(this allows setting Content-Length header and similar)
So are you sure that the server actually starts
sending data?Thanks again Arne for the quick response. You guys are really good.There are two modes. I am getting the response back in One-Time
mode. I am getting time-out error in Continuous mode. I increased
the timeout to 10 minutes and still I get time-out error. As the data
is supposed to be sent continuously, web server can not be waiting
till all data is completed. I should be getting the response back in
seconds.You have a very good point though. May be for some reason, web server
is just not sending the response back in Continuous mode.I am going to check with the web site owner to see if they have an
explanation.
You can try testing with a browser.
For POST with values just make a local form that submit
to the correct URL.
If the browser does not get a reply, then the program will
not either.
Arne
Want to reply to this thread or ask your own question?
You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.