Opening up in RichTextBox2 ?

[Keith (Southend)G]

No problem, I like doing stuff like this.

On the long delay for loading the file, I had a similar experience today
when testing the program, but only the first time I read the file. Every
time I tried to read it after that, it read it very quickly. Since I
couldn't duplicate it, I wasn't able to track it down. It's possible that
the program is loading a library into memory, or the O/S is building a list
of all the files in the folder, or something. I'm pretty sure the delay is
happening outside the program, but I'm not sure where. If it is happening
outside the program, your form probably won't respond until eith .NET or the
O/S is done doing whatever it needs to do. My guess is that you'll see a bit
of a wait, then all of a sudden you'll see the text box filling with data,
fairly quickly. If your experience is like mine, if you close the program
and start it up again, it will read the file almost immediately the second
time.

I went ahead and changed the program so that the data class raises an event
every time it finishes reading a line of text, and also right before it
creates an output file. To do that, I had to move the code that does the
actual reading of the data from the New sub into a separate one, and callit
separately. The events weren't being processed inside the constructor for
the class. I changed the form so that it intercepts the events and populates
the text boxes from them, rather than looping through the results later. In
the event handlers, I called a method named DoEvents, which makes sure that
the textbox updates are visible immediately, and lets the rest of the form
respond to movement, if needed. I also added to the program to have it
disable the buttons and change the cursor to the "Wait" mode (usually an
hourglass) while it's reading; that way, you know it's working, and you know
when it's done. The updated code is here:

http://www.CaptainJack3s.com/Code/ogprog4.txt

As for grabbing the data off the web, that may take a bit. I've done thata
few times before, using the WebBrowser control. I'm not certain what we need
to do, and it may take some playing with. I'll take a look at it, but it's
the kind of thing I probably won't get to until the weekend, and it may take
a couple of weekends of playing with it to get it right. The WebBrowser
control raises events every time it gets a web page, and if the page is
sending multiple bits (which will happen a lot with ASP pages and sometimes
with PHP) there can be a lot of event occurences to sift through before you
get the data you want. I'm working on an animation right now (one of my
other hobbies) which will fill up my nights this week, but I should have
some time either Saturday or Sunday (whichever day my team isn't playing....
go Colts and all that <G>). I'll try to create a component that goes to the
web site and grabs some data, then we can look at how to track date ranges,
countries read, and so on.

http://www.captainjack3d.com/Code/ogprog4.txt

Ah, found it

 

[Keith (Southend)G]

Sorry, I'm typing with one eye closed (cataract surgery for me in a couple
of weeks, and I'll be as right as rain). That should have been 3d, not 3s,
as in:

http://www.CaptainJack3d.com/Code/ogprog4.txt

Man, my whole day has been like that, one little thing after another.
Probably should have stayed in bed... :-D

No problem

Glad to hear you enjoy this stuff.
Running the new version now, the text appears straight away when I
click the first button and the hour glass is showing with the both
buttons grayed out. Still 5 minutes+, the 2nd bit is almost instant.
Out of interest how much RAM do you have on the computer you're using?

Animation is very a very rewarding pastime, you must have lots of
patience. For many years I was on the committee of out local Video
Club, before that it was Cine (Super 8), I was one for making holiday
films (good for people who can't sleep!), but we had one member George
Daniels, who's been dead for some years, that made super animations. I
imagine it's all computer based now?

Keith (Southend)

 

[Keith (Southend)G]

Glad to hear you enjoy this stuff.
Running the new version now, the text appears straight away when I
click the first button and the hour glass is showing with the both
buttons grayed out. Still 5 minutes+, the 2nd bit is almost instant.
Out of interest how much RAM do you have on the computer you're using?

Jack,

Not sure if there's anything here that means more to you with regards
to the time it takes to append the ogimet.txt to RichTextBox1. I tried
an even larger file (18 hours worth), which took 11 minutes. The 2nd
bit again was almost instant to run. Even so, I can just go and make
myself a cup of tea and relax for a few minutes.

http://www.eggheadcafe.com/software/aspnet/31952804/fastest-way-to-append-tex.aspx

<snip>
' How about 10,000 lines, 5 seconds:

Private Sub Button1_Click(ByVal sender As System.Object, _
ByVal e As System.EventArgs) Handles Button1.Click
Dim n As Integer
Dim s As String = "Hello, World, how is it going these days, " _
+ "is this quick enough for you?" + vbCrLf

Dim sb As New System.Text.StringBuilder

Dim t As Date = Now

With Me.TextBox1
For n = 1 To 10000
sb.Append(n.ToString("0000") + " ")
sb.Append(s)
If n Mod 500 = 0 Then
..Text = sb.ToString
..SelectionStart = .Text.Length
..SelectedText = " "
Me.Text = n.ToString
My.Application.DoEvents()
End If
Next n
..Text = sb.ToString
..SelectionStart = .Text.Length

<snip>

ps: I hope the Indianapolis Colts play well against the N Y Jets
tomorrow (Sunday).

Keith (Southend)

 

[Keith (Southend)G]

I don't think that would help with the problem we're having. That code
sample is for a quick way to build a really big string from other string
variables (and the StringBuilder is definitely the way to go for that). I'm
pretty sure the delay we're getting is happening somwhere when the
StreamReader is being created, which is a kettle of fish of a different
color.




They did indeed do well, although they got off to a rocky start. They won
the game, which sends them to the Big Game in two weeks. There was much
cheering (and too much consumption of potato chips, I regret to say) in my
household.

I set up a WebBrowser control to point to the data site on Saturday
afternoon, and I was able to load the page in a program. I need to do some
more work on it... I want to build the program so you can set up a list of
locations to pull data from, have it record the last date and time each
location was run for, and automatically download the data for each batch in
turn. The idea would be an application that would automatically get all of
the data you want since the last time you ran it, filter and sort the data
automatically, save its state, then shut down. My hope is that we'll have a
progam that you can run once a day (or as often as you like) and not have to
pay any attention to. From there, we could even set the program to go to
sleep every 24 hours, wake up and run, then go back to sleep until the next
day. If you start it when your computer boots up, you'd never have to mess
with collecting data again. At least, until they change the web site, which
some places have an annoying habit of doing.

I'll try to work on the project over the coming week and see what I can do.
If I get a chance, I'll try to isolate that start up issue with the inbound
file. If it is in the opening of the file, I'll see if there's another
reading option. If it's just slow on the reading because it's not buffering,
I can change to read in large blocks, and peel the lines off in memory (I've
done that with a lot of other unbuffered systems and it works well).

What version of VB are you using? I've got both 2005 and 2008 here. I'll
probably have more than one form before we're done, and it'll be easier if I
send you an entire solution zipped up, rather than just some code in text.

Hi Jack,

Good to hear the Colts won, when I looked they were trailing
behind. You had some rain in Minneapolis, I guess that's why they
put the roof over, but not as cold as it's been !

Sounds like you've given a lot of thought to the WebBrowser control
for downloading that data. I'm quietly confident that the website I
get the data from is unlikely to change it's address as I have had
emails with the author in the past. In fact the address with
"www1.ogimet...." he gave me as this presides on the University of
Granada (Spain) website, the version without the '1' is on his home
computer and regularily gets overloaded. So there are two variants of
the same website:
http://www1.ogimet.com/synopsc.phtml.en (best one to use)
http://www.ogimet.com/synopsc.phtml.en
I like the automatic running idea, certainly useful if I have to work
away or go on holiday, always a nightmare catching up, I must admit I
grab and use the data at different times of the day, although
generally evenings. What if my internet went down, as it's on 24/7,
would it just pick up from where it left? As long as I can set dates/
times manually if I want to I don't see a problem. A 24 hour download
file would be about 3.5Meg.

I have installed 'Visual Basic 2008 Express', it came with a book I
bought "Build a Program Now." http://www.microsoft.com/learning/en/us/book.aspx?ID=12282&locale=en-us
Ironically, the main exercise is for a weather reporting programme

Thanks again.

Keith (Southend)

 

[Keith (Southend)G]

What I was thinking of would be a table that would be kept in a text fileon
your computer that would list each of the locations that get looked up, and
what the last date and time was for each one. When the program runs, it will
grab anthing after that date and time for each location up to the current
date and time, so that it would always be getting whatever hadn't been
retrieved before, regardless of how long it had been run. That would also
allow for adding new locations, or catching up if data was not available for
a certain location or if the download gets interrupted somehow.

There'll probably be a need to set up a couple of folders for working space
on the system, too, so the program has a safe place to store temporary
files, and a permanent place to store its logs. I'm thinking of making a
little window that will allow doing that kind of set up, although it would
be pretty easy to just put it into a text file and edit it by hand, too.

I think we could build in some monitoring for the WebBrowser that would
check the preferred web site, and only try the alternate site if the primary
site times out more than a specified number of times. Come to think of it,
we'll probably need a screen that shows the log of recent activity, so that
we'll know if no data is being grabbed.

That sounds all to good to be true Jack, I think you're ahead of me
now ;-)

The text file list sounds excellent, as you say you add more locations
if required.
Thought I'd mention this, on my C:/ drive I currently have a folder
called 'SynopData' and under that 'Formatted', 'Raw' & 'StationLists',
basically the formatted folder would be the place for the newly
created hourly files. Any text files, exe file etc, could possibly sit
in 'SynopData'.

Keith (Southend)

 

[Keith (Southend)G]

What I was thinking of would be a table that would be kept in a text fileon
your computer that would list each of the locations that get looked up, and
what the last date and time was for each one. When the program runs, it will
grab anything after that date and time for each location up to the current
date and time, so that it would always be getting whatever hadn't been
retrieved before, regardless of how long it had been run. That would also
allow for adding new locations, or catching up if data was not available for
a certain location or if the download gets interrupted somehow.

One thing worth mentioning, the way the data on that site updates
means that the full complement of data is normally 6 hours behind.
eg, data up to say 14:00, won't all be in until 20:00, so I grab the
data up to 14:00 and pick the rest the next day and so on. It updates
every 6 hours as a rule, but you will find data there, but not all
prior to this time, so I never actually go as far as the current time.
That's probably as clear as mud
It just dawned on me as I was thinking about the automation of the
programme.

Keith (Southend)

 

[Keith (Southend)G]

That shouldn't be a problem, we can just add an offset to the automation
that can be specified in the set up. To calculate the target time, the
program would just have to subtract the offset from the current time before
requesting the data.

Come to think of it, we might want a maximum time to download data, too. If
for some reason the program doesn't get run for a week, it might be better
to restrict a given download to a manageable size, like 24 or 48 hours. That
would keep the program from downloading too much data at one time, but still
let it catch up.

That sounds about the right balance Jack, I'll be lurking in
anticipation

Many thanks

Keith (Southend)