How to prevent 404 error when a page is updated via ftp?

[Robm]

Hello,

In my FP2000 web I use a link to a htm file in a non-published subweb. This
file is auto-refreshed once a minute by means of the meta tag: <meta
http-equiv="Refresh" content="60">. The file is also overwritten once a
minute via ftp. The idea of course is to get a real-time data display. The
problem is that if the refresh comes while the file is being updated, a 404
error is produced. Any idea how to prevent or handle this?

 

[Net55]

The are many ways to handle that problem, the easiest is
to make a custom error page that refreshs to your
original page. this will create a loop until the page
loads.
custom error page:
<meta http-equiv="refresh"
content="5;URL=http://www.yourdomain.com/subweb/yourpage.h
tml">

 

[Jim Buyens]

I would avoid displaying the automatically FTP'ed
directly. Instead, I would write an ASP or ASP.NET page
that read the file from the server's file system and
transmitted it to the Web server.

This provides a way that you can trap the error and then
supply alternate content, or retry for a little while at
some frequency. For example, you could retry at 5, 10,
and 15 seconds following an initial failure.

It might also help for your FTP process to first upload
the new file to a temporary name, then delete or rename
the old file, then rename the new file. This shortens the
window when the file is unavailable.

Jim Buyens
Microsoft FrontPage MVP
http://www.interlacken.com
Author of:
*----------------------------------------------------
|\---------------------------------------------------
|| Microsoft Office FrontPage 2003 Inside Out
||---------------------------------------------------
|| Web Database Development Step by Step .NET Edition
|| Microsoft FrontPage Version 2002 Inside Out
|| Faster Smarter Beginning Programming
|| (All from Microsoft Press)
|/---------------------------------------------------
*----------------------------------------------------

 

[Robm]

Thanks. I already tried to reduce the time by ftp-ing to a .tmp file and
then removing the original and renaming the .tmp file. Even so I find that
the page is unavailable for too long a time. I had this working originally
as a Visual Interdev web site, with a database connection to the data
server, but my customer doesn't allow this any more. Why would the page be
unavailable for such a long time (multiples of tens of seconds)? The file
size is under 4K!

 

[Sparky Polastri]

Not that I know any better, but I think you are taking the overall wrong
approach to getting real-time display. All other displays I know of use a
database and a query, where something else is feeding the database on the
back end in "real time".

For your particular solution, you might try setting up a scheduled command
line script to move the files in place, but then upload to a temp folder.
Using the script, copy files from the temp folder and over write the ones
that are "live" using the OS... it'll be much faster.

Your data could be up to two minutes old of course.

But if it's really important that it be real time, you need to re-think your
solution.

 

[Jim Buyens]

Thanks. I already tried to reduce the time by ftp-ing to a .tmp file and
then removing the original and renaming the .tmp file. Even so I find that
the page is unavailable for too long a time. I had this working originally
as a Visual Interdev web site, with a database connection to the data
server, but my customer doesn't allow this any more. Why would the page be
unavailable for such a long time (multiples of tens of seconds)? The file
size is under 4K!

Sometimes IIS holds files open, to save time reopening them if
additional requests arrive.

This would lead me toward the solution of having an ASP or ASP.NET
page open and transmit the automatically-updated file, rather than
letting IIS deliver it directly. That way, you have more control over
file opening and closing.

How did the database connection play into this? Were you putting the
automatically-updated content in a database?

Jim Buyens
Microsoft FrontPage MVP
http://www.interlacken.com
Author of:
*----------------------------------------------------
|\---------------------------------------------------
|| Microsoft Office FrontPage 2003 Inside Out
||---------------------------------------------------
|| Web Database Development Step by Step .NET Edition
|| Microsoft FrontPage Version 2002 Inside Out
|| Faster Smarter Beginning Programming
|| (All from Microsoft Press)
|/---------------------------------------------------
*----------------------------------------------------

 

[Robm]

Jim, regarding your question "How did the database connection play into
this? Were you putting the automatically-updated content in a database?":

In the original design there an MS-Access database resided on the web
server, and the web site simply filled tables from that database, as
requested by the users. The data server, on the other side of the firewall,
would use temprary mapping and a database link to place new records in the
tables of the web server's database. This worked fine with one customer, who
wanted it this way.

The new design, for a customer who does not allow temporary drive mapping,
is much more restrictive, in that we only have fixed pages with data. These
pages are prepared in html format on the data server, and are then ftp-ed to
the web server. The problem, as reportes in this thread, is that the files
get locked up for considerable time, possibly due to virus checking,
possibly die to the server deciding to keep them locked, two suggestions
supplied by others in this thread.

I have no experience with ASP.NET and am worried about the learning curve. I
have lots of Visual Interdev ASP experience, and could use that for sure.
Thanks for the suggestions!

Rob M.

 

[Jim Buyens]

Can you describe the application a little more? For example:

o What kind of data is this?
o How often do updates occur?
o When updates occurs, how many files are affected?
o Are the updated files HTML, GIF, JPEG, or what?
o Why do the Web pages need to auto-refresh every 60 seconds?

Jim Buyens
Microsoft FrontPage MVP
http://www.interlacken.com
Author of:
*----------------------------------------------------
|\---------------------------------------------------
|| Microsoft Office FrontPage 2003 Inside Out
||---------------------------------------------------
|| Web Database Development Step by Step .NET Edition
|| Microsoft FrontPage Version 2002 Inside Out
|| Faster Smarter Beginning Programming
|| (All from Microsoft Press)
|/---------------------------------------------------
*---------------------------------------------------

 

[Robm]

Jim, the data is 1-minute averages and hourly average values for about 10
parameters in an air quality monitoring application. So the source data
updates once a minute. The data server formats this data in a single html
file and uses ftp to send it to the web server, also once a minute. The page
autorefreshes so that the user can just have the page open and see the most
current data at all times.

I am now working on a scheme where the ftp-ed files are read by a separate
process which places the data in tables of an MS-Access database. This
database is then used by the Visual Interdev website to display the data. So
I am using the original Visual Interdev web design but I am replacing the
direct database links between data server and web server by ftp-ed files. I
haven't completed this yet, but I believe that it should work fine.

Rob M

 

[Jim Buyens]

Too bad you're not into .NET -- this would be a great application for
a Web service. The Web service would run on the Web server and receive
update requests from a client program on the data server.

Using only ASP, you can get much the same effect by writing an ASP
page that receives updates via an HTML form or a request body
formatted as XML. However, you wouldn't submit the HTML form input
from a browser; you would write a program that runs on the dataserver
and emulates a brower. There are methods for doing this without too
much work.

Another possibility is a program named "curl". This program makes HTTP
requests from the command line, and is available for a wide variety of
operating systems. Therefore, on the data server, you could run a
command like

curl http://my.server.com/updateparms.asp?p1=123&p2=456&p3=789

and invoke an updateparms.asp page on your Web server that updated the
database. Of course, you would need to secure the updateparms.asp
page.

You can get curl form http://curl.haxx.se/

Otherwise, you could FTP a data file -- in CSV format, perhaps -- and
then have the Web page read the CSV file. However, you will still have
file-locking problems because rewriting the CSV file requries
exclusive control.

Jim Buyens
Microsoft FrontPage MVP
http://www.interlacken.com
Author of:
*----------------------------------------------------
|\---------------------------------------------------
|| Microsoft Office FrontPage 2003 Inside Out
||---------------------------------------------------
|| Web Database Development Step by Step .NET Edition
|| Microsoft FrontPage Version 2002 Inside Out
|| Faster Smarter Beginning Programming
|| (All from Microsoft Press)
|/---------------------------------------------------
*----------------------------------------------------