Problem Upsizing Large Table to SQL Server 2000

  • Thread starter David W. Fenton
  • Start date
D

David W. Fenton

I've upsized the back ends of a few relatively small apps to SQL
Server in the past and had no problems whatsoever with the process.
Today I was testing the upsizing of the biggest back end any of my
clients have (it's c. 400MBs compacted). Most of it went OK, but the
two largest tables had major problems. One of the tables has nearly
600K records and the other nearly 400K. The larger table wouldn't
upsize at all and was skipped. The second one upsized, but omitted
importing the data.

So, I created the first table structure alone and then tried to run
append queries to append the data. The first attempt crapped out
with "Overflow," so I assumed some records were screwed up, and did
the 50:50 split, using TOP 50% to append the first half of the
table, and then the second half of the table failed. Then I split
that half, and so forth, until I got to the last 1/4 of the table,
which gave me the overflow error no matter how I parsed it. I
started appending 10K records at a time, and that worked for a
while, until it failed, and then I started appending in 1K chunks,
and then in chunks of 500 and then 100. This worked sporadically.
Eventually, I wrote code to append 100 records at a time, and
appended everything that would work.

I ended up short about 12K records, and can't figure out why.

I then created a brand-new table with those records and tried to
upsize it, and got the same result.

Now, so far as I can see, all the data types are exactly correct
(that was my first assumption, that something was the wrong data
type), and there are no memo fields in either of the tables. Given
that I recreated the table (actually more than once) and compacted
several times (which takes excruciatingly long with a file that
large), I can't think what kind of corruption would cause this.

So, I'm stuck with 12K records that aren't going in and can't figure
out why. Any ideas?

And, oh, BTW -- I did the upsizing with A2K3, even though the app
runs in A2K. This is because the A2K upsizing wizard can't handle
SQL Server 2000, and gives, wait for it, an OVERFLOW error. This was
somewhat inconvenient as the server had only A2K installed, so I had
to use A2K3 from my laptop. I tried both, but in both versions the
problematic records gave the overflow error.

The other table that was empty, I just ran one append and it caused
no problems at all.

BTW, there was no DRI on either of these tables when all this
happened, so it's not a failure there (if it had been, all the
records in the larger table would have been rejected, since they
were supposed to be children of the second table, which I only later
discovered was entirely empty).

Any ideas on this?

What am I missing in terms of possibilities here? Remember that I
was able to upsize a temp table created from a MAKETABLE query
containing the problematic records with no problem, but then
couldn't append from that table into the destination table (using
either Access or Enterprise Manager's query builder).

Help?
 
A

AnandaSim

Ok, brainstorming time. I like playing Dr House when I'm detached.
Possible reasons why something won't fit:
a. integrity and validation rules
b. bugs in the upsizing software
c. capacity limits (database, table, field)
d. bandwidth limits
e. corruption at the source and/or the destination

The SQL Server DTS tool seems like a good idea.

To give you more room to move, why not take the sql server database
offline, copy the files to your local desktop for playing. You can
install sql server desktop edition on the local desktop and try as
much as you want without having bandwidth constraints, people looking
over your shoulder, missing software etc...
 
S

Sylvain Lafontaine

Suggestion: delete column(s) until you find the offending field. Taking a
look with the SQL-Profiler could also be a good idea.
 
T

Tony Toews [MVP]

David W. Fenton said:
Most of it went OK, but the
two largest tables had major problems. One of the tables has nearly
600K records and the other nearly 400K. The larger table wouldn't
upsize at all and was skipped. The second one upsized, but omitted
importing the data.

Date fields somehow before 1899? Or whatever the SQL Server limit is
on the date field? A client had 12 such errors in a table with 100K
entries. Users fat fingers had keyed in a century somehow and their
date format did expose the century. Also there were no calculations
done on that date so no one ever noticed.

Tony
--
Tony Toews, Microsoft Access MVP
Please respond only in the newsgroups so that others can
read the entire thread of messages.
Microsoft Access Links, Hints, Tips & Accounting Systems at
http://www.granite.ab.ca/accsmstr.htm
Tony's Microsoft Access Blog - http://msmvps.com/blogs/access/
 
D

David W. Fenton

Ok, brainstorming time. I like playing Dr House when I'm detached.
Possible reasons why something won't fit:
a. integrity and validation rules

I've checked them field by field and they are exactly the same (or
equivalent in SQL Server terms).
b. bugs in the upsizing software

I've tried just appending the records directly with Access/ODBC, and
importing via the Enterprise Manager import tools, so the problem
exists outside the upsizing wizard.
c. capacity limits (database, table, field)

There's plenty of disk space (new server), and the field types match
exactly (or use the equivalents), so I can't see anything here that
would apply.
d. bandwidth limits

I've run it both across the LAN and running in a remote desktop
session directly on the server, so there's no bandwidth involved in
that one.
e. corruption at the source and/or the destination

The copy of the data I'm importing is in a brand-new MDB, imported
from the production data. I've compacted and repaired. I've appended
the problematic records into a brand new table. I've imported that
table into SQL Server, but the problem appears to be with the data
somehow, but I can't figure out what the problem is.

The SQL Server database is brand-new, so I can't see how it could be
corrupted already (I've shrunk it, just to make sure that wasn't
causing a problem, and it did shrink from 1.6GBs to .5GBs).
The SQL Server DTS tool seems like a good idea.

I'm just not familiar with that, so I'll have to look at it, thanks.
To give you more room to move, why not take the sql server
database offline, copy the files to your local desktop for
playing. You can install sql server desktop edition on the local
desktop and try as much as you want without having bandwidth
constraints, people looking over your shoulder, missing software
etc...

I don't have to worry about bandwidth, as I'm able to run everything
directly on the server.
 
D

David W. Fenton

Have you tried using SQL Server's Data Transformation Services?

I've never used them (never had this kind of problem before -- every
other back end I've upsized went smoothly with no need for
processing on the SQL Server side), so I'll have a look. What
capability will DTS have that the other tools I've used lack?
 
D

David W. Fenton

"Sylvain Lafontaine" <sylvain aei ca (fill the blanks, no spam
please)> wrote in
Suggestion: delete column(s) until you find the offending field.
Taking a look with the SQL-Profiler could also be a good idea.

Will the profiler record what goes wrong in the import, if run from
Enterprise Manager? Or from Access/ODBC?
 
D

David W. Fenton

Date fields somehow before 1899? Or whatever the SQL Server limit
is on the date field?

Now *that* is a good suggestion! I have encountered many 3-digit
years in some of the legacy data in this app (not since 2003, when I
put in a year check).
A client had 12 such errors in a table with 100K
entries. Users fat fingers had keyed in a century somehow and
their date format did expose the century. Also there were no
calculations done on that date so no one ever noticed.

There are two date fields filled out by users in this table, so this
is a good suggestion.

Wouldn't it be nice if the upsizing wizard tested your data for
compatibility before you imported and reported errors?

Anyway, I have to log off this machine to check, but thanks to
everyone for the useful suggestions. I'll report back later,
hopefully after I've solved the problem.
 
P

Pieter Wijnen

DTS works ;-)

For your purpose - it batch processes the data appending x number of records
at a time. you can control it pretty well if you save the "job" to a
vbScript File
It can map fields etc, etc.
I've used it succesfully a lot of times, including moving 25GB + databases
from SQLServer to Oracle (and vice versa)

I'm wondering a bit about the setup of your SQL Server Database.
Have you checked the default growth characteristics of the Data & Log Files,
The defaults *suck* for large amounts of data (read: any database) , and can
indeed stop the xfer

Pieter
 
S

Sylvain Lafontaine

Well, the profiler will show you what Access is doing and what it's
attempting to do. Hopefully, it will show you the wrong command or the one
just before it.

The error could be anything (out of range value: for example trying to
store -1 in a bit field, out of range decimal value, out of range datetime,
out of range float/real value, INT/BIGINT, etc.); a corrupted/missing
foreign key in the Access database, an unreadable record, etc.
 
T

Tony Toews [MVP]

I didn't even realize it was you, David, asking the question. I just
banged away at the answer. Funny how focused I get sometimes.

BTW SQL Server has two date formats. I don't recall the details but
the short one only handles "recent" dates and thus is not a good
choice for genealogy databases.
Wouldn't it be nice if the upsizing wizard tested your data for
compatibility before you imported and reported errors?

It would but .... The SSMA might do a better job there.

Tony
--
Tony Toews, Microsoft Access MVP
Please respond only in the newsgroups so that others can
read the entire thread of messages.
Microsoft Access Links, Hints, Tips & Accounting Systems at
http://www.granite.ab.ca/accsmstr.htm
Tony's Microsoft Access Blog - http://msmvps.com/blogs/access/
 
D

David W. Fenton

I didn't even realize it was you, David, asking the question. I
just banged away at the answer. Funny how focused I get
sometimes.

You get the gold star, Tony -- it was, in fact, the dates. Somewhere
along the line, somebody (it might even have been me :( ) ran an
update query that marked a bunch of records with dates that were
supposed to be in 2002, but came out as sometime during the late
Roman Empire, i.e., the year 200.

There were a handful of records with data entry errors of the same
kind (about 30) in two other date fields. Once I cleared those out
and did a DateSerial() update of the year 200 records, all imported
just fine.
BTW SQL Server has two date formats. I don't recall the details
but the short one only handles "recent" dates and thus is not a
good choice for genealogy databases.

Well, I don't need any dates after 1990 or so in this database. I
checked all the dates while I was at it and didn't find any other
problems.
It would but .... The SSMA might do a better job there.

Well, crap. I didn't even know about that. It probably would have
saved me a lot of time.
 
D

David W. Fenton

"Pieter Wijnen"
<it.isi.llegal.to.send.unsollicited.mail.wijnen.nospam.please@online.
replace.with.norway> wrote in
I'm wondering a bit about the setup of your SQL Server Database.
Have you checked the default growth characteristics of the Data &
Log Files, The defaults *suck* for large amounts of data (read:
any database) , and can indeed stop the xfer

Well, I will give that a look, but given Tony's hint about dates, I
solved it first thing this morning, as there were just a bunch of
records with years like 200. Once those were fixed, they all
imported without a hitch.

Thanks to all -- I'll keep all of the advice in mind as I work
through converting this big project.

So far, the things I thought would be easy have been hard, and the
things I just *knew* would be terribly difficult were easy (a task
I'd allocated 3-4 hours for in the project proposal took 20
minutes, mostly because the code I'd written had been designed with
conversion in mind, so that it wasn't hard at all to fix it up to
work).
 
P

Pieter Wijnen

should have thought of that one myself ; one of those items on the automatic
checklist

Pieter
 
T

Tony Toews [MVP]

David W. Fenton said:
You get the gold star, Tony -- it was, in fact, the dates.

Glad I could help.
Somewhere
along the line, somebody (it might even have been me :( ) ran an
update query that marked a bunch of records with dates that were
supposed to be in 2002, but came out as sometime during the late
Roman Empire, i.e., the year 200.

I don't know if you were in the newsgroups but back in '98 or '99,
whoa that sounds ancient, I discovered a bug in A97. When you
changed the machine date to 2000 entered a date with just the day and
month Access (actually oleaut32.dll so it was a OS bug) put in the
century of 0. Not 2000 as you'd expect. Or something very similar.
There were a handful of records with data entry errors of the same
kind (about 30) in two other date fields. Once I cleared those out
and did a DateSerial() update of the year 200 records, all imported
just fine.

Glad it was so easy.
Well, crap. I didn't even know about that. It probably would have
saved me a lot of time.

Well, you didn't ask. <smile>

Tony
--
Tony Toews, Microsoft Access MVP
Please respond only in the newsgroups so that others can
read the entire thread of messages.
Microsoft Access Links, Hints, Tips & Accounting Systems at
http://www.granite.ab.ca/accsmstr.htm
Tony's Microsoft Access Blog - http://msmvps.com/blogs/access/
 
T

Tony Toews [MVP]

DTS is still one of the very best things microsoft ever came up with

But isn't DTS more for doing bulk copies between databases? Will it
create tables, indexes and relationships?

Tony
--
Tony Toews, Microsoft Access MVP
Please respond only in the newsgroups so that others can
read the entire thread of messages.
Microsoft Access Links, Hints, Tips & Accounting Systems at
http://www.granite.ab.ca/accsmstr.htm
Tony's Microsoft Access Blog - http://msmvps.com/blogs/access/
 
P

Pieter Wijnen

It will create tables.
Personally I find it better to copy the data first & create the indexes etc.
later especially when dealing with huge amounts of data; I like to pretend
to have a life <g>. scripting the drop / create index & relations &
executing takes far less time than the time involved in waiting for the data
to be processed.
I must admit to having uttered quite a few #¤%!¤% when the data did not
comply with the constraints after the fact, but ...

Pieter
 
D

David W. Fenton

Glad I could help.


I don't know if you were in the newsgroups but back in '98 or '99,
whoa that sounds ancient, I discovered a bug in A97. When you
changed the machine date to 2000 entered a date with just the day
and month Access (actually oleaut32.dll so it was a OS bug) put in
the century of 0. Not 2000 as you'd expect. Or something very
similar.


Glad it was so easy.


Well, you didn't ask. <smile>

Well, I downloaded it and it wouldn't have helped, as it is for SQL
2005, and I'm working with 2000. It was also one of the most
annoying installations I've ever seen, as it first told me I didn't
have the .NET 2.0 CLR installed, but didn't offer to install it. And
then, to top it off, after installing that, it complained I didn't
have the J# runtime installed. Thankfully, it gave me the option of
installing it, so it wasn't *quite* as annoying.

And then it couldn't read secured MDBs, which is just bloody stupid,
so it was a complete waste of my time.

If there's a SQL Server 2000 version of it, I'd like to know about
it, but I couldn't find it in my extensive Googling.

Oh, BTW, the Access 2K3 upsizing wizard did a *terrible* job. It
failed to cluster the PK indexes, which resulted in table scans
*snort* on some of my queries! By revising the indexes to match what
the source MDB had had, the query plans were vastly improved and
tons of performance increases resulted.

That exercise did raise an issue for me that I don't encounter with
Jet back ends, and that's the option of clustering on something
other than my surrogate key PK (i.e., Autonumber). A couple of my
tables are only ever retrieved by PK, but certain others are
retrieved only by a foreign key, and others still by non-key fields
(like name fields). Actually, here are the main tables:

Most common retrieval fields:

tblDonor -- primary retrieval on name
DonorID PK Clustered
FirstName non-unique index with Nulls OK
LastName non-unique index with Nulls OK

tblContributions -- primary retrieval on DonorID
ContributionID PK Clustered
DonorID FK non-null
SubSourceID FK non-null -- used in joins only

tblSubSource -- primary retrieval on SubSourceID
SubSourceID PK Clustered
SourceID FK non-null -- used in joins only

tblSource -- retrieval on SourceID
SourceID PK Clustered
SRC_CODE unique index (int data type -- used only in dropdowns, so
no real need for lookup on this index, since it always uses
SourceID as the dropdown bound column)

Any thoughts on whether tblDonor would be better to cluster on the
name fields (assuming clustering can be done on compound keys that
allow Nulls)? I'm pretty sure the other tables are set up
appropriately.

On the other hand, performance on tblDonor name lookups as outlined
above is just as fast as it was with Access. Perhaps it would be
faster if clustered on the name fields?
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top