serialize vs database

[mp]

Hi all,
My current project is evolving over time as are my self inflicted
requirements.<g>
I'm now seeing the need to store info over time for comparisons

The project gathers daily info on stocks from web.
Up to 85 criteria can be chosen for what to look at.
at this point I am only looking at 10 or so but just starting
so that could evolve with time.
if i get data for 10 or so stocks that's between 100-850 'data points'
daily, and i need to watch trends over time so even at 100
that could be from 36500 to 365000 in a year(approx)
(even if i dont' want to see all 85 criteria today, i may find in the
future
that i wished i'd collected past data)

i found serializing one object to be easy.
i'm not so confident about being able to learn database.

I have a feeling however that xml files will become quickly unweildy
as compared a single .db file.

and obviously data mining/relationship comparisons would be much richer with
sql compared to comparing thousands of xml files.

Any thoughts? or simple tutorials on database in .net?
obviously i can find 100000 hits on google, but someone here may
already have culled some good ones
thanks
mark

 

[Jason Keats]

i found serializing one object to be easy.
i'm not so confident about being able to learn database.

I have a feeling however that xml files will become quickly unweildy
as compared a single .db file.

and obviously data mining/relationship comparisons would be much richer with
sql compared to comparing thousands of xml files.

Any thoughts? or simple tutorials on database in .net?
obviously i can find 100000 hits on google, but someone here may
already have culled some good ones

Your hunch that a database might be the way to go is, I believe, correct.

Have you thought of buying a book?

For example:
http://www.amazon.com/Beginning-C-Databases-Novice-Professional/dp/1590594339/

 

[Arne Vajhøj]

My current project is evolving over time as are my self inflicted
requirements.<g>
I'm now seeing the need to store info over time for comparisons

The project gathers daily info on stocks from web.
Up to 85 criteria can be chosen for what to look at.
at this point I am only looking at 10 or so but just starting
so that could evolve with time.
if i get data for 10 or so stocks that's between 100-850 'data points'
daily, and i need to watch trends over time so even at 100
that could be from 36500 to 365000 in a year(approx)
(even if i dont' want to see all 85 criteria today, i may find in the
future
that i wished i'd collected past data)

i found serializing one object to be easy.
i'm not so confident about being able to learn database.

Database is not so hard.

I have a feeling however that xml files will become quickly unweildy
as compared a single .db file.

and obviously data mining/relationship comparisons would be much richer with
sql compared to comparing thousands of xml files.

Correct.

It is not easy/fast to update part of a single big XML file.

It is not easy/fast to search in many small XML files.

Database is the way to go.

Any thoughts? or simple tutorials on database in .net?
obviously i can find 100000 hits on google, but someone here may
already have culled some good ones

Order of learning:
SQL
ADO.NET - SqlCommand and SqlDataReader (for SQLServer, similar for
other databases)
ORM framework - LINQ to EF

As you have found out then there are plenty of stuff
available on the net - I don't have anything particular
tutorials to recommend.

Arne

 

[Registered User]

Hi all,
My current project is evolving over time as are my self inflicted
requirements.<g>
I'm now seeing the need to store info over time for comparisons

The project gathers daily info on stocks from web.
Up to 85 criteria can be chosen for what to look at.
at this point I am only looking at 10 or so but just starting
so that could evolve with time.
if i get data for 10 or so stocks that's between 100-850 'data points'
daily, and i need to watch trends over time so even at 100
that could be from 36500 to 365000 in a year(approx)
(even if i dont' want to see all 85 criteria today, i may find in the
future
that i wished i'd collected past data)

I see where you're going and my advise is don't overwhelm yourself and
your project. For the moment concentrate on gathering and storing the
transactional data. There are other tools which can be used to extract
business intelligence (tracking trends, historical comparisons etc.)
from the transactional data.

i found serializing one object to be easy.
i'm not so confident about being able to learn database.

There is a lot to learn but you don't have to learn everything all at
once.

I have a feeling however that xml files will become quickly unweildy
as compared a single .db file.
yep
and obviously data mining/relationship comparisons would be much richer with
sql compared to comparing thousands of xml files.
yep
Any thoughts? or simple tutorials on database in .net?

These links may or may not be useful to you but
http://msdn.microsoft.com/
is always a good place to start.

http://code.msdn.microsoft.com/SQLTutorials
http://msdn.microsoft.com/en-us/library/bb655891(v=VS.90).aspx

Don't worry about extracting business intelligence from your data just
yet. Once you have an understanding of relational databases and are
comfortable using them, then you can learn about MS SQL Integration
Services, Analysis Services and Reporting Services. These are powerful
tools and the way you're thinking, you'll really appreciate them.

regards
A.G.

 

[Jeff Johnson]

Any thoughts? or simple tutorials on database in .net?

Let's step back. Do you have any SQL experience at all? Did you ever use ADO
in VB?

 

[mp]

I see where you're going and my advise is don't overwhelm yourself and
your project. For the moment concentrate on gathering and storing the
transactional data. There are other tools which can be used to extract
business intelligence (tracking trends, historical comparisons etc.)
from the transactional data.

There is a lot to learn but you don't have to learn everything all at
once.


These links may or may not be useful to you but
http://msdn.microsoft.com/
is always a good place to start.

http://code.msdn.microsoft.com/SQLTutorials
http://msdn.microsoft.com/en-us/library/bb655891(v=VS.90).aspx

Don't worry about extracting business intelligence from your data just
yet. Once you have an understanding of relational databases and are
comfortable using them, then you can learn about MS SQL Integration
Services, Analysis Services and Reporting Services. These are powerful
tools and the way you're thinking, you'll really appreciate them.

regards
A.G.

thanks for the input
I'll check those links out
mark

 

[mp]

Let's step back. Do you have any SQL experience at all? Did you ever use
ADO in VB?

just a teensy bit, tried sqlite and olaf helped a lot
did a tiny bit with ado and iirc olaf had some helper classes
that hit the sqlite engine
i have a base concept of normalizing
i think i can figure out which tables to break into
primary and foreign keys etc
i think i can learn how to do in ado.net
but it's just another whole learning process that will take
another month or two from the actual project...
so i'm hoping to find some close examples i can tweak
for my use...that's usually the way i learn fastest...
not fast <g> ... guess i should say least slowest
;-)
i downloaded sqlserver expressa year ago or so and it was
way over my ability ...
i can probably look back at my vb6 db samples and get
some handle on the sqlite db option.
i think i need to dl a dotnet sqlite lib or something
mark

 

[Arne Vajhøj]

just a teensy bit, tried sqlite and olaf helped a lot
did a tiny bit with ado and iirc olaf had some helper classes
that hit the sqlite engine
i have a base concept of normalizing
i think i can figure out which tables to break into
primary and foreign keys etc
i think i can learn how to do in ado.net
but it's just another whole learning process that will take
another month or two from the actual project...
so i'm hoping to find some close examples i can tweak
for my use...that's usually the way i learn fastest...
not fast<g> ... guess i should say least slowest
;-)
i downloaded sqlserver expressa year ago or so and it was
way over my ability ...
i can probably look back at my vb6 db samples and get
some handle on the sqlite db option.
i think i need to dl a dotnet sqlite lib or something

SQLite support does not come with MS .NET, so you will
need to download the ADO.NET provider for SQLite.

It really should not be that difficult to learn.

I think I have two simple SQLite examples on the
shelf. Ping me if interested.

Arne

 

[SNK]

( http://www.shahriarnk.com/Shahriar-N-K-Research-Embedding-SQL-in-C-Sharp-Java.html )

At the above link, you will find a very useful tutorial on developing database applications using C# as well as Java.

It describes in detail how to connect to the database, how to pass parametrized queries, call stored procedures and much more.

Do let us know if you find it useful.

Submitted via EggHeadCafe
ASP.NET In-Memory Image Control with Built-In Resizing of Posted File
http://www.eggheadcafe.com/tutorial...rol-with-builtin-resizing-of-posted-file.aspx

 

[mp]

[]


I think I have two simple SQLite examples on the
shelf. Ping me if interested.

Arne

examples are always appreciated!
I found one class on the web and have been looking at it

here's what i used to do in vb6
i'd created a field object with some properties
then sent a collection of those to this function to create the table
would i do somewhat similar in dotnet?

Public Sub CreateTable(TableName As String, FieldObjectsCollection As
Collection)
On Error GoTo CreateTable_Error
Dim ADOXtable As Table
Dim oField As cField
Set ADOXtable = New ADOX.Table
ADOXtable.Name = TableName
Set ADOXtable.ParentCatalog = moCatalog

With ADOXtable
'Add Columns to the table
With .Columns
For Each oField In FieldObjectsCollection
.Append oField.Name, oField.DataType
Next oField
End With 'columns

For Each oField In FieldObjectsCollection
If oField.AutoIncrement Then
.Columns(oField.Name).Properties("AutoIncrement") = True
End If
If oField.IsNullable Then
.Columns(oField.Name).Properties("Nullable") = True
End If
Next oField
End With 'table

'append tables to database`
moCatalog.Tables.Append ADOXtable

'logEntry "Check primary key "
For Each oField In FieldObjectsCollection
If oField.IsPrimary Then
Dim oKey As ADOX.Key
Set oKey = New ADOX.Key
With oKey
.Name = "PrimaryKey"
.Type = adKeyPrimary
.RelatedTable = TableName
.Columns.Append oField.Name
End With
ADOXtable.Keys.Append oKey
Exit For 'just one primary key
End If
Next oField

'Clean up
Set ADOXtable = Nothing
msDbTableName = TableName

ExitHere:
On Error GoTo 0
Exit Sub

CreateTable_Error:
LogError ProcName
End Sub

 

[Arne Vajhøj]

examples are always appreciated!
I found one class on the web and have been looking at it

See below.

here's what i used to do in vb6
i'd created a field object with some properties
then sent a collection of those to this function to create the table
would i do somewhat similar in dotnet?

Probably, but it is not how it is typical done in .NET.

Suggested order for learning:
1) Connection, Command and DataReader
2) DataAdapter and DataSet
3) Entity Framework

Arne

=====

using System;
using System.Data;

using System.Data.SQLite;

namespace E
{
public class TestClass
{
public static void Main(string[] args)
{
SQLiteConnection con = new SQLiteConnection(@"Data
Source=C:\TestEmb.db;Version=3;Password=hemmeligt;New=True");
con.Open();
SQLiteCommand cre = new SQLiteCommand("CREATE TABLE tbl (id
INTEGER NOT NULL, txt NVARCHAR(50), PRIMARY KEY(id))", con);
cre.ExecuteNonQuery();
for(int i = 0; i < 3; i++)
{
SQLiteCommand ins = new SQLiteCommand("INSERT INTO tbl
VALUES(" + i + ",'Test" + i + "')", con);
ins.ExecuteNonQuery();
}
con.Close();
}
}
}

using System;
using System.Data;
using System.Data.SQLite;

namespace E
{
public class TestClass
{
public static void Main(string[] args)
{
SQLiteConnection con = new SQLiteConnection(@"Data
Source=C:\TestEmb.db;Version=3;Password=hemmeligt");
con.Open();
SQLiteCommand sel = new SQLiteCommand("SELECT * FROM tbl",
con);
SQLiteDataReader rdr = sel.ExecuteReader();
while(rdr.Read())
{
Console.WriteLine((long)rdr[0] + " " + (string)rdr[1]);
}
rdr.Close();
con.Close();
}
}
}

 

[mp]

[] ?

These links may or may not be useful to you but
http://msdn.microsoft.com/
is always a good place to start.

http://code.msdn.microsoft.com/SQLTutorials
http://msdn.microsoft.com/en-us/library/bb655891(v=VS.90).aspx

Don't worry about extracting business intelligence from your data just
yet. Once you have an understanding of relational databases and are
comfortable using them, then you can learn about MS SQL Integration
Services, Analysis Services and Reporting Services. These are powerful
tools and the way you're thinking, you'll really appreciate them.

regards
A.G.

been following the links, it's taken me to an example to download
http://www.4shared.com/file/TnR6qUzE/data.html
when i go there i get to 4shared site which says 4shared extension not
installed
is this a safe site and exe as far as you know?
thanks
mark

 

[mp]

[]

?

been following the links, it's taken me to an example to download
http://www.4shared.com/file/TnR6qUzE/data.html
when i go there i get to 4shared site which says 4shared extension not
installed
is this a safe site and exe as far as you know?
thanks
mark

nevermind, turns out i was able to download without the extension
don't know what that was
thanks
mark

 

[mp]

examples are always appreciated!
I found one class on the web and have been looking at it

See below.

here's what i used to do in vb6
i'd created a field object with some properties
then sent a collection of those to this function to create the table
would i do somewhat similar in dotnet?

Probably, but it is not how it is typical done in .NET.

Suggested order for learning:
1) Connection, Command and DataReader
2) DataAdapter and DataSet
3) Entity Framework

Arne

=====

using System;
using System.Data;

using System.Data.SQLite;

namespace E
{
public class TestClass
{
public static void Main(string[] args)
{
SQLiteConnection con = new SQLiteConnection(@"Data
Source=C:\TestEmb.db;Version=3;Password=hemmeligt;New=True");
con.Open();
SQLiteCommand cre = new SQLiteCommand("CREATE TABLE tbl (id
INTEGER NOT NULL, txt NVARCHAR(50), PRIMARY KEY(id))", con);
cre.ExecuteNonQuery();
for(int i = 0; i < 3; i++)
{
SQLiteCommand ins = new SQLiteCommand("INSERT INTO tbl
VALUES(" + i + ",'Test" + i + "')", con);
ins.ExecuteNonQuery();
}
con.Close();
}
}
}

using System;
using System.Data;
using System.Data.SQLite;

namespace E
{
public class TestClass
{
public static void Main(string[] args)
{
SQLiteConnection con = new SQLiteConnection(@"Data
Source=C:\TestEmb.db;Version=3;Password=hemmeligt");
con.Open();
SQLiteCommand sel = new SQLiteCommand("SELECT * FROM tbl",
con);
SQLiteDataReader rdr = sel.ExecuteReader();
while(rdr.Read())
{
Console.WriteLine((long)rdr[0] + " " + (string)rdr[1]);
}
rdr.Close();
con.Close();
}
}
}

thanks Arne,
will commence 'playing'

mark

 

[mp]

[]
cool i ran your sample
so i can just expand this command and add my 85(or however many)
fields?
new SQLiteCommand("CREATE TABLE tbl (id INTEGER NOT NULL, txt NVARCHAR(50),
PRIMARY KEY(id))"

(that's assuming for now i put them all in one table)

seems like it will become one big line, but i guess thats okay?

thanks

mark

 

[Arne Vajhøj]

[]
cool i ran your sample
so i can just expand this command and add my 85(or however many)
fields?
new SQLiteCommand("CREATE TABLE tbl (id INTEGER NOT NULL, txt NVARCHAR(50),
PRIMARY KEY(id))"

(that's assuming for now i put them all in one table)

seems like it will become one big line, but i guess thats okay?

That should be OK.

I think you can get GUI tools for creating tables for
SQLite so you don't need to do it programmtically.

Arne

 

[mp]

On 26-01-2011 15:38, mp wrote:
I think I have two simple SQLite examples on the
shelf. Ping me if interested.

examples are always appreciated!

[]
cool i ran your sample
so i can just expand this command and add my 85(or however many)
fields?
new SQLiteCommand("CREATE TABLE tbl (id INTEGER NOT NULL, txt
NVARCHAR(50),
PRIMARY KEY(id))"

(that's assuming for now i put them all in one table)

seems like it will become one big line, but i guess thats okay?

That should be OK.

I think you can get GUI tools for creating tables for
SQLite so you don't need to do it programmtically.

Arne

i may look into that when i get time, but programmatcaly might even be
faster than gui. the other day when i decided to make the class with the 85
properties, i wrote a quick app to write the class .cs file by reading my
list of 85 items and turning each one into a private variable name and
matching public property with corresponding gets and sets...faster than
typing all that in by hand, let alone doing from a gui.

 

[Jeff Johnson]

the other day when i decided to make the class with the 85 properties, i
wrote a quick app to write the class .cs file by reading my list of 85
items and turning each one into a private variable name and matching
public property with corresponding gets and sets...faster than typing all
that in by hand, let alone doing from a gui.

Ah, you've discovered the joy of code generation!

 

[mp]

[]
Ah, you've discovered the joy of code generation!

And soon he will discover that for the most part, code that is generated
by other code should be used only by other code. At least, not when
the code-generation input doesn't require explicitly typing in all 85
members by hand _somewhere_ (e.g. .xsd schema turned into a container
type).

Ironically, I suspect that in this particular situation, there never
really should have been a type with 85 properties.

i instinctively knew i was doing something wrong!
the dictionary is obviously far superior

Instead, it probably

would have made more sense to use a dictionary/indexer/etc. and have the
caller just provide the name as a string, rather than having it baked into
the code.
[]


Live and learn.

Pete

i'm trying to, and you guys are making it much less
painful than it would be if i didn't have this miraculous
ng source of assistance.
mark

 

[Arne Vajhøj]

( http://www.shahriarnk.com/Shahriar-N-K-Research-Embedding-SQL-in-C-Sharp-Java.html )

At the above link, you will find a very useful tutorial on developing database applications using C# as well as Java.

It describes in detail how to connect to the database, how to pass parametrized queries, call stored procedures and much more.

Do let us know if you find it useful.

It gives an OK overview of .NET data access in 2004 and Java
data access in 1998.

Unfortunately the calendar says 2011.

Arne