C# database project

[Jelle de Jong]

Hi,

I want to start a new project, I want to build my own database server and
start very simple. I want to create a file-based database server, with
XML-files. Does anyone know a good site or book to get some more information
about this kind of project (like 'How to build your own database server')?

Or can someone help me start...?

I want to build it in C# (.NET) and save files in XML.

I want to use XML-files to use as my database-definition files, like:

<data>
<database name="TestDB">
<table name="TestTable">
<field type="PrimaryKey" name="ID" />
<field type="String" name="Name" />
<field type="DateTime" name="DateOfBirth" />
<field type="Bool" name="Male" default="True" />
</table>
</database>
</data>

I also want to use/create a SQL like query language to use for querying this
database.

Hopefully someone can help me.

Thanks in advance.



Greetings,

Jelle

 

[Jelle de Jong]

By the way, I want to use it for webapplications and windowsapplications.

 

[Nicholas Paldino [.NET/C# MVP]]

Jelle,

This is a ^tremendous^ undertaking that you are considering. I don't
know if you are doing it because you have to (in which case, I would say
find some other data source) or if you are just doing it for the hell of it,
but either way, it's not easy.

The major areas you have to deal with are the storage of the data, and
the querying of the data. While these are the two main concerns, there are
going to be many things that you have to consider:

- How will storing data structure/information in XML affect the performance
of the database?
- How will you affect concurrent access to the database?
- How will you handle indexes (you have a primary key, which implies an
index in most/all DB systems)
- How will you connect to the database and execute commands/return
information
- What will the query language look like?

These are just ^some^ of the issues you have to deal with when writing a
DB server. There are many, many more, but the questions posed above are
probably good places to start.

 

[Jelle de Jong]

Nicholas,

I know it's not an easy project, but I just want to create something for my
own use. And this is a real challenge! ;-)

I want to start very simple. My idea was to work with some form of
Serialization. But I don't know what the performance will be when I want to
write back to the filesystem.

It's just for fun, and maybe someone also wants to create such a thing or
did it before?

Gr.,
Jelle

Jelle,

This is a ^tremendous^ undertaking that you are considering. I don't
know if you are doing it because you have to (in which case, I would say
find some other data source) or if you are just doing it for the hell of
it, but either way, it's not easy.

The major areas you have to deal with are the storage of the data, and
the querying of the data. While these are the two main concerns, there
are going to be many things that you have to consider:

- How will storing data structure/information in XML affect the
performance of the database?
- How will you affect concurrent access to the database?
- How will you handle indexes (you have a primary key, which implies an
index in most/all DB systems)
- How will you connect to the database and execute commands/return
information
- What will the query language look like?

These are just ^some^ of the issues you have to deal with when writing
a DB server. There are many, many more, but the questions posed above are
probably good places to start.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Hi,

I want to start a new project, I want to build my own database server and
start very simple. I want to create a file-based database server, with
XML-files. Does anyone know a good site or book to get some more
information about this kind of project (like 'How to build your own
database server')?

Or can someone help me start...?

I want to build it in C# (.NET) and save files in XML.

I want to use XML-files to use as my database-definition files, like:

<data>
<database name="TestDB">
<table name="TestTable">
<field type="PrimaryKey" name="ID" />
<field type="String" name="Name" />
<field type="DateTime" name="DateOfBirth" />
<field type="Bool" name="Male" default="True" />
</table>
</database>
</data>

I also want to use/create a SQL like query language to use for querying
this database.

Hopefully someone can help me.

Thanks in advance.



Greetings,

Jelle

 

[Nicholas Paldino [.NET/C# MVP]]

Jelle,

If you are going to use it for web-based applications, then I really
have to say that you are going to have to do plenty of dirty little things
to make sure it performs well, and you are going to have to make sure that
transaction consistency is bullet-proof. With the number of people that
will hammer the database at the same time in a web-based app, those things
are certainly going to be put to the test.

Additionally, if the database is file-based, you simply are not going to
get the performance you can get with keeping the data in memory and paging
out to disk when necessary. The reason for this is that if all of your
operations are done on the file each time they occur, your disk access
becomes a HUGE bottleneck for you.

 

[cbmeeks]

Hi,

I want to start a new project, I want to build my own database server and
start very simple. I want to create a file-based database server, with
XML-files. Does anyone know a good site or book to get some more information
about this kind of project (like 'How to build your own database server')?

Or can someone help me start...?

I want to build it in C# (.NET) and save files in XML.

I want to use XML-files to use as my database-definition files, like:

<data>
<database name="TestDB">
<table name="TestTable">
<field type="PrimaryKey" name="ID" />
<field type="String" name="Name" />
<field type="DateTime" name="DateOfBirth" />
<field type="Bool" name="Male" default="True" />
</table>
</database>
</data>

I also want to use/create a SQL like query language to use for querying this
database.

Hopefully someone can help me.

Thanks in advance.

Greetings,

Jelle

Don't get put off. Check out SimpleDB from Amazon. Or CouchDB. You
can serialize objects into them. They are more of an "document
database" than a "relational database".

cbmeeks
http://codershangout.com

 

[Jelle de Jong]

Nicholas,

So maybe it's better to use XmlSerialization to save objects to the
filesystem and Deserialize them to use them as objects again. So no
query-language or in-memory data or whatsoever. Or use MySQL, SQL Server or
Oracle instead ;-)

Or do you have other nice ideas about how people should store their data
into a data-structure or a filesystem?


Gr.,
Jelle

Jelle,

If you are going to use it for web-based applications, then I really
have to say that you are going to have to do plenty of dirty little things
to make sure it performs well, and you are going to have to make sure that
transaction consistency is bullet-proof. With the number of people that
will hammer the database at the same time in a web-based app, those things
are certainly going to be put to the test.

Additionally, if the database is file-based, you simply are not going
to get the performance you can get with keeping the data in memory and
paging out to disk when necessary. The reason for this is that if all of
your operations are done on the file each time they occur, your disk
access becomes a HUGE bottleneck for you.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

By the way, I want to use it for webapplications and windowsapplications.

 

[Nicholas Paldino [.NET/C# MVP]]

Jelle,

Why is it that you are adamant about using the filesystem? I ask
because you had the desire to have this used in web applications, and a
file-based database is just going to be crushed under the load in a web-app
with any real load.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Nicholas,

So maybe it's better to use XmlSerialization to save objects to the
filesystem and Deserialize them to use them as objects again. So no
query-language or in-memory data or whatsoever. Or use MySQL, SQL Server
or Oracle instead ;-)

Or do you have other nice ideas about how people should store their data
into a data-structure or a filesystem?


Gr.,
Jelle

Jelle,

If you are going to use it for web-based applications, then I really
have to say that you are going to have to do plenty of dirty little
things to make sure it performs well, and you are going to have to make
sure that transaction consistency is bullet-proof. With the number of
people that will hammer the database at the same time in a web-based app,
those things are certainly going to be put to the test.

Additionally, if the database is file-based, you simply are not going
to get the performance you can get with keeping the data in memory and
paging out to disk when necessary. The reason for this is that if all of
your operations are done on the file each time they occur, your disk
access becomes a HUGE bottleneck for you.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

By the way, I want to use it for webapplications and
windowsapplications.



Hi,

I want to start a new project, I want to build my own database server
and start very simple. I want to create a file-based database server,
with XML-files. Does anyone know a good site or book to get some more
information about this kind of project (like 'How to build your own
database server')?

Or can someone help me start...?

I want to build it in C# (.NET) and save files in XML.

I want to use XML-files to use as my database-definition files, like:

<data>
<database name="TestDB">
<table name="TestTable">
<field type="PrimaryKey" name="ID" />
<field type="String" name="Name" />
<field type="DateTime" name="DateOfBirth" />
<field type="Bool" name="Male" default="True" />
</table>
</database>
</data>

I also want to use/create a SQL like query language to use for querying
this database.

Hopefully someone can help me.

Thanks in advance.



Greetings,

Jelle

 

[Jelle de Jong]

Nicholas,

Why is it that you are adamant about using the filesystem? I ask
because you had the desire to have this used in web applications, and a
file-based database is just going to be crushed under the load in a
web-app with any real load.

I want to use the filesystem because it's easy en cheap. When I'm creating
simple websites for maybe 50 - 100 visitors a day, I want to have an easy
way of saving data. That's why. So it's pure for saving some content for
webpages, or feedback from visitors, or mailaddresses form some kind of
newsletter/mailinglist.


Jelle

 

[Nicholas Paldino [.NET/C# MVP]]

Jelle,

There is nothing wrong with saving data to the file system, but keep in
mind that ^always^ accessing the data from the files in the file system is
going to be a huge bottleneck for you.

 

[Jelle de Jong]

Nicholas,

There is nothing wrong with saving data to the file system, but keep in
mind that ^always^ accessing the data from the files in the file system is
going to be a huge bottleneck for you.

But when I'm using Serialization, my application should perform better, then
the object will be in-memory while it's being used, isn't it? And in-memory
is the fastest way, I presume?



Jelle

 

[Nicholas Paldino [.NET/C# MVP]]

Jelle,

Yes, it will be in memory, but you want to ^keep^ it in memory for as
long as possible. If you save it back to the filesystem after an update,
for example, and then reload it again the next time you need it, that's
going to cause a bottleneck. If you already have it lying around in memory,
you wouldn't have to load it from disc.

 

[Charles Calvert]

I want to start a new project, I want to build my own database server

For the love of ghu, why? Databases are a seriously thorny area.
There are all sorts of concerns, even for a very basic implementation.

and start very simple. I want to create a file-based database server,
with XML-files.

Because you want to guarantee that it's as slow as possible? XML was
designed for data exchange, not as a data repository. I know that
people constantly abuse it that way (especially Microsoft), but that
doesn't mean that you should repeat the mistake. A binary
representation would be easy enough to serialize and likely a good bit
faster in real world use.

Does anyone know a good site or book to get some more information
about this kind of project (like 'How to build your own database server')?

You'd be better off asking in comp.databases or a similar group.

[snip]

I also want to use/create a SQL like query language to use for querying this
database.

I hope that you're comfortable with lexx and yacc or similar tools, as
you'll need to build a fairly sophisticated parser.

I admire the drive to build something, but I think you'd be better off
picking a less complex project. If you're interested in how databases
work, grab the source for one of the open source or public domain
light databases, such as SQLite <http://www.sqlite.org/> and dig in.
Once you've got a handle on that, you can move up to something larger
like PostgreSQL <http://www.postgresql.org/>.

 

[Robert Woodliff]

Concerning file a database file system. Use a cache to keep the database object in memory which if not in use after a time span the cache will release the object and resources.

I believe a better solution to db caching is table caching (which MYSQL is famous for because the actual dbo contains the table information in three file formats) which in hence reduces having to load all the tables in memory but only the tables that are being called managed by the dbo information schema.

XML is far by any means meant for a database system and should not be.

Hi,

I want to start a new project, I want to build my own database server and
start very simple. I want to create a file-based database server, with
XML-files. Does anyone know a good site or book to get some more information
about this kind of project (like 'How to build your own database server')?

Or can someone help me start...?

I want to build it in C# (.NET) and save files in XML.

I want to use XML-files to use as my database-definition files, like:

<data>
<database name="TestDB">
<table name="TestTable">
<field type="PrimaryKey" name="ID" />
<field type="String" name="Name" />
<field type="DateTime" name="DateOfBirth" />
<field type="Bool" name="Male" default="True" />
</table>
</database>
</data>

I also want to use/create a SQL like query language to use for querying this
database.

Hopefully someone can help me.

Thanks in advance.



Greetings,

Jelle

On Wednesday, February 13, 2008 2:58 PM Jelle de Jong wrote:
By the way, I want to use it for webapplications and windowsapplications.

On Wednesday, February 13, 2008 2:59 PM Nicholas Paldino [.NET/C# MVP] wrote:
Jelle,

This is a ^tremendous^ undertaking that you are considering. I don't
know if you are doing it because you have to (in which case, I would say
find some other data source) or if you are just doing it for the hell of it,
but either way, it's not easy.

The major areas you have to deal with are the storage of the data, and
the querying of the data. While these are the two main concerns, there are
going to be many things that you have to consider:

- How will storing data structure/information in XML affect the performance
of the database?
- How will you affect concurrent access to the database?
- How will you handle indexes (you have a primary key, which implies an
index in most/all DB systems)
- How will you connect to the database and execute commands/return
information
- What will the query language look like?

These are just ^some^ of the issues you have to deal with when writing a
DB server. There are many, many more, but the questions posed above are
probably good places to start.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

news:[email protected]...

On Wednesday, February 13, 2008 3:13 PM Nicholas Paldino [.NET/C# MVP] wrote:
Jelle,

If you are going to use it for web-based applications, then I really
have to say that you are going to have to do plenty of dirty little things
to make sure it performs well, and you are going to have to make sure that
transaction consistency is bullet-proof. With the number of people that
will hammer the database at the same time in a web-based app, those things
are certainly going to be put to the test.

Additionally, if the database is file-based, you simply are not going to
get the performance you can get with keeping the data in memory and paging
out to disk when necessary. The reason for this is that if all of your
operations are done on the file each time they occur, your disk access
becomes a HUGE bottleneck for you.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

news:[email protected]...

On Wednesday, February 13, 2008 3:33 PM Nicholas Paldino [.NET/C# MVP] wrote:
Jelle,

Why is it that you are adamant about using the filesystem? I ask
because you had the desire to have this used in web applications, and a
file-based database is just going to be crushed under the load in a web-app
with any real load.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

news:[email protected]...

On Wednesday, February 13, 2008 4:12 PM Nicholas Paldino [.NET/C# MVP] wrote:
Jelle,

There is nothing wrong with saving data to the file system, but keep in
mind that ^always^ accessing the data from the files in the file system is
going to be a huge bottleneck for you.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

news:[email protected]...

On Wednesday, February 13, 2008 4:43 PM Nicholas Paldino [.NET/C# MVP] wrote:
Jelle,

Yes, it will be in memory, but you want to ^keep^ it in memory for as
long as possible. If you save it back to the filesystem after an update,
for example, and then reload it again the next time you need it, that's
going to cause a bottleneck. If you already have it lying around in memory,
you wouldn't have to load it from disc.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

news:[email protected]...

48:39 +0100, "Jelle de Jong"
<[email protected]>:


For the love of ghu, why? Databases are a seriously thorny area.
There are all sorts of concerns, even for a very basic implementation.


Because you want to guarantee that it's as slow as possible? XML was
designed for data exchange, not as a data repository. I know that
people constantly abuse it that way (especially Microsoft), but that
doesn't mean that you should repeat the mistake. A binary
representation would be easy enough to serialize and likely a good bit
faster in real world use.


You'd be better off asking in comp.databases or a similar group.

[snip]


I hope that you're comfortable with lexx and yacc or similar tools, as
you'll need to build a fairly sophisticated parser.

I admire the drive to build something, but I think you'd be better off
picking a less complex project. If you're interested in how databases
work, grab the source for one of the open source or public domain
light databases, such as SQLite <http://www.sqlite.org/> and dig in.
Once you've got a handle on that, you can move up to something larger
like PostgreSQL <http://www.postgresql.org/>.
--
Charles Calvert | Software Design/Development
Celtic Wolf, Inc. | Project Management
http://www.celticwolf.com/ | Technical Writing
(703) 580-0210 | Research