DataGridView with LARGE dataset

M

Mike Oliszewski

We have the need to allow the user to quickly scroll through a log with over
100 million records. Accessing the data quickly and with low memory is no
problem, but displaying it seems to have no answer. DataGridView supports a
Virtual Mode but from our tests it is needing about 10 bytes of memory for
each record making it unable to handle lists of the size we need to display.

Does anyone know of a product or solution to this issue, we really don't
want to have to recreate the DataGridView control from scratch in order to
display this data. Please don't tell me to filter the data, I tried and
users DO NOT like that answer for what we are doing.
 
Z

Zhi-Xin Ye [MSFT]

Hi mikeoliszewski,

Thank you for using Microsoft Managed Newsgroup, I'm Zhi-Xin Ye, it's my
pleasure to work with you on this issue.

In fact implement virtual mode in DataGridView can retrieve data only as it
is needed which improves the scrolling performance. For example, say the
DataGridView you use displays 30 rows in the display area, then you can
divide the data into pages, each page contains more than 30 rows, when the
applications starts up, loads only the first page of data from the
database, and when user scrolls the DataGridView, loads another page of
data when neccessary. This approach is called just-in-time data loading,
using this approach, it won't use up much of memory. For more details
about how to just-in-time data loading, see:

Implementing Virtual Mode with Just-In-Time Data Loading in the Windows
Forms DataGridView Control
http://msdn.microsoft.com/en-us/library/ms171624.aspx

Best Regards,
Zhi-Xin Ye
Microsoft Managed Newsgroup Support Team

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.

Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
 
M

Mike Oliszewski

My posting explains that I need to scroll through a list of greater than
100,000,000 records and the DataGridView seems to allocate 10 bytes per
record despite being in the virtual mode. This makes it UNABLE to handle
lists of this magnitude.
 
N

Norman Yuan

Ah, you said you tried to tell your users...but here I go again.

I surely do not want to use an application that display 100 million records
and be expected to scroll it through quickly (it could be days!). No human
being should be doing this kind of things. If the records have to be handled
in some way by human, it should be human-handleable by the capacity of eyes,
brain and finger... How many rows a screen can show with smallest font that
is readable, 50? 100? OK, 100. So, you got 10 million screens. How fast the
user needs to scan the screen? 10 screens a second (a superman, indeed,
also, buy the user the best video card you cna find), 1,000,000 seconds,
which is more than 12 days.

So, your users are super-superman. Not only there is no existing,
good-enough control for this, I doubt you can "recreate" one. I'd rather
find another job.
 
M

Mike Oliszewski

First, we are able to load this dataset in a few seconds using only a couple
meg of RAM total despite the physical storage taking up many gigabytes of
storage.

Second, we are able to scroll to and fully retrieve any thousand sequential
records in under a 1/10th of a second. Performance on the back end is simply
not an issue.

Third, given the nature of what were doing it is truly more convenient for
our users to have instantaneouos access to the entire dataset not a filtered
section of it.

Analytical processes of complex data often have seemingly retarded methods
end up actually being the very best.
 
Z

Zhi-Xin Ye [MSFT]

Hi mikeoliszewski,

Are you trying to load all the 100milion records at the same time? If so,
it would certainly run out of memory. As I said in my last reply, you can
load only a "page" of data at a time, and loads another "page" of data when
necessary. Please check the just-in-time data loading in the following
sample:

Implementing Virtual Mode with Just-In-Time Data Loading in the Windows
Forms DataGridView Control
http://msdn.microsoft.com/en-us/library/ms171624.aspx

Best Regards,
Zhi-Xin Ye
Microsoft Managed Newsgroup Support Team

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

This posting is provided "AS IS" with no warranties, and confers no rights.
 
M

Mike Oliszewski

NO! I am using the DataGridView's Virtual mode, and it (in virtual mode)
uses 10 or more bytes per entry. Its extremely easy to reproduce. Simply
build a DataGridView put it in Virtual Mode, and set the record count to
10,000,000. Before it returns from setting the value the memory consumption
will have climbed to around 100M.
 
M

Mike Oliszewski

The code below is a full application demonstrating the problem. It consumes
150Meg of RAM on my system.


using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;

namespace DataGridTest
{
public partial class Form1 : Form
{
/// <summary>
/// The main entry point for the application.
/// </summary>
[STAThread]
static void Main()
{
Application.EnableVisualStyles();
Application.SetCompatibleTextRenderingDefault(false);
Application.Run(new Form1());
}

public Form1()
{
InitializeComponent();
}

private void Form1_Load(object sender, EventArgs e)
{
dataGridView1.RowCount = 10000000;
}
}
partial class Form1
{
/// <summary>
/// Required designer variable.
/// </summary>
private System.ComponentModel.IContainer components = null;

/// <summary>
/// Clean up any resources being used.
/// </summary>
/// <param name="disposing">true if managed resources should be
disposed; otherwise, false.</param>
protected override void Dispose(bool disposing)
{
if (disposing && (components != null))
{
components.Dispose();
}
base.Dispose(disposing);
}

#region Windows Form Designer generated code

/// <summary>
/// Required method for Designer support - do not modify
/// the contents of this method with the code editor.
/// </summary>
private void InitializeComponent()
{
this.dataGridView1 = new System.Windows.Forms.DataGridView();
this.Column1 = new
System.Windows.Forms.DataGridViewTextBoxColumn();
this.Column2 = new
System.Windows.Forms.DataGridViewTextBoxColumn();
this.Column3 = new
System.Windows.Forms.DataGridViewTextBoxColumn();

((System.ComponentModel.ISupportInitialize)(this.dataGridView1)).BeginInit();
this.SuspendLayout();
//
// dataGridView1
//
this.dataGridView1.ColumnHeadersHeightSizeMode =
System.Windows.Forms.DataGridViewColumnHeadersHeightSizeMode.AutoSize;
this.dataGridView1.Columns.AddRange(new
System.Windows.Forms.DataGridViewColumn[] {
this.Column1,
this.Column2,
this.Column3});
this.dataGridView1.Location = new System.Drawing.Point(12, 21);
this.dataGridView1.Name = "dataGridView1";
this.dataGridView1.Size = new System.Drawing.Size(240, 150);
this.dataGridView1.TabIndex = 0;
this.dataGridView1.VirtualMode = true;
//
// Column1
//
this.Column1.HeaderText = "Column1";
this.Column1.Name = "Column1";
//
// Column2
//
this.Column2.HeaderText = "Column2";
this.Column2.Name = "Column2";
//
// Column3
//
this.Column3.HeaderText = "Column3";
this.Column3.Name = "Column3";
//
// Form1
//
this.AutoScaleDimensions = new System.Drawing.SizeF(6F, 13F);
this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;
this.ClientSize = new System.Drawing.Size(292, 273);
this.Controls.Add(this.dataGridView1);
this.Name = "Form1";
this.Text = "Form1";
this.Load += new System.EventHandler(this.Form1_Load);

((System.ComponentModel.ISupportInitialize)(this.dataGridView1)).EndInit();
this.ResumeLayout(false);

}

#endregion

private System.Windows.Forms.DataGridView dataGridView1;
private System.Windows.Forms.DataGridViewTextBoxColumn Column1;
private System.Windows.Forms.DataGridViewTextBoxColumn Column2;
private System.Windows.Forms.DataGridViewTextBoxColumn Column3;
}
}
 
Z

Zhi-Xin Ye [MSFT]

Hi mikeoliszewski,

Thanks for the feedback.

Yes, setting RowCount property to 10,000,000 consumes around 100M memory,
but why do you load 10,000,000 rows at a time? You can divide the data into
pages, each page contains for example 1000 rows, then load first page of
data when the form loads, and load another page of data from database when
necessary. This way can avoid high memory usage, because there're only a
page of data in memory each time.
Have you tried the just-in-time data loading in your DataGridView?

Best Regards,
Zhi-Xin Ye
Microsoft Managed Newsgroup Support Team

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

This posting is provided "AS IS" with no warranties, and confers no rights.
 
M

Mike Oliszewski

I posted the application in this thread, please explain how it could be done
differently. Virtual mode is just-in-time loading, as it calls out to
retrieve the data, how can I make it any more virtualized than that ?
 
M

Mike Oliszewski

I have found an answer. The System.Windows.Forms.ListView has a virtual mode
that works VERY well. It CAN handle these enormous lists. The only problem
is that the ListView isn't very visually pleasing as it seems to be flicker a
lot when scrolling, but "working" is more important at this point and I see
that there are postings on ways to address the flicker issue.

Fortunately MS does have an answer that works.
 
Joined
Jul 21, 2009
Messages
1
Reaction score
0
Reason for memory use

I do not have a solution to the problem. But I can inform you why the memory is being used. I believe, anyway. It seems that this was an issue in understanding the problem.

When you use the DataGridView in VirtualMode you have to set the RowCount of the DGV. Doing this creates that many rows in the DGV. These rows are what are using the memory. I have read that this is done to allow the DGV to know how to display the scroll bar. 10 bytes for an empty row isn't too bad in my opinion.

In any case, I tend to agree that 100,000,000 rows is largely useless for a human. I'd love to know what these humans are hoping to find in a haystack that large. You wrote that it is a log file... But of what use such a mass could be is beyond me. Either you are searching carefully locally, or your jumping around a million records with each click along the scroll bar. Lotsa room to miss whatever you're looking for.

-Jerome
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top