We have the need to allow the user to quickly scroll through a log with over
100 million records. Accessing the data quickly and with low memory is no
problem, but displaying it seems to have no answer. DataGridView supports a
Virtual Mode but from our tests it is needing about 10 bytes of memory for
each record making it unable to handle lists of the size we need to display.

Does anyone know of a product or solution to this issue, we really don't
want to have to recreate the DataGridView control from scratch in order to
display this data. Please don't tell me to filter the data, I tried and
users DO NOT like that answer for what we are doing.

Hi mikeoliszewski,

Thank you for using Microsoft Managed Newsgroup, I'm Zhi-Xin Ye, it's my
pleasure to work with you on this issue.

In fact implement virtual mode in DataGridView can retrieve data only as it
is needed which improves the scrolling performance. For example, say the
DataGridView you use displays 30 rows in the display area, then you can
divide the data into pages, each page contains more than 30 rows, when the
applications starts up, loads only the first page of data from the
database, and when user scrolls the DataGridView, loads another page of
data when neccessary. This approach is called just-in-time data loading,
using this approach, it won't use up much of memory. For more details
about how to just-in-time data loading, see:

Implementing Virtual Mode with Just-In-Time Data Loading in the Windows
Forms DataGridView Control
http://msdn.microsoft.com/en-us/library/ms171624.aspx

Best Regards,
Zhi-Xin Ye
Microsoft Managed Newsgroup Support Team

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(E-Mail Removed).

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subs...#notifications.

Note: MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 2 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions. Issues of this
nature are best handled working with a dedicated Microsoft Support Engineer
by contacting Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subs.../aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

My posting explains that I need to scroll through a list of greater than
100,000,000 records and the DataGridView seems to allocate 10 bytes per
record despite being in the virtual mode. This makes it UNABLE to handle
lists of this magnitude.

"Zhi-Xin Ye [MSFT]" wrote:

> Hi mikeoliszewski,
>
> Thank you for using Microsoft Managed Newsgroup, I'm Zhi-Xin Ye, it's my
> pleasure to work with you on this issue.
>
> In fact implement virtual mode in DataGridView can retrieve data only as it
> is needed which improves the scrolling performance. For example, say the
> DataGridView you use displays 30 rows in the display area, then you can
> divide the data into pages, each page contains more than 30 rows, when the
> applications starts up, loads only the first page of data from the
> database, and when user scrolls the DataGridView, loads another page of
> data when neccessary. This approach is called just-in-time data loading,
> using this approach, it won't use up much of memory. For more details
> about how to just-in-time data loading, see:
>
> Implementing Virtual Mode with Just-In-Time Data Loading in the Windows
> Forms DataGridView Control
> http://msdn.microsoft.com/en-us/library/ms171624.aspx
>
> Best Regards,
> Zhi-Xin Ye
> Microsoft Managed Newsgroup Support Team
>
> Delighting our customers is our #1 priority. We welcome your comments and
> suggestions about how we can improve the support we provide to you. Please
> feel free to let my manager know what you think of the level of service
> provided. You can send feedback directly to my manager at:
> (E-Mail Removed).
>
> ==================================================
> Get notification to my posts through email? Please refer to
> http://msdn.microsoft.com/en-us/subs...#notifications.
>
> Note: MSDN Managed Newsgroup support offering is for non-urgent issues
> where an initial response from the community or a Microsoft Support
> Engineer within 2 business day is acceptable. Please note that each follow
> up response may take approximately 2 business days as the support
> professional working with you may need further investigation to reach the
> most efficient resolution. The offering is not appropriate for situations
> that require urgent, real-time or phone-based interactions. Issues of this
> nature are best handled working with a dedicated Microsoft Support Engineer
> by contacting Microsoft Customer Support Services (CSS) at
> http://msdn.microsoft.com/en-us/subs.../aa948874.aspx
> ==================================================
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
>

Ah, you said you tried to tell your users...but here I go again.

I surely do not want to use an application that display 100 million records
and be expected to scroll it through quickly (it could be days!). No human
being should be doing this kind of things. If the records have to be handled
in some way by human, it should be human-handleable by the capacity of eyes,
brain and finger... How many rows a screen can show with smallest font that
is readable, 50? 100? OK, 100. So, you got 10 million screens. How fast the
user needs to scan the screen? 10 screens a second (a superman, indeed,
also, buy the user the best video card you cna find), 1,000,000 seconds,
which is more than 12 days.

So, your users are super-superman. Not only there is no existing,
good-enough control for this, I doubt you can "recreate" one. I'd rather
find another job.


"Mike Oliszewski" <(E-Mail Removed)> wrote in message
news:199917FB-F68B-4F7C-891C-(E-Mail Removed)...
> We have the need to allow the user to quickly scroll through a log with
> over
> 100 million records. Accessing the data quickly and with low memory is no
> problem, but displaying it seems to have no answer. DataGridView supports
> a
> Virtual Mode but from our tests it is needing about 10 bytes of memory for
> each record making it unable to handle lists of the size we need to
> display.
>
> Does anyone know of a product or solution to this issue, we really don't
> want to have to recreate the DataGridView control from scratch in order to
> display this data. Please don't tell me to filter the data, I tried and
> users DO NOT like that answer for what we are doing.
>

First, we are able to load this dataset in a few seconds using only a couple
meg of RAM total despite the physical storage taking up many gigabytes of
storage.

Second, we are able to scroll to and fully retrieve any thousand sequential
records in under a 1/10th of a second. Performance on the back end is simply
not an issue.

Third, given the nature of what were doing it is truly more convenient for
our users to have instantaneouos access to the entire dataset not a filtered
section of it.

Analytical processes of complex data often have seemingly retarded methods
end up actually being the very best.


"Norman Yuan" wrote:

> Ah, you said you tried to tell your users...but here I go again.
>
> I surely do not want to use an application that display 100 million records
> and be expected to scroll it through quickly (it could be days!). No human
> being should be doing this kind of things. If the records have to be handled
> in some way by human, it should be human-handleable by the capacity of eyes,
> brain and finger... How many rows a screen can show with smallest font that
> is readable, 50? 100? OK, 100. So, you got 10 million screens. How fast the
> user needs to scan the screen? 10 screens a second (a superman, indeed,
> also, buy the user the best video card you cna find), 1,000,000 seconds,
> which is more than 12 days.
>
> So, your users are super-superman. Not only there is no existing,
> good-enough control for this, I doubt you can "recreate" one. I'd rather
> find another job.
>
>
> "Mike Oliszewski" <(E-Mail Removed)> wrote in message
> news:199917FB-F68B-4F7C-891C-(E-Mail Removed)...
> > We have the need to allow the user to quickly scroll through a log with
> > over
> > 100 million records. Accessing the data quickly and with low memory is no
> > problem, but displaying it seems to have no answer. DataGridView supports
> > a
> > Virtual Mode but from our tests it is needing about 10 bytes of memory for
> > each record making it unable to handle lists of the size we need to
> > display.
> >
> > Does anyone know of a product or solution to this issue, we really don't
> > want to have to recreate the DataGridView control from scratch in order to
> > display this data. Please don't tell me to filter the data, I tried and
> > users DO NOT like that answer for what we are doing.
> >
>
>

Hi mikeoliszewski,

Are you trying to load all the 100milion records at the same time? If so,
it would certainly run out of memory. As I said in my last reply, you can
load only a "page" of data at a time, and loads another "page" of data when
necessary. Please check the just-in-time data loading in the following
sample:

Implementing Virtual Mode with Just-In-Time Data Loading in the Windows
Forms DataGridView Control
http://msdn.microsoft.com/en-us/library/ms171624.aspx

Best Regards,
Zhi-Xin Ye
Microsoft Managed Newsgroup Support Team

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(E-Mail Removed).

This posting is provided "AS IS" with no warranties, and confers no rights.

NO! I am using the DataGridView's Virtual mode, and it (in virtual mode)
uses 10 or more bytes per entry. Its extremely easy to reproduce. Simply
build a DataGridView put it in Virtual Mode, and set the record count to
10,000,000. Before it returns from setting the value the memory consumption
will have climbed to around 100M.

"Zhi-Xin Ye [MSFT]" wrote:

> Hi mikeoliszewski,
>
> Are you trying to load all the 100milion records at the same time? If so,
> it would certainly run out of memory. As I said in my last reply, you can
> load only a "page" of data at a time, and loads another "page" of data when
> necessary. Please check the just-in-time data loading in the following
> sample:
>
> Implementing Virtual Mode with Just-In-Time Data Loading in the Windows
> Forms DataGridView Control
> http://msdn.microsoft.com/en-us/library/ms171624.aspx
>
> Best Regards,
> Zhi-Xin Ye
> Microsoft Managed Newsgroup Support Team
>
> Delighting our customers is our #1 priority. We welcome your comments and
> suggestions about how we can improve the support we provide to you. Please
> feel free to let my manager know what you think of the level of service
> provided. You can send feedback directly to my manager at:
> (E-Mail Removed).
>
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
>
>
>

The code below is a full application demonstrating the problem. It consumes
150Meg of RAM on my system.


using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;

namespace DataGridTest
{
public partial class Form1 : Form
{
/// <summary>
/// The main entry point for the application.
/// </summary>
[STAThread]
static void Main()
{
Application.EnableVisualStyles();
Application.SetCompatibleTextRenderingDefault(false);
Application.Run(new Form1());
}

public Form1()
{
InitializeComponent();
}

private void Form1_Load(object sender, EventArgs e)
{
dataGridView1.RowCount = 10000000;
}
}
partial class Form1
{
/// <summary>
/// Required designer variable.
/// </summary>
private System.ComponentModel.IContainer components = null;

/// <summary>
/// Clean up any resources being used.
/// </summary>
/// <param name="disposing">true if managed resources should be
disposed; otherwise, false.</param>
protected override void Dispose(bool disposing)
{
if (disposing && (components != null))
{
components.Dispose();
}
base.Dispose(disposing);
}

#region Windows Form Designer generated code

/// <summary>
/// Required method for Designer support - do not modify
/// the contents of this method with the code editor.
/// </summary>
private void InitializeComponent()
{
this.dataGridView1 = new System.Windows.Forms.DataGridView();
this.Column1 = new
System.Windows.Forms.DataGridViewTextBoxColumn();
this.Column2 = new
System.Windows.Forms.DataGridViewTextBoxColumn();
this.Column3 = new
System.Windows.Forms.DataGridViewTextBoxColumn();

((System.ComponentModel.ISupportInitialize)(this.dataGridView1)).BeginInit();
this.SuspendLayout();
//
// dataGridView1
//
this.dataGridView1.ColumnHeadersHeightSizeMode =
System.Windows.Forms.DataGridViewColumnHeadersHeightSizeMode.AutoSize;
this.dataGridView1.Columns.AddRange(new
System.Windows.Forms.DataGridViewColumn[] {
this.Column1,
this.Column2,
this.Column3});
this.dataGridView1.Location = new System.Drawing.Point(12, 21);
this.dataGridView1.Name = "dataGridView1";
this.dataGridView1.Size = new System.Drawing.Size(240, 150);
this.dataGridView1.TabIndex = 0;
this.dataGridView1.VirtualMode = true;
//
// Column1
//
this.Column1.HeaderText = "Column1";
this.Column1.Name = "Column1";
//
// Column2
//
this.Column2.HeaderText = "Column2";
this.Column2.Name = "Column2";
//
// Column3
//
this.Column3.HeaderText = "Column3";
this.Column3.Name = "Column3";
//
// Form1
//
this.AutoScaleDimensions = new System.Drawing.SizeF(6F, 13F);
this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;
this.ClientSize = new System.Drawing.Size(292, 273);
this.Controls.Add(this.dataGridView1);
this.Name = "Form1";
this.Text = "Form1";
this.Load += new System.EventHandler(this.Form1_Load);

((System.ComponentModel.ISupportInitialize)(this.dataGridView1)).EndInit();
this.ResumeLayout(false);

}

#endregion

private System.Windows.Forms.DataGridView dataGridView1;
private System.Windows.Forms.DataGridViewTextBoxColumn Column1;
private System.Windows.Forms.DataGridViewTextBoxColumn Column2;
private System.Windows.Forms.DataGridViewTextBoxColumn Column3;
}
}

"Zhi-Xin Ye [MSFT]" wrote:

> Hi mikeoliszewski,
>
> Are you trying to load all the 100milion records at the same time? If so,
> it would certainly run out of memory. As I said in my last reply, you can
> load only a "page" of data at a time, and loads another "page" of data when
> necessary. Please check the just-in-time data loading in the following
> sample:
>
> Implementing Virtual Mode with Just-In-Time Data Loading in the Windows
> Forms DataGridView Control
> http://msdn.microsoft.com/en-us/library/ms171624.aspx
>
> Best Regards,
> Zhi-Xin Ye
> Microsoft Managed Newsgroup Support Team
>
> Delighting our customers is our #1 priority. We welcome your comments and
> suggestions about how we can improve the support we provide to you. Please
> feel free to let my manager know what you think of the level of service
> provided. You can send feedback directly to my manager at:
> (E-Mail Removed).
>
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
>
>
>

Hi mikeoliszewski,

Thanks for the feedback.

Yes, setting RowCount property to 10,000,000 consumes around 100M memory,
but why do you load 10,000,000 rows at a time? You can divide the data into
pages, each page contains for example 1000 rows, then load first page of
data when the form loads, and load another page of data from database when
necessary. This way can avoid high memory usage, because there're only a
page of data in memory each time.
Have you tried the just-in-time data loading in your DataGridView?

Best Regards,
Zhi-Xin Ye
Microsoft Managed Newsgroup Support Team

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(E-Mail Removed).

This posting is provided "AS IS" with no warranties, and confers no rights.

I posted the application in this thread, please explain how it could be done
differently. Virtual mode is just-in-time loading, as it calls out to
retrieve the data, how can I make it any more virtualized than that ?

"Zhi-Xin Ye [MSFT]" wrote:

> Hi mikeoliszewski,
>
> Thanks for the feedback.
>
> Yes, setting RowCount property to 10,000,000 consumes around 100M memory,
> but why do you load 10,000,000 rows at a time? You can divide the data into
> pages, each page contains for example 1000 rows, then load first page of
> data when the form loads, and load another page of data from database when
> necessary. This way can avoid high memory usage, because there're only a
> page of data in memory each time.
> Have you tried the just-in-time data loading in your DataGridView?
>
> Best Regards,
> Zhi-Xin Ye
> Microsoft Managed Newsgroup Support Team
>
> Delighting our customers is our #1 priority. We welcome your comments and
> suggestions about how we can improve the support we provide to you. Please
> feel free to let my manager know what you think of the level of service
> provided. You can send feedback directly to my manager at:
> (E-Mail Removed).
>
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
>