Virtualizing List Views to Handle Large Amounts of Data

In a project I have recently undertook, a window was needed where it was neccessary to display large amounts of data (over 150,000 items) in a single CListCtrl. Futhermore, a second and even third window was needed, each with their own CListCtrl. These additional windows worked off the same data source as the first window, displaying different views or other alternate information.

The technique I use is made possible with the LVS_OWNERDATA style. Basically what that style does, is tell the CListCtrl that we are in charge of all memory handling. You never call CListCtrl::InsertItem() when this style is set. Everytime the control needs to paint or display text, Windows will ask you for the text located at item, subitem (in the form of a LVN_GETDISPINFO message). It is then your job to supply this information.

An example of this:

BEGIN_MESSAGE_MAP(CMyListCtrl, CListCtrl)
 //{{AFX_MSG_MAP(CMyListCtrl)
 ON_NOTIFY_REFLECT(LVN_GETDISPINFO, OnGetdispinfo)
 //}}AFX_MSG_MAP
END_MESSAGE_MAP()

//Windows is asking for the text to display, 
//at item(iItemIndex), subitem(pItem->iSubItem)
void CMyListCtrl::OnGetdispinfo(NMHDR* pNMHDR, LRESULT* pResult)
{
 LV_DISPINFO* pDispInfo = (LV_DISPINFO*)pNMHDR;
 LV_ITEM* pItem= &(pDispInfo)->item;
 int iItemIndex= pItem->iItem;

 if(pItem->mask & LVIF_TEXT)
 {
  if(pItem->iSubItem == 0) //first column
   lstrcpyn(pItem->pszText, 
            "This is the first columns data",
            pItem->cchTextMax);
  else
  if(pItem->iSubItem == 1) //second column
   lstrcpyn(pItem->pszText, 
            "Second column data", 
            pItem->cchTextMax);
 }
 *pResult = 0;
}

In the project I described earlier, I have a single linked list, with hundreds of thousands of elements in it. I have three different windows, each displaying different information about the data within this massive linked list, each in its own particular sorting order. I knew right away I didn't want to call InsertItem() and DeleteItem() hundreds of thousands of times, each time the user wanted to sort. And I definitely wasn't about to sort this entire linked list, or make duplicate copies for the other windows. So the technique I decided on was to simply have each CListCtrl object maintain its own dynamic array of pointers within this linked list. Everytime a new node is added to the linked list, I pass a pointer of the newly created node to the CListCtrl object, which stores the pointer in its array. I then simply call CListCtrl::SetItemCount(new_count);, which forces a redraw of the control. No new memory is allocated (unless the array needs to grow), and there is no delay at all from inserting the item. If many nodes are created at the same time (as is done with this example), then I pass them all to the view in one function call, so only a single screen update is needed.

This technique has several advantages. The first being that I can now easily have many CListCtrls working off the same data source, maintaining their own personal sort order, while minimizing the amount of memory required. Sorting is a snap: simply call the ANSI C function qsort() passing in the address of your array, along with your own compare function. After qsort() completes, call CListCtrl::SetItemCount(current_count) to redisplay the data. Done!

Another advantage is pure, raw speed. In the example code provided, I tested the window with 1,000,000 items in it, and it displays and scrolls in real time. It will scroll as fast as you can drag the mouse down. The only delay you will notice is the time spent to initially allocate and create a linked list with 1,000,000 elements in it. Of course that is only done once, and if your linked list is already created, with this technique you could create and display a CListCtrl object with well over 1,000,000 elements in less than a second! Sorting is also faster, as the only memory being moved around are the tiny 4byte pointers being stored in the array.

Downloads

Download demo project source code - 25 Kb
Download demo application (sans source) - 37 Kb


Comments

  • Using a CListCtrl in a DLL

    Posted by Legacy on 01/05/2004 12:00am

    Originally posted by: Ras

    Have you used a CListCtrl in a DLL. I've been trying it and with no luck. The dialog comes out with the outline of the list but no data. Any ideas ?

    • I have this working for a CListCtrl (actually a CListView)

      Posted by Mr. X on 04/01/2005 04:17am

      I have this working.  Display was not an issue for me (more issues around parsing, saving & sorting data!).
      
      Do you have something this line:
      ON_NOTIFY_REFLECT_EX(LVN_GETDISPINFO, GetDispInfo) // AJY
      
      and something like this method:
      
      BOOL DigitTableRightView::GetDispInfo(NMHDR* pNMHDR, LRESULT* pResult) 
      {
      
      	LV_DISPINFO* pDispInfo = (LV_DISPINFO*)pNMHDR;
      	LV_ITEM* pItem= &(pDispInfo)->item;
      	CLabelItem rLabel = m_arLabels.ElementAt(pItem->iItem);
      	CCLIExceptionTableRecord *pCLIExceptionTableRecord = (CCLIExceptionTableRecord *)(rLabel.m_dwCLI);
         
      	if (pItem->mask & LVIF_TEXT) //valid text buffer?
      	{
      		// then display the appropriate column
      		switch(pItem->iSubItem)
      		{
      		case 0:
      			lstrcpy(pItem->pszText, (char *)(pCLIExceptionTableRecord->Datas.record.cli_exception));
      			break;
      		case 1:
      			lstrcpy(pItem->pszText, rLabel.m_strServiceProvider);
      			break;
      		default:
      			ASSERT(0);
      			break;
      		}
      	}
      
      		pItem->iSubItem = LVIF_IMAGE;
      	pItem->iImage = 0; //H_ICON;
      	pItem->lParam = (unsigned long) pCLIExceptionTableRecord;
      	
      	*pResult = 0;
      
      	return(TRUE);
      }

      Reply
    Reply
  • Selection Problem

    Posted by Legacy on 11/25/2003 12:00am

    Originally posted by: Malik

    Thanks for a very nice demo.
    After making some selection (assume multiple selection) and then sorting the list, the selection seems to be incorrect.
    Any solution to this problem?

    Reply
  • You are a king ! THANKS A LOT !!!

    Posted by Legacy on 09/26/2003 12:00am

    Originally posted by: Yair Konfino

    in two minutes of work you saved me a brain damage for sure

    Reply
  • Select whole row in this case.... See here how.....

    Posted by Legacy on 08/05/2003 12:00am

    Originally posted by: Vaibhav Patle

    Thanks to you for such a beautiful idea.. However, to get the complete feel of list control, it is necessary to select the whole row. For this add following code to OnCreate() function of CMyListCtrl. 
    
    

    DWORD dwStyle = SendMessage(LVM_GETEXTENDEDLISTVIEWSTYLE);
    dwStyle |= LVS_EX_FULLROWSELECT;
    dwStyle |= LVS_EX_GRIDLINES ;
    SendMessage(LVM_SETEXTENDEDLISTVIEWSTYLE, 0, (LPARAM)dwStyle);


    Hope that it helps someone..

    Reply
  • When OnGetdispinfo() run??

    Posted by Legacy on 04/15/2002 12:00am

    Originally posted by: Dong-Jin Jeong

    Wating curser column's header?
    or click with any key?

    I Don't know...

    What event can be run OnGetdispinfo()function?
    anyone say to me..... please.

    Reply
  • And how for WTL CLIstView?

    Posted by Legacy on 04/03/2002 12:00am

    Originally posted by: Mishturak Ruslan

    I have problem by implementing virtual list view in WTL. Message notifycation was do not send to superclass. Can you help me?

    • ...did you do this?

      Posted by Mr. X on 04/01/2005 04:41am

      ON_NOTIFY_REFLECT_EX(LVN_GETDISPINFO, GetDispInfo)

      Reply
    Reply
  • Solution!

    Posted by Legacy on 02/13/2002 12:00am

    Originally posted by: Hyun-woo Jung

    Solution!

    --------------------------------------------------------------------------------

    I face your problem, but I found the solution.
    Don't call "SetWindowLong" with LVS_OWNERDATA function after create the control(CListCtrl).

    Instead, you can set LVS_OWNERDATA in Create Function.

    For example,

    CListCtrl m_List;
    m_List.Create(WS_CHILD | WS_VISIBLE | LVS_OWNERDATA,
    ~~~~~);


    P.S I can't explain English well... ^^; Sorry.
    If you can't understand, please email me.
    I'll give you sample source about this solution.

    • ...we know what you mean...

      Posted by Mr. X on 04/01/2005 04:38am

      ...an alternative is to override the PreCreate() method as shown in my earlier reply. For CViews you usually don't get to call Create directly.

      Reply
    Reply
  • What Wrong about LVS_OWNERDATA?

    Posted by Legacy on 02/06/2002 12:00am

    Originally posted by: kai

    My Application use m_wndSplitter split window into two views,Left view is CTreeView,Right view is CListView,I use
    LVS_OWNERDATA in my Right view,but it result in "Access Violation"

    I do this in these steps:
    1.in CRightView::OnInitialUpdate()
    CListCtrl &rListCtrl = GetListCtrl();
    LONG lStyle;
    lStyle=GetWindowLong(rListCtrl.m_hWnd,GWL_STYLE);
    lStyle&=~LVS_TYPEMASK;
    lStyle|=LVS_REPORT|LVS_OWNERDATA;
    SetWindowLong(rListCtrl.m_hWnd,GWL_STYLE,lStyle);
    2.void CRightView::fuc1()
    {
    CListCtrl & rListCtrl = GetListCtrl();
    int count = 10000;
    rListCtrl.SetItemCount(count);
    }
    What is Wrong with it?

    and when error occur,in Call Stack is:

    COMCTL32! 70d79931()
    COMCTL32! 70d7512c()
    USER32! 77e42c1c()
    USER32! 77e435a3()
    CWnd::DefWindowProcA(unsigned int 4143, unsigned int 10000, long 0) line 1000 + 32 bytes
    CWnd::WindowProc(unsigned int 4143, unsigned int 10000, long 0) line 1586 + 26 bytes
    AfxCallWndProc(CWnd * 0x00482fa0 {CRightView hWnd=???}, HWND__ * 0x001b0ae6, unsigned int 4143, unsigned int 10000, long 0) line 215 + 26 bytes
    AfxWndProc(HWND__ * 0x001b0ae6, unsigned int 4143, unsigned int 10000, long 0) line 368
    AfxWndProcBase(HWND__ * 0x001b0ae6, unsigned int 4143, unsigned int 10000, long 0) line 220 + 21 bytes
    USER32! 77e42381()
    USER32! 77e42be6()
    CListCtrl::SetItemCount(int 10000) line 216 + 77 bytes
    CRightView::fuc1() line 224
    CMainFrame::OnFileInfo(unsigned int 0, long 4726832) line 213
    CWnd::OnWndMsg(unsigned int 1034, unsigned int 0, long 4726832, long * 0x0012fd90) line 1815 + 17 bytes
    CWnd::WindowProc(unsigned int 1034, unsigned int 0, long 4726832) line 1585 + 30 bytes
    AfxCallWndProc(CWnd * 0x00356e38 {CMainFrame hWnd=???}, HWND__ * 0x001e0ad2, unsigned int 1034, unsigned int 0, long 4726832) line 215 + 26 bytes
    AfxWndProc(HWND__ * 0x001e0ad2, unsigned int 1034, unsigned int 0, long 4726832) line 368
    AfxWndProcBase(HWND__ * 0x001e0ad2, unsigned int 1034, unsigned int 0, long 4726832) line 220 + 21 bytes
    USER32! 77e41777()
    004820

    Thanks!!

    • For CListView OWNERDATA you need to ...

      Posted by Mr. X on 04/01/2005 04:25am

      ..set this attribute before creation. 
      [Microsoft don't mention that in their method descriptions currently & the methods return success!].  
      
      Here is one of the preferred ways to do this:
      
      BOOL RightView::PreCreateWindow(CREATESTRUCT& cs)
      {
      	cs.lpszName = WC_LISTVIEW;
      	cs.style &= ~LVS_TYPEMASK;
      	cs.style |= LVS_REPORT;
      	cs.style |= LVS_EDITLABELS;
      	cs.style |= LVS_OWNERDATA;
      
              return true;
      	//return( CListView::PreCreateWindow(cs) ); 
      }

      Reply
    • For CListView OWNERDATA you to ...

      Posted by Mr. X on 04/01/2005 04:24am

      ..set this attributye before creation [Microsoft don't mention that in their method descriptions currently & the methods return success!].  Here is one of the preferred ways to do this:
      
      BOOL RightView::PreCreateWindow(CREATESTRUCT& cs)
      {
      	cs.lpszName = WC_LISTVIEW;
      	cs.style &= ~LVS_TYPEMASK;
      	cs.style |= LVS_REPORT;
      	cs.style |= LVS_EDITLABELS;
      	cs.style |= LVS_OWNERDATA;
      
              return true;
      	//return( CListView::PreCreateWindow(cs) ); }

      Reply
    Reply
  • Yes, but how do i with STL list

    Posted by Legacy on 12/06/2001 12:00am

    Originally posted by: We

    I went through your code, it was helpful, but i have my list control data from a stl list, from which i cannot take the index and populate my list control. can u help me in doing this.

    • ...suggest you abandon STL and use a CArray then...

      Posted by Mr. X on 04/01/2005 04:36am

      CArray can be resized easily (& you can pre-allocate memory to avoid doing this for each iterations, it can be accessed by index but perhaps more importantly, you can run qsort & bsearch on it for high performance.

      Reply
    Reply
  • Change fonts for specific rows

    Posted by Legacy on 11/13/2001 12:00am

    Originally posted by: Tomas Keri

    How to change fonts for specific rows using a Virtual List. SO far I have been able to change all rows using CustomDraw but not item by item!

    Reply
  • Loading, Please Wait ...

Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • Live Event Date: October 29, 2014 @ 11:00 a.m. ET / 8:00 a.m. PT Are you interested in building a cognitive application using the power of IBM Watson? Need a platform that provides speed and ease for rapidly deploying this application? Join Chris Madison, Watson Solution Architect, as he walks through the process of building a Watson powered application on IBM Bluemix. Chris will talk about the new Watson Services just released on IBM bluemix, but more importantly he will do a step by step cognitive …

  • Live Event Date: November 13, 2014 @ 2:00 p.m. ET / 11:00 a.m. PT APIs can be a great source of competitive advantage. The practice of exposing backend services as APIs has become pervasive, however their use varies widely across companies and industries. Some companies leverage APIs to create internal, operational and development efficiencies, while others use them to drive ancillary revenue channels. Many companies successfully support both public and private programs from the same API by varying levels …

Most Popular Programming Stories

More for Developers

Latest Developer Headlines

RSS Feeds