How to allocate a large memory of 648791892 bytes?

[jameschuang]

How to allocate a large memory of 648791892 bytes?
I use the API of VirtuallAloc(), but fail to allocate the memory.
Could anyone helps me to do this? thanks a lot!

GetLastError() return 8. (Not enough storage is available to process this command.)
Why? and how to solve this?

PS: I have 2GB of RAM.

[Marc G]

Maybe the problem is that you don't have such a big block of unfragmented memory.
But the question is, do you really need to allocate such a big block of memory?

[0xC0000005]

What are you passing for flAllocationType (parameter 3) and flProtect (parameter 4)? I am assuming that parameter 1 is NULL.

[jameschuang]

Here is my sample codes.
PVOID ptrVirt, ptrMem;
ptrMem = VirtualAlloc(NULL, 999999999, MEM_COMMIT, PAGE_READWRITE);

[jameschuang]

Maybe the problem is that you don't have such a big block of unfragmented memory.
But the question is, do you really need to allocate such a big block of memory?
How to have a big block unfragmented memory?
Yes, I need such a big block of memory to do a 2400 dpi image rotation/deskew of image processing with 8.5 X 14 inch. (Maybe 3200dpi or higher...)

[Zaccheus]

Can you work in chunks? You could try allocating several 1MB blocks instead of a single huge memory block.

[JohnW@Wessex]

You could maybe create an image class that stores the image on disc and caches a limited number of regions in memory, loading and saving the regions as required. You'd have to load a region when there was a cache miss and keep track of the least used regions to write back to the the file.

[jameschuang]

Can you work in chunks? You could try allocating several 1MB blocks instead of a single huge memory block.
No, I cannot allocate such a large size of 999999999. But if 1MB, it is okay for 1024 times.
PVOID ptrMemory[1024];
memset(ptrMemory, 0, sizeof(PVOID) * 1024);
for (int i=0;i<1024;i++) {
ptrMemory[i] = VirtualAlloc(NULL, 1024*1024, MEM_COMMIT, PAGE_READWRITE);
}
It is okay.
But our image procssing library needs to be used a continuous memory to do this. The above are not continuous memories for our library.
And we do not have the library source codes to change it.

Could anybody give me a suggestion or hint to do this?

[Paul McKenzie]

No, I cannot allocate such a large size of 999999999. But if 1MB, it is okay for 1024 times.
PVOID ptrMemory[1024];
memset(ptrMemory, 0, sizeof(PVOID) * 1024);
for (int i=0;i<1024;i++) {
ptrMemory[i] = VirtualAlloc(NULL, 1024*1024, MEM_COMMIT, PAGE_READWRITE);
}
It is okay.
But our image procssing library needs to be used a continuous memory to do this. The above are not continuous memories for our library.Do you really expect the system to have this much contiguous memory available?

Is this image processing library a commercial library? If it is, what is the name of the library? If it isn't a commercial library, why don't you have the source to it?

Could anybody give me a suggestion or hint to do this?If the system doesn't have the memory, no hint or suggestion is possible, except to do things differently. You can't magically make memory appear. That's why it would be helpful if you give us more information on the library that you're using (i.e. the name of the library if it's a commercial library, whether the library has other functions that work in chunks, etc.)

Regards,

Paul McKenzie

[Zaccheus]

As this is for Windows, then the issue is address space fragmentation rather than physical memory fragmentation.

Each process receives its own full address range, so there might be a solution:

You could write a small program which is launched by the main application. The small program loads the image into memory, does specified image modifications using the library, and then saves the image again. The small program would then close.

[0xC0000005]

Virtual memory is completely different from physical memory. Windows does not need 2GB of unfragmented physical memory available to allocate 2GB of virtual memory even though virtual memory looks like that to your program. Only a part of the virtual memory is held in physical memory at any one time - the rest being swapped out to disk via virtual address translation, done automatically, on-the-fly and transparently to your program.

Having said that, you should have no problem allocating 2GB of virtual memory if you do have enough physical and disk space available. If VirtualAlloc() fails with the parameters you are using I would ask whether or not you are allocating more than one block and failing to free all of them when done using the memory. In other words, are you certain that you are calling VirtualAlloc() and VirtualFree() in pairs?

[MikeAThon]

No, I cannot allocate such a large size of 999999999. But if 1MB, it is okay for 1024 times.
PVOID ptrMemory[1024];
memset(ptrMemory, 0, sizeof(PVOID) * 1024);
for (int i=0;i<1024;i++) {
ptrMemory[i] = VirtualAlloc(NULL, 1024*1024, MEM_COMMIT, PAGE_READWRITE);
}
It is okay. ...
Are you certain that this resulted in 1024 successful allocations of memory? According to the docs, it should not have worked, since you failed to reserve memory before, or at the same time as, committing it. See http://msdn2.microsoft.com/en-us/library/aa366887.aspx :
To reserve and commit pages in one step, call VirtualAlloc with MEM_COMMIT | MEM_RESERVE.

The VirtualAlloc function fails if you attempt to commit a page that has not been reserved. The resulting error code is ERROR_INVALID_ADDRESS.
So, are you positive that you were able to confirm that the above code definitely worked? There is no error-checking in the code you show.

Mike

[jameschuang]

typedef struct
{
// INT16 iAngle; //Angle for rotate
float iAngle; //+: clockwise
//-: counter clockwise
LONGRECT stInRect; //Source area in coordinates
//referenced to full area of
//source image requed by library.

LONGRECT stOutRect; //Destination area in coordinates
//referenced to full area of
//Destination image requed by library.
BYTE byFillColor[6];

UINT32 dwCount; //Number of counts.

UINT32 dwMaxSrcDataSize;
UINT32 dwMaxDstDataSize;
UINT32 dwMaxSrcMaskDataSize;
UINT32 dwMaxDstMaskDataSize;

IMAGEBUF stSrcBuf;
IMAGEBUF stDstBuf;
IMAGEBUF stSrcMaskBuf;
IMAGEBUF stDstMaskBuf;

INT16 iSourceSequential;

void * pParameters;

}ROTATEINFO2, *PROTATEINFO2, RotateInfo2, *pRotateInfo2, **hRotateInfo2;

typedef struct {
INT16 depth; // input image type: Lineart(1), Binary (2), Gray (8) (16),
// Color (24, 30, 36, 48)
INT16 format; // used for color image only: Planar (0), // Gray (0),
// chuncky RGB (1), chuncky 0rgb (2),
// chuncky rgb0 (3), chuncky bgr (4),
// chuncky bgr0 (5), chuncky 0bgr (6)

//format:0 depth:1 -> lineart
//format:0 depth:8 -> Gray 8
//format:0 depth:16 -> Gray 16

//format:1 depth:24 -> RGB 24
//format:1 depth:48 -> RGB 48

INT32 rowBytes; // offset to next row
INT32 width; // source image width and height in pixels
INT32 height;
pUINT8 buffer1; // image buffer pointer and planar rgb
pUINT8 buffer2; // image use all addresses.
pUINT8 buffer3;
// Handles are used if NewImageBufHdl() is called.
Handle handle1; // image buffer handle storage
Handle handle2;
Handle handle3;
} IMAGEBUF, *pIMAGEBUF, **hIMAGEBUF, *PIMAGEBUF, **HIMAGEBUF;

ROTATEINFO2 stRotateInfo;
IPRotateContinue2(&stRotateInfo);

So if we want to call rotate library, we need the stSrcBuf and stDstBuf.
I found that stSrcBuf allocate successfully, but stDstBuf allocate fail.
Maybe my program total allocate size (stSrcBuf + stDstBuf) > 2GB.

The rotate library is a .lib. And the source codes is lost.

Thanks for all of your reply. Maybe I need to try another way to do this.

[Ajay Vijay]

I don't the that memory is the problem (or solution).
The problem is with data-structures used and algorithms applied.

Any modern, efficient and commercial image-processing software will not do this way. You may need to rotate (or any other graphical-transalation) the object in chunks. These blocks can be rows and columns, or lines (outermost to innermost) or other way around.