Process looping in VIP kernal runtime
Posted: 20 Jul 2016 15:35
Part of our processing engine is compiled with VIP 7.4. Since we changed the project to compile as 64-bit we are now experiencing an issue with the running process randomly hanging, or at least it appears that way.
In reviewing the process with Process Explorer we can see that it is still executing, but appears to be stuck in a loop with the VIP kernel and the CPU usage of the one task increases from it's normal 5 to 10% to an average 25%. All other threads within the same process go to "not responding" due to lack of cpu cycles.
Following is a sample of the run stack from Process Explorer at the time it's "hung".
ntdll.dll!memmove+0x25e
ntdll.dll!RtlUnwindEx+0xef
KERNEL32.dll!RtlUnwindEx+0x1e
vip7kernel.x64.dll!VIP::Kernel::allocastack+0xd452
ntdll.dll!RtlDecodePointer+0xad
ntdll.dll!RtlUnwindEx+0xbbf
ntdll.dll!KiUserExceptionDispatcher+0x2e
vip7kernel.x64.dll!VIP::Kernel::allocastack+0xd6a1
vip7kernel.x64.dll!VIP::Kernel::allocastack+0xb28b
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x5e68
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x8f20
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x90c1
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x1d20
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x2582
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x275d
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x2b2c
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x308
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x404
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x5bf
vip7kernel.x64.dll!VIP::Kernel::AllocAtomicHeapWithContext+0x80
vip7kernel.x64.dll!VIP::Kernel::CopyBinaryToHeap+0x24
rmbintf.dll+0x5b3e06
rmbintf.dll+0x5b3e17
rmbintf.dll+0x5b3d6b
rmbintf.dll+0x587e77
[Native Frame: IL Method without Metadata]
[Managed to Unmanaged Transition]
McKesson.PCON.Prolog.dll!McKesson.PCON.Prolog.clsRmbIntf.PdsModel+0xa7
. . .
We can see the stack changing above the AllocAtomicHeapWithContext procedure, so it is still executing, but never seems to return from it regardless of how fast I refresh the stack display.
Search our code for calls to CopyBinaryToHeap or AllocAtomicHeapWithContext, but could only find one or two references in the PFC compiled code (memory.pro). So we are guessing it has something to do with memory management. But, unfortunately that's used generally across the entire code. And because the run stack doesn't show any procedure names (only memory address) below the CopyBinaryToHeap, it's difficult to tell where in our code it's being executed. Although the memory addresses listed in the stack below the CopyBinaryToHeap are always the same.
The curious thing is that it's only happening when it's compiled as 64-bit. Take the exact same code, compile and run as 32-bit, it doesn't exhibit the issue.
Also tried running the code interactively in the debugger. While it does still "hang" as when it's running normally, I was hoping the debugger might give me a better idea as to where in the code it is currently executing. But, when it's "hung" and able to click the Break Program toolbar button (doesn't always break when clicked) it displays a disassembly view, which doesn't help very much.
Also tried this compiling with VIP 7.5, but still same issue.
Has this been previously reported and/or can someone give me some guidance on what types of functions/predicates/procedures to research in our code, so that I can determine what/where the problem is?
Regards.
In reviewing the process with Process Explorer we can see that it is still executing, but appears to be stuck in a loop with the VIP kernel and the CPU usage of the one task increases from it's normal 5 to 10% to an average 25%. All other threads within the same process go to "not responding" due to lack of cpu cycles.
Following is a sample of the run stack from Process Explorer at the time it's "hung".
ntdll.dll!memmove+0x25e
ntdll.dll!RtlUnwindEx+0xef
KERNEL32.dll!RtlUnwindEx+0x1e
vip7kernel.x64.dll!VIP::Kernel::allocastack+0xd452
ntdll.dll!RtlDecodePointer+0xad
ntdll.dll!RtlUnwindEx+0xbbf
ntdll.dll!KiUserExceptionDispatcher+0x2e
vip7kernel.x64.dll!VIP::Kernel::allocastack+0xd6a1
vip7kernel.x64.dll!VIP::Kernel::allocastack+0xb28b
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x5e68
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x8f20
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x90c1
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x1d20
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x2582
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x275d
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x2b2c
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x308
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x404
vip7kernel.x64.dll!VIP::Kernel::allocastack+0x5bf
vip7kernel.x64.dll!VIP::Kernel::AllocAtomicHeapWithContext+0x80
vip7kernel.x64.dll!VIP::Kernel::CopyBinaryToHeap+0x24
rmbintf.dll+0x5b3e06
rmbintf.dll+0x5b3e17
rmbintf.dll+0x5b3d6b
rmbintf.dll+0x587e77
[Native Frame: IL Method without Metadata]
[Managed to Unmanaged Transition]
McKesson.PCON.Prolog.dll!McKesson.PCON.Prolog.clsRmbIntf.PdsModel+0xa7
. . .
We can see the stack changing above the AllocAtomicHeapWithContext procedure, so it is still executing, but never seems to return from it regardless of how fast I refresh the stack display.
Search our code for calls to CopyBinaryToHeap or AllocAtomicHeapWithContext, but could only find one or two references in the PFC compiled code (memory.pro). So we are guessing it has something to do with memory management. But, unfortunately that's used generally across the entire code. And because the run stack doesn't show any procedure names (only memory address) below the CopyBinaryToHeap, it's difficult to tell where in our code it's being executed. Although the memory addresses listed in the stack below the CopyBinaryToHeap are always the same.
The curious thing is that it's only happening when it's compiled as 64-bit. Take the exact same code, compile and run as 32-bit, it doesn't exhibit the issue.
Also tried running the code interactively in the debugger. While it does still "hang" as when it's running normally, I was hoping the debugger might give me a better idea as to where in the code it is currently executing. But, when it's "hung" and able to click the Break Program toolbar button (doesn't always break when clicked) it displays a disassembly view, which doesn't help very much.
Also tried this compiling with VIP 7.5, but still same issue.
Has this been previously reported and/or can someone give me some guidance on what types of functions/predicates/procedures to research in our code, so that I can determine what/where the problem is?
Regards.