[SafeCall Exception]: Out of memory.

Hello all,

We have been getting "[SafeCall Exception]: Out of memory." occasionally for the last month, in the afternoon on very busy days. It starts to appear sporadically on the Leads screen when we click 'edit'. The solution is a recycle of the app pool, which clearly that kicks users out, so I'd prefer to avoid this. Memory leak? Any tips tracking this down? Elf logs below.

I have already upped the virtual memory settings on the box.

EurekaLog 6.0.25

Application:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1.1 Start Date : Tue, 6 Mar 2018 06:54:19 +0000
1.2 Name/Description: w3wp.exe - (IIS Worker Process)
1.3 Version Number : 8.0.9200.16384
1.4 Parameters : -ap "CRMTNG App Pool" -v "v4.0" -l "webengine4.dll" -a \\.\pipe\iisipm64b235b5-a8ee-437a-b6bd-7459769e01ba -h "C:\inetpub\temp\apppools\CRM App Pool\CRM App Pool.config" -w "" -m 0
1.5 Compilation Date: Tue, 27 Jan 2015 10:50:58 +0000
1.6 Up Time : 9 hours, 49 minutes, 19 seconds

Exception:
----------------------------------------------------------
2.1 Date : Tue, 6 Mar 2018 16:43:39 +0000
2.2 Address : 1A0067E7
2.3 Module Name : eware.dll
2.4 Module Version:
2.5 Type : EOutOfMemory
2.6 Message : [SafeCall Exception]: Out of memory.
2.7 ID : EA37
2.8 Count : 1
2.9 Status : New
2.10 Note :

User:
-------------------------------------------------------
3.1 ID : SYSTEM
3.2 Name : Windows User
3.3 Email :
3.4 Company :
3.5 Privileges: SeAssignPrimaryTokenPrivilege - OFF
SeIncreaseQuotaPrivilege - OFF
SeTcbPrivilege - ON
SeBackupPrivilege - OFF
SeRestorePrivilege - OFF
SeDebugPrivilege - ON
SeAuditPrivilege - ON
SeChangeNotifyPrivilege - ON
SeImpersonatePrivilege - ON
SeCreateGlobalPrivilege - ON

Computer:
---------------------------------------------------------------
5.1 Name : CRM
5.2 Total Memory : 16384 Mb
5.3 Free Memory : 9266 Mb
5.4 Total Disk : 129.66 Gb
5.5 Free Disk : 21.66 Gb
5.6 System Up Time: 11 days, 20 hours, 41 minutes, 38 seconds
5.7 Processor : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
5.8 Display Mode : 1024 x 768, 32 bit
5.9 Display DPI : 96
5.10 Video Card : (driver - RAM 4 MB)
5.11 Printer :

  • 0

    Magic

    What version of Sage CRM are you on? (Memory management changed in 2018 R1; is this an upgrade?

    Is this a multi-server install?

  • 0

    Jeff!

    7.2.g I'm afraid old bean. I need to schedule in the upgrade. I've had a look at the new interface, and you know what, I think we could live with it. With other work going on (we are rolling out ANOTHER new, green field, CRM implementation for a new business) I don't think we will be able to get to the upgrade for six months. Anything you can suggest to ease the App Pool blowing up?

  • 0

    Try changing change your app pool recycling to fixed rather than timed and set it alittle before the normal working day. While not a fix it might alleviate the issue.

  • 0

    Thanks, but we already do that as good house keeping. Check out the leak in CRM - it gets to 6gbs of private memory (red line) and boom.

    On code like this: QueryCols = CRM.CreateQueryObj(sColSQL); Is there a dispose method to use to clean this up?

  • 0

    Magical

    Sage CRM within its server side code uses the Windows Script Host to actually process the JavaScript. This means that JavaScript memory management is under its control.

    Giving the example

    QueryCols = CRM.CreateQueryObj(sColSQL);

    opens up the whole question of whether the Record object maybe a better object for you to use.

    But having said that when we run the code

    QueryCols = CRM.CreateQueryObj(sColSQL);

    At this point the JavaScript engine allocates memory for the object.

    You can set an object to null. When you do this it means that there are no references to it and so should ensure that it can be 'garbage collected'.

    Another thing to consider is whether there are any issues on the database - for example deadlocks or blocks could basically be choking the system.

  • 0

    Hi,

    Just a couple of notes here.

    The memory management change in 2018 R1 mainly affects metadata refreshes (it makes them more reliable). The old memory manager wasn't causing a leak per se, but it potentially resulted in errors happening as a result of a refresh.

    We fixed at least one memory leak in the eWare DLL in 7.3. Here's the case reference:

    0-165629-QA (fixed in 7.3 c hf3): This memory leak was in the function used to get the text of JavaScript functions in the page head. If this is the cause, you'd expect to see a steady increase in memory usage over time, which seems to be pretty much what you've got in your graph.

    I think that this might have also been back-fixed in case 0-166485-QA for 7.2j hotfix 1, so I'd suggest putting the latest 7.2j hotfix on there (hotfixes are cumulative), and see if you get an improvement.

    There's a bunch of other issues that might cause out of memory errors (DB blocks / deadlocks being one - unfulfilled user requests can pile up waiting on a critical section to become free), but I reckon that in the absence of any other catastrophic errors you've just got a fairly steady, standard memory leak.

    Hope this helps,

    Rob

  • 0

    Hello all,

    Thank you for the full response from both training and development at Sage - that level of support is one of the reasons we have continued investment into Sage CRM with another big instance coming online in summer.

    We have upgraded (with trepidation and much testing) to 7.3 SP3.8 and I am observing the private bytes of the App pool today. As you rightly said, with the older version, the bytes would creep up cumulatively over the day until they reached crisis point at circa 4:30 in the afternoon and the App Pool needed to be torched.

    I can see, even now, the base line for the app pool is much smaller. The 32 bit app pool of CRM can support a maximum of 2gb before it falls over, when the old version fired up, it would immediately take 1Gb, which would ramp up gently over the day as below (In the below days trace, we didn't quite blow the pool).

    In this version we are running at circa 200 mb, and I can see more proactive memory management going on (its been up to 218 mbs and its now come back down to 194mbs), giving lots of overhead to work with.

    Thanks again for the assist.

  • 0

    That's our pleasure.

    And after Sage CRM 7.3sp3 you'll be able to skip on to Sage CRM 2018 very easily!

  • 0 in reply to Sage CRM

    We are getting these errors again on a 2018 R3 - any suggestions? We are getting them about once a week... 

  • 0 in reply to MagicalArmchair

    And this has started to appear again?  I take it that is was working fine on your install of 2018 R3 but has just recently started to generate the error message.

  • 0 in reply to Sage CRM

    Correct, as we have purchased more users and the system gets busier, it appears to be becoming an issue again. We nightly recycled the app pool, however it’s generally around 4pm in the afternoon the app pool has run out of its space in the 32 bit app pool. If you watch it in perfmon you can see the space gradually get eaten up over the course of the day. Clearly something is leaking memory. 

  • 0 in reply to MagicalArmchair

    Is SQL on the same server? If the server is close to 100% RAM I tend to tell SQL to limit its Max RAM, on a different issue we've found the Sage 200 Classic Integration fails is the server is almost out of RAM so limiting the SQL server seems to have helped in those situations and wonder if this could be the same

  • 0 in reply to Matthew Shaw

    Hello Matthew, SQL is on the server and it is an Azure server with plenty of grunt. The issue is the private bytes maximum that the 32 bit application pool can use, and that is around 2gb. Over the course of the day, you can see this edge up as garbage collection fails to dispose entirely of everything loaded into memory until you reach the 2gb tipping point, at which point CRM gives the "Out of Memory" error and then needs to be recycled.

    A lot of applications handle recycles gracefully - CRM however logs everyone out and rebuilds metadata on first load, so it is painfully apparent when we have to recycle and gives all users 5 minutes of downtime and egg on our faces. 

    Any suggestions? We have made sure we dispose of any ActiveX objects fully and the like.