ColdFusion Server crash after thousands of HTTP requests

coldfusionhttpwindows-server-2003

We are running ColdFusion 8 on a windows server 2003 VPS with an API that exposes student records to a partner API through a connector. Our API returns around 50k student records serialized in XML format pretty seamlessly. My question originates when something very frightening happened today when we tested our connector to our partners API. Our entire website and web host went down. We assumed that our host was just having some issues and after 4 hours with no resolution and no response from their customer service we finally got a response from them claiming that they had an "unauthorized user" in their network. After our server was back up we were unable to connect to our website as if the web service or coldfusion itself had froze. This is really where my concern comes from as I fear we may have overloaded the web service.

As I mentioned before we tried sending over 50k HTTP POST requests over to our partner's API, however everything stopped after around 1.6k

Is this bad practice or is there some sort of rate limiting I can relax somewhere in server configuration? We managed to find a workaround, but it bypasses our connector which is critical to our design.

This would have been a one time deal as the purpose of so many requests was to populate our partner's website with current data, after that hourly syncs will keep requests down to around 100 per hour.

UPDATE

Our partner API is owned and operated by Pardot. We are converting students to prospects by passing student data to their API which unfortunately only seems to accept one student at a time. For that reason we have to do all 50k requests individually.

Our server has 4GB of RAM, an Intel Core 2 Duo @ 2.8GHz running Windows Server 2003 SP2.

I monitored the server during a 100 student sync, a 400 student sync, and a 1.4k student sync with the following results:

100 students – 2.25GB of Memory, 30-40% CPU utilization, 0.2-0.3% network bandwidth

400 students – 2.30GB of Memory, 30-50% CPU utilization, 0.2-1.0% network bandwidth

1.4k students – 2.30GB of Memory, 30-70% CPU utilization, 0.2-1.0% network bandwidth

I know this is a far cry from 50k students, but I don't want to risk taking down our CMS system again assuming that was the cause.

To give you a look at our code:

<cfif (#getStudents.statusCode# eq "200 OK")>   
    <cftry>
        <cfloop index="StudentXML" array="#XmlSearch(responseSTUD,'/students/student')#">
            <cfset StudentXML = XmlParse(StudentXML)>
            <cfhttp
                url="#PARDOT_CMS_UPSERT#"
                method="post" 
                timeout="10000" >
                <cfhttpparam type="url" name="user_key" value="#PARDOT_CMS_USERKEY#">
                <cfhttpparam type="url" name="api_key" value="#api_key#">
                <cfhttpparam type="url" name="email" value="#StudentXML.student.email.XmlText#">
                <cfhttpparam type="url" name="first_name" value="#StudentXML.student.first.XmlText#">
                <cfhttpparam type="url" name="last_name" value="#StudentXML.student.last.XmlText#">
                <cfhttpparam type="url" name="in_cms" value="#StudentXML.student.studentid.XmlText#">
                <cfhttpparam type="url" name="company" value="#StudentXML.student.agencyname.XmlText#">
                <cfhttpparam type="url" name="country" value="#StudentXML.student.countryname.XmlText#">
                <cfhttpparam type="url" name="address_one" value="#StudentXML.student.address.XmlText#">
                <cfhttpparam type="url" name="address_two" value="#StudentXML.student.address2.XmlText#">
                <cfhttpparam type="url" name="city" value="#StudentXML.student.city.XmlText#">
                <cfhttpparam type="url" name="state" value="#StudentXML.student.state_province.XmlText#">
                <cfhttpparam type="url" name="zip" value="#StudentXML.student.postalcode.XmlText#">
                <cfhttpparam type="url" name="phone" value="#StudentXML.student.phone.XmlText#">
                <cfhttpparam type="url" name="fax" value="#StudentXML.student.fax.XmlText#">
                <cfhttpparam type="url" name="output" value="simple">
            </cfhttp>
        </cfloop>
    <cfcatch type="any">
        <cfdump var="#cfcatch.Message#">
    </cfcatch>
    </cftry>
</cfif>

UPDATE 2

I checked the CF logs and found a couple of these:

"Error","jrpp-8","06/06/13","16:10:18","CMS-API","Java heap space The specific sequence   of files included or processed is: D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm, line: 675 "
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.io.CharArrayWriter.write(CharArrayWriter.java:105)
at coldfusion.runtime.CharBuffer.replace(CharBuffer.java:37)
at coldfusion.runtime.CharBuffer.replace(CharBuffer.java:50)
at coldfusion.runtime.NeoBodyContent.write(NeoBodyContent.java:254)
at         cfapi2ecfm292155732._factor30(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:675)
at   cfapi2ecfm292155732._factor31(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:662)
at cfapi2ecfm292155732._factor36(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:659)
at cfapi2ecfm292155732._factor42(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:657)
at cfapi2ecfm292155732._factor37(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm)
at cfapi2ecfm292155732._factor44(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:456)
at cfapi2ecfm292155732._factor38(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm)
at cfapi2ecfm292155732._factor46(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:455)
at cfapi2ecfm292155732._factor39(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm)
at cfapi2ecfm292155732._factor47(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:453)
at cfapi2ecfm292155732.runPage(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:1)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:192)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:366)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.CfmServlet.service(CfmServlet.java:175)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)

Looks like I might have crashed the JVM in CF, is there a better way to do this? We are thinking of just exporting all records initially as a CSV file and importing it into Pardot seeing as we will never have to do a request this large again.

Best Answer

Have you monitored CPU, disk and bandwidth usage on your server? The first thing I'd do is make sure none of these resources are being overloaded. Perfmon is an obvious place to start and check out your CF logs.

Were you making 50K post requests (i.e. separate requests) one after the other or is this a single request for the 50K records in an XML file? If the former then why would you make 50K requests? Could it be done by requesting all 50K records at once as that would be just 1 request and 1 file transfer? If the 50K requests are sequential (i.e. the next request isn't made until the current one is complete) then that seems like a gentle way of making a large number of requests.

Related Topic