Like you, I am seeing variable response times for my same orchestration to run. My test used a simple HelloWorld type of orchestration which featured a call to a custom .NET assembly which only returned a different string every time when the orch calls it.
My orch consistently took 6 to 8 seconds on the first run, and then just under 1 second on all successive calls. I tried it with two kinds of ports: a file-drop, and a web service. Both port types showed roughly the same speeds for the request-response cycle. The same pattern always occurs here: The first run is slow, and the rest of the invocations are fast. Everything is running on one basic development workstation on WinXP with no network hops.
You might check that your paging file in WinXP is smaller (preferably much smaller) than 1 GB during your tests because I have seen low memory conditions and paging to disk really bog my computer. Also might check that your antivirus is not hogging the cpu and disk resources during tests.