You are right, SQL Server does play well with others. It is reluctant to release memory once allocated.
There are options in SQL Server where you can limit the maximum memory it will use. By default this is all the memory in the system.
You could try to limit SQL to 1GB of memory. This may slow SQL somewhat (smaller cache = more disk hits) but may speed up the machine as a whole by allowing other processes more memory and reducing page faults.