Been noticing quite a bit of network slowness in Lion, particularly Lion Server. If you are too, try this:
Open your Network System Preference. Are you getting your IP info from a DHCP Server? Click Advanced, and click the DNS tab. Even if the DNS server address and subnet are correct, click the + and type them in again, just to make sure they stick. It helped me. Server app and ServerAdmin are a lot faster now.