You can always do that if you trust the person enough. The idea with exchanges is essentially to act as an escrow service, but one where the two parties don't have to directly know who each other is. I don't think you can decentralize trust, realistically. Someone ultimately has to be responsible for a bad transaction or handling of BTC.