Do your access rules work in both directions? Do you have any strange routing going on? Do you need to configure a static route for the returning traffic?
Sorry, knowing very little about your setup means i can only suggest vague possibilities based on networks i work with.
Oh yeah! That too! Flick the bean!