Not sure about whether it's the ACL that would prevent that - thought the fact that the problem appears when you change it would be suspicious, I agree... I'd guess that it's using the loopback address and not the "real" IP address, which is normal for a Linux system. That means it's using 127.0.0.1, so try adding a rule for that as well.
You can confirm whether it's the ACL in the debug messages, as I think something is written out to the log if wrong credentials are given - in this case, a not-allowed address.