VsFTPd – LDAP and PAM Integration

I am trying to configure a VsFTPd server to authenticate agains an LDAP server. It may be easy, but since it is the first time that I am using both LDAP and PAM, I have some difficulties. VsFTPd runs on an Ubuntu Server 11.04 and the LDAP is OpenLDAP on an 10.10 Ubuntu Server. I disabled AppArmor on the first one.
VsFTPd cannot connect to the LDAP server, in my syslog I have:

vsftpd: pam_ldap: ldap_simple_bind Can't contact LDAP server

The LDAP server is OK since I can do an ldapsearch.

Here is my /etc/pam.d/vsftpd file:

auth required pam_listfile.so item=user sense=deny file=/etc/ftpusers onerr=succeed
@include common-account
@include common-session
@include common-auth
auth required pam_ldap.so
account required pam_ldap.so
session required pam_ldap.so
password required pam_ldap.so

And here is my /etc/ldap.conf file:

base dc=example,dc=com
uri ldapi:///ldap.example.com
ldap_version 3
rootbinddn cn=admin,dc=example,dc=com
pam_password md5
nss_initgroups_ignoreusers a_bunch_of_system_users

Can anyone help me please ?
Thank you.

EDIT: new version of the /etc/pam.d/vsftpd file:

auth    required    pam_listfile.so item=user sense=deny file=/etc/ftpusers onerr=succeed

account required    pam_unix.so
account sufficient  pam_ldap.so

session required    pam_limits.so
session required    pam_unix.so
session optimal     pam_ldap.so

auth    required    pam_env.so
auth    sufficient  pam_unix.so nullok_secure
auth    sufficient  pam_ldap.so use_first_pass

auth    required    pam_shells.so