Hi,
please check this note 305201. Also please check when the documents are created in FI side then this table BSIP should contain the entry for that document so that next time if similar document is getting created then it will display the error or warning message based on the setting.
Note: In the vendor master data 'Double invoice check' field should be marked Payment transaction accounting tab. More over you can debugg this function module FI_DUPLICATE_INVOICE_CHECK
to find the reason why duplicate invoice is not working. This shold solves the issue.
Best regards,
Mithlesh