PIN lockout - the number of pin guessing attempts is limited. Malware could attempt user and admin PIN guesses and lock the card making it useless.
PIN capture - the host using the card can be compromised and allow capture of the PIN. This would give access to the token for as long as it is connected.
Data capture - private keys are secured on the device and cannot be copied. Other data (e.g. web user names and passwords) may be stored on the card and it may be subject to compromise.
Token loss
Not a big deal for signature keys. Just replace them and revoke the old one.
Data loss may occur if encryption keys are lost. Any encryption system needs to consider data recovery.
Data loss of application data. Application data stored on the card (e.g. web user names and passwords) would be lost if not backed up.
Tools are available to monitor card access, e.g. pkcs11-spy.
General Issues
How bad is it to leave your token plugged in all the time?
Middleware may have a configurable PIN timeout, e.g. after 1/2 hour you have to enter your PIN again.
A plugged-in token is vulnerable to being locked out by a malicious user.
Tokens (and all USB devices) Can Be Shared Over The Network
What does this mean to our understanding of what a signed document really proves?
Trust
The integrity of the host platform must be assured or an attacker who controls the platform could sign documents if the token is present and the PIN has been entered.
External readers with secure pin pads eliminate some of this risk.
The user has no way to know what data he is signing. What is displayed may not be what he is actually signing.
Can biometric auth be used rather than certificates on tokens
PB: Still provides something better than PID and password.
PB: Two big issues with tokens is that users will likely leave them plugged in all the time or they will not have them with them.