One of my main problems while connecting to a serial device was a software called ModemManager which automatically opened a connection to my devices on plug-in.
The solution was to disable or remove ModemManager since I never used one of these old dialup modems with my PCs anyways.
apt-get purge modemmanager
or simply stop it:
systemctl stop ModemManager