One of my main problems while connecting to a serial device was a software called ModemManager which automatically opened a connection to my devices on plug-in.

The solution was to disable or remove ModemManager since I never used one of these old dialup modems with my PCs anyways.

Remove ModemManager:

apt-get purge modemmanager

or simply stop it:

systemctl stop ModemManager