What's probably happening here is your adapter is signaling some button press when you connect it that's popping up the Assistant. Only way to fix that is to get a new adapter since you can't manually disable that input from triggering the Assistant without root.
However, you can completely disable the Google Assistant from appearing at all via:
Settings > Apps > Default Apps > Digital Assistant App > Default Digital Assistant App > None
That should fully disable it in all regards and it'll never pop up again.