Τελικά ήταν ευκολότερο από ότι φανταζόμουνα :
Installer και ένα ακόμη custom action με τα εξής overrides :

Public Overrides Sub Install(ByVal stateSaver As System.Collections.IDictionary)
   Dim appId As String = Nothing
   Dim typeLib As String = Nothing
   Dim MyAssembly As String = Me.GetType.Assembly.Location
   Dim RegHelp As New RegistrationHelper

   With stateSaver
      .Add("AppID", appId)
      .Add("Assembly", MyAssembly)
   End With

End Sub

Public Overrides Sub Uninstall(ByVal savedState As System.Collections.IDictionary)
   Dim appId As String = CStr(savedState("AppID"))
   Dim MyAssembly As String = CStr(savedState("Assembly"))
   Dim RegHelp As New RegistrationHelper

      RegHelp.UninstallAssembly(MyAssembly, appId)

End Sub

Κύριοι έχω τις ευλογίες σας ; ή ζω illusions of grandure;

Συγνώμη ξέχασα ... προσθήκη στο GAC ...
Στο setup project : 
Add Special folder (Global Assembly Cache folder) και προσθήκη εκεί μέσα του assembly μας ....