Italy's appeal courts have deemed returning migrants to Libya illegal after an Italian captain allegedly forced over 100 people back to Libya.