A group must be void of employees before it can be deleted. First assign the group’s employees to other groups. After a group has zero employees, the [delete] button will show up on the Groups page.
Deleting a group won’t remove the Manager from your account. You can assign them as Manager of another group. To remove a Manager, read this article for more information.