to drift apart

 to become less close to someone.

» Glossary 3 Friends