Considering the VisualStudio gitignore file contains a line "**/[Pp]ackages/*", these 2 lines from Umbraco gitignore never get applied properly to the desired files, even though Umbraco lines were added after the VS lines. I guess the VS's line is more specific, so it gets a priority of some sort.
# Don't ignore Umbraco packages (VisualStudio.gitignore mistakes this for a NuGet packages folder)
# Make sure to include details from VisualStudio.gitignore BEFORE this
-!**/App_Data/[Pp]ackages/
-!**/[Uu]mbraco/[Dd]eveloper/[Pp]ackages
+!**/App_Data/[Pp]ackages/*
+!**/[Uu]mbraco/[Dd]eveloper/[Pp]ackages/*
# ImageProcessor DiskCache
**/App_Data/cache/