You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

GcOrphanFilesTest.java 3.1KB

Ensure that GC#deleteOrphans respects pack lock If pack or index files are guarded by a pack lock (.keep file) deleteOrphans() should not touch the respective files protected by the lock file. Otherwise it may interfere with PackInserter concurrently inserting a new pack file and its index. The problem was caused by the following race. All mentioned files are located in "objects/pack/". File endings relevant in "pack" dir: .pack .keep .idx .bitmap When ReceivePack receives a pack file it executes the following steps: ReceivePack.service(): receivePackAndCheckConnectivity(): receivePack(): receive the pack parse the pack, returns packLock (.keep file) PackInserter.flush(): write tmpPck file: "insert_<random>.pack" write tmpIdx file: "insert_<random>.idx" real pack name: "pack-<SHA1>.pack" real index name: "pack-<SHA1>.idx" atomic rename tmpPack to realPack atomic rename tmpIdx to tmpIdx execute commands unlock pack by removing .keep file trigger auto gc if enabled When PackInserter.flush() renames the temporary pack to the final "pack-xxx.pack" file the temporary pack index file "insert_xxx.idx" has no matching .pack file with the same base name for a short interval. If deleteOrphans() ran during that interval it deduced the pack index file was orphaned. Subsequently the missing pack index caused MissingObjectExceptions since objects contained in the pack couldn't be looked up anymore. Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=13544 Change-Id: I559c81e4b1d7c487f92a751bd78b987d32c98719 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 년 전
Ensure that GC#deleteOrphans respects pack lock If pack or index files are guarded by a pack lock (.keep file) deleteOrphans() should not touch the respective files protected by the lock file. Otherwise it may interfere with PackInserter concurrently inserting a new pack file and its index. The problem was caused by the following race. All mentioned files are located in "objects/pack/". File endings relevant in "pack" dir: .pack .keep .idx .bitmap When ReceivePack receives a pack file it executes the following steps: ReceivePack.service(): receivePackAndCheckConnectivity(): receivePack(): receive the pack parse the pack, returns packLock (.keep file) PackInserter.flush(): write tmpPck file: "insert_<random>.pack" write tmpIdx file: "insert_<random>.idx" real pack name: "pack-<SHA1>.pack" real index name: "pack-<SHA1>.idx" atomic rename tmpPack to realPack atomic rename tmpIdx to tmpIdx execute commands unlock pack by removing .keep file trigger auto gc if enabled When PackInserter.flush() renames the temporary pack to the final "pack-xxx.pack" file the temporary pack index file "insert_xxx.idx" has no matching .pack file with the same base name for a short interval. If deleteOrphans() ran during that interval it deduced the pack index file was orphaned. Subsequently the missing pack index caused MissingObjectExceptions since objects contained in the pack couldn't be looked up anymore. Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=13544 Change-Id: I559c81e4b1d7c487f92a751bd78b987d32c98719 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 년 전
Ensure that GC#deleteOrphans respects pack lock If pack or index files are guarded by a pack lock (.keep file) deleteOrphans() should not touch the respective files protected by the lock file. Otherwise it may interfere with PackInserter concurrently inserting a new pack file and its index. The problem was caused by the following race. All mentioned files are located in "objects/pack/". File endings relevant in "pack" dir: .pack .keep .idx .bitmap When ReceivePack receives a pack file it executes the following steps: ReceivePack.service(): receivePackAndCheckConnectivity(): receivePack(): receive the pack parse the pack, returns packLock (.keep file) PackInserter.flush(): write tmpPck file: "insert_<random>.pack" write tmpIdx file: "insert_<random>.idx" real pack name: "pack-<SHA1>.pack" real index name: "pack-<SHA1>.idx" atomic rename tmpPack to realPack atomic rename tmpIdx to tmpIdx execute commands unlock pack by removing .keep file trigger auto gc if enabled When PackInserter.flush() renames the temporary pack to the final "pack-xxx.pack" file the temporary pack index file "insert_xxx.idx" has no matching .pack file with the same base name for a short interval. If deleteOrphans() ran during that interval it deduced the pack index file was orphaned. Subsequently the missing pack index caused MissingObjectExceptions since objects contained in the pack couldn't be looked up anymore. Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=13544 Change-Id: I559c81e4b1d7c487f92a751bd78b987d32c98719 Signed-off-by: Matthias Sohn <matthias.sohn@sap.com>
3 년 전
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107
  1. /*
  2. * Copyright (C) 2017 Ericsson and others
  3. *
  4. * This program and the accompanying materials are made available under the
  5. * terms of the Eclipse Distribution License v. 1.0 which is available at
  6. * https://www.eclipse.org/org/documents/edl-v10.php.
  7. *
  8. * SPDX-License-Identifier: BSD-3-Clause
  9. */
  10. package org.eclipse.jgit.internal.storage.file;
  11. import static org.junit.Assert.assertFalse;
  12. import static org.junit.Assert.assertTrue;
  13. import java.io.File;
  14. import java.io.IOException;
  15. import org.junit.Before;
  16. import org.junit.Test;
  17. public class GcOrphanFilesTest extends GcTestCase {
  18. private static final String PACK = "pack";
  19. private static final String BITMAP_File_1 = PACK + "-1.bitmap";
  20. private static final String BITMAP_File_2 = PACK + "-2.bitmap";
  21. private static final String IDX_File_2 = PACK + "-2.idx";
  22. private static final String IDX_File_malformed = PACK + "-1234idx";
  23. private static final String KEEP_File_2 = PACK + "-2.keep";
  24. private static final String PACK_File_2 = PACK + "-2.pack";
  25. private static final String PACK_File_3 = PACK + "-3.pack";
  26. private File packDir;
  27. @Override
  28. @Before
  29. public void setUp() throws Exception {
  30. super.setUp();
  31. packDir = repo.getObjectDatabase().getPackDirectory();
  32. }
  33. @Test
  34. public void bitmapAndIdxDeletedButPackNot() throws Exception {
  35. createFileInPackFolder(BITMAP_File_1);
  36. createFileInPackFolder(IDX_File_2);
  37. createFileInPackFolder(PACK_File_3);
  38. gc.gc();
  39. assertFalse(new File(packDir, BITMAP_File_1).exists());
  40. assertFalse(new File(packDir, IDX_File_2).exists());
  41. assertTrue(new File(packDir, PACK_File_3).exists());
  42. }
  43. @Test
  44. public void bitmapDeletedButIdxAndPackNot() throws Exception {
  45. createFileInPackFolder(BITMAP_File_1);
  46. createFileInPackFolder(IDX_File_2);
  47. createFileInPackFolder(PACK_File_2);
  48. createFileInPackFolder(PACK_File_3);
  49. gc.gc();
  50. assertFalse(new File(packDir, BITMAP_File_1).exists());
  51. assertTrue(new File(packDir, IDX_File_2).exists());
  52. assertTrue(new File(packDir, PACK_File_2).exists());
  53. assertTrue(new File(packDir, PACK_File_3).exists());
  54. }
  55. @Test
  56. public void malformedIdxNotDeleted() throws Exception {
  57. createFileInPackFolder(IDX_File_malformed);
  58. gc.gc();
  59. assertTrue(new File(packDir, IDX_File_malformed).exists());
  60. }
  61. @Test
  62. public void keepPreventsDeletionOfIndexFilesForMissingPackFile()
  63. throws Exception {
  64. createFileInPackFolder(BITMAP_File_1);
  65. createFileInPackFolder(IDX_File_2);
  66. createFileInPackFolder(BITMAP_File_2);
  67. createFileInPackFolder(KEEP_File_2);
  68. createFileInPackFolder(PACK_File_3);
  69. gc.gc();
  70. assertFalse(new File(packDir, BITMAP_File_1).exists());
  71. assertTrue(new File(packDir, BITMAP_File_2).exists());
  72. assertTrue(new File(packDir, IDX_File_2).exists());
  73. assertTrue(new File(packDir, KEEP_File_2).exists());
  74. assertTrue(new File(packDir, PACK_File_3).exists());
  75. }
  76. private void createFileInPackFolder(String fileName) throws IOException {
  77. if (!packDir.exists() || !packDir.isDirectory()) {
  78. assertTrue(packDir.mkdirs());
  79. }
  80. assertTrue(new File(packDir, fileName).createNewFile());
  81. }
  82. @Test
  83. public void noSuchPackFolder() throws Exception {
  84. assertTrue(packDir.delete());
  85. gc.gc();
  86. }
  87. }