You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

ObjectIdOwnerMapTest.java 5.9KB

ObjectIdOwnerMap: More lightweight map for ObjectIds OwnerMap is about 200 ms faster than SubclassMap, more friendly to the GC, and uses less storage: testing the "Counting objects" part of PackWriter on 1886362 objects: ObjectIdSubclassMap: load factor 50% table: 4194304 (wasted 2307942) ms spent 36998 36009 34795 34703 34941 35070 34284 34511 34638 34256 ms avg 34800 (last 9 runs) ObjectIdOwnerMap: load factor 100% table: 2097152 (wasted 210790) directory: 1024 ms spent 36842 35112 34922 34703 34580 34782 34165 34662 34314 34140 ms avg 34597 (last 9 runs) The major difference with OwnerMap is entries must extend from ObjectIdOwnerMap.Entry, where the OwnerMap has injected its own private "next" field into each object. This allows the OwnerMap to use a singly linked list for chaining collisions within a bucket. By putting collisions in a linked list, we gain the entire table back for the SHA-1 bits to index their own "private" slot. Unfortunately this means that each object can appear in at most ONE OwnerMap, as there is only one "next" field within the object instance to thread into the map. For types that are very object map heavy like RevWalk (entity RevObject) and PackWriter (entity ObjectToPack) this is sufficient, these entity types are only put into one map by their container. By introducing a new map type, we don't break existing applications that might be trying to use ObjectIdSubclassMap to track RevCommits they obtained from a RevWalk. The OwnerMap uses less memory. Each object uses 1 reference more (so we're up 1,886,362 references), but the table is 1/2 the size (2^20 rather than 2^21). The table itself wastes only 210,790 slots, rather than 2,307,942. So OwnerMap is wasting 200k fewer references. OwnerMap is more friendly to the GC, because it hardly ever generates garbage. As the map reaches its 100% load factor target, it doubles in size by allocating additional segment arrays of 2048 entries. (So the first grow allocates 1 segment, second 2 segments, third 4 segments, etc.) These segments are hooked into the pre-allocated directory of 1024 spaces. This permits the map to grow to 2 million objects before the directory itself has to grow. By using segments of 2048 entries, we are asking the GC to acquire 8,204 bytes in a 32 bit JVM. This is easier to satisfy then 2,307,942 bytes (for the 512k table that is just an intermediate step in the SubclassMap). By reusing the previously allocated segments (they are re-hashed in-place) we don't release any memory during a table grow. When the directory grows, it does so by discarding the old one and using one that is 4x larger (so the directory goes to 4096 entries on its first grow). A directory of size 4096 can handle up to 8 millon objects. The second directory grow (16384) goes to 33 million objects. At that point we're starting to really push the limits of the JVM heap, but at least its many small arrays. Previously SubclassMap would need a table of 67108864 entries to handle that object count, which needs a single contiguous allocation of 256 MiB. That's hard to come by in a 32 bit JVM. Instead OwnerMap uses 8192 arrays of about 8 KiB each. This is much easier to fit into a fragmented heap. Change-Id: Ia4acf5cfbf7e9b71bc7faa0db9060f6a969c0c50 Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
13 years ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217
  1. /*
  2. * Copyright (C) 2011, Google Inc.
  3. * and other copyright owners as documented in the project's IP log.
  4. *
  5. * This program and the accompanying materials are made available
  6. * under the terms of the Eclipse Distribution License v1.0 which
  7. * accompanies this distribution, is reproduced below, and is
  8. * available at http://www.eclipse.org/org/documents/edl-v10.php
  9. *
  10. * All rights reserved.
  11. *
  12. * Redistribution and use in source and binary forms, with or
  13. * without modification, are permitted provided that the following
  14. * conditions are met:
  15. *
  16. * - Redistributions of source code must retain the above copyright
  17. * notice, this list of conditions and the following disclaimer.
  18. *
  19. * - Redistributions in binary form must reproduce the above
  20. * copyright notice, this list of conditions and the following
  21. * disclaimer in the documentation and/or other materials provided
  22. * with the distribution.
  23. *
  24. * - Neither the name of the Eclipse Foundation, Inc. nor the
  25. * names of its contributors may be used to endorse or promote
  26. * products derived from this software without specific prior
  27. * written permission.
  28. *
  29. * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
  30. * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
  31. * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
  32. * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
  33. * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
  34. * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
  35. * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
  36. * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
  37. * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
  38. * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
  39. * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
  40. * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
  41. * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  42. */
  43. package org.eclipse.jgit.lib;
  44. import static org.junit.Assert.assertEquals;
  45. import static org.junit.Assert.assertFalse;
  46. import static org.junit.Assert.assertNotNull;
  47. import static org.junit.Assert.assertSame;
  48. import static org.junit.Assert.assertTrue;
  49. import static org.junit.Assert.fail;
  50. import java.util.Iterator;
  51. import java.util.NoSuchElementException;
  52. import org.junit.Before;
  53. import org.junit.Test;
  54. public class ObjectIdOwnerMapTest {
  55. private MutableObjectId idBuf;
  56. private SubId id_1, id_2, id_3, id_a31, id_b31;
  57. @Before
  58. public void init() {
  59. idBuf = new MutableObjectId();
  60. id_1 = new SubId(id(1));
  61. id_2 = new SubId(id(2));
  62. id_3 = new SubId(id(3));
  63. id_a31 = new SubId(id(31));
  64. id_b31 = new SubId(id((1 << 8) + 31));
  65. }
  66. @Test
  67. public void testEmptyMap() {
  68. ObjectIdOwnerMap<SubId> m = new ObjectIdOwnerMap<SubId>();
  69. assertTrue(m.isEmpty());
  70. assertEquals(0, m.size());
  71. Iterator<SubId> i = m.iterator();
  72. assertNotNull(i);
  73. assertFalse(i.hasNext());
  74. assertFalse(m.contains(id(1)));
  75. }
  76. @Test
  77. public void testAddGetAndContains() {
  78. ObjectIdOwnerMap<SubId> m = new ObjectIdOwnerMap<SubId>();
  79. m.add(id_1);
  80. m.add(id_2);
  81. m.add(id_3);
  82. m.add(id_a31);
  83. m.add(id_b31);
  84. assertFalse(m.isEmpty());
  85. assertEquals(5, m.size());
  86. assertSame(id_1, m.get(id_1));
  87. assertSame(id_1, m.get(id(1)));
  88. assertSame(id_1, m.get(id(1).copy()));
  89. assertSame(id_2, m.get(id(2).copy()));
  90. assertSame(id_3, m.get(id(3).copy()));
  91. assertSame(id_a31, m.get(id(31).copy()));
  92. assertSame(id_b31, m.get(id_b31.copy()));
  93. assertTrue(m.contains(id_1));
  94. }
  95. @Test
  96. public void testClear() {
  97. ObjectIdOwnerMap<SubId> m = new ObjectIdOwnerMap<SubId>();
  98. m.add(id_1);
  99. assertSame(id_1, m.get(id_1));
  100. m.clear();
  101. assertTrue(m.isEmpty());
  102. assertEquals(0, m.size());
  103. Iterator<SubId> i = m.iterator();
  104. assertNotNull(i);
  105. assertFalse(i.hasNext());
  106. assertFalse(m.contains(id(1)));
  107. }
  108. @Test
  109. public void testAddIfAbsent() {
  110. ObjectIdOwnerMap<SubId> m = new ObjectIdOwnerMap<SubId>();
  111. m.add(id_1);
  112. assertSame(id_1, m.addIfAbsent(new SubId(id_1)));
  113. assertEquals(1, m.size());
  114. assertSame(id_2, m.addIfAbsent(id_2));
  115. assertEquals(2, m.size());
  116. assertSame(id_a31, m.addIfAbsent(id_a31));
  117. assertSame(id_b31, m.addIfAbsent(id_b31));
  118. assertSame(id_a31, m.addIfAbsent(new SubId(id_a31)));
  119. assertSame(id_b31, m.addIfAbsent(new SubId(id_b31)));
  120. assertEquals(4, m.size());
  121. }
  122. @Test
  123. public void testAddGrowsWithObjects() {
  124. int n = 16384;
  125. ObjectIdOwnerMap<SubId> m = new ObjectIdOwnerMap<SubId>();
  126. m.add(id_1);
  127. for (int i = 32; i < n; i++)
  128. m.add(new SubId(id(i)));
  129. assertEquals(n - 32 + 1, m.size());
  130. assertSame(id_1, m.get(id_1.copy()));
  131. for (int i = 32; i < n; i++)
  132. assertTrue(m.contains(id(i)));
  133. }
  134. @Test
  135. public void testAddIfAbsentGrowsWithObjects() {
  136. int n = 16384;
  137. ObjectIdOwnerMap<SubId> m = new ObjectIdOwnerMap<SubId>();
  138. m.add(id_1);
  139. for (int i = 32; i < n; i++)
  140. m.addIfAbsent(new SubId(id(i)));
  141. assertEquals(n - 32 + 1, m.size());
  142. assertSame(id_1, m.get(id_1.copy()));
  143. for (int i = 32; i < n; i++)
  144. assertTrue(m.contains(id(i)));
  145. }
  146. @Test
  147. public void testIterator() {
  148. ObjectIdOwnerMap<SubId> m = new ObjectIdOwnerMap<SubId>();
  149. m.add(id_1);
  150. m.add(id_2);
  151. m.add(id_3);
  152. Iterator<SubId> i = m.iterator();
  153. assertTrue(i.hasNext());
  154. assertSame(id_1, i.next());
  155. assertTrue(i.hasNext());
  156. assertSame(id_2, i.next());
  157. assertTrue(i.hasNext());
  158. assertSame(id_3, i.next());
  159. assertFalse(i.hasNext());
  160. try {
  161. i.next();
  162. fail("did not fail on next with no next");
  163. } catch (NoSuchElementException expected) {
  164. // OK
  165. }
  166. i = m.iterator();
  167. assertSame(id_1, i.next());
  168. try {
  169. i.remove();
  170. fail("did not fail on remove");
  171. } catch (UnsupportedOperationException expected) {
  172. // OK
  173. }
  174. }
  175. private AnyObjectId id(int val) {
  176. idBuf.setByte(0, val & 0xff);
  177. idBuf.setByte(3, (val >>> 8) & 0xff);
  178. return idBuf;
  179. }
  180. private static class SubId extends ObjectIdOwnerMap.Entry {
  181. SubId(AnyObjectId id) {
  182. super(id);
  183. }
  184. }
  185. }