kafka.log

OffsetIndex

class OffsetIndex extends Logging

An index that maps offsets to physical file locations for a particular log segment. This index may be sparse: that is it may not hold an entry for all messages in the log.

The index is stored in a file that is pre-allocated to hold a fixed maximum number of 8-byte entries.

The index supports lookups against a memory-map of this file. These lookups are done using a simple binary search variant to locate the offset/location pair for the greatest offset less than or equal to the target offset.

Index files can be opened in two ways: either as an empty, mutable index that allows appends or an immutable read-only index file that has previously been populated. The makeReadOnly method will turn a mutable file into an immutable one and truncate off any extra bytes. This is done when the index file is rolled over.

No attempt is made to checksum the contents of this file, in the event of a crash it is rebuilt.

The file format is a series of entries. The physical format is a 4 byte "relative" offset and a 4 byte file location for the message with that offset. The offset stored is relative to the base offset of the index file. So, for example, if the base offset was 50, then the offset 55 would be stored as 5. Using relative offsets in this way let's us use only 4 bytes for the offset.

The frequency of entries is up to the user of this class.

All external APIs translate from relative offsets to full offsets, so users of this class do not interact with the internal storage format.

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. OffsetIndex
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new OffsetIndex(file: File, baseOffset: Long, maxIndexSize: Int = -1)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def append(offset: Long, position: Int): Unit

    Append an entry for the given offset/location pair to the index.

    Append an entry for the given offset/location pair to the index. This entry must have a larger offset than all subsequent entries.

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. val baseOffset: Long

  9. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  10. def close(): Unit

    Close the index

  11. def debug(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  12. def debug(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  13. def debug(msg: ⇒ String): Unit

    Definition Classes
    Logging
  14. def delete(): Boolean

    Delete this index file

  15. def entries(): Int

    The number of entries in this index

  16. def entry(n: Int): OffsetPosition

    Get the nth offset mapping from the index

    Get the nth offset mapping from the index

    n

    The entry number in the index

    returns

    The offset/position pair at that entry

  17. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  18. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  19. def error(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  20. def error(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  21. def error(msg: ⇒ String): Unit

    Definition Classes
    Logging
  22. def fatal(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  23. def fatal(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  24. def fatal(msg: ⇒ String): Unit

    Definition Classes
    Logging
  25. var file: File

  26. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  27. def flush(): Unit

    Flush the data in the index to disk

  28. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  29. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  30. def info(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  31. def info(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  32. def info(msg: ⇒ String): Unit

    Definition Classes
    Logging
  33. def isFull: Boolean

    True iff there are no more slots available in this index

  34. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  35. var lastOffset: Long

  36. var logIdent: String

    Attributes
    protected
    Definition Classes
    Logging
  37. lazy val logger: Logger

    Definition Classes
    Logging
  38. val loggerName: String

    Definition Classes
    Logging
  39. def lookup(targetOffset: Long): OffsetPosition

    Find the largest offset less than or equal to the given targetOffset and return a pair holding this offset and it's corresponding physical file position.

    Find the largest offset less than or equal to the given targetOffset and return a pair holding this offset and it's corresponding physical file position.

    targetOffset

    The offset to look up.

    returns

    The offset found and the corresponding file position for this offset. If the target offset is smaller than the least entry in the index (or the index is empty), the pair (baseOffset, 0) is returned.

  40. var maxEntries: Int

    The maximum number of eight-byte entries this index can hold

  41. val maxIndexSize: Int

  42. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  43. final def notify(): Unit

    Definition Classes
    AnyRef
  44. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  45. def readLastEntry(): OffsetPosition

    The last entry in the index

  46. def renameTo(f: File): Boolean

    Rename the file that backs this offset index

    Rename the file that backs this offset index

    returns

    true iff the rename was successful

  47. def resize(newSize: Int): Unit

    Reset the size of the memory map and the underneath file.

    Reset the size of the memory map and the underneath file. This is used in two kinds of cases: (1) in trimToValidSize() which is called at closing the segment or new segment being rolled; (2) at loading segments from disk or truncating back to an old segment where a new log segment became active; we want to reset the index size to maximum index size to avoid rolling new segment.

  48. def sanityCheck(): Unit

    Do a basic sanity check on this index to detect obvious problems

    Do a basic sanity check on this index to detect obvious problems

    Exceptions thrown
    IllegalArgumentException

    if any problems are found

  49. def sizeInBytes(): Int

    The number of bytes actually used by this index

  50. def swallow(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  51. def swallowDebug(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  52. def swallowError(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  53. def swallowInfo(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  54. def swallowTrace(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  55. def swallowWarn(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  56. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  57. def toString(): String

    Definition Classes
    AnyRef → Any
  58. def trace(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  59. def trace(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  60. def trace(msg: ⇒ String): Unit

    Definition Classes
    Logging
  61. def trimToValidSize(): Unit

    Trim this segment to fit just the valid entries, deleting all trailing unwritten bytes from the file.

  62. def truncate(): Unit

    Truncate the entire index, deleting all entries

  63. def truncateTo(offset: Long): Unit

    Remove all entries from the index which have an offset greater than or equal to the given offset.

    Remove all entries from the index which have an offset greater than or equal to the given offset. Truncating to an offset larger than the largest in the index has no effect.

  64. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  65. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  66. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  67. def warn(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  68. def warn(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  69. def warn(msg: ⇒ String): Unit

    Definition Classes
    Logging

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped