Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Corruption of index files (double get and IndexOutOfBounds errors) #101

Open
dchenbecker opened this issue Jan 31, 2013 · 8 comments
Open

Comments

@dchenbecker
Copy link

We're using JDBM with a large number (15k+) of distinct DBs (each with a single TreeMap instance) and while overall they're working well, we're getting what appears to be intermittent corruption of the files. When they corrupt, we can open and read values from the files just fine, but if we attempt to add values to the map and then commit, we get one of a number of errors:

java.lang.Error: double get for page 101
at org.apache.jdbm.PageFile.get(PageFile.java:141) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.PhysicalRowIdManager.allocNew(PhysicalRowIdManager.java:163) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.PhysicalRowIdManager.alloc(PhysicalRowIdManager.java:138) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.PhysicalRowIdManager.insert(PhysicalRowIdManager.java:57) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.insert2(DBStore.java:255) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.insert(DBStore.java:219) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBCache.insert(DBCache.java:75) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.writeValues(BTreeNode.java:1040) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.serialize(BTreeNode.java:990) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.serialize(BTreeNode.java:43) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.update2(DBStore.java:333) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.update(DBStore.java:317) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBCache.updateCacheEntries(DBCache.java:146) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBCache.commit(DBCache.java:100) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]

java.lang.IndexOutOfBoundsException: null
at java.nio.Buffer.checkIndex(Buffer.java:538) ~[na:1.7.0_09]
at java.nio.DirectByteBuffer.getShort(DirectByteBuffer.java:584) ~[na:1.7.0_09]
at org.apache.jdbm.PageIo.readShort(PageIo.java:172) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.RecordHeader.getAvailableSize(RecordHeader.java:72) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.PhysicalRowIdManager.allocNew(PhysicalRowIdManager.java:179) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.PhysicalRowIdManager.alloc(PhysicalRowIdManager.java:138) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.PhysicalRowIdManager.insert(PhysicalRowIdManager.java:57) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.insert2(DBStore.java:255) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.insert(DBStore.java:219) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBCache.insert(DBCache.java:75) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.writeValues(BTreeNode.java:1040) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.serialize(BTreeNode.java:990) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.serialize(BTreeNode.java:43) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.update2(DBStore.java:333) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.update(DBStore.java:317) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBCache.updateCacheEntries(DBCache.java:146) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBCache.commit(DBCache.java:100) ~[shard-assembly-2.3.0-1-g9ca4462.jar:na]

I can provide copies of the corrupted files if that helps.

@jankotek
Copy link
Owner

JDBM3 is no longer supported. Please migrate to MapDB (aka JDBM4)

On Wednesday 30 January 2013 16:45:37 Derek Chen-Becker wrote:

We're using JDBM with a large number (15k+) of distinct DBs (each with a
single TreeMap instance) and while overall they're working well, we're getting
what appears to be intermittent corruption of the files. When they corrupt, we
can open and read values from the files just fine, but if we attempt to add
values to the map and then commit, we get one of a number of errors:
java.lang.Error: double get for page 101
at org.apache.jdbm.PageFile.get(PageFile.java:141) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at
org.apache.jdbm.PhysicalRowIdManager.allocNew(PhysicalRowIdManager.java:163)
~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.PhysicalRowIdManager.alloc(PhysicalRowIdManager.java:138)
~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.PhysicalRowIdManager.insert(PhysicalRowIdManager.java:57)
~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.insert2(DBStore.java:255) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at org.apache.jdbm.DBStore.insert(DBStore.java:219) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at org.apache.jdbm.DBCache.insert(DBCache.java:75) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.writeValues(BTreeNode.java:1040) ~[shard-
assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.serialize(BTreeNode.java:990) ~[shard-
assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.serialize(BTreeNode.java:43) ~[shard-
assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.update2(DBStore.java:333) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at org.apache.jdbm.DBStore.update(DBStore.java:317) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at org.apache.jdbm.DBCache.updateCacheEntries(DBCache.java:146) ~[shard-
assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBCache.commit(DBCache.java:100) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
java.lang.IndexOutOfBoundsException: null
at java.nio.Buffer.checkIndex(Buffer.java:538) ~[na:1.7.0_09]
at java.nio.DirectByteBuffer.getShort(DirectByteBuffer.java:584) ~[na:1.7.0_09]
at org.apache.jdbm.PageIo.readShort(PageIo.java:172) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at org.apache.jdbm.RecordHeader.getAvailableSize(RecordHeader.java:72)
~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at
org.apache.jdbm.PhysicalRowIdManager.allocNew(PhysicalRowIdManager.java:179)
~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.PhysicalRowIdManager.alloc(PhysicalRowIdManager.java:138)
~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.PhysicalRowIdManager.insert(PhysicalRowIdManager.java:57)
~[shard-assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.insert2(DBStore.java:255) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at org.apache.jdbm.DBStore.insert(DBStore.java:219) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at org.apache.jdbm.DBCache.insert(DBCache.java:75) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.writeValues(BTreeNode.java:1040) ~[shard-
assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.serialize(BTreeNode.java:990) ~[shard-
assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.BTreeNode.serialize(BTreeNode.java:43) ~[shard-
assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBStore.update2(DBStore.java:333) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at org.apache.jdbm.DBStore.update(DBStore.java:317) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
at org.apache.jdbm.DBCache.updateCacheEntries(DBCache.java:146) ~[shard-
assembly-2.3.0-1-g9ca4462.jar:na]
at org.apache.jdbm.DBCache.commit(DBCache.java:100) ~[shard-assembly-2.3.0-1-
g9ca4462.jar:na]
I can provide copies of the corrupted files if that helps.

Reply to this email directly or view it on GitHub.

@ShaileshMangayil
Copy link

Hi Jan,

Does the API have known issues like getting runtime ArrayIndexOutOfBounds Exception while doing an insert. How do you resolve these. Is the only solution migrating to JDBM4 and how will this mitigate the issue. It will be helpful in knowing how JDBM4 fixes these issues.

Thanks,
Shailesh

@jankotek
Copy link
Owner

MapDB is complete rewrite. So it does not
share any issue with JDBM3, it brings its own :-)

The difference is that it had much more testing
than JDBM3 and I am eager to fix all problems
you might encounter.

@ShaileshMangayil
Copy link

Hi Jan,

Thanks for the reply . I like the API and am still discovering the features. Could you let me know why I could be getting this when I do an insert

java.lang.ArrayIndexOutOfBoundsException: 32
at net.kotek.jdbm.HTreeBucket.addElement(HTreeBucket.java:152)
at net.kotek.jdbm.HTreeDirectory.put(HTreeDirectory.java:222)
at net.kotek.jdbm.HTree.put(HTree.java:167)

Thanks,
Shailesh

@jankotek
Copy link
Owner

Honestly no idea. It is almost
two years since I touched
JDBM3 last time.

@ShaileshMangayil
Copy link

It looks like a bug and I am getting it often. Would you be able to suggest what I need to do . Should I continue with this API or something else. The documentation is sketchy. Could you direct me to any documentation on the API or some known issues etc...

Whatever help will be really useful...

@jankotek
Copy link
Owner

My suggestion is to move to
JDBM3...

@ShaileshMangayil
Copy link

I am using JDBM3 . The issue is with JDBM3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants