- Patch es6 iterators not having Symbol.iterator function #479 #249 via @kasbah
- Fixed
find-datom
on empty DB #477 thx @RCmerci
- Regression: transacting many ref value as a set of inline maps #476
- Regression: :db.fn/call returning entity without :db/id #474 #475 via @DerGuteMoritz
- Stable sorting of sequences of various types #470
- Correctly restore
:max-tx
from storage - Fixed tempid/upsert resolution when multiple tempids are added first #472
- Allow upsert by implicit tuple when only tuple components are specified #473
- Allow lookup-refs inside tuples used for lookup-refs #452
- Regression: fixed some OR queries broken in 1.6.4 #468 #469
- Implement “constant substitution” optimization for queries #462
- Fixed :max-eid for dangling entities during reader-based serialization #463
- Fixed tempid in unique refs #464
- Pass through BigInteger/BigDeciman to freeze-fn in serializable #465 #466
- Fix regression in 1.6.2 #460 via @galdre
- Query: shortcircuit clause resolution when result is guaranteed to be empty #459 via @galdre
- Fixed JS version bug caused by Conn rewrite
- [ BREAKING ] Implement Conn as a single atom with extend-clj
- Only breaking if you relied on getting listneres from Conn meta
- Before:
@(:listeners (meta conn))
- After:
(:listeners @(:atom conn))
- prevent explosion of redundant tuples during rule solving #457, thx @RutledgePaulV
- Fixed:
:xform
is not called on ref attributes #455
- Added:
d/reset-schema!
,d/with-schema
- JVM: Return
nil
fromd/restore-conn
if there is no db - Do not throw from d/touch when finding hanging refs
- JVM: Fixed
d/collect-garbage
- JVM: Return nil instead of throwing from
d/restore
if there is no db
- CLJS: Fixed shadow-cljs warning
- JVM: Storage!
d/storage
,d/store
,d/restore
,d/addresses
,d/collect-garbage
,d/file-storage
,d/restore-conn
. See docs/storage.md for details d/settings
and per-database:branching-factor
(passed via :opts)- New API:
d/find-datom
. Works liked/datoms
, but only returns single datom, but is faster than(first (d/datoms ...))
- CLJS: Optimized various parts of CLJS version related to compilation and index access
- [BREAKING]
d/entity
now returnsnil
for numeric eids if entity doesn’t exist #447 #448 via @lynaghk
- Allow the use of lookup refs in get-else and get-some #446 via @jjttjj
- Bump persistent-sorted-set to 0.2.3
- Improve performance of getter-fn and tuple-key-fn #438 via @bsless
- Performance improvement in JVM version #436 via @bsless
- Additionaly compare db identities when
=
/hash
entities #433 #435 via @ggeoffrey
- Check for nils in upsert attributes
- Fixed pull with reverse component references #430
- Fixed serializable/from-serializable for JS interop #429
- Allow symbols as data #425 #426 via @zoren
- When parsing repeated pull attrs, last definition wins
- Add get to built-ins for binding on map values #423 thx @logseq-cldwalker
- Fixed :tupleAttrs in JS API #422
- Query comparison fns work for all data type and cross-types #420 #421 thx @zoren
- Fixed pull on filtered dbs #419
- Fixed pull on deserialized DB
- Allow symbols as constants #416 #417 via @zoren
- Fixed multiple reverse refs bug #412
- Fixed reverse recursive pulls #411
- New pull implementation:
- Up to 7 times faster than old impl
- Up to 2,5 times faster than doing pull through entities
- Up to 4 times faster than Datomic in-memory
- Support :xform
- Cached parse results
- Fixed wildcard + recursion bug:
[* {:child ...}]
did not work before - A callback to track all entities touched by pull (usable for reactive dependency tracking)
- Simplified benchmark code
- Ability to override query parse cache:
datascript.query/*query-cache*
- Fix broken wildcard pull on non-matching ref #410 thx @joodie
- Allow :db/valueType :db.type/tuple #407 #408 thx @joodie
- Data retracted incorrectly in CLJS #404
- Tempid resolution fails in some cases #403
- Special case Inf/-Inf and NaN in serializer since JSON can’t hold them #402
- Allow specifying tuples in inserts/upserts when their value fully matches all the attrs in DB #400
- Bump persistent-sorted-set to 0.1.4
- Lookup ref should not throw in pull #315 #397 thx @timothypratley
- Bump persistent-sorted-set to 0.1.3 in deps.edn too #396 thx @fversnel
- Bump persistent-sorted-set to 0.1.3, fixes
(d/conn-from-datoms (d/datoms @*conn :eavt) schema)
on empty database tonsky/persistent-sorted-set#4
- New namespace:
datascript.serialize
. Prepares database for fast to/from JSON serialization
- [ Breaking ] Remove
ISeqable
fromDB
/FilteredDB
as it was breakingkeys
#393
- Ensure transitive property for hash value comparisons #274 #356 #386 #388 thx @filipesilva
- Fixed empty non-queried relation not affecting results #385
- Implement nav/datafy for entities #325 thx @IamDrowsy
- Temporary disable transient indices #373
- Fixed db diffs with different types of the same attribute #369 #372 thx @darkleaf
- Fixed upsert incorectly failing tempid check #363
- Support composite tuples #323
- Support return maps (:keys/:syms/:strs) in query #322 #345
- Support any hashable values in cardinality-many and indexed attributes #274 #356
- Throw when init-db is used with anything but datoms #276
- Throw on using unindexed attribute in :avet index access #344
- Throw on referencing undefined rule #319
- Fix
[:db/retract e a false]
working as[:db/retract e a]
- Validate tempids only used as values in transaction #304
- Throw if transacted entity id is out of range #292
- Better validate rules syntax #300
- Add
keyword
to the built-ins #231 - Throw when pred/fn called on unbound variable #309
- Validate inputs count in
:in
andd/q
#297
- Fix
empty?
builtin #349, thx @ash14
- In
or
branches, only check that all free vars match, not bound ones (#348)
- Ignore tx when diff datoms (#347, thx @darkleaf)
- Support: db/retract without value (#339, #340, thx @ericdscott)
- Report in :tempids all newly created entities, even if they don’t have :db/id assigned in tx (#246, #337, thx @filipesilva)
empty
preserves meta (#331, #332, thx @darkleaf)- Filtered db can be pprinted (#330, #333, thx @darkleaf)
- Allow aliasing map keys in pull entry (#328, thx @jamesnvc)
- Enable :unique/identity and upserts for cardinality-many values (#321, thx @cjsauer)
- Extracted pprinting to separate ns (#318, thx @zajac)
- Fixed core.cljc NS declaration to require cljs.reader (#316, thx @Quezion)
- Added re-pattern and str/blank? str/includes? str/starts-with? str/ends-with? to built-ins (#306)
- Fixed vector misbehaving in queries in CLJS (#262)
- Fixed regression when transient indexes were mutated and iterated during single tx (#294).
- Bumped persistent-sorted-set to 0.1.2 (throw if transient set is mutated and iterated at the same time).
- Fixed regression when unique ref tempid was not resolved during transaction (#295).
- Bumped persistent-sorted-set to 0.1.1 (Java 8 bytecode)
- Fixed DB and Datom pprinting (#287)
- Fixed cases when upsert resolves to tempid (#285)
- Throw on tempid in
:db.fn/cas
(#264) distinct
aggregate returns set not a vector (thx @jdf-id-au)- Ability to run tests with Kaocha
- [ BREAKING ] Some internals of
datascript.arrays
,datascript.btset
anddatascript.Datom
type has changed
Performance optimizations for JVM version:
- Reimplemented btset in Java with transients and better performance
- Extracted btset to
[persistent-sorted-set "0.1.0"]
- Used raw ints in Datom intead of wrapped Integers, added stored in tx sign
Numbers I get on my 3.2 GHz i7-8700B (median time per test, ms):
version | add-1 | add-5 | add-all | init | retract-5 | q1 | q2 | q3 | q4 | qpred1 | qpred2 |
---|---|---|---|---|---|---|---|---|---|---|---|
0.17.1-jvm | 795.8 | 670.7 | 651.8 | 79.4 | 617.5 | 2.3 | 5.4 | 8.2 | 13.1 | 7.1 | 27.3 |
0.18.0-jvm | 625.2 | 450.9 | 401.8 | 21.8 | 389.5 | 1.9 | 5.4 | 8.2 | 13.3 | 7.3 | 28.9 |
0.9.5703-datomic | 1693.9 | 737.9 | 528.5 | --- | 1420.9 | 2.8 | 5.2 | 7.3 | 9.3 | 12.8 | 15.5 |
0.18.0-v8 | 1231.6 | 963.1 | 930.3 | 76.5 | 809.1 | 6.4 | 15.2 | 23.8 | 33.6 | 24.2 | 24.5 |
Tests are as follows:
Test | Description |
---|---|
add-1 | Add 100k datoms to an empty DB, one datom per transaction |
add-5 | Add 20k entities to an empty DB, 5 datoms per transaction, 100k datoms total |
add-all | Add 20k entities to an empty DB in a single transaction, 100k datoms total |
init | “Fast” datascript DB creation from an already sorted array of datoms (used in DB deserialization), 100k datoms |
retract-5 | Retract 20k entities from a DB with 100k datoms. Each entity removes 5 datoms. 1 entity per tx. |
q1 | Query with 1 clause over a DB with 100k datoms, ~12k tuples in resultset [:find ?e :where [?e :name "Ivan"]] |
q2 | Query with 2 clauses, 1 join, ~12k tuples [:find ?e ?a :where [?e :name "Ivan"] [?e :age ?a]] |
q3 | Query with 3 clauses, 2 joins, ~6k tuples [:find ?e ?a :where [?e :name "Ivan"] [?e :age ?a] [?e :sex :male]] |
q4 | Query with 4 clauses, 3 joins, ~6k tuples [:find ?e ?l ?a :where [?e :name "Ivan"] [?e :last-name ?l] [?e :age ?a] [?e :sex :male]] |
qpred1 | Query with a predicate, ~50k tuples [:find ?e ?s :where [?e :salary ?s] [(> ?s 50000)]] |
qpred2 | Query with a predicate and dynamic input, ~50k tuples [:find ?e ?s :in $ ?min_s :where [?e :salary ?s] [(> ?s ?min_s)]] |
For Datomic an datomic:mem://
database was used.
What we see:
- 20..40% faster transactions,
- 75% faster deserialization (db-init),
- No significant change on queries,
- JVM transactions are more than twice as fast as V8,
- JVM queries are 3-4 times as fast as V8,
- DataScript transactions are 25..70% faster that Datomic in-memory. Query times vary.
or
,or-join
,not
andnot-jon
support in queries (#238, #50)
- Implement
clojure.data/diff
ondatascript/DB
(#281) - Drop Clojure 1.7 and 1.8 support
- Fix externs.js syntax (PR #216, thx @thheller)
- Support
:as
in Pull API an attr-with-opts syntax (#270, PR #271, thx @Jumblemuddle) - Support idents expansion (PR #245, thx bamarco)
- JS API correctly handles nested maps with
{":db/id"}
in transactions (#228, thx @serebrianyi) - Calling transaction fns through idents directly (PR #185, thx @refset)
- AOT artifacts are now deployed with classifiers (
0.16.9-aot1.7
,0.16.9-aot1.8
,0.16.9-aot1.9
). Main DataScript artifact has no AOTed code (related: #241, #279, #280)
- Docstrings for https://cljdoc.org/d/datascript/datascript
- Removed references to
perf
fromdatascript.query-v3
(closes #272) - Fixed compile error under java 11 (PR #273, thx Ryan Belohlavek)
- Add support for renamed DB functions :db/retractEntity and :db/cas (#265, PR #256, thx Kenny Williams)
- Provision AOT-compiled classes in datascript.jar (closes #241)
- Fixed direct-linking compatibility (rolled back #197, fixed #219, closes #259)
- Support string tempids (#251, PR #252, thx Kenny Williams)
- Better compatibility with JS / CLJS 1.9.946 (aget, aset on plain objects)
- Added
rseek-datoms
(#253, PR #254, thx Jeremy Taylor) - Object.equals for entites (PR #255, thx Camilo Roca)
- Silently skip nils in transaction (was stopping processing without an error before)
- Fixed compatibility with CLJS 1.9.660 (#233, #236)
- Fixed rules ignoring false arguments passed to them (#218)
- Fixed infinite loop in parser on CLJS 1.9.456+ (#210)
- Added
contains?
to built-ins (#211) - Fixed handling of false values in entity cache (PR #198, thx Brandon Bloom)
- Fixed issue when string values were interpreted as lookup refs (#214)
- Speed up rschema calculation (#192, thx Andre R.)
- Optimize generated JS code by declaring fn arities (#197)
- [ BREAKING ] Removed ^:exports from
datascript.core
to enable dead code elimination (#191). This should only affect you if you were using DataScript from JS and were importingdatascript.core
directly
- Add
vector
,hash-map
,array-map
,list
andset
to built-in query functions (PR #186, thx Linus Ericsson) - When filtering already filtered DB, filters were applied in the wrong order
- ~30% better performance for predicate and fn calls in queries (see #180)
- JS API now supports lookup refs in index lookups and queries (see #179)
- Lookup refs now work with
:db.unique/value
too (issue #171)
- Compatibility with 1.9 records (issue #176)
- Resolve query functions and predicates on JVM (issue #177, PR #178, thx Luke VanderHart)
- large long entity ids were breaking
init-db
(issue #165)
get-else
throws ifnil
is used for default value- JS API: Do not keywordize attribute names in schema (PR #155, issue #153, thx typeetfunc)
init-db
sets correctmax-eid
when processing facts about transactions (PR #164, issue #163, thx Nick Alexander)
- Better error messages for upsert conflicts
- Backtracking of tempids when redefining them later with upserts (issue #76)
- Upsert works in vector form too (issue #99, #109)
- Can specify transaction number in
:db/add
- Can put datoms into transaction directly (issue #121: supports both addition and retration datoms, keeps tx number)
- Added all
datascript.core
symbols to externs so they can be called directly from JS (e.g. in combination with mori, issue #139) - Added
reset-conn
,conn-from-datoms
andconn-from-db
(issue #45)
- Unify fn binding results with existing relations (PR #138, issue #141, thx Aaron Brooks)
- [ BREAKING ]
get-some
returns[attr value]
instead of justvalue
(#143, same as Datomic) - [ BREAKING ] Returning
nil
from query function filters out tuple from result - Throw when calling unbound predicate/function inside a query (#111)
- Added several built-ins, including
count
(#142) andsubs
(#111) - Lookups with nil value (
[<attr> nil]
) resolve to nil (#140) - Allow pull-pattern to be specified with a input variable without
?
prefix (#136, #122) - Retract functions do not fail if lookup ref is pointing nowhere (#131)
- Fixed BTSet implementation of IReduceInit (#124)
- Accept
nil
as old value in:db.fn/cas
(PR #127, thx Petter Eriksson)
- Fixed issue #120 with binding of large collections
- Guard
conn?
will check that value is derefable instead of checking for(instance? Atom)
- [ BREAKING ] Main namespace to include is now
datascript.core
, notdatascript
- [ BREAKING ] Old
datascript.core
(internal namespace) was renamed todatascript.db
- [ BREAKING ]
datascript.shim
(internal namespace) was renamed todatascript.arrays
Motivations:
- Usage of top-level namespaces is discouraged and even generates a warning in CLJS.
- Better sooner than later.
- 0.13 seems like a great number for this sort of release.
Migration procedure:
- Just change
(require '[datascript :as d])
to(require '[datascript.core :as d])
and you’re good to go.
For the sake of easy migration, there’re no other changes in this release. Just renamings.
- Fix null pointer exception when
contains?
is called with an entity (PR #114, thx Kevin Lynagh)
db-init
respects:db/index
property
[ BREAKING ] Introducing new :db/index
schema attribute:
-
Attributes are not put to AVET by default anymore
-
Following attributes are put to AVET:
:db/index true
:db/unique :db.unique/identity
:db/unique :db.unique/value
:db/valueType :db.type/ref
-
All attributes put to AVET should be comparable. Note: maps and list are not comparable by default, vectors are compared value-by-value
-
[ BREAKING ] Min/max aggregation functions will only work on a comaparable values
Benefits:
- You can finally store any trash easily and reliably in DataScript database: maps, vectors, functions, JS objects. It doesn’t even have to be comparable. Just do not mark it as
:db/index
and do not make it:db.cardinality/many
- Faster transactions. There’s ⅓ less indexes to fill
- Good defaults. Most cases where you’ll probably use AVET (lookup by value)—lookup refs, external ids, references—they are all indexed by default
Caveats:
- Your code may break (see below)
- Queries can do lookup by value event without
:db/index
attribute, but it’ll be slower
Migration procedure:
- Check your code for
datoms
,seek-datoms
calls with:avet
index, andindex-range
call - Check your queries to see if they utilize AVET index. It happens when you use this pattern:
[?free-var <constant-attr> <constant-value>]
- Mark necessary attributes with
:db/index true
in the schema
- msec argument to squuid (#95, #97)
- lookup refs in JS API
pull
,pull_many
,entity
(#94, thx Matt Senior) - fix in Pull API for reverse non-component attributes (#91, thx Matt Senior)
- Node.js and Browser repls for dev profile (#93)
- Preconditions to validate db/conn arguments (#101)
- Id allocation bug (#66)
- Ported BTSet to JVM
- BTSet and BTSetIter implement ChunkedSeq
- New benchmark runner
This release brings a significant performance boost for JVM version of DataScript (numbers are in comparison to JS/v8 version):
- queries with single clause, no join: ~5–6 times faster
- queries with joins: ~3–4.5 times faster
transact
: ~3 times fasterinit-db
: ~3–4 times faster- rules: ~3-4 times faster
Experimental support for Clojure on JVM:
sorted-set
instead of BTSet- polymorphic
get
lookup in query - no sorting optimisation for
init-db
Performance numbers so far (raw data):
- queries with single clause, no join: 30–50% faster
- queries with joins: ~50% slower
transact
: ~3.5–4 times fasterinit-db
: ~2–2.5 times slower- rules: ~2–5 times faster
- db
equiv
: ~5 times slower - db
hash
: ~10 times slower
- Fixed UUID constructor warning under CLJS 0.0-3291 and later
- Support reverse attribute refs in combination with wildcards (issue #80)
*.cljc
sources do not conflict with Clojure now. Code compiles under[lein-cljsbuild "1.0.6"]
- Return nil from entity when passed nil eid (issue #75)
- Fixed
init_db
in js (issue #73) - Preliminary work for Clojure port (PR #70, #71)
- [ BREAKING ] DS now requires CLJS 0.0-3196 or later
- New parser for query and rules
- Added a lot of meaningful error messages and validations to query parser
- Support for _ (ignore binding) in
:in
andfn-clause
bindings - Fixed id allocation bug when nested maps are used for referenced entities (issue #59)
- Fixed squuids omitting leading zeros (issue #60)
- Fixed a bug when cross-referencing components caused infinite loop in Pull API (isuue #58, pull request #61)
- Pull API handles recursion more like Datomic does (issue #62)
- Exposed DB filter API to js (pull request #65)
- Added ICounted, ISeqable, IEmptyableCollection to
datascript.core/DB
- Force put
Datom
andDB
tag readers tocljs.reader/*tag-table*
, do not rely ondata_readers.clj
- Lookup refs can be inlined in query
:where
patterns (issue #53) - Pull API (issue #37, pull request #51) by David Thomas Hume
pull
form in:find
clause
:db/unique
constraints- Upserts
- Lookup refs
entid
function
- Find specifications: collection
:find [?e ...]
, tuple:find [?e ?v]
, and scalar:find ?e .
- Support for
:db/isComponent
(issue #3) - Support for nested maps in
transact!
(also fixes #38) - [ BREAKING ] Custom aggregate fns must be called via special syntax (
aggregate
keyword):(q '[:find (aggregate ?myfn ?e) :in $ ?myfn ...])
. Built-in aggregates work as before:(q '[:find (count ?e) ...]
- Return nil from
entity
when passed nil eid (issue #47) - Transaction data is now validated, with proper error messages (also fixes #48)
- Fixed a bug with function call after function relation was filtered out (issue #44)
- Fixed a bug when emtpy coll in input was not creating empty relation
- BTSet and its slices (returned by
datoms
/seek-datoms
/index-range
calls) now support fast reverse iteration viareverse
. (datascript/datom e a v & [tx added])
call for creating new datoms- Added missing aggregate funs:
avg
,median
,variance
,stddev
andcount-distinct
(issue #42, thx @montyxcantsin) min
andmax
aggregates use comparator instead of default js<
comparison- Fixed a bug when fn inside a query on empty relation returned non-empty result
- Filtered DB support via
filter
- [ BREAKING ] Entity ids and transaction ids now have to be numbers, attributes can only be keywords (strings if used from JS)
- New method
init-db
for fast creation of DB from existing datoms (useful for e.g. initialization from serialized state) - Improved performance of
transact!
(~20%) and initial database population (init-db
,db-from-reader
, ~800%)
- Externs provided via
deps.cljs
— no need to manually specify externs when using datascript dependency
- Ability to pass inputs (predicates/functions bindings) down to rules (issues #28 #29)
- Javascript version is now packaged as a proper CommonJS/RequireJS module (include via script tag still supported) (issue #39)
- Published to npm: npmjs.org/package/datascript
- [ BREAKING ] Javascript namespace is renamed from
datascript.js
todatascript
- Reference transaction id inside transaction by using
:db/current-tx
instead of entity id (issue #32) - You can get id of transaction from
TxReport
:(get-in tx-report [:tempids :db/current-tx])
- Ability to pass arbitrary metadata along with transaction (third argument to
transact!
) and read it in listener callback (:tx-meta
key ofTxReport
) (issue #32)
- Added
:db.fn/cas
transaction function (issue #20, thx @montyxcantsin) - Added
get-some
,get-else
,misssing?
query functions (issue #21, thx @montyxcantsin) - Fixed
touch
breaking entity cache (issue #17) - Added
ground
fn (issue #25) - Fixed query predicate/fn calls without free variables
- Retract incoming entity references on
:db.fn/retractEntity
(issue #18) - You can now use reverse relation name when transacting entity as a map
Cosmetic changes to better mimic Datomic API. Useful for sharing code between Datomic and DataScript:
- Added
tempid
,resolve-tempid
,db
,transact
,transact-async
,index-range
,squuid
,squuid-time-millis
- [ BREAKING ] renamed
transact
towith
,with
todb-with
- Optimized speed of DB’s
equiv
andhash
, Datom’shash
- Entity’s
touch
call accessible throughdatascript
namespace - Accept sets in entity maps as values for
:db.cardinality/many
attributes
Proper entities implementation:
- Entities are now lazy and implement usual Map protocols
- [ BREAKING ] When accessing attribute of
:db/valueType :db.type/ref
, its value will be automatically expanded to entites, allowing for recursive exploration of entities graphs (e.g.(-> (d/entity db 42) :parent :parent :children)
) - Entities support backwards navigation (e.g.
(:person/_friends (d/entity db 42))
)
- Externs file now can be referred as
:externs [datascript/externs.js"]
Big performance improvements:
- New B-tree based indexes
- New set-at-a-time, hash-join powered query and rules engine
- Queries now up to 10× times faster
- You can specify DB for rule call (like
($db follows ?e1 ?e2)
) - Datoms are required to have integer id and keyword attributes, but no restriction on types of values
- Clojure reader support (pr/read) for DB and Datom
datoms
andseek-datoms
API calls- referencing other entities’ tempids in transaction data (issue #10)
- Transactor functions via
:db.fn/call
(thx @thegeez) - Vanilla JS API bindings
- [ BREAKING ] Schema keywords namespaced on a par with Datomic schema
entity
added