Skip to content

Releases: dbt-labs/dbt-core

dbt 0.14.1

04 Sep 13:51
Compare
Choose a tag to compare

dbt 0.14.1 - (September 3, 2019)

Overview

This release fixes a number of bugs and includes minor improvements to the dbt workflow. These changes include UI improvements for dbt's console output, updates to the dbt Docs website, and general performance improvements.

Note: this release includes an important bugfix for the check snapshot strategy on Snowflake and BigQuery. See this guide for more information about the issue and resolution.

Links:

Breaking changes

  • The undocumented macros attribute was removed from the graph context variable (#1615)

Features:

  • Summarize warnings at the end of dbt runs (#1597, #1654)
  • Speed up catalog generation on postgres by using avoiding use of the information_schema (#1540)
  • Docs site updates (#1621)
    • Fix for incorrect node selection logic in DAG view (docs#38)
    • Update page title, meta tags, and favicon (docs#39)
    • Bump the version of dbt-styleguide, changing file tree colors from orange to black :)
  • Add environment variables for macro debugging flags (#1628, #1629)
  • Speed up node selection by making it linear, rather than quadratic, in complexity (#1611, #1615)
  • Specify the application field in Snowflake connections (#1622, #1623)
  • Add support for clustering on Snowflake (#634, #1591, #1689) (docs)
  • Add support for job priority on BigQuery (#1456, #1673) (docs)
  • Add node.config and node.tags to the generate_schema_name and generate_alias_name macro context (#1700, #1701)

Fixes:

  • Fix for reused check_cols values in snapshots (#1614, #1709)
  • Fix for rendering column descriptions in sources (#1619, #1633)
  • Fix for is_incremental() returning True for models that are not materialized as incremental models (#1249, #1608)
  • Fix for serialization of BigQuery results which contain nested or repeated records (#1626, #1638)
  • Fix for loading seed files which contain non-ascii characters (#1632, #1644)
  • Fix for creation of user cookies in incorrect directories when --profile-dir or $DBT_PROFILES_DIR is provided (#1645, #1656)
  • Fix for error handling when transactions are being rolled back (#1647)
  • Fix for incorrect references to dbt.exceptions in jinja code (#1569, #1609)
  • Fix for duplicated schema creation due to case-sensitive comparison (#1651, #1663)
  • Fix for "schema stub" created automatically by dbt (#913, #1663)
  • Fix for incremental merge query on old versions of postgres (<=9.6) (#1665, #1666)
  • Fix for serializing results of queries which return TIMESTAMP_TZ columns on Snowflake in the RPC server (#1670)
  • Fix typo in InternalException (#1640, #1672)
  • Fix typo in CLI help for snapshot migration subcommand (#1664)
  • Fix for error handling logic when empty queries are submitted on Snowflake (#1693, #1694)
  • Fix for non-atomic column expansion logic in Snowflake incremental models and snapshots (#1687, #1690)
  • Fix for unprojected count(*) expression injected by custom data tests (#1688)
  • Fix for dbt run and dbt docs generate commands when running against Panoply Redshift (#1479, #1686)

Contributors:

Thanks for your contributions to dbt!

dbt 0.14.0

10 Jul 14:11
4569c90
Compare
Choose a tag to compare

dbt 0.14.0 - Wilt Chamberlain (July 10, 2019)

Overview

This release adds an rpc server to dbt (docs), replaces Archives with Snapshots (docs, migration guide) and adds new dbt subcommands:

  • dbt ls (docs)
  • dbt run-operation (docs)

Tests can now be defined with a severity (either warning or error) (docs), in addition to a number of other workflow and quality of life improvements.

Links:

Breaking changes

  • Stub out adapter methods at parse-time to speed up parsing (#1413)
  • Removed support for the --non-destructive flag (#1419, #1415)
  • Removed support for the sql_where config to incremental models (#1408, #1351)
  • Changed expand_target_column_types to take a Relation instead of a string (#1478)
  • Replaced Archives with Snapshots
    • Normalized meta-column names in Snapshot tables (#1361, #251)

Features

  • Add run-operation command which invokes macros directly from the CLI (#1328) (docs)
  • Add a dbt ls command which lists resources in your project (#1436, #467) (docs)
  • Add Snapshots, an improvement over Archives (#1361, #1175) (docs)
    • Add the 'check' snapshot strategy (#1361, #706)
    • Support Snapshots across logical databases (#1455)
    • Implement Snapshots using a merge statement where supported (#1478)
    • Support Snapshot selection using --select (#1520, #1512)
  • Add an RPC server via dbt rpc (#1301, #1274) (docs)
  • Made printer width configurable (#1026, #1247) (docs)
  • Retry package downloads from the hub.getdbt.com (#1451, #1491)
  • Add a test "severity" level, presented as a keyword argument to schema tests (#1410, #1005) (docs)
  • Add a generate_alias_name macro to configure alias names dynamically (#1363) (docs)
  • Add a node argument to generate_schema_name to configure schema names dynamically (#1483, #1463) (docs)
  • Use create or replace on Snowflake to rebuild tables and views atomically (#1101, #1409)
  • Use merge statement for incremental models on Snowflake (#1414, #1307, #1409) (docs)
  • Add support seed CSV files that start with a UTF-8 Byte Order Mark (BOM) (#1452, #1177)
  • Add a warning when git packages are not pinned to a version (#1453, #1446)
  • Add logging for on-run-start and on-run-end hooks to console output (#1440, #696)
  • Add modules and tracking information to the rendering context for configuration files (#1441, #1320)
  • Add support for null vars, and distinguish null vars from unset vars (#1426, #608)
  • Add support for the search_path configuration in Postgres/Redshift profiles (#1477, #1476) (docs (postgres), docs (redshift))
  • Add support for persisting documentation as descriptions for tables and views on BigQuery (#1031, #1285) (docs)
  • Add a --project-dir path which will invoke dbt in the specified directory (#1549, #1544)

dbt docs Changes

  • Add searching by tag name (#32)
  • Add context menu link to export graph viz as a PNG (#34)
  • Fix for clicking models in left-nav while search results are open (#31)

Fixes

  • Fix for unduly long timeouts when anonymous event tracking is blocked (#1445, #1063)
  • Fix for error with mostly-duplicate git urls in packages, picking the one that came first. (#1428, #1084)
  • Fix for unrendered description field as jinja in top-level Source specification (#1484, #1494)
  • Fix for API error when very large temp tables are created in BigQuery (#1423, #1478)
  • Fix for compiler errors that occurred if jinja code was present outside of a docs blocks in .md files (#1513, #988)
  • Fix TEXT handling on postgres and redshift (#1420, #781)
  • Fix for compiler error when vars are undefined but only used in disabled models (#1429, #434)
  • Improved the error message when iterating over the results of a macro that doesn't exist (#1425, [#1424](https://github.com/fishtown-analyti...
Read more

dbt 0.13.1

13 May 19:25
Compare
Choose a tag to compare

dbt 0.13.1 (May 13, 2019)

Overview

This release primarily fixes minor bugs introduced in v0.13.0

Features

Fixes

  • Add "MaterializedView" relation type to the Snowflake adapter (#1430, #1432) (@adriank-convoy)
  • Quote databases properly (#1396, #1402)
  • Use "ilike" instead of "=" for database equality when listing schemas (#1411, #1412)
  • Pass the model name along in get_relations (#1384, #1388)

dbt docs

Contributors:

Thanks for your contributions to dbt!

dbt 0.13.0

22 Mar 14:54
da4c135
Compare
Choose a tag to compare

dbt 0.13.0 - Stephen Girard (March 21, 2019)

Overview

This release provides a stable API for building new adapters and reimplements dbt's adapters as "plugins". Additionally, a new adapter for Presto was added using this architecture. Beyond adapters, this release of dbt also includes Sources which can be used to document and test source data tables. See the full list of features added in 0.13.0 below.

Links:

Installation instructions

dbt v0.13.0 is available on Homebrew, PyPi, and in dbt Cloud.

# Homebrew
$ brew install dbt

# pip
$ pip install dbt

Note: If you're upgrading from an earlier version of dbt using pip, you may need to install dbt with the -I flag. This flag is required due to changes in the structure of the dbt package.

$ pip install -I dbt

For full installation instructions, consult the docs

Breaking Changes

  • Version 1 schema.yml specs are no longer implemented. Please use the version 2 spec instead (migration guide)
  • {{this}} is no longer implemented for on-run-start and on-run-end hooks. Use {{ target }} or an on-run-end context variable instead (#1176, implementing #878)
  • A number of materialization-specific adapter methods have changed in breaking ways. If you use these adapter methods in your macros or materializations, you may need to update your code accordingly.

Deprecations

  • The following adapter methods are now deprecated, and will be removed in a future release:

Features

  • Add sources to dbt, use them to calculate source data freshness (docs ) (#814, #1240)
  • Add support for Presto (docs, repo) (#1106)
  • Add require-dbt-version option to dbt_project.yml to state the supported versions of dbt for packages (docs) (#581)
  • Add an output line indicating the installed version of dbt to every run (#1134)
  • Add a new model selector (@) which build models, their children, and their children's parents (docs) (#1156)
  • Add support for Snowflake Key Pair Authentication (docs) (#1232)
  • Support SSO Authentication for Snowflake (docs) (#1172)
  • Add support for Snowflake's transient tables (default=true) (docs) (#946)
  • Capture build timing data in run_results.json to visualize project performance (#1179)
  • Add CLI flag to toggle warnings as errors (docs) (#1243)
  • Add tab completion script for Bash (docs) (#1197)
  • Added docs on how to build a new adapter (docs) (#560)
  • Use new logo (#1349)

Fixes

  • Fix for Postgres character columns treated as string types (#1194)
  • Fix for hard to reach edge case in which dbt could hang (#1223)
  • Fix for dbt deps in non-English shells (#1222)
  • Fix for over eager schema creation when models are run with --models (#1239)
  • Fix for dbt seed --show (#1288)
  • Fix for is_incremental() which should only return True if the target relation is a table (#1292)
  • Fix for error in Snowflake table materializations with custom schemas (#1316)
  • Fix errored out concurrent transactions on Redshift and Postgres (#1356)
  • Fix out of order execution on model select (#1354, #1355)
  • Fix adapter macro namespace issue (#1352, #1353)
  • Re-add CLI flag to toggle warnings as errors (#1347)
  • Fix release candidate regression that runs run hooks on test invocations (#1346)
  • Fix Snowflake source quoting (#1338, #1317, #1332)
  • Handle unexpected max_loaded_at types (#1330)

Under the hood

  • Replace all SQL in Python code with Jinja in macros (#1204)
  • Loosen restrictions of boto3 dependency (#1234)
  • Rewrote Postgres introspective queries to be faster on large databases (#1192)

Contributors:

Thanks for your contributions to dbt!

Stephen Girard

Can you imagine buying the Federal Reserve? That’s how rich Stephen Girard was. The First Bank of the United States, Alexander Hamilton’s brainchild, ran out its charter in 1811 and Girard decided it was a steal. Estimated to be the 4th richest American ever, he just up and bought it.

Not satisfied with one bank, Girard went for more. With the British invading in 1812, Girard put everything on the line and underwrote nearly all the US war bonds. Without this capital, we would likely all have British accents today. In return, the US made Girard a major shareholder of the Second Bank of the United States when it was founded.

Buying government bonds: dépassé.
Buying governments: branché.

Girard, who had no children, was one of America’s biggest philanthropists. He single-handedly endowed the 43-acre Girard College to educate “poor, male, white orphans,” inspired by his own hard-scrabble early life. Thankfully, its target demographic has since broadened—its first African-American student was admitted in 1968 and its first female student was admitte...

Read more

dbt 0.12.2

08 Jan 15:53
Compare
Choose a tag to compare

Overview

This release reduces the runtime of dbt projects by improving dbt's approach to model running. Additionally, a number of workflow improvements have been added. You can find more information about dbt's new approach to model running in this blog post.

Deprecations

Features

  • More intelligently order and execute nodes in the graph. This significantly speeds up the runtime of most dbt projects (#813)
  • Add -m flag as an alias for --models (#1160)
  • Add post_hook and pre_hook as aliases for post-hook and pre-hook, respectively (#1124) (docs)
  • Better handling of git errors in dbt deps + full support for Windows (#994, #778, #895)
  • Add support for specifying a location in BigQuery datasets (#969) (docs)
  • Add support for Jinja expressions using the {% do ... %} block (#1113)
  • The dbt debug command is actually useful now (#1061)
  • The config function can now be called multiple times in a model (#558)
  • Source the latest version of dbt from PyPi instead of GitHub (#1122)
  • Add a peformance profiling mechnanism to dbt (#1001)
  • Add caching for dbt's macros-only manifest to speedup parsing (#1098)

Fixes

  • Fix for custom schemas used alongside the generate_schema_name macro (#801)
  • Fix for silent failure of tests that reference nonexistent models (#968)
  • Fix for generate_schema_name macros that return whitespace-padded schema names (#1074)
  • Fix for incorrect relation type for backup tables on Snowflake (#1103)
  • Fix for incorrectly cased values in the relation cache (#1140)
  • Fix for JSON decoding error on Python2 installed with Anaconda (#1155)
  • Fix for unhandled exceptions that occur in anonymous event tracking (#1180)
  • Fix for analysis files that contain raw tags (#1152)
  • Fix for packages which reference the hubsite (#1095)

Grace Kelly

Grace Kelly (1929-1982) was a Philadelphian film actress who became the Princess of Monaco. In her early life, Kelly took an interest in acting and so moved to New York City to attend the American Academy of Dramatic Arts. As her career progressed, Kelly acted in theater productions, television shows, and ultimately gained stardom on the silver screen. She was awarded the Academy Award for Best Actress for her role in The Country Girl (1954).

While operating as a U.S. delegate to the Cannes Film Festival, Kelly came to meet the prince of Monaco. After their eventual wedding, Kelly thusly became the Princess of Monaco. In this role, Kelly adopted a life of philanthropy. Kelly founded and was involved in numerous non-profits dedicated to advancing the arts and the wellbeing of citizens of Monaco and the world alike.

dbt 0.12.1

15 Nov 17:08
0264386
Compare
Choose a tag to compare

Overview

This is a bugfix release.

Fixes

  • Fix for relation caching when views outside of a dbt schema depend on relations inside of a dbt schema (#1119)

dbt 0.12.0

13 Nov 15:40
Compare
Choose a tag to compare

dbt 0.12.0 - Guion Bluford (November 12, 2018)

Overview

This release adds caching for some introspective queries on all adapters. Additionally, custom tags can be supplied for models, along with many other minor improvements and bugfixes.

Breaking Changes

  • Support for the repositories: block in dbt_project.yml (deprecated in 0.10.0) was removed. Use the packages.yml file instead (docs)

tl;dr

  • Make runs faster by caching introspective queries
  • Support model tags
  • Add a list of schemas to the on-run-end context
  • Set your profiles directory with an environment variable

Features

  • Cache the existence of relations to speed up dbt runs (#1025)
  • Add support for tag configuration and selection (#1014)
    • Add tags to the model and graph views in the docs UI (#7)
  • Add the set of schemas that dbt built models into in the on-run-end hook context (#908)
  • Warn for unused resource config paths in dbt_project.yml (#725)
  • Add more information to the dbt --help output (#1058)
  • Add support for configuring the profiles directory with an env var (#1055)
  • Add support for cli and env vars in most dbt_project.yml and profiles.yml fields (#1033)
  • Provide a better error message when seed file loading fails on BigQuery (#1079)
  • Improved error handling and messaging on Redshift (#997)
  • Include datasets with underscores when listing BigQuery datasets (#954)
  • Forgo validating the user's profile for dbt deps and dbt clean commands (#947, #1022)
  • Don't read/parse CSV files outside of the dbt seed command (#1046)

Fixes

  • Fix for incorrect model selection with the --models CLI flag when projects and directories share the same name (#1023)
  • Fix for table clustering configuration with multiple columns on BigQuery (#1013)
  • Fix for incorrect output when a single row fails validation in dbt test (#1040)
  • Fix for unwieldly Jinja errors regarding undefined variables at parse time (#1086, #1080, #935)
  • Fix for incremental models that have a line comment on the last line of the file (#1018)
  • Fix for error messages when ephemeral models fail to compile (#1053)

Under the hood

  • Create adapters as singleton objects instead of classes (#961)
  • Combine project and profile into a single, coherent object (#973)
  • Investigate approaches for providing more complete compilation output (#588)

Contributors

Thanks for contributing!

Guion Bluford

Dr. Guion “Guy” Stewart Bluford Jr. is a true Philadelphian, through and through. He was born in Philadelphia, PA November 22nd, 1942 and attended Overbrook High School. He grew up in a home that encouraged academic success, which inspired him to eventually receive his Ph. D in Aerospace Engineering from the Air Force Institute of Technology in 1978.

Not only an academic, Bluford flew 144 combat missions during the Vietnam War, and earned numerous medals in the process. He became the first African American in space in 1983 as a specialist for mission STS-8 aboard the space shuttle Challenger. He was inducted into the International Space Hall of Fame in 1997 and the United States Astronaut Hall of Fame in 2010.

In addition to his professional success, Bluford has been seen as a role model for many African Americans. He has been awarded the NCAAP Image Award, Ebony Black Achievement Award and the 1991 Black Engineer of the Year award, to name a few.

A family man, a war veteran, an astronaut and a role model; Guion Bluford.

dbt 0.11.1

19 Sep 00:01
Compare
Choose a tag to compare

dbt 0.11.1 - Lucretia Mott (September 18, 2018)

This is a patch release containing a few bugfixes and quality of life changes, especially around dbt docs.

Features

  • dbt
    • Add --port parameter to dbt docs serve (#987)
    • Add support for BigQuery clustering (#978)

Fixes

  • dbt
    • Fix hooks in model configs not running (#985)
    • Fix integration test on redshift catalog generation (#977)
    • Snowflake: Fix docs generation errors when QUOTED_IDENTIFIER_IGNORE_CASE is set (#998)
    • Translate empty strings to null in seeds (#995)
    • Filter out null schemas during catalog generation (#992)
    • Fix quoting on drop, truncate, and rename (#991)
  • dbt-docs
    • Fix for non-existent column in schema.yml (#3)
    • Fixes for missing tests in docs UI when columns are upcased (#2)
    • Fix "copy to clipboard" (#4)

Thanks to @ericalouie for the biography of Lucretia Mott, below.

Lucretia Mott

Lucretia Mott (1793-1880) was a women’s rights activist, abolitionist, and reformer. As a powerful orator, she dedicated her life to achieving equality for all of America’s disadvantaged and disenfranchised, including Indians, women, slaves, and free blacks. During an era where women were to be seen and never heard, Mott’s public speaking certainly defied that. And while we now have the luxury to admire that, her behavior was often denounced as controversial and inappropriate. Nevertheless, she persisted.

To pinpoint Mott’s accomplishments into a single entity is almost impossible as her influence expanded to so many areas: speaking for women’s rights at the Seneca Falls Convention, authoring Discourse on Woman, co-founding the Philadelphia Female Anti-Slavery Society.

Originally born in Nantucket, Massachusetts, she was raised as a Quaker. And although her impact on the feminist discourse and abolitionist movement spread throughout America and American history, her legacy is rooted in Philadelphia. Having moved to Philadelphia at 16, she certainly built her roots here. She married James Mott, also an abolitionist and social reformer, at Front and 2nd in Old City. Together, they fought for the same causes, housed several runaway slaves, and raised 5 children who all became active in the anti-slavery movement. In 1869 Mott and her husband were active in the plans for the opening of Swarthmore College, a Quaker institute for higher learning. When the college had been chartered in 1864, she and James had insisted it be coeducational.

Because she had a strong presence as a fighter against gender and racial injustices, mobs would often target her home in Philadelphia. Mott was often seen waiting in her parlor, willing to face her violent opponents. And although she didn’t live to see the day women won the right to vote, Mott is credited with igniting the women’s rights movement and serving as mentor to Elizabeth Cady Stanton, who continued Mott’s work after she died.

dbt 0.11.0

06 Sep 14:39
0d074c3
Compare
Choose a tag to compare

dbt v0.11.0, Isaac Asimov - tl;dr

This release introduces auto-generated dbt documentation, adds a new syntax for schema.yml files, and fixes a number of minor bugs. With the exception of planned changes to Snowflake's default quoting strategy, this release should not contain any breaking changes. Check out the release blog post for more information about this release.

Breaking Changes

  • Change default Snowflake quoting strategy to "unquoted" (docs)

Features

  • Add autogenerated dbt project documentation (docs)
  • Version 2 of schema.yml, which allows users to create table and column comments that end up in the manifest (docs)
  • Remove distinction between this.table and this.schema by refactoring materialization SQL

For a full list of changes, check out the changelog.

Resources for upgrading to 0.11:

Contributors

Thanks for contributing!

Isaac Asimov

Isaac Asimov (1920-1992) was a prolific writer of science fiction. He is best known for his "Foundation" series, although he is credited with writing and editing thousands of works of both fiction and nonfiction. Though he was born in New York, he lived in West Philadelphia from 1942 to 1949 with his wife, Gertrude. During this time he wrote a handful of short stories which would later become "Foundation," the first novel in the famous series.

In "Foundation," the mathematician Hari Seldon uses the advanced science of psychohistory to predict the future of human civilization. Concerned by what he foresees, Seldon is allowed to gather the most gifted scientific minds in the galaxy onto a remote planet to create the Encyclopedia Galactica, a compendium of all human knowledge, in the hopes that this knowledge will be preserved for tens of thousands of years and prevent the future downfall of human civilization. The Encyclopedists spend hundreds of years compiling the Encyclopedia, and in the process, they create a rich, new civilization with scientific prowess at the forefront.

Inspired by Hari Seldon's effort, we hope that dbt 0.11 helps you capture and share the important knowledge in your data warehouse.

dbt 0.10.2

03 Aug 18:22
b4480cb
Compare
Choose a tag to compare

This release makes it possible to alias relation names, rounds out support for BigQuery with incremental, archival, and hook support, adds the IAM Auth method for Redshift, and builds the foundation for autogenerated dbt project documentation, to come in the next release.

Additionally, a number of bugs have been fixed including intermittent BigQuery 404 errors, Redshift "table dropped by concurrent query" errors, and a probable fix for Redshift connection timeout issues. For a complete list of changes, check out the Changelog.

tl;dr

Contributors

We want to extend a big thank you to our outside contributors for this release! You all are amazing.

A word on future releases

Going forwards, we'll be naming planned released after famous Philadelphians. We'll include a brief biography of these historical figures in the dbt release notes. This is our little way of exporting Philly culture across the world. You can read a short bio of Betsy Ross, courtesy of @jtcohen6, below 🔔

Betsy Ross

Betsy Ross (1752-1836) was an American seamstress and upholsterer famously associated with three legends: having sewn the ‘first’ American flag for General George Washington in 1776; having kept Hessian Colonel Carl von Dunup from fighting in the Battle of Trenton, the one for which Washington crossed the Delaware; and having lived in a still-standing, built-in-1740 house near 2nd & Arch that remains one of Philadelphia’s most visited tourist attractions.

The first-flag story dates to the 1876 American centennial (celebrated where else but in Philly), when a grandson invented details of Ross’s fateful, five-pointed-star meeting with Washington—though not out of whole cloth, since Ross did sew naval ensigns and later designs. Who knows about von Dunup. As far as 239 Arch, she most likely lived right next door.

We would be wrong, however, to dismiss mythologies as mere falsehoods. Though Ross’s stature in the pantheon of American revolutionaries is somewhat a fabrication, her place in Philadelphia was very real. She grew up in a strict, Quaker household, eloped with a fellow apprentice (and nephew of a signer of the Declaration of Independence), and they attended Christ Church in Old City. She married three times, her first two husbands both casualties of the revolution against Britain. She lived in the heart of the capital of the brand-new United States until, as all good Philadelphians must, she moved to be with her children in Montgomery County.

The Flag Resolution of 1777, officially replacing the Continental Colors / Grand Union Flag with the Stars and Stripes, is one sentence and 31 words long. Ross’s story wraps Old Glory in a more compelling narrative of it-takes-a-city ingenuity. The historical Betsy Ross was a significant Philadelphian whose life spanned several transformative decades; the mythical Ross helped stitch together the tapestry of a nation.