11 Sep 16:39

iprentic

68ae3bf

v0.11.0

Release Highlights

Potential Downtime

This release introduces substantial improvements to search ranking which require reindexing indices.

During the reindexing:

a system-update job will set indices to read-only and create a backup/clone of each index
new components will be prevented from start-up until the reindex completes
Helm deployments will go into read-only mode and new ingestion runs will fail

This process can take anywhere from 5 minutes to multiple hours; as a rough estimate, please expect it to take 1 hour for every 2.3 million entities. After the reindex is complete, please check your ingestion run to re-run any that did not complete.

User Experience

New Search and Browse Experience

We have some really exciting improvements to the DataHub user experience in this release! The new search and browse experience, which was first made available in the previous release behind a feature flag, is now on by default. Check out our release notes for v0.10.5 to get more information and documentation on this new Browse experience.

Learn all about the new Search and Browse experience!

Improvements to Search

In addition to the ranking changes mentioned above, this release includes changes to the highlighting of search entities to understand why they match your query. You can also sort your results alphabetically or by last updated times, in addition to relevance. In this release, we suggest a correction if your query has a typo in it.

See the Search improvements in action!

Manage Home Page Posts

In this release we now enable you to create and delete pinned announcements on your DataHub homepage! If you have the “Manage Home Page Posts” platform privilege you’ll see a new section in settings called “Home Page Posts” where you can create and delete text posts and link posts that your users see on the home page.

OpenAPI Endpoints Expanded

OpenAPI entity and aspect endpoints expanded to improve developer experience when using this API with additional aspects to be added in the near future.

Metadata ingestion

Added support for Confluent S3 Sink Connector, extracting stored procedures and jobs from mssql, and snowflake shares. Additionally, sql parsing source now converts query logs into CLL and usage.

Developer Experience

The CLI now supports recursive deletes.

Versioned documentation

Starting from this release, we support versioned documentation on the datahub docs site! Select the version you’re on and browse docs specifically at that version.

Performance Improvements

Batching of default aspects on initial ingestion (SQL)
Improvements to multi-threading. Ingestion recipes, if previously reduced to 1 thread, can be restored to the 15 thread default.
Gradle 7 upgrade moderately improves build speed
DataHub Ingestion slim images reduced in size by 2GB+

Important Bug Fixes

Glue Schema Registry fixed

Deprecation Notice

MAE Events are no longer produced. MAE events have been deprecated for over a year.

What's Changed

feat(ingest/presto-on-hive): enable partition key for presto-on-hive by @zheyu001 in #8380
feat(classification): allow parallelisation to reduce time by @mayurinehate in #8368
feat(ingest): Add metabase database id to platform instance mapping by @k-popov in #8359
feat(ingest): add ability to read other method types than GET for OAS ingest recipes by @jsmilkstein in #8303
fix(ingest): fix data platform urn in dataset_urn_to_key and dataset_key_to_urn by @Masterchen09 in #8209
fix(ingest/s3): wrong sorting in case of multi-partition key by @anshbansal in #8536
fix(ingest/presto): fix presto on hive test failures by @hsheth2 in #8548
Cypress test for managing groups by @kkorchak in #8520
feat(ingest/kafka-connect): add support for Confluent S3 Sink Connector by @tusharm in #8298
Variable rename - Allows deselection of members in add members modal for a group by @Sukeerthi31 in #8529
fix(ingest/s3): catch no such bucket exception instead of failing by @anshbansal in #8549
fix(ingest): add tableau sqlglot dep by @hsheth2 in #8552
fix(ingetion/mssql): convert dataset urns to lowercase by @siddiquebagwan in #8551
Fix flaky add_user smoke test by @kkorchak in #8471
feat(ci): use docker registry cache by @hsheth2 in #8544
fix(glue): restore glue configurations by @RyanHolstien in #8533
build(release): Update files for 0.10.5 release by @iprentic in #8556
docs(release): Update updating-datahub.md for 0.10.5 release by @iprentic in #8557
feat(ingestion/snowflake): use user email-id in urn generation for top users stat by @siddiquebagwan in #8513
docs(development.md): Minor grammatical error by @PauloGoncalvesLima in #8558
fix(usage): Update index lifecycle policy to not delete old datahub usage events by @iprentic in #8565
fix(ui): Simplify background color for Entity Health Status popover by @jjoyce0510 in #8559
fix: add --write args on pre-commit prettier by @yoonhyejin in #8560
docs(observe): Add feature doc for Freshness Assertions by @jjoyce0510 in #8547
docs(updating): add details on Unified Search & Browse experience by @maggiehays in #8568
fix: fix features section by @yoonhyejin in #8571
feat(ingest): allow lower freq profiling based on date of month/day of week by @anshbansal in #8489
fix(stats): default to 3 months by @anshbansal in #8566
fix(aspect): count query only for relevant aspect index by @iprentic in #8569
feat(quickstart): bump quickstart start periods more by @hsheth2 in #8573
Origin/cypress test for managing policies by @kkorchak in #8554
feat(ui) Show source documentation when editing entity documentation by @chriscollins3456 in #8516
fix(ingest): handle redaction of configs with int keys by @hsheth2 in #8545
fix(ingest/snowflake): maintain qualified name casing, do not lowercase by @mayurinehate in #8574
feat(docs): add github repo links to readme and docs by @yoonhyejin in #8422
feat(ebean): Add metric in ebean aspect DAO for failed tries, as well as failed operation… by @iprentic in #8576
refactor(search) Use search across multiple-entities API, deprecate Aggregator classes by @iprentic in #8498
feat(siblings): dont show multiple platform icons if the siblings are ghost nodes by @gabe-lyons in #8543
docs(lineage): Add description to make_lineage_mce by @eboneil in #8596
doc(ingest/log): failure log at pipeline level document by @anshbansal in #8591
Dataset ownership test by @kkorchak in #8583
doc(release): release notes for 0.2.10 by @anshbansal in #8599
docs(release): fix typo by @anshbansal in #8600
feat(ui): apply views to: domains, containers, terms by @eboneil in #8572
feat(search): embedded view dropdown by @joshuaeilers in #8598
fix(ingest/file): remove entity_type_counts and aspect_counts by @hsheth2 in #8586
fix(ingest): use hive pure_sasl variant by @hsheth2 in #8570
Feat(ingest/ldap)fix list index out of range error by @alplatonov in https://githu...

Contributors

tusharm, githendrik, and 40 other contributors

Assets 2

02 Aug 03:58

david-leifker

v0.10.5

4f9fc67

v0.10.5

Release Highlights

NEW: Unified Search and Browse Experience

It’s here, it’s here! We are incredibly excited to roll out our re-designed, streamlined Search and Browse experience. End-users now have a one-stop-shop to search for specific data entities and browse across systems, making it easier than ever to find the most relevant and meaningful resources within DataHub.

Checkout the screenshot below and get a full walk-through in this video!