fix(report): too long report causes MSG_SIZE_TOO_LARGE in kafka

[sgomezvillamor]

While investigating following error:

cimpl.KafkaException: KafkaError{code=MSG_SIZE_TOO_LARGE,val=10,str="Unable to produce message: Broker: Message size too large"}

we found that was produced because of the following messages including call stacks were added to the reports:

              'products_events_behaviour.formsaveerror': ['unable to get column information due to an error -> (pyhive.exc.OperationalError) TExecuteStatementResp(status=TStatus(statusCode=3, infoMessages=["*org.apache.hive.service.cli.HiveSQLException:Error running query: org.apache.spark.sql.AnalysisException: s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist:140:139", \'org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$:runningQueryError:HiveThriftServerErrors.scala:47\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute:SparkExecuteStatementOperation.scala:435\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:runInternal:SparkExecuteStatementOperation.scala:209\', \'org.apache.hive.service.cli.operation.Operation:run:Operation.java:383\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkOperation$$super$run:SparkExecuteStatementOperation.scala:52\', \'org.apache.spark.sql.hive.thriftserver.SparkOperation:run:SparkOperation.scala:106\', \'org.apache.spark.sql.hive.thriftserver.SparkOperation:run$:SparkOperation.scala:104\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:run:SparkExecuteStatementOperation.scala:52\', \'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:573\', \'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatement:HiveSessionImpl.java:549\', \'org.apache.hive.service.cli.CLIService:executeStatement:CLIService.java:305\', \'org.apache.hive.service.cli.thrift.OSSTCLIServiceIface:ExecuteStatement:ThriftCLIService.java:1059\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.QueryRejectionThriftHandler:ExecuteStatement:QueryRejectionThriftHandler.scala:45\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SQLGatewaySessionConfThriftHandler:ExecuteStatement:SQLGatewaySessionConfThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SQLGatewaySessionStateThriftHandler:ExecuteStatement:SQLGatewaySessionStateThriftHandler.scala:307\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.FetchProtocolThriftHandler:ExecuteStatement:FetchProtocolThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.GetResultSetMetadataThriftHandler:ExecuteStatement:GetResultSetMetadataThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.InitialNamespaceThriftHandler:ExecuteStatement:InitialNamespaceThriftHandler.scala:33\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SessionCapabilitiesHandler:ExecuteStatement:SessionCapabilitiesHandler.scala:27\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.AsyncMetadataOperationsThriftHandler:ExecuteStatement:AsyncMetadataOperationsThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.GetInfoInOpenSessionThriftHandler:ExecuteStatement:GetInfoInOpenSessionThriftHandler.scala:36\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ResultSetMetadataInFetchResultsThriftHandler:ExecuteStatement:ResultSetMetadataInFetchResultsThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DirectResultsThriftHandler:ExecuteStatement:DirectResultsThriftHandler.scala:58\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ProtocolVersionThriftHandler:ExecuteStatement:ProtocolVersionThriftHandler.scala:29\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.FilterJDBCStatementBlockListThriftHandler:ExecuteStatement:FilterJDBCStatementBlockListThriftHandler.scala:54\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:super$ExecuteStatement:LocalPropertiesThriftHandler.scala:90\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:$anonfun$ExecuteStatement$1:LocalPropertiesThriftHandler.scala:90\', \'org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties:withLocalProperties:ThriftLocalProperties.scala:123\', \'org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties:withLocalProperties$:ThriftLocalProperties.scala:48\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:withLocalProperties:LocalPropertiesThriftHandler.scala:30\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:ExecuteStatement:LocalPropertiesThriftHandler.scala:90\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:super$ExecuteStatement:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:$anonfun$ExecuteStatement$1:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.logging.Log4jUsageLoggingShim$:$anonfun$withAttributionContext$1:Log4jUsageLoggingShim.scala:33\', \'scala.util.DynamicVariable:withValue:DynamicVariable.scala:62\', \'com.databricks.logging.AttributionContext$:withValue:AttributionContext.scala:94\', \'com.databricks.logging.Log4jUsageLoggingShim$:withAttributionContext:Log4jUsageLoggingShim.scala:31\', \'com.databricks.logging.UsageLogging:withAttributionContext:UsageLogging.scala:205\', \'com.databricks.logging.UsageLogging:withAttributionContext$:UsageLogging.scala:204\', \'com.databricks.spark.util.PublicDBLogging:withAttributionContext:DatabricksSparkUsageLogger.scala:22\', \'com.databricks.logging.UsageLogging:withAttributionTags:UsageLogging.scala:240\', \'com.databricks.logging.UsageLogging:withAttributionTags$:UsageLogging.scala:225\', \'com.databricks.spark.util.PublicDBLogging:withAttributionTags:DatabricksSparkUsageLogger.scala:22\', \'com.databricks.spark.util.PublicDBLogging:withAttributionTags0:DatabricksSparkUsageLogger.scala:63\', \'com.databricks.spark.util.DatabricksSparkUsageLogger:withAttributionTags:DatabricksSparkUsageLogger.scala:163\', \'com.databricks.spark.util.UsageLogging:$anonfun$withAttributionTags$1:UsageLogger.scala:493\', \'com.databricks.spark.util.UsageLogging$:withAttributionTags:UsageLogger.scala:605\', \'com.databricks.spark.util.UsageLogging$:withAttributionTags:UsageLogger.scala:614\', \'com.databricks.spark.util.UsageLogging:withAttributionTags:UsageLogger.scala:493\', \'com.databricks.spark.util.UsageLogging:withAttributionTags$:UsageLogger.scala:491\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:withAttributionTags:SetNextOperationIdThriftHandler.scala:37\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:ExecuteStatement:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SetSessionHandleProtocolVersionThriftHandler:ExecuteStatement:SetSessionHandleProtocolVersionThriftHandler.scala:51\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:super$ExecuteStatement:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:$anonfun$ExecuteStatement$1:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:createThriftThreadLocalContext:ThreadLocalContextThriftHandler.scala:45\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:ExecuteStatement:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:super$ExecuteStatement:ErrorPropagationThriftHandler.scala:73\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:$anonfun$ExecuteStatement$1:ErrorPropagationThriftHandler.scala:73\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:runSafely:ErrorPropagationThriftHandler.scala:119\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:ExecuteStatement:ErrorPropagationThriftHandler.scala:73\', \'org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:425\', \'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1437\', \'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1422\', \'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:38\', \'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39\', \'org.apache.thrift.server.TServlet:doPost:TServlet.java:83\', \'org.apache.hive.service.cli.thrift.ThriftHttpServlet:doPost:ThriftHttpServlet.java:95\', \'javax.servlet.http.HttpServlet:service:HttpServlet.java:523\', \'javax.servlet.http.HttpServlet:service:HttpServlet.java:590\', \'org.eclipse.jetty.servlet.ServletHolder:handle:ServletHolder.java:799\', \'org.eclipse.jetty.servlet.ServletHandler:doHandle:ServletHandler.java:550\', \'org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:143\', \'org.eclipse.jetty.security.SecurityHandler:handle:SecurityHandler.java:602\', \'org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:235\', \'org.eclipse.jetty.server.session.SessionHandler:doHandle:SessionHandler.java:1624\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:233\', \'org.eclipse.jetty.server.handler.ContextHandler:doHandle:ContextHandler.java:1434\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:188\', \'org.eclipse.jetty.servlet.ServletHandler:doScope:ServletHandler.java:501\', \'org.eclipse.jetty.server.session.SessionHandler:doScope:SessionHandler.java:1594\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:186\', \'org.eclipse.jetty.server.handler.ContextHandler:doScope:ContextHandler.java:1349\', \'org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:141\', \'org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127\', \'org.eclipse.jetty.server.Server:handle:Server.java:516\', \'org.eclipse.jetty.server.HttpChannel:lambda$handle$1:HttpChannel.java:388\', \'org.eclipse.jetty.server.HttpChannel:dispatch:HttpChannel.java:633\', \'org.eclipse.jetty.server.HttpChannel:handle:HttpChannel.java:380\', \'org.eclipse.jetty.server.HttpConnection:onFillable:HttpConnection.java:277\', \'org.eclipse.jetty.io.AbstractConnection$ReadCallback:succeeded:AbstractConnection.java:311\', \'org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105\', \'org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint:onFillable:SslConnection.java:555\', \'org.eclipse.jetty.io.ssl.SslConnection:onFillable:SslConnection.java:410\', \'org.eclipse.jetty.io.ssl.SslConnection$2:succeeded:SslConnection.java:164\', \'org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105\', \'org.eclipse.jetty.io.ChannelEndPoint$1:run:ChannelEndPoint.java:104\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:runTask:EatWhatYouKill.java:338\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:doProduce:EatWhatYouKill.java:315\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:tryProduce:EatWhatYouKill.java:173\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:run:EatWhatYouKill.java:131\', \'org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread:run:ReservedThreadExecutor.java:386\', \'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149\', \'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624\', \'java.lang.Thread:run:Thread.java:750\', "*org.apache.spark.sql.AnalysisException:s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist:140:1", \'org.apache.spark.sql.AnalysisException:copy:AnalysisException.scala:71\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute:SparkExecuteStatementOperation.scala:427\'], sqlState=None, errorCode=0, errorMessage="Error running query: org.apache.spark.sql.AnalysisException: s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist"), operationHandle=None)\n[SQL: DESCRIBE `products_events_behaviour`.`formsaveerror`]\n(Background on this error at: https://sqlalche.me/e/14/e3q8)', 'Ingestion error: (pyhive.exc.OperationalError) TExecuteStatementResp(status=TStatus(statusCode=3, infoMessages=["*org.apache.hive.service.cli.HiveSQLException:Error running query: org.apache.spark.sql.AnalysisException: s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist:140:139", \'org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$:runningQueryError:HiveThriftServerErrors.scala:47\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute:SparkExecuteStatementOperation.scala:435\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:runInternal:SparkExecuteStatementOperation.scala:209\', \'org.apache.hive.service.cli.operation.Operation:run:Operation.java:383\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkOperation$$super$run:SparkExecuteStatementOperation.scala:52\', \'org.apache.spark.sql.hive.thriftserver.SparkOperation:run:SparkOperation.scala:106\', \'org.apache.spark.sql.hive.thriftserver.SparkOperation:run$:SparkOperation.scala:104\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:run:SparkExecuteStatementOperation.scala:52\', \'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:573\', \'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatement:HiveSessionImpl.java:549\', \'org.apache.hive.service.cli.CLIService:executeStatement:CLIService.java:305\', \'org.apache.hive.service.cli.thrift.OSSTCLIServiceIface:ExecuteStatement:ThriftCLIService.java:1059\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.QueryRejectionThriftHandler:ExecuteStatement:QueryRejectionThriftHandler.scala:45\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SQLGatewaySessionConfThriftHandler:ExecuteStatement:SQLGatewaySessionConfThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SQLGatewaySessionStateThriftHandler:ExecuteStatement:SQLGatewaySessionStateThriftHandler.scala:307\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.FetchProtocolThriftHandler:ExecuteStatement:FetchProtocolThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.GetResultSetMetadataThriftHandler:ExecuteStatement:GetResultSetMetadataThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.InitialNamespaceThriftHandler:ExecuteStatement:InitialNamespaceThriftHandler.scala:33\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SessionCapabilitiesHandler:ExecuteStatement:SessionCapabilitiesHandler.scala:27\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.AsyncMetadataOperationsThriftHandler:ExecuteStatement:AsyncMetadataOperationsThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.GetInfoInOpenSessionThriftHandler:ExecuteStatement:GetInfoInOpenSessionThriftHandler.scala:36\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ResultSetMetadataInFetchResultsThriftHandler:ExecuteStatement:ResultSetMetadataInFetchResultsThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DirectResultsThriftHandler:ExecuteStatement:DirectResultsThriftHandler.scala:58\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ProtocolVersionThriftHandler:ExecuteStatement:ProtocolVersionThriftHandler.scala:29\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.FilterJDBCStatementBlockListThriftHandler:ExecuteStatement:FilterJDBCStatementBlockListThriftHandler.scala:54\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:super$ExecuteStatement:LocalPropertiesThriftHandler.scala:90\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:$anonfun$ExecuteStatement$1:LocalPropertiesThriftHandler.scala:90\', \'org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties:withLocalProperties:ThriftLocalProperties.scala:123\', \'org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties:withLocalProperties$:ThriftLocalProperties.scala:48\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:withLocalProperties:LocalPropertiesThriftHandler.scala:30\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:ExecuteStatement:LocalPropertiesThriftHandler.scala:90\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:super$ExecuteStatement:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:$anonfun$ExecuteStatement$1:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.logging.Log4jUsageLoggingShim$:$anonfun$withAttributionContext$1:Log4jUsageLoggingShim.scala:33\', \'scala.util.DynamicVariable:withValue:DynamicVariable.scala:62\', \'com.databricks.logging.AttributionContext$:withValue:AttributionContext.scala:94\', \'com.databricks.logging.Log4jUsageLoggingShim$:withAttributionContext:Log4jUsageLoggingShim.scala:31\', \'com.databricks.logging.UsageLogging:withAttributionContext:UsageLogging.scala:205\', \'com.databricks.logging.UsageLogging:withAttributionContext$:UsageLogging.scala:204\', \'com.databricks.spark.util.PublicDBLogging:withAttributionContext:DatabricksSparkUsageLogger.scala:22\', \'com.databricks.logging.UsageLogging:withAttributionTags:UsageLogging.scala:240\', \'com.databricks.logging.UsageLogging:withAttributionTags$:UsageLogging.scala:225\', \'com.databricks.spark.util.PublicDBLogging:withAttributionTags:DatabricksSparkUsageLogger.scala:22\', \'com.databricks.spark.util.PublicDBLogging:withAttributionTags0:DatabricksSparkUsageLogger.scala:63\', \'com.databricks.spark.util.DatabricksSparkUsageLogger:withAttributionTags:DatabricksSparkUsageLogger.scala:163\', \'com.databricks.spark.util.UsageLogging:$anonfun$withAttributionTags$1:UsageLogger.scala:493\', \'com.databricks.spark.util.UsageLogging$:withAttributionTags:UsageLogger.scala:605\', \'com.databricks.spark.util.UsageLogging$:withAttributionTags:UsageLogger.scala:614\', \'com.databricks.spark.util.UsageLogging:withAttributionTags:UsageLogger.scala:493\', \'com.databricks.spark.util.UsageLogging:withAttributionTags$:UsageLogger.scala:491\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:withAttributionTags:SetNextOperationIdThriftHandler.scala:37\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:ExecuteStatement:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SetSessionHandleProtocolVersionThriftHandler:ExecuteStatement:SetSessionHandleProtocolVersionThriftHandler.scala:51\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:super$ExecuteStatement:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:$anonfun$ExecuteStatement$1:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:createThriftThreadLocalContext:ThreadLocalContextThriftHandler.scala:45\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:ExecuteStatement:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:super$ExecuteStatement:ErrorPropagationThriftHandler.scala:73\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:$anonfun$ExecuteStatement$1:ErrorPropagationThriftHandler.scala:73\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:runSafely:ErrorPropagationThriftHandler.scala:119\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:ExecuteStatement:ErrorPropagationThriftHandler.scala:73\', \'org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:425\', \'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1437\', \'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1422\', \'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:38\', \'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39\', \'org.apache.thrift.server.TServlet:doPost:TServlet.java:83\', \'org.apache.hive.service.cli.thrift.ThriftHttpServlet:doPost:ThriftHttpServlet.java:95\', \'javax.servlet.http.HttpServlet:service:HttpServlet.java:523\', \'javax.servlet.http.HttpServlet:service:HttpServlet.java:590\', \'org.eclipse.jetty.servlet.ServletHolder:handle:ServletHolder.java:799\', \'org.eclipse.jetty.servlet.ServletHandler:doHandle:ServletHandler.java:550\', \'org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:143\', \'org.eclipse.jetty.security.SecurityHandler:handle:SecurityHandler.java:602\', \'org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:235\', \'org.eclipse.jetty.server.session.SessionHandler:doHandle:SessionHandler.java:1624\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:233\', \'org.eclipse.jetty.server.handler.ContextHandler:doHandle:ContextHandler.java:1434\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:188\', \'org.eclipse.jetty.servlet.ServletHandler:doScope:ServletHandler.java:501\', \'org.eclipse.jetty.server.session.SessionHandler:doScope:SessionHandler.java:1594\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:186\', \'org.eclipse.jetty.server.handler.ContextHandler:doScope:ContextHandler.java:1349\', \'org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:141\', \'org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127\', \'org.eclipse.jetty.server.Server:handle:Server.java:516\', \'org.eclipse.jetty.server.HttpChannel:lambda$handle$1:HttpChannel.java:388\', \'org.eclipse.jetty.server.HttpChannel:dispatch:HttpChannel.java:633\', \'org.eclipse.jetty.server.HttpChannel:handle:HttpChannel.java:380\', \'org.eclipse.jetty.server.HttpConnection:onFillable:HttpConnection.java:277\', \'org.eclipse.jetty.io.AbstractConnection$ReadCallback:succeeded:AbstractConnection.java:311\', \'org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105\', \'org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint:onFillable:SslConnection.java:555\', \'org.eclipse.jetty.io.ssl.SslConnection:onFillable:SslConnection.java:410\', \'org.eclipse.jetty.io.ssl.SslConnection$2:succeeded:SslConnection.java:164\', \'org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105\', \'org.eclipse.jetty.io.ChannelEndPoint$1:run:ChannelEndPoint.java:104\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:runTask:EatWhatYouKill.java:338\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:doProduce:EatWhatYouKill.java:315\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:tryProduce:EatWhatYouKill.java:173\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:run:EatWhatYouKill.java:131\', \'org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread:run:ReservedThreadExecutor.java:386\', \'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149\', \'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624\', \'java.lang.Thread:run:Thread.java:750\', "*org.apache.spark.sql.AnalysisException:s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist:140:1", \'org.apache.spark.sql.AnalysisException:copy:AnalysisException.scala:71\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute:SparkExecuteStatementOperation.scala:427\'], sqlState=None, errorCode=0, errorMessage="Error running query: org.apache.spark.sql.AnalysisException: s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist"), operationHandle=None)\n[SQL: DESCRIBE FORMATTED `products_events_behaviour`.`formsaveerror`]\n(Background on this error at: https://sqlalche.me/e/14/e3q8)'],

This is preventing (or reducing the change of) the issue by limiting the reason message when adding a warn or a failure to the report, while keeping the full message in the logger.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[sgomezvillamor]

Alternatively, cutting the reason message could be done in the report_warning method

datahub/metadata-ingestion/src/datahub/ingestion/api/source.py

Line 97 in 85fa5a1

warnings.append(reason)

To be decided by the reviewer 😅

[jjoyce0510]

This is looking great to me. Awaiting final review from @hsheth2 and team!

[hsheth2]

LGTM

[hsheth2]

CI failure is unrelated.