From a9f73811a4f00bc91c9410d4e1a6cf0e1cff7ec1 Mon Sep 17 00:00:00 2001 From: Karthik Kalyanaraman <105607645+karthikscale3@users.noreply.github.com> Date: Fri, 17 May 2024 09:03:01 -0700 Subject: [PATCH] Release 1.4.0 (#108) * Pagination bug * Bug fix * chore: add docker cmd * Compatibility fixes for SDK version 2.0.0 (#69) * Pagination bug * Bug fix * Fix for schema changes * Render tool calling * Support for Langgraph, Qdrant & Groq (#73) * Pagination bug * Bug fix * Add langgraph support * QDrant support * Add Groq support * update README * update README * feat: optimise docker image for self host setup * adding api access to traces endpoint * clean up * refactor * feat: add clickhouse db create on app start (#79) * docs: add railway deploy, fix sdk badges (#81) * untrack .env * Revert "untrack .env" This reverts commit 4551d7eb33a22b4ba94be8eb1e20d620c6e0a567. * Playground and Prompt Management (#83) * Pagination bug * Bug fix * Playground - basic implementation * Playground - streaming and nonstreaming * Move playground inside project * API key flow * Api key * Playground refactor * Add chat hookup * anthropic streaming support * Bug fixes to openai playground * Anthropic bugfixes * Anthropic bugfix * Cohere first iteration * Cohere role fixes * Cohere api fix * Parallel running * Playground cost calculation non streaming * playground - streaming token calculation * latency and cost * Support for Groq * Add model name * Prompt management views * Remove current promptset flow * Prompt management API hooks * Prompt registry final * Playground bugfixes * Bug fix playground * Rearrange project nav * Fix playground * Fix prompts * Bugfixes * Minor fix * Prompt versioning bugfix * Bugfix * fix: clickhouse table find queries (#82) * Fix to surface multiple LLM requests inside LLM View (#84) * Pagination bug * Bug fix * Fix for surfacing multiple LLM requests in LLMView * Minor bugfixes (#86) * Pagination bug * Bug fix * Bugfixes * api to fetch promptset with prompt filters * bug fixes * fix invalid redirect * fix invalid status code * Project Switcher (#90) * Pagination bug * Bug fix * Project switcher * Feat: dataset download (#60) * API: download dataset * API: Download dataset * updated download-dataset api * Updated: download_dataset api * Updated download dataset API * Updated Download API: changed Response to Next Response, add a condition to ensure max page size is 500 * updated the download-dataset API: fixed the format and removed redundant lines of code * Updated download_daatset API: file name and removed 'id' param * Added the Download dataset button. * Merged developemnt into my branch * Updated button size * Fixes --------- Co-authored-by: Karthik Kalyanaraman * Update prompt registry with instructions to fetch prompts (#91) * Pagination bug * Bug fix * Update prompt registry * Minor bugfix (#94) * Pagination bug * Bug fix * Minor bugfix * chore: update github repo badges * optimizing token count function * Add GPT4-O Pricing and Playground (#98) * Pagination bug * Bug fix * Add GPT4-O support * Add GPT4-O support * Update cost * Dylan/s3en 2234 add perplexity support to playground (#89) * adding perplexity to playground types * adding ui stuff:' * adding perplexity chat api * fixing perplexity model dropdown --------- Co-authored-by: Karthik Kalyanaraman * api changes * add api access to get api and fix all bugs * bug fix * bug fix * updating descriptions to optional * prio python * cleanup and fixes * more bug fixes * more fixes * remove console log * updating trace_service functions * add migration * add format function, updating from day to hour * adding dropwdown menu * updating query key * updating query keys v2 * clean up * fix bug * Minor bugfix (#102) * Pagination bug * Bug fix * Minor bugfix --------- Co-authored-by: Darshit Suratwala Co-authored-by: darshit-s3 <119623510+darshit-s3@users.noreply.github.com> Co-authored-by: dylan Co-authored-by: dylanzuber-scale3 <116033320+dylanzuber-scale3@users.noreply.github.com> Co-authored-by: Rohit Kadhe Co-authored-by: Rohit Kadhe <113367036+rohit-kadhe@users.noreply.github.com> Co-authored-by: MayuriS24 <159064413+MayuriS24@users.noreply.github.com> --- .../evaluations/[test_id]/page.tsx | 22 +- app/api/dataset/download/route.ts | 2 - app/api/evaluation/route.ts | 434 ++++++++++-------- app/api/metrics/accuracy/route.ts | 118 +++-- app/api/metrics/latency/trace/route.ts | 11 +- app/api/metrics/tests/route.ts | 4 +- app/api/metrics/usage/cost/route.ts | 2 +- app/api/metrics/usage/span/route.ts | 2 +- app/api/metrics/usage/token/route.ts | 2 +- app/api/metrics/usage/trace/route.ts | 2 +- components/charts/eval-chart.tsx | 63 +-- components/charts/latency-chart.tsx | 18 +- components/charts/model-accuracy-chart.tsx | 126 ----- components/charts/token-chart.tsx | 27 +- components/charts/trace-chart.tsx | 27 +- components/evaluations/create-test.tsx | 6 +- components/evaluations/eval-dialog.tsx | 13 +- components/evaluations/evaluation-row.tsx | 19 +- components/evaluations/evaluation-table.tsx | 1 + components/project/create.tsx | 6 +- components/project/dataset/create.tsx | 12 +- components/project/metrics.tsx | 64 ++- components/shared/setup-instructions.tsx | 44 +- .../scale3_clickhouse/client/client.ts | 2 +- lib/services/trace_service.ts | 130 +++--- lib/utils.ts | 24 +- package.json | 4 +- .../migration.sql | 13 + .../migration.sql | 9 + prisma/schema.prisma | 29 +- scripts/create-clickhouse-db.ts | 2 +- 31 files changed, 632 insertions(+), 606 deletions(-) delete mode 100644 components/charts/model-accuracy-chart.tsx create mode 100644 prisma/migrations/20240513220159_rename_score_and_userid/migration.sql create mode 100644 prisma/migrations/20240513234910_evaluation_fields_optional/migration.sql diff --git a/app/(protected)/project/[project_id]/evaluations/[test_id]/page.tsx b/app/(protected)/project/[project_id]/evaluations/[test_id]/page.tsx index da6bb2e5..1d1596bf 100644 --- a/app/(protected)/project/[project_id]/evaluations/[test_id]/page.tsx +++ b/app/(protected)/project/[project_id]/evaluations/[test_id]/page.tsx @@ -52,7 +52,7 @@ export default function Page() { }, }); - const [score, setScore] = useState(testData?.test?.min || -1); + const [score, setScore] = useState(testData?.test?.min ?? -1); const [scorePercent, setScorePercent] = useState(0); const [color, setColor] = useState("red"); const [span, setSpan] = useState(null); @@ -179,7 +179,7 @@ export default function Page() { } const result = await response.json(); const sc = - result.evaluations.length > 0 ? result.evaluations[0].score : -1; + result.evaluations.length > 0 ? result.evaluations[0].ltUserScore ?? -1 : -1; onScoreSelected(sc); return result; }, @@ -220,7 +220,7 @@ export default function Page() { // Check if an evaluation already exists if (evaluationsData?.evaluations[0]?.id) { - if (evaluationsData.evaluations[0].score === score) { + if (evaluationsData.evaluations[0].ltUserScore === score) { // setBusy(false); return; } @@ -232,7 +232,8 @@ export default function Page() { }, body: JSON.stringify({ id: evaluationsData.evaluations[0].id, - score: score, + ltUserScore: score, + testId }), }); queryClient.invalidateQueries({ @@ -249,12 +250,7 @@ export default function Page() { projectId: projectId, spanId: span.span_id, traceId: span.trace_id, - spanStartTime: span?.start_time - ? new Date(correctTimestampFormat(span.start_time)) - : new Date(), - score: score, - model: model, - prompt: systemPrompt, + ltUserScore: score, testId: testId, }), }); @@ -365,7 +361,7 @@ export default function Page() { ))} ) : ( -
diff --git a/app/api/dataset/download/route.ts b/app/api/dataset/download/route.ts index ae5640d0..36564551 100644 --- a/app/api/dataset/download/route.ts +++ b/app/api/dataset/download/route.ts @@ -60,8 +60,6 @@ export async function GET(req: NextRequest) { const timestamp = new Date().toISOString().slice(0, 19).replace(/[-:]/g, ''); const filename = `${datasetName}_${timestamp}.csv`; - console.log(`CSV file '${filename}' `); - return new NextResponse(csv, { headers: { 'Content-Type': 'text/csv', diff --git a/app/api/evaluation/route.ts b/app/api/evaluation/route.ts index c248ff11..918b1566 100644 --- a/app/api/evaluation/route.ts +++ b/app/api/evaluation/route.ts @@ -1,121 +1,168 @@ import { authOptions } from "@/lib/auth/options"; import prisma from "@/lib/prisma"; +import { authApiKey, convertToDateTime64 } from "@/lib/utils"; import { getServerSession } from "next-auth"; -import { redirect } from "next/navigation"; import { NextRequest, NextResponse } from "next/server"; export async function POST(req: NextRequest) { - const session = await getServerSession(authOptions); - if (!session || !session.user) { - redirect("/login"); - } - const email = session?.user?.email as string; - if (!email) { - return NextResponse.json( - { - error: "email not found", + const apiKey = req.headers.get("x-api-key"); + if(apiKey!==null) { + const response = await authApiKey(apiKey); + if(response.status!==200) { + return NextResponse.json({ error: "Unauthorized" }, { status: 401 } ); + } + const projectData = await response.json(); + const projectId = projectData.data.project.id; + const data = await req.json(); + const { + traceId, + spanId, + userScore, + userId + } = data; + // check if an evaluation already exists for the spanId + const existingEvaluation = await prisma.evaluation.findFirst({ + where: { + spanId, }, - { status: 404 } - ); - } + }); - const user = await prisma.user.findUnique({ - where: { - email, - }, - include: { - Team: true, - }, - }); + if (existingEvaluation) { + return NextResponse.json( + { + error: "Evaluation already exists for this span", + }, + { status: 400 } + ); + } - if (!user) { - return NextResponse.json( - { - error: "user not found", + const evaluation = await prisma.evaluation.create({ + data: { + spanId, + traceId, + projectId, + userId, + userScore }, - { status: 404 } - ); + }); + return NextResponse.json({ + data: evaluation, + }); + } else { + const session = await getServerSession(authOptions); + if (!session || !session.user) { + NextResponse.json({ error: "Unauthorized" }, { status: 401 } ); + } + const email = session?.user?.email as string; + if (!email) { + return NextResponse.json( + { + error: "email not found", + }, + { status: 404 } + ); } - const data = await req.json(); - const { - traceId, - spanId, - projectId, - model, - score, - spanStartTime, - prompt, - testId, - } = data; + const user = await prisma.user.findUnique({ + where: { + email, + }, + include: { + Team: true, + }, + }); - // check if this user has access to this project - const project = await prisma.project.findFirst({ - where: { - id: projectId, - teamId: user.teamId, - }, - }); + if (!user) { + return NextResponse.json( + { + error: "user not found", + }, + { status: 404 } + ); + } - if (!project) { - return NextResponse.json( - { - error: "User does not have access to this project", + const data = await req.json(); + const { + traceId, + spanId, + projectId, + ltUserScore, + testId, + } = data; + + // check if this user has access to this project + const project = await prisma.project.findFirst({ + where: { + id: projectId, + teamId: user.teamId, }, - { status: 403 } - ); - } + }); - // check if an evaluation already exists for the spanId - const existingEvaluation = await prisma.evaluation.findFirst({ - where: { - spanId, - }, - }); + if (!project) { + return NextResponse.json( + { + error: "User does not have access to this project", + }, + { status: 403 } + ); + } - if (existingEvaluation) { - return NextResponse.json( - { - error: "Evaluation already exists for this span", + // check if an evaluation already exists for the spanId + const existingEvaluation = await prisma.evaluation.findFirst({ + where: { + spanId, }, - { status: 400 } - ); - } + }); - const evaluation = await prisma.evaluation.create({ - data: { - spanStartTime, - spanId, - traceId, - userId: user.id, - projectId, - model, - score, - prompt, - testId, - }, - }); + if (existingEvaluation) { + return NextResponse.json( + { + error: "Evaluation already exists for this span", + }, + { status: 400 } + ); + } + + const evaluation = await prisma.evaluation.create({ + data: { + spanId, + traceId, + ltUserId: user.id, + projectId, + ltUserScore, + testId, + }, + }); - return NextResponse.json({ - data: evaluation, - }); + return NextResponse.json({ + data: evaluation, + }); + } } export async function GET(req: NextRequest) { try { const session = await getServerSession(authOptions); - if (!session || !session.user) { - redirect("/login"); + const apiKey = req.headers.get("x-api-key"); + let projectId = req.nextUrl.searchParams.get("projectId") as string; + if ((!session || !session.user) && !apiKey) { + return NextResponse.json({ error: "Unauthorized" }, { status: 401 } ); + } + if(apiKey){ + const response = await authApiKey(apiKey); + if(response.status!==200) { + return NextResponse.json({ error: "Unauthorized" }, { status: 401 } ); + } + const projectData = await response.json(); + projectId = projectData.data.project.id; } - const projectId = req.nextUrl.searchParams.get("projectId") as string; const spanId = req.nextUrl.searchParams.get("spanId") as string; - const prompt = req.nextUrl.searchParams.get("prompt") as string; - if (!projectId && !spanId && !prompt) { + if (!projectId && !spanId) { return NextResponse.json( { - message: "Please provide a projectId or spanId or prompt", + message: "Please provide a projectId or spanId", }, { status: 400 } ); @@ -139,56 +186,37 @@ export async function GET(req: NextRequest) { }); } - if (prompt) { - const evaluations = await prisma.evaluation.findMany({ + // check if this user has access to this project + if(session) { + const email = session?.user?.email as string; + const user = await prisma.user.findUnique({ where: { - prompt, + email, }, }); - - if (!evaluations) { - return NextResponse.json({ - evaluations: [], - }); + if (!user) { + return NextResponse.json( + { + message: "user not found", + }, + { status: 404 } + ); } - - return NextResponse.json({ - evaluations: [evaluations], - }); - } - - // check if this user has access to this project - const email = session?.user?.email as string; - const user = await prisma.user.findUnique({ - where: { - email, - }, - }); - - if (!user) { - return NextResponse.json( - { - message: "user not found", - }, - { status: 404 } - ); - } - - // check if this user has access to this project - const project = await prisma.project.findFirst({ - where: { - id: projectId, - teamId: user.teamId, - }, - }); - - if (!project) { - return NextResponse.json( - { - message: "User does not have access to this project", + // check if this user has access to this project + const project = await prisma.project.findFirst({ + where: { + id: projectId, + teamId: user.teamId, }, - { status: 403 } - ); + }); + if (!project) { + return NextResponse.json( + { + message: "User does not have access to this project", + }, + { status: 403 } + ); + } } const evaluations = await prisma.evaluation.findMany({ @@ -216,60 +244,104 @@ export async function GET(req: NextRequest) { export async function PUT(req: NextRequest) { const session = await getServerSession(authOptions); - if (!session || !session.user) { - redirect("/login"); - } - const email = session?.user?.email as string; - if (!email) { - return NextResponse.json( - { - error: "email not found", - }, - { status: 404 } - ); - } - - const user = await prisma.user.findUnique({ - where: { - email, - }, - include: { - Team: true, - }, - }); + const apiKey = req.headers.get("x-api-key"); + if(apiKey!==null) { + const response = await authApiKey(apiKey); + if(response.status!==200) { + return NextResponse.json({ error: "Unauthorized" }, { status: 401 } ); + } + const projectData = await response.json(); + const projectId = projectData.data.project.id; - if (!user) { - return NextResponse.json( - { - error: "user not found", + let {spanId, userScore, userId} = await req.json().catch(() => ({})); + if(!spanId || !userScore || !userId) { + return NextResponse.json({ error: "spanId, userId and userScore are required in the request body" }, { status: 400 } ); + } + userScore = Number(userScore); + if(Number.isNaN(userScore)) { + return NextResponse.json({ error: "userScore must be a number" }, { status: 400 } ); + } + if(userScore!==1 && userScore!==-1) { + return NextResponse.json({ error: "userScore must be 1 or -1" }, { status: 400 } ); + } + if(userId?.length === 0) { + return NextResponse.json({ error: "userId must be a non-empty string" }, { status: 400 } ); + } + const evaluation = await prisma.evaluation.findFirst({ + where: { + projectId, + spanId, }, - { status: 404 } - ); - } + }); + if(!evaluation) { + return NextResponse.json({ error: "Evaluation not found" }, { status: 404 } ); + } + const updatedEvaluation = await prisma.evaluation.update({ + where: { + id: evaluation.id, + }, + data: { + userScore, + userId, + }, + }); + return NextResponse.json({ data: updatedEvaluation }); + } else { + if (!session || !session.user) { + NextResponse.json({ error: "Unauthorized" }, { status: 401 } ); + } + const email = session?.user?.email as string; + if (!email) { + return NextResponse.json( + { + error: "email not found", + }, + { status: 404 } + ); + } - const data = await req.json(); - const { id, score } = data; + const user = await prisma.user.findUnique({ + where: { + email, + }, + include: { + Team: true, + }, + }); - const evaluation = await prisma.evaluation.update({ - where: { - id, - }, - data: { - userId: user.id, - score, - }, - }); + if (!user) { + return NextResponse.json( + { + error: "user not found", + }, + { status: 404 } + ); + } - if (!evaluation) { - return NextResponse.json( - { - error: "No evaluation found", + const data = await req.json(); + const { id, ltUserScore, testId } = data; + const evaluation = await prisma.evaluation.update({ + where: { + id, }, - { status: 404 } - ); - } + data: { + ltUserId: user.id, + ltUserScore, + testId + }, + }); + + if (!evaluation) { + return NextResponse.json( + { + error: "No evaluation found", + }, + { status: 404 } + ); + } - return NextResponse.json({ - data: evaluation, - }); + return NextResponse.json({ + data: evaluation, + }); + } } diff --git a/app/api/metrics/accuracy/route.ts b/app/api/metrics/accuracy/route.ts index 8a0e34ed..257f4e02 100644 --- a/app/api/metrics/accuracy/route.ts +++ b/app/api/metrics/accuracy/route.ts @@ -1,5 +1,6 @@ import { authOptions } from "@/lib/auth/options"; import prisma from "@/lib/prisma"; +import { TraceService } from "@/lib/services/trace_service"; import { Evaluation } from "@prisma/client"; import { getServerSession } from "next-auth"; import { redirect } from "next/navigation"; @@ -13,7 +14,12 @@ export async function GET(req: NextRequest) { const projectId = req.nextUrl.searchParams.get("projectId") as string; const testId = req.nextUrl.searchParams.get("testId") as string; - const byModel = req.nextUrl.searchParams.get("by_model") as string; + let lastNDays = Number(req.nextUrl.searchParams.get("lastNDays")); + let overallAccuracy = 0 + + if(Number.isNaN(lastNDays) || lastNDays < 0){ + lastNDays = 7; + } if (!projectId) { return NextResponse.json( @@ -59,76 +65,66 @@ export async function GET(req: NextRequest) { } let evaluations: Evaluation[] = []; - let allEvaluations: Evaluation[] = []; - let average: number = 0; - - // get evalutaion for the last 7 days + const traceService = new TraceService(); + //Fetch last 7 days of spanIds from clickhouse + const spans = await traceService.GetSpansInProject( + projectId, + lastNDays + ); + + // get evalutaion for the lastNDays + // and all evaluations where score is 1 or -1 evaluations = await prisma.evaluation.findMany({ where: { projectId, testId, - spanStartTime: { - gte: new Date(new Date().getTime() - 7 * 24 * 60 * 60 * 1000), - }, - }, - }); - - // get average evaluation for all evaluations - // get all evaluations where score is 1 or -1 - allEvaluations = await prisma.evaluation.findMany({ - where: { - projectId, - testId, - score: { + spanId: { in: [...spans.map((span) => span.span_id)]}, + ltUserScore: { in: [1, -1], - }, + } }, }); - - const totalPositive = allEvaluations.reduce((acc, evaluation) => { - if (evaluation.score === 1) { - return acc + 1; - } - return acc; - }, 0); - - const totalNegative = allEvaluations.reduce((acc, evaluation) => { - if (evaluation.score === -1) { - return acc + 1; - } - return acc; - }, 0); - - // calculate average - average = (totalPositive / (totalPositive + totalNegative)) * 100; - - if (byModel === "true") { - const evaluationsByModel = evaluations.reduce( - (acc: any, evaluation: Evaluation) => { - if (!evaluation.model) { - return acc; - } - if (acc[evaluation.model]) { - acc[evaluation.model].push(evaluation); - } else { - acc[evaluation.model] = [evaluation]; - } - return acc; - }, - {} - ); - - return NextResponse.json({ - evaluations: evaluationsByModel, - }); - } - if (!evaluations) { - return NextResponse.json({ evalutions: [], average: 0 }, { status: 200 }); + return NextResponse.json({ accuracyPerDay: [], overallAccuracy: null }, { status: 200 }); } + const evalsByDate: Record = {}; + evaluations.forEach((evaluation, index) => { + const span = spans[index]; + const date = span.start_time.split("T")[0]; + if(evalsByDate[date]){ + evalsByDate[date].push(evaluation); + } else { + evalsByDate[date] = [evaluation]; + } + }) + let totalPositive = 0; + let totalNegative = 0; + + const accuracyPerDay = Object.entries(evalsByDate).map(([date, scores]) => { + let totalPositivePerDay = 0; + let totalNegativePerDay = 0; + + scores.forEach((score) => { + if (score.ltUserScore === 1) { + totalPositivePerDay += 1; + totalPositive += 1; + } else { + totalNegativePerDay+= 1; + totalNegative += 1; + } + }); + const accuracy = (totalPositive / (totalPositive + totalNegative)) * 100; + return { + date, + accuracy, + }; + }); + accuracyPerDay.sort((a, b) => new Date(a.date).getTime() - new Date(b.date).getTime()); + // calculate average + overallAccuracy = (totalPositive / (totalPositive + totalNegative)) * 100; return NextResponse.json({ - evaluations, - average, + overallAccuracy, + accuracyPerDay }); } diff --git a/app/api/metrics/latency/trace/route.ts b/app/api/metrics/latency/trace/route.ts index cb8fdcc4..081e198c 100644 --- a/app/api/metrics/latency/trace/route.ts +++ b/app/api/metrics/latency/trace/route.ts @@ -22,16 +22,13 @@ export async function GET(req: NextRequest) { } const traceService = new TraceService(); - const totalTracesPerDay: any = - await traceService.GetTotalTracePerDayPerProject( - projectId, - 7 // last 7 days - ); + const totalTracesPerHour: any = + await traceService.GetTotalTracePerHourPerProject(projectId, 168); // last 7 days in hours const { averageLatencies, p99Latencies, p95Latencies } = - await traceService.GetAverageTraceLatenciesPerDayPerProject(projectId); + await traceService.GetAverageTraceLatenciesPerHourPerProject(projectId); return NextResponse.json( { - totalTracesPerDay, + totalTracesPerHour, averageLatencies, p99Latencies, p95Latencies, diff --git a/app/api/metrics/tests/route.ts b/app/api/metrics/tests/route.ts index 5360975a..33efd470 100644 --- a/app/api/metrics/tests/route.ts +++ b/app/api/metrics/tests/route.ts @@ -71,14 +71,14 @@ export async function GET(req: NextRequest) { }); const totalPositive = evaluations.reduce((acc, evaluation) => { - if (evaluation.score === 1) { + if (evaluation.ltUserScore === 1) { return acc + 1; } return acc; }, 0); const totalNegative = evaluations.reduce((acc, evaluation) => { - if (evaluation.score === -1) { + if (evaluation.ltUserScore === -1) { return acc + 1; } return acc; diff --git a/app/api/metrics/usage/cost/route.ts b/app/api/metrics/usage/cost/route.ts index a7e00a48..2531481d 100644 --- a/app/api/metrics/usage/cost/route.ts +++ b/app/api/metrics/usage/cost/route.ts @@ -14,7 +14,7 @@ export async function GET(req: NextRequest) { const projectId = req.nextUrl.searchParams.get("projectId") as string; const traceService = new TraceService(); const total = await traceService.GetTokensCostPerProject(projectId); - const cost = await traceService.GetTokensCostPerDayPerProject(projectId); + const cost = await traceService.GetTokensCostPerHourPerProject(projectId); return NextResponse.json( { cost, diff --git a/app/api/metrics/usage/span/route.ts b/app/api/metrics/usage/span/route.ts index 7cd00ed8..c0a9e35a 100644 --- a/app/api/metrics/usage/span/route.ts +++ b/app/api/metrics/usage/span/route.ts @@ -22,7 +22,7 @@ export async function GET(req: NextRequest) { } const traceService = new TraceService(); - const spans: any = await traceService.GetTotalSpansPerDayPerProject( + const spans: any = await traceService.GetTotalSpansPerHourPerProject( projectId, 7 // last 7 days ); diff --git a/app/api/metrics/usage/token/route.ts b/app/api/metrics/usage/token/route.ts index fca436dd..ad50118d 100644 --- a/app/api/metrics/usage/token/route.ts +++ b/app/api/metrics/usage/token/route.ts @@ -24,7 +24,7 @@ export async function GET(req: NextRequest) { const traceService = new TraceService(); const total = await traceService.GetTokensUsedPerProject(projectId); - const usage = await traceService.GetTokensUsedPerDayPerProject(projectId); + const usage = await traceService.GetTokensUsedPerHourPerProject(projectId); return NextResponse.json( { usage, diff --git a/app/api/metrics/usage/trace/route.ts b/app/api/metrics/usage/trace/route.ts index 796cabc6..e656c80f 100644 --- a/app/api/metrics/usage/trace/route.ts +++ b/app/api/metrics/usage/trace/route.ts @@ -22,7 +22,7 @@ export async function GET(req: NextRequest) { } const traceService = new TraceService(); - const traces: any = await traceService.GetTotalTracePerDayPerProject( + const traces: any = await traceService.GetTotalTracePerHourPerProject( projectId, 7 // last 7 days ); diff --git a/components/charts/eval-chart.tsx b/components/charts/eval-chart.tsx index 57fd4292..7bbd8410 100644 --- a/components/charts/eval-chart.tsx +++ b/components/charts/eval-chart.tsx @@ -1,5 +1,6 @@ "use client"; +import { formatDurationForDisplay } from "@/lib/utils"; import { Test } from "@prisma/client"; import { AreaChart } from "@tremor/react"; import { useQuery } from "react-query"; @@ -9,16 +10,18 @@ import LargeChartLoading from "./large-chart-skeleton"; export function EvalChart({ projectId, test, + lastNHours = 168, chartDescription = "Evaluated Average(%) over time (last 7 days) for LLM interactions aggregated by day.", info = "Average is calculated based on the score of evaluated llm interactions in the Evaluation tab of the project. Span's start_time is used for day aggregation.", }: { projectId: string; test: Test; + lastNHours?: number; chartDescription?: string; info?: string; }) { const fetchAccuracy = useQuery({ - queryKey: [`fetch-accuracy-${projectId}-${test.id}-query`], + queryKey: ["fetch-accuracy", projectId, test.id, lastNHours], queryFn: async () => { const response = await fetch( `/api/metrics/accuracy?projectId=${projectId}&testId=${test.id}` @@ -31,53 +34,9 @@ export function EvalChart({ if (fetchAccuracy.isLoading || !fetchAccuracy.data) { return ; } else { - // aggregate accuracy per day and return the data - const evaluations = fetchAccuracy?.data?.evaluations; - const accuracyPerDay = evaluations.reduce((acc: any, evaluation: any) => { - const date = evaluation.spanStartTime.split("T")[0]; - if (acc[date]) { - acc[date].push(evaluation.score); - } else { - acc[date] = [evaluation.score]; - } - return acc; - }, {}); - - // calculate accuracy by dividing the sum of scores that are only 1s by the number of evaluations times 100 - const data = Object.keys(accuracyPerDay).map((date) => { - const scores = accuracyPerDay[date]; - - // count +1s and -1s - const totalPositive = scores.reduce((acc: number, score: number) => { - if (score === 1) { - return acc + 1; - } - return acc; - }, 0); - - const totalNegative = scores.reduce((acc: number, score: number) => { - if (score === -1) { - return acc + 1; - } - return acc; - }, 0); - - // calculate accuracy - const accuracy = (totalPositive / (totalPositive + totalNegative)) * 100; - - return { - date, - "Evaluated Accuracy(%)": accuracy, - }; - }); - - // sort the data by date - data.sort((a: any, b: any) => { - return (new Date(a.date) as any) - (new Date(b.date) as any); - }); - - // calculate the overall accuracy - const overallAccuracy = fetchAccuracy?.data?.average; + const data: Array> = + fetchAccuracy?.data?.accuracyPerDay; + const overallAccuracy: number = fetchAccuracy?.data?.overallAccuracy; return ( <> @@ -93,7 +52,10 @@ export function EvalChart({

({ + date: dat.date, + "Evaluated Accuracy(%)": dat.accuracy, + }))} index="date" categories={["Evaluated Accuracy(%)"]} colors={["purple"]} @@ -103,7 +65,8 @@ export function EvalChart({ noDataText="Get started by sending traces to your project." />

- Evaulated Accuracy(%) over time (last 7 days) + Evaulated Accuracy(%) over time{" "} + {formatDurationForDisplay(lastNHours)}

diff --git a/components/charts/latency-chart.tsx b/components/charts/latency-chart.tsx index b2d1d3fc..c249a098 100644 --- a/components/charts/latency-chart.tsx +++ b/components/charts/latency-chart.tsx @@ -1,19 +1,28 @@ "use client"; +import { formatDurationForDisplay } from "@/lib/utils"; import { AreaChart } from "@tremor/react"; import { useQuery } from "react-query"; import { toast } from "sonner"; import { Info } from "../shared/info"; import LargeChartLoading from "./large-chart-skeleton"; -export function TraceLatencyChart({ projectId }: { projectId: string }) { +export function TraceLatencyChart({ + projectId, + lastNHours = 168, +}: { + projectId: string; + lastNHours?: number; +}) { const { data: metricsLatencyAverageTracePerDay, isLoading: metricsLatencyAverageTracePerDayLoading, error: metricsLatencyAverageTracePerDayError, } = useQuery({ queryKey: [ - `fetch-metrics-latency-average-trace-per-day-${projectId}-query`, + "fetch-metrics-latency-average-trace-per-day", + projectId, + lastNHours, ], queryFn: async () => { const response = await fetch( @@ -61,7 +70,7 @@ export function TraceLatencyChart({ projectId }: { projectId: string }) { }) ); - const countData = metricsLatencyAverageTracePerDay?.totalTracesPerDay?.map( + const countData = metricsLatencyAverageTracePerDay?.totalTracesPerHour?.map( (data: any, index: number) => ({ date: data?.date || "", "Trace Count": data?.traceCount || 0, @@ -113,7 +122,8 @@ export function TraceLatencyChart({ projectId }: { projectId: string }) { noDataText="Get started by sending traces to your project." />

- Average trace latency per day(ms) for the last 7 days + Average trace latency per day(ms) for the{" "} + {formatDurationForDisplay(lastNHours)}

diff --git a/components/charts/model-accuracy-chart.tsx b/components/charts/model-accuracy-chart.tsx deleted file mode 100644 index ec3fdb83..00000000 --- a/components/charts/model-accuracy-chart.tsx +++ /dev/null @@ -1,126 +0,0 @@ -"use client"; - -import { Test } from "@prisma/client"; -import { BarChart } from "@tremor/react"; -import { useQuery } from "react-query"; -import { Info } from "../shared/info"; -import LargeChartLoading from "./large-chart-skeleton"; - -export function ModelAccuracyChart({ - projectId, - test, -}: { - projectId: string; - test: Test; -}) { - const fetchAccuracy = useQuery({ - queryKey: [`fetch-accuracy-model-${projectId}-${test.id}-query`], - queryFn: async () => { - const response = await fetch( - `/api/metrics/accuracy?projectId=${projectId}&testId=${test.id}&by_model=true` - ); - const result = await response.json(); - return result; - }, - }); - - if (fetchAccuracy.isLoading || !fetchAccuracy.data) { - return ; - } else { - // for each key aggregate accuracy per day and return the data - const evaluations: any[] = fetchAccuracy?.data?.evaluations; - const result = Object.entries(evaluations).map(([key, evaluations]) => { - const accuracyPerDay: any = {}; - let totalCorrect = 0; // To calculate overall accuracy - - evaluations.forEach((evaluation: any) => { - const date = evaluation.spanStartTime.split("T")[0]; - if (!accuracyPerDay[date]) { - accuracyPerDay[date] = { correct: 0, total: 0 }; - } - accuracyPerDay[date].total++; - if (evaluation.score === 1) { - accuracyPerDay[date].correct++; - totalCorrect++; - } - }); - - const data = Object.entries(accuracyPerDay) - .map(([date, { correct, total }]: any) => ({ - date, - [`${key} Evaluated Accuracy(%)`]: (correct / total) * 100, - })) - .sort((a, b) => (new Date(a.date) as any) - (new Date(b.date) as any)); - - const overallAccuracy = (totalCorrect / evaluations.length) * 100 || 0; - - return { - model: key, - data, - overallAccuracy, - }; - }); - - const finalData: any = {}; - result.forEach((model) => { - model.data.forEach((entry: any) => { - const { date } = entry; - const accuracyKey = Object.keys(entry).find((key) => - key.includes("Evaluated Accuracy") - ); - if (!finalData[date]) { - finalData[date] = { date }; - } - if (accuracyKey) { - finalData[date][accuracyKey] = entry[accuracyKey]; - } - }); - }); - - const sortedArray = Object.values(finalData).sort( - (a: any, b: any) => - new Date(a.date).getTime() - new Date(b.date).getTime() - ); - const colors = ["purple", "blue", "green", "red", "yellow", "orange"]; - - return ( - <> -
-
-

- Evaluated Accuracy(%) over time (last 7 days) for LLM interactions - aggregated by day. - -

-
- {result.map((r, i) => ( -

- {r.model} Overall Accuracy: {r.overallAccuracy?.toFixed(2)}% -

- ))} - `${r.model} Evaluated Accuracy(%)`) || [] - } - colors={result.map((r) => { - // get a random color from the colors array - const index = Math.floor(Math.random() * colors.length); - const color = colors[index]; - return color; - })} - showAnimation={true} - showTooltip={true} - yAxisWidth={33} - noDataText="Get started by sending traces to your project." - /> -

- Evaulated Accuracy(%) over time (last 7 days) -

-
- - ); - } -} diff --git a/components/charts/token-chart.tsx b/components/charts/token-chart.tsx index 10a7c58b..df5a4ca4 100644 --- a/components/charts/token-chart.tsx +++ b/components/charts/token-chart.tsx @@ -1,20 +1,27 @@ "use client"; +import { formatDurationForDisplay } from "@/lib/utils"; import { BarChart } from "@tremor/react"; import { useQuery } from "react-query"; import { toast } from "sonner"; import SmallChartLoading from "./small-chart-skeleton"; -export function TokenChart({ projectId }: { projectId: string }) { +export function TokenChart({ + projectId, + lastNHours = 168, +}: { + projectId: string; + lastNHours?: number; +}) { const { data: tokenUsage, isLoading: tokenUsageLoading, error: tokenUsageError, } = useQuery({ - queryKey: [`fetch-metrics-usage-token-${projectId}-query`], + queryKey: ["fetch-metrics-usage-token", projectId, lastNHours], queryFn: async () => { const response = await fetch( - `/api/metrics/usage/token?projectId=${projectId}` + `/api/metrics/usage/token?projectId=${projectId}&lastNHours=${lastNHours}` ); if (!response.ok) { const error = await response.json(); @@ -66,7 +73,7 @@ export function TokenChart({ projectId }: { projectId: string }) { noDataText="Get started by sending traces to your project." />

- Total tokens over time (last 7 days) + Total tokens over time {formatDurationForDisplay(lastNHours)}

@@ -74,13 +81,19 @@ export function TokenChart({ projectId }: { projectId: string }) { } } -export function CostChart({ projectId }: { projectId: string }) { +export function CostChart({ + projectId, + lastNHours = 168, +}: { + projectId: string; + lastNHours?: number; +}) { const { data: costUsage, isLoading: costUsageLoading, error: costUsageError, } = useQuery({ - queryKey: [`fetch-metrics-usage-cost-${projectId}-query`], + queryKey: ["fetch-metrics-usage-cost", projectId, lastNHours], queryFn: async () => { const response = await fetch( `/api/metrics/usage/cost?projectId=${projectId}` @@ -139,7 +152,7 @@ export function CostChart({ projectId }: { projectId: string }) { noDataText="Get started by sending traces to your project." />

- Total cost over time (last 7 days) + Total cost over time {formatDurationForDisplay(lastNHours)}

diff --git a/components/charts/trace-chart.tsx b/components/charts/trace-chart.tsx index 7e0a2db5..26ac00d6 100644 --- a/components/charts/trace-chart.tsx +++ b/components/charts/trace-chart.tsx @@ -1,17 +1,24 @@ "use client"; +import { formatDurationForDisplay } from "@/lib/utils"; import { BarChart } from "@tremor/react"; import { useQuery } from "react-query"; import { toast } from "sonner"; import SmallChartLoading from "./small-chart-skeleton"; -export function TraceSpanChart({ projectId }: { projectId: string }) { +export function TraceSpanChart({ + projectId, + lastNHours = 168, +}: { + projectId: string; + lastNHours?: number; +}) { const { data: traceUsage, isLoading: traceUsageLoading, error: traceUsageError, } = useQuery({ - queryKey: [`fetch-metrics-usage-trace-${projectId}-query`], + queryKey: ["fetch-metrics-usage-trace", projectId, lastNHours], queryFn: async () => { const response = await fetch( `/api/metrics/usage/trace?projectId=${projectId}` @@ -35,7 +42,7 @@ export function TraceSpanChart({ projectId }: { projectId: string }) { isLoading: spanUsageLoading, error: spanUsageError, } = useQuery({ - queryKey: [`fetch-metrics-usage-span-${projectId}-query`], + queryKey: ["fetch-metrics-usage-span", projectId, lastNHours], queryFn: async () => { const response = await fetch( `/api/metrics/usage/span?projectId=${projectId}` @@ -110,7 +117,7 @@ export function TraceSpanChart({ projectId }: { projectId: string }) { noDataText="Get started by sending traces to your project." />

- Total traces over time (last 7 days) + Total traces over time {formatDurationForDisplay(lastNHours)}

@@ -118,13 +125,19 @@ export function TraceSpanChart({ projectId }: { projectId: string }) { } } -export function SpanChart({ projectId }: { projectId: string }) { +export function SpanChart({ + projectId, + lastNHours = 168, +}: { + projectId: string; + lastNHours?: number; +}) { const { data: spanUsage, isLoading: spanUsageLoading, error: spanUsageError, } = useQuery({ - queryKey: [`fetch-metrics-usage-span-${projectId}-query`], + queryKey: ["fetch-metrics-usage-span", projectId, lastNHours], queryFn: async () => { const response = await fetch( `/api/metrics/usage/span?projectId=${projectId}` @@ -172,7 +185,7 @@ export function SpanChart({ projectId }: { projectId: string }) { noDataText="Get started by sending traces to your project." />

- Total spans over time (last 7 days) + Total spans over time {formatDurationForDisplay(lastNHours)}

diff --git a/components/evaluations/create-test.tsx b/components/evaluations/create-test.tsx index 8d1bfcd7..c0ab8b75 100644 --- a/components/evaluations/create-test.tsx +++ b/components/evaluations/create-test.tsx @@ -50,7 +50,7 @@ export function CreateTest({ const [step, setStep] = useState(2); const schema = z.object({ name: z.string().min(2, "Too short").max(30, "Too long"), - description: z.string().min(2, "Too short").max(200, "Too long"), + description: z.string().max(200, "Too long").optional(), min: z .string() .refine((val) => !Number.isNaN(parseInt(val, 10))) @@ -102,7 +102,9 @@ export function CreateTest({ }, body: JSON.stringify({ name: data.name, - description: data.description.toLowerCase(), + description: data.description + ? data.description.toLowerCase() + : "", min: data.min ? parseInt(data.min, 10) : -1, max: data.max ? parseInt(data.max, 10) : 1, step: data.step ? parseInt(data.step, 10) : 2, diff --git a/components/evaluations/eval-dialog.tsx b/components/evaluations/eval-dialog.tsx index 881f814c..aea7839d 100644 --- a/components/evaluations/eval-dialog.tsx +++ b/components/evaluations/eval-dialog.tsx @@ -184,7 +184,7 @@ function EvalContent({ const response = await fetch(`/api/evaluation?spanId=${span?.span_id}`); const result = await response.json(); const sc = - result.evaluations.length > 0 ? result.evaluations[0].score : min; + result.evaluations.length > 0 ? result.evaluations[0].ltUserScore : min; onScoreSelected(sc); return result; }, @@ -213,7 +213,7 @@ function EvalContent({ // Check if an evaluation already exists if (evaluationsData?.evaluations[0]?.id) { - if (evaluationsData.evaluations[0].score === score) { + if (evaluationsData.evaluations[0].ltUserScore === score) { setBusy(false); return; } @@ -225,7 +225,7 @@ function EvalContent({ }, body: JSON.stringify({ id: evaluationsData.evaluations[0].id, - score: score, + ltUserScore: score, }), }); queryClient.invalidateQueries([ @@ -243,12 +243,7 @@ function EvalContent({ projectId: projectId, spanId: span.span_id, traceId: span.trace_id, - spanStartTime: span?.start_time - ? new Date(correctTimestampFormat(span.start_time)) - : new Date(), - score: score, - model: model, - prompt: systemPrompt, + ltUserScore: score, testId: test.id, }), }); diff --git a/components/evaluations/evaluation-row.tsx b/components/evaluations/evaluation-row.tsx index d54fa68c..2f468619 100644 --- a/components/evaluations/evaluation-row.tsx +++ b/components/evaluations/evaluation-row.tsx @@ -59,7 +59,7 @@ export default function EvaluationRow({ const result = await response.json(); setEvaluation(result.evaluations.length > 0 ? result.evaluations[0] : {}); setScore( - result.evaluations.length > 0 ? result.evaluations[0].score : -100 + result.evaluations.length > 0 ? result.evaluations[0].ltUserScore : -100 ); return result; }, @@ -80,7 +80,8 @@ export default function EvaluationRow({ if (!attributes) return null; // extract the metrics - const userScore = attributes["user.feedback.rating"] || ""; + const userScore = evaluation?.userScore || ""; + const userId = evaluation?.userId || ""; const startTimeMs = new Date( correctTimestampFormat(span.start_time) ).getTime(); @@ -110,12 +111,12 @@ export default function EvaluationRow({ } piiDetected = piiDetected || - detectPII( + (responses && detectPII( JSON.parse(responses)[0]?.message?.content || JSON.parse(responses)[0]?.text || JSON.parse(responses)[0]?.content || "" - ).length > 0; + ).length > 0); // score evaluation const evaluateSpan = async (newScore: number) => { @@ -130,10 +131,7 @@ export default function EvaluationRow({ projectId: projectId, spanId: span.span_id, traceId: span.trace_id, - spanStartTime: new Date(correctTimestampFormat(span.start_time)), - score: newScore, - model: model, - prompt: promptContent, + ltUserScore: newScore, testId: testId, }), }); @@ -145,7 +143,7 @@ export default function EvaluationRow({ }, body: JSON.stringify({ id: evaluation?.id, - score: newScore, + ltUserScore: newScore, }), }); } @@ -242,6 +240,9 @@ export default function EvaluationRow({

{userScore ? userScore : "Not evaluated"}

+

+ {userId} +

{addedToDataset ? ( diff --git a/components/evaluations/evaluation-table.tsx b/components/evaluations/evaluation-table.tsx index b48b367b..4ef9cd17 100644 --- a/components/evaluations/evaluation-table.tsx +++ b/components/evaluations/evaluation-table.tsx @@ -162,6 +162,7 @@ export default function EvaluationTable({

Duration

Evaluated Score

User Score

+

User Id

Added to Dataset

)} diff --git a/components/project/create.tsx b/components/project/create.tsx index b65535f4..d8a39059 100644 --- a/components/project/create.tsx +++ b/components/project/create.tsx @@ -42,7 +42,7 @@ export function Create({ const [busy, setBusy] = useState(false); const schema = z.object({ name: z.string().min(2, "Too short").max(30, "Too long"), - description: z.string().min(2, "Too short").max(100, "Too long"), + description: z.string().max(100, "Too long").optional(), }); const CreateProjectForm = useForm({ resolver: zodResolver(schema), @@ -73,7 +73,9 @@ export function Create({ }, body: JSON.stringify({ name: data.name, - description: data.description.toLowerCase(), + description: data.description + ? data.description.toLowerCase() + : "", teamId, }), }); diff --git a/components/project/dataset/create.tsx b/components/project/dataset/create.tsx index c0c56233..573c7db7 100644 --- a/components/project/dataset/create.tsx +++ b/components/project/dataset/create.tsx @@ -42,7 +42,7 @@ export function CreateDataset({ const [busy, setBusy] = useState(false); const schema = z.object({ name: z.string().min(2, "Too short").max(30, "Too long"), - description: z.string().min(2, "Too short").max(100, "Too long"), + description: z.string().max(100, "Too long").optional(), }); const CreateDatasetForm = useForm({ resolver: zodResolver(schema), @@ -73,7 +73,9 @@ export function CreateDataset({ }, body: JSON.stringify({ name: data.name, - description: data.description.toLowerCase(), + description: data.description + ? data.description.toLowerCase() + : "", projectId, }), }); @@ -172,7 +174,7 @@ export function CreatePromptset({ const [busy, setBusy] = useState(false); const schema = z.object({ name: z.string().min(2, "Too short").max(30, "Too long"), - description: z.string().min(2, "Too short").max(100, "Too long"), + description: z.string().max(100, "Too long").optional(), }); const CreatePromptsetForm = useForm({ resolver: zodResolver(schema), @@ -203,7 +205,9 @@ export function CreatePromptset({ }, body: JSON.stringify({ name: data.name, - description: data.description.toLowerCase(), + description: data.description + ? data.description.toLowerCase() + : "", projectId, }), }); diff --git a/components/project/metrics.tsx b/components/project/metrics.tsx index aa853ea8..2864144c 100644 --- a/components/project/metrics.tsx +++ b/components/project/metrics.tsx @@ -1,23 +1,77 @@ "use client"; +import { + DropdownMenu, + DropdownMenuContent, + DropdownMenuRadioGroup, + DropdownMenuRadioItem, + DropdownMenuSeparator, + DropdownMenuTrigger, +} from "@radix-ui/react-dropdown-menu"; +import { ChevronDownIcon } from "@radix-ui/react-icons"; import { useParams } from "next/navigation"; +import { useState } from "react"; import { TraceLatencyChart } from "../charts/latency-chart"; import { CostChart, TokenChart } from "../charts/token-chart"; import { TraceSpanChart } from "../charts/trace-chart"; import { Separator } from "../ui/separator"; +const timeRanges = [ + { label: "12 hours", value: 12 }, + { label: "1 day", value: 24 }, + { label: "3 days", value: 72 }, + { label: "5 days", value: 120 }, + { label: "7 days", value: 168 }, + { label: "30 days", value: 720 }, + { label: "60 days", value: 1440 }, +]; + export default function Metrics({ email }: { email: string }) { const project_id = useParams()?.project_id as string; + const [lastNHours, setLastNHours] = useState(timeRanges[0].value); + + const handleTimeRangeChange = (value: number) => { + setLastNHours(value); + }; + + const selectedLabel = + timeRanges.find((range) => range.value === lastNHours)?.label || "12 hours"; return (
-

Usage

+
+

Usage

+ + + + + + + handleTimeRangeChange(Number(value))} + > + {timeRanges.map((range) => ( + + {range.label} + + ))} + + + +
- - - + + +
@@ -26,7 +80,7 @@ export default function Metrics({ email }: { email: string }) {

Latency

- +
diff --git a/components/shared/setup-instructions.tsx b/components/shared/setup-instructions.tsx index 4126b732..5eb61adb 100644 --- a/components/shared/setup-instructions.tsx +++ b/components/shared/setup-instructions.tsx @@ -4,7 +4,7 @@ import { Button } from "../ui/button"; import GenerateApiKey from "./api-key"; export function SetupInstructions({ project_id }: { project_id: string }) { - const [sdk, setSdk] = useState("typescript"); + const [sdk, setSdk] = useState("python"); const [apiKey, setApiKey] = useState(""); const handleApiKeyGenerated = (newApiKey: string) => { @@ -19,18 +19,18 @@ export function SetupInstructions({ project_id }: { project_id: string }) {
@@ -104,7 +104,7 @@ export function SetupInstructions({ project_id }: { project_id: string }) { } export function TestSetupInstructions({ testId }: { testId: string }) { - const [sdk, setSdk] = useState("typescript"); + const [sdk, setSdk] = useState("python"); const copyToClipboard = (code: string) => { navigator.clipboard.writeText(code); @@ -115,18 +115,18 @@ export function TestSetupInstructions({ testId }: { testId: string }) {
@@ -210,7 +210,7 @@ def test_function(): } export function PromptInstructions({ id }: { id: string }) { - const [sdk, setSdk] = useState("typescript"); + const [sdk, setSdk] = useState("python"); const copyToClipboard = (code: string) => { navigator.clipboard.writeText(code); @@ -220,13 +220,6 @@ export function PromptInstructions({ id }: { id: string }) { return (
- +

diff --git a/lib/clients/scale3_clickhouse/client/client.ts b/lib/clients/scale3_clickhouse/client/client.ts index 6cff26fd..4b05fb1d 100644 --- a/lib/clients/scale3_clickhouse/client/client.ts +++ b/lib/clients/scale3_clickhouse/client/client.ts @@ -37,7 +37,7 @@ export class ClickhouseBaseClient implements IBaseChClient { constructor(database = CLICK_HOUSE_CONSTANTS.database) { this.client = createClient({ database, - host: process.env.CLICK_HOUSE_HOST, + url: process.env.CLICK_HOUSE_HOST, username: process.env.CLICK_HOUSE_USER, password: process.env.CLICK_HOUSE_PASSWORD, compression: { diff --git a/lib/services/trace_service.ts b/lib/services/trace_service.ts index d5938d79..4f6d8336 100644 --- a/lib/services/trace_service.ts +++ b/lib/services/trace_service.ts @@ -9,34 +9,39 @@ import { QueryBuilderService, } from "./query_builder_service"; -// may want to think about how we want to store account_id in table or table name to prevent -// having to pass in project_ids to get total spans per account - export interface PaginationResult { result: T[]; metadata?: { page?: number; page_size?: number; total_pages: number }; } +function getFormattedTime(lastNHours: number): string { + const nHoursAgo = format( + new Date(Date.now() - lastNHours * 60 * 60 * 1000), + "yyyy-MM-dd HH:mm:ss" + ); + return nHoursAgo; +} + export interface ITraceService { - GetTotalTracePerDayPerProject: ( + GetTotalTracePerHourPerProject: ( project_id: string, - lastNDays?: number + lastNHours?: number ) => Promise; - GetTotalSpansPerDayPerProject: ( + GetTotalSpansPerHourPerProject: ( project_id: string, - lastNDays?: number + lastNHours?: number ) => Promise; - GetTokensUsedPerDayPerProject: ( + GetTokensUsedPerHourPerProject: ( project_id: string, - lastNDays?: number + lastNHours?: number ) => Promise; - GetTokensCostPerDayPerProject: ( + GetTokensCostPerHourPerProject: ( project_id: string, - lastNDays?: number + lastNHours?: number ) => Promise; - GetAverageTraceLatenciesPerDayPerProject( + GetAverageTraceLatenciesPerHourPerProject( project_id: string, - lastNDays?: number + lastNHours?: number ): Promise; GetTokensCostPerProject: (project_id: string) => Promise; GetTotalTracesPerProject: (project_id: string) => Promise; @@ -52,7 +57,7 @@ export interface ITraceService { pageSize: number, filters?: AttributesFilter[] ) => Promise>; - GetSpansInProject: (project_id: string) => Promise; + GetSpansInProject: (project_id: string, lastNDays: number) => Promise; GetTracesInProjectPaginated: ( project_id: string, page: number, @@ -174,14 +179,11 @@ export class TraceService implements ITraceService { } } - async GetTotalSpansPerDayPerProject( + async GetTotalSpansPerHourPerProject( project_id: string, - lastNDays = 7 + lastNHours = 168 ): Promise { - const nDaysAgo = format( - new Date(Date.now() - lastNDays * 24 * 60 * 60 * 1000), - "yyyy-MM-dd" - ); + const nHoursAgo = getFormattedTime(lastNHours); try { const tableExists = await this.client.checkTableExists(project_id); if (!tableExists) { @@ -194,7 +196,7 @@ export class TraceService implements ITraceService { `count(*) AS spanCount`, ]) .from(project_id) - .where(sql.gte("start_time", nDaysAgo)) + .where(sql.gte("start_time", nHoursAgo)) .groupBy(`toDate(parseDateTimeBestEffort(start_time))`) .orderBy(`toDate(parseDateTimeBestEffort(start_time))`); const result = await this.client.find(query); @@ -226,14 +228,11 @@ export class TraceService implements ITraceService { } } - async GetTotalTracePerDayPerProject( + async GetTotalTracePerHourPerProject( project_id: string, - lastNDays = 7 + lastNHours = 168 ): Promise { - const nDaysAgo = format( - new Date(Date.now() - lastNDays * 24 * 60 * 60 * 1000), - "yyyy-MM-dd" - ); + const nHoursAgo = getFormattedTime(lastNHours); try { const tableExists = await this.client.checkTableExists(project_id); if (!tableExists) { @@ -246,7 +245,7 @@ export class TraceService implements ITraceService { `COUNT(DISTINCT trace_id) AS traceCount`, ]) .from(project_id) - .where(sql.gte("start_time", nDaysAgo)) + .where(sql.gte("start_time", nHoursAgo)) .groupBy(`toDate(parseDateTimeBestEffort(start_time))`) .orderBy(`toDate(parseDateTimeBestEffort(start_time))`); const result = await this.client.find(query); @@ -399,11 +398,20 @@ export class TraceService implements ITraceService { } } - async GetSpansInProject(project_id: string): Promise { + async GetSpansInProject( + project_id: string, + lastNHours = 168 + ): Promise { try { const query = sql.select().from(project_id); - const spans: Span[] = await this.client.find(query); - return spans; + if (!lastNHours) { + return await this.client.find(query); + } else { + const nHoursAgo = getFormattedTime(lastNHours); + query.where(sql.gte("start_time", nHoursAgo)); + + return await this.client.find(query); + } } catch (error) { throw new Error( `An error occurred while trying to get the spans ${error}` @@ -483,9 +491,9 @@ export class TraceService implements ITraceService { } } - async GetAverageTraceLatenciesPerDayPerProject( + async GetAverageTraceLatenciesPerHourPerProject( project_id: string, - lastNDays = 7 + lastNHours = 168 ): Promise { try { const tableExists = await this.client.checkTableExists(project_id); @@ -497,10 +505,7 @@ export class TraceService implements ITraceService { }; } - const nDaysAgo = format( - new Date(Date.now() - lastNDays * 24 * 60 * 60 * 1000), - "yyyy-MM-dd" - ); + const nHoursAgo = getFormattedTime(lastNHours); // Directly embedding the ClickHouse-specific functions within string literals let innerSelect = sql @@ -511,7 +516,7 @@ export class TraceService implements ITraceService { "(toUnixTimestamp(max(parseDateTime64BestEffort(end_time))) - toUnixTimestamp(min(parseDateTime64BestEffort(start_time)))) * 1000 AS duration" ) .from(project_id) - .where(sql.gte("start_time", nDaysAgo)) + .where(sql.gte("start_time", nHoursAgo)) .groupBy("trace_id"); // Assembling the outer query @@ -572,9 +577,9 @@ export class TraceService implements ITraceService { } } - async GetTokensUsedPerDayPerProject( + async GetTokensUsedPerHourPerProject( project_id: string, - lastNDays = 7 + lastNHours = 168 ): Promise { try { const tableExists = await this.client.checkTableExists(project_id); @@ -582,10 +587,7 @@ export class TraceService implements ITraceService { return []; } - const nDaysAgo = format( - new Date(Date.now() - lastNDays * 24 * 60 * 60 * 1000), - "yyyy-MM-dd" - ); + const nHoursAgo = getFormattedTime(lastNHours); const query = sql .select([ @@ -595,14 +597,14 @@ export class TraceService implements ITraceService { .from(project_id) .where( sql.like("attributes", "%total_tokens%"), - sql.gte("start_time", nDaysAgo) + sql.gte("start_time", nHoursAgo) ) .groupBy("date") .orderBy("date"); const result = await this.client.find(query); // calculate total tokens used per day - const tokensUsedPerDay = result.map((row: any) => { + const tokensUsedPerHour = result.map((row: any) => { let totalTokens = 0; let inputTokens = 0; let outputTokens = 0; @@ -638,7 +640,7 @@ export class TraceService implements ITraceService { }; }); - return tokensUsedPerDay; + return tokensUsedPerHour; } catch (error) { throw new Error( `An error occurred while trying to get the tokens used ${error}` @@ -646,9 +648,9 @@ export class TraceService implements ITraceService { } } - async GetTokensCostPerDayPerProject( + async GetTokensCostPerHourPerProject( project_id: string, - lastNDays = 7 + lastNHours = 168 // Default to 168 hours (7 days) ): Promise { try { const tableExists = await this.client.checkTableExists(project_id); @@ -656,10 +658,7 @@ export class TraceService implements ITraceService { return []; } - const nDaysAgo = format( - new Date(Date.now() - lastNDays * 24 * 60 * 60 * 1000), - "yyyy-MM-dd" - ); + const nHoursAgo = getFormattedTime(lastNHours); const query = sql .select([ @@ -669,14 +668,14 @@ export class TraceService implements ITraceService { .from(project_id) .where( sql.like("attributes", "%total_tokens%"), - sql.gte("start_time", nDaysAgo) + sql.gte("start_time", nHoursAgo) ) .groupBy("date") .orderBy("date"); const result = await this.client.find(query); // calculate total tokens used per day - const costPerDay = result.map((row: any) => { + const costPerHour = result.map((row: any) => { let costs = { total: 0, input: 0, output: 0 }; row.attributes_list.forEach((attributes: any) => { const parsedAttributes = JSON.parse(attributes); @@ -697,7 +696,7 @@ export class TraceService implements ITraceService { }; }); - return costPerDay; + return costPerHour; } catch (error) { throw new Error( `An error occurred while trying to get the tokens used ${error}` @@ -761,20 +760,19 @@ export class TraceService implements ITraceService { totalOutputTokens: 0, }; } - - const query = sql - .select() - .from(project_id) - .where(sql.like("attributes", "%total_tokens%")); - const spans: Span[] = await this.client.find(query); + const query = sql.select( + `JSONExtractRaw(attributes, 'llm.token.counts') AS token_counts FROM ${project_id}` + ); + const spans: any[] = await this.client.find(query); let totalTokens = 0; let totalInputTokens = 0; let totalOutputTokens = 0; spans.forEach((span) => { - const parsedAttributes = JSON.parse(span.attributes || "{}"); - const llmTokenCounts = parsedAttributes["llm.token.counts"] - ? JSON.parse(parsedAttributes["llm.token.counts"]) - : {}; + const parsedAttributes = JSON.parse(span.token_counts || "{}"); + const llmTokenCounts = + typeof parsedAttributes === "string" + ? JSON.parse(parsedAttributes) + : parsedAttributes; const token_count = llmTokenCounts.total_tokens || 0; totalTokens += token_count; diff --git a/lib/utils.ts b/lib/utils.ts index 695d7743..c625952a 100644 --- a/lib/utils.ts +++ b/lib/utils.ts @@ -4,6 +4,7 @@ import { createHash, randomBytes } from "crypto"; import { TiktokenEncoding, getEncoding } from "js-tiktoken"; import { NextResponse } from "next/server"; import { prettyPrintJson } from "pretty-print-json"; +import qs from "qs"; import { twMerge } from "tailwind-merge"; import { Span } from "./clients/scale3_clickhouse/models/span"; import { @@ -15,7 +16,6 @@ import { PERPLEXITY_PRICING, SpanStatusCode, } from "./constants"; -import qs from "qs"; export function cn(...inputs: ClassValue[]) { return twMerge(clsx(inputs)); @@ -160,7 +160,7 @@ function convertDurationToMicroseconds(duration: [number, number]): number { return totalMicroseconds; } -function convertToDateTime64(dateTime: [number, number]): string { +export function convertToDateTime64(dateTime: [number, number]): string { // Extract seconds and nanoseconds from the input const [seconds, nanoseconds] = dateTime; @@ -280,14 +280,17 @@ export function prepareForClickhouse(spans: Normalized[]): Span[] { }); } -export function fillPromptStringTemplate(template: string, variables: { [key: string]: string }): string { +export function fillPromptStringTemplate( + template: string, + variables: { [key: string]: string } +): string { return template.replace(/\{(\w+)\}/g, (match, key) => { - return variables[key] || match; + return variables[key] || match; }); } //TODO: Move to a middleware -export function parseQueryString(url: string): Record{ +export function parseQueryString(url: string): Record { return qs.parse(url.split("?")[1], { decoder(str) { if (str === "true") return true; @@ -461,3 +464,14 @@ export function calculateTokens(content: string): number { return estimateTokens(content); // Fallback method } } + +export function formatDurationForDisplay(hours: number): string { + if (hours === 12) { + return "last 12 hours"; + } else if (hours % 24 === 0) { + const days = hours / 24; + return `last ${days} day${days > 1 ? "s" : ""}`; + } else { + return `last ${hours} hours`; + } +} diff --git a/package.json b/package.json index d7274189..06f08854 100644 --- a/package.json +++ b/package.json @@ -9,6 +9,8 @@ "create-clickhouse-db": "npx tsx scripts/create-clickhouse-db.ts", "predev": "npm run create-tables && npm run create-clickhouse-db", "dev": "next dev", + "create-migration": "npx prisma migrate dev --create-only", + "migration": "npx prisma migrate dev", "prestart": "npm run create-tables && npm run create-clickhouse-db", "start": "next start", "lint": "next lint" @@ -128,4 +130,4 @@ "browser": { "crypto": false } -} +} \ No newline at end of file diff --git a/prisma/migrations/20240513220159_rename_score_and_userid/migration.sql b/prisma/migrations/20240513220159_rename_score_and_userid/migration.sql new file mode 100644 index 00000000..e81f1736 --- /dev/null +++ b/prisma/migrations/20240513220159_rename_score_and_userid/migration.sql @@ -0,0 +1,13 @@ +-- DropForeignKey +ALTER TABLE "Evaluation" DROP CONSTRAINT "Evaluation_userId_fkey"; +-- AlterTable +ALTER TABLE "Evaluation" + RENAME COLUMN "userId" TO "ltUserId"; +ALTER TABLE "Evaluation" + RENAME COLUMN "score" TO "ltUserScore"; +ALTER TABLE "Evaluation" +ADD COLUMN "userId" TEXT, + ADD COLUMN "userScore" INTEGER; +-- AddForeignKey +ALTER TABLE "Evaluation" +ADD CONSTRAINT "Evaluation_ltUserId_fkey" FOREIGN KEY ("ltUserId") REFERENCES "User"("id") ON DELETE CASCADE ON UPDATE CASCADE; \ No newline at end of file diff --git a/prisma/migrations/20240513234910_evaluation_fields_optional/migration.sql b/prisma/migrations/20240513234910_evaluation_fields_optional/migration.sql new file mode 100644 index 00000000..57577de1 --- /dev/null +++ b/prisma/migrations/20240513234910_evaluation_fields_optional/migration.sql @@ -0,0 +1,9 @@ +-- Remove columns +ALTER TABLE "Evaluation" DROP COLUMN "model", + DROP COLUMN "prompt", + DROP COLUMN "spanStartTime"; +-- Alter columns to be nullable +ALTER TABLE "Evaluation" +ALTER COLUMN "ltUserId" DROP NOT NULL, + ALTER COLUMN "ltUserScore" DROP NOT NULL, + ALTER COLUMN "testId" DROP NOT NULL; \ No newline at end of file diff --git a/prisma/schema.prisma b/prisma/schema.prisma index ba236868..8f33f2e1 100644 --- a/prisma/schema.prisma +++ b/prisma/schema.prisma @@ -73,21 +73,20 @@ model Test { } model Evaluation { - id String @id @default(cuid()) - spanStartTime DateTime? - spanId String? - traceId String? - model String? - userId String - projectId String - testId String - score Int - prompt String? - createdAt DateTime @default(now()) - updatedAt DateTime @updatedAt - User User @relation(fields: [userId], references: [id], onDelete: Cascade) - Project Project @relation(fields: [projectId], references: [id], onDelete: Cascade) - Test Test @relation(fields: [testId], references: [id], onDelete: Cascade) + id String @id @default(cuid()) + spanId String? + traceId String? + ltUserId String? + ltUserScore Int? + userId String? + userScore Int? + projectId String + testId String? + createdAt DateTime @default(now()) + updatedAt DateTime @updatedAt + User User? @relation(fields: [ltUserId], references: [id], onDelete: Cascade) + Project Project @relation(fields: [projectId], references: [id], onDelete: Cascade) + Test Test? @relation(fields: [testId], references: [id], onDelete: Cascade) } model Prompt { diff --git a/scripts/create-clickhouse-db.ts b/scripts/create-clickhouse-db.ts index e03aa219..9845a1c0 100644 --- a/scripts/create-clickhouse-db.ts +++ b/scripts/create-clickhouse-db.ts @@ -5,7 +5,7 @@ loadEnvConfig(process.cwd()); const chClient = createClient({ database: "default", - host: process.env.CLICK_HOUSE_HOST, + url: process.env.CLICK_HOUSE_HOST, username: process.env.CLICK_HOUSE_USER, password: process.env.CLICK_HOUSE_PASSWORD, compression: {