Transform for Code Profiling

[pankajskku]

This tranform extracts the base syntactic concepts from the multi-language source codes and represent these concepts in an unified langauge-agnostic representation that can be further used for multi-lnaguage data profiling. While programming languages expose similar syntactic building blocks to represent programming intent, such as importing packages/libraries, functions, classes, loops, conditionals, comments and others, these concepts are expressed through language-specific grammar, defined by distinct keywords and syntactic form.

Why are these changes needed?

Data profiling, in the context of machine learning, is the process of examining and analyzing data to create
useful statistics. These statistics are used both as an aid for better comprehension of the properties of data as
well as for a variety of downstream data processing tasks such as data valuation (assessing the value of data
relative to the business objectives at hand) and data curation (filtering and prioritizing training data based on
derived thresholds). In the Large Language Model (LLM) setting, training data is typically unstructured in
nature comprising natural language text, images, and code. In this work, we specifically focus on code-LLMs,
where the quality of code training data substantially affects the model accuracy of LLM-based coding tasks
such as code generation and summarization. Therefore, having the capabilities to characterize code data in
terms of programming language concepts aids in both deriving insights related to code training/evaluation
data and in the downstream curation of code training data. In this work, we address the problem of profiling
multi-lingual code datasets by extracting an extensible user-defined set of syntactic concepts
over arbitrary programming languages.

Related issue number (if any).

[pankajskku]

@daw3rd Please let me know your opinion on the updated PR.

[touma-I]

@pankajskku Please slack me when you have a chance. Internal ID: touma@us.ibm.com