The Center for Emerging Technology Identification for Roadway Infrastructure Construction & Maintenance has developed a radar and rubric to evaluate emerging technologies for VTRC scientists to pilot and research.
This rubric has undergone several iterations of refinement through discussions with faculty and iterations with LLMs. The existing version of the radar has been created using a retrieval augmented generation (RAG) concept. The LLM compiles 30 URLs for each technology and uses them to rate the technology against the rubric. These scores are then used to place the technology on the radar. Around 90 technologies have been rated; these were identified using a VTRC Report on Emerging Technologies in 2004 and research conducted by the Center's postdoctoral researchers.
As the Center shifts toward generating pilot projects for VTRC scientists, postdoctoral researchers, and undergraduate students to work on, there is still a need to verify the LLM output and to build a more extensive database for each technology. The goal of this document is to outline a plan for undergraduate researchers hired by the Center to conduct this literature review and generate these scores with the supervision of the postdoctoral researchers.
Outline:
Undergraduate researchers hired by the Center are expected to conduct the following tasks:
- Verify the URLs provided by the LLM. If these URLs are sufficient, no additional information is required.
- For URLs to be considered sufficient, they must meet the following criteria: Provide information about the performance of the technology
- Provide information about the cost of the technology
- Provide information such that the technology can be compared with conventional practice
- Come from credible sources (e.g., .gov or .edu sites or official vendor sites)
If the URLs are not sufficient, the undergraduate researchers must conduct an additional literature search using academic papers or the websites of industry vendors offering the technology.
- Generate a summary based on the literature review. The summary should define the technology and provide context for the rubric ratings. The summary should contain references and be no longer than one page.
- Rate technologies based on the literature review. The ratings should be self-supporting (justifiable from the summary alone). If ratings are based on judgment due to a lack of information, justification must be provided.
- Justification must also be provided on any rubric rows that were omitted from scoring.
- Provide a confidence flag for each technology (high/medium/low) to guide additional review. If confidence is medium or low, postdoctoral researchers will conduct an additional literature review before signing off on scoring.
One of the postdoctoral researchers will sign off on each technology before the score is committed to the radar. If the postdoctoral researcher disagrees with the rating, the two will meet to compare the reasoning behind each score and determine whether additional information should be added to the summary.
Deliverables:
Undergraduate researchers will submit for each technology, a one-page PDF summary along with a rubric sheet highlighting the chosen scores and justification. A template will be provided for undergraduate researchers to fill out to ensure consistency. Both files will be stored in a folder named after each technology.
Criteria:
Undergraduate researchers must be able to synthesize information and provide summaries that allow the Center team to assess each technology without referring to external sources.
Experience: Preference is for candidates with engineering backgrounds, but other adjacent fields will also be considered.
Undergrad student wage position for less than 20 hours per week.
Compensation: $16.50/hr.
MINIMUM REQUIREMENTS:
Resume required.
Education: Current UVA Undergraduate Student.
Experience: None
Licensure: None
Physical Demands: This position may require the ability to sit, stand, walk, bend, stoop, squat, climb stairs, use fine finger manipulation, etc., while conducting, observing, analyzing and/or recording research data.
Under the direct supervision of a faculty member or research supervisor, conducts research, follows protocols, records research results, etc. Will use various methods to collect, organize and analyze data, maintain records of research and/or experiments and document results. Ability to follow oral and written instructions, work effectively individually and in a team environment. Students are required to follow all policies and procedures required by the position, including but not limited to environmental, safety, information security/data privacy, etc.
The University of Virginia is an equal opportunity employer. All interested persons are encouraged to apply, including veterans and individuals with disabilities. Learn more about UVA's commitment to non-discrimination and equal opportunity employment.
