Responsibility
- Design and implement an automated script for generating and updating data dictionary.
- Scann all schemas in the Catalog to select tables into path list;
- Extract detailed information at both table and column levels into data frame;
- Address missingand irregular values using PySpark; Integrated with the Notion API to populate the data dictionary in a prescribed format on Notion pages.
- Deploy the script on Databricks Workflow and set regular execution rules;
- Enhance data quality and consistency as well as reduced manual maintenance costs.
- Utilize Selenium and ChromeDriver to auto-load SuperSet pages on a schedule basis and generate cache, shortening user-perceived loading time.
Requirements
- Bachelor's degree in Computer Science, Data Science, Statistics, or a closely related field.
- Minimum of 6 months of practical experience in data platforms, data analytics, or related projects.
- Experience in using Python ( NumPy, Pandas, PySpark, Scikit learn, Matplotlib, Plotly, Selenium , Pytest Bash
- Experience in Hadoop, Hive, Spark Databricks Airflow Argo Docker Redshift Athena, BigQuery
- Exhibit strong expertise in data visualization tools like Tableau, Power BI, or similar, enabling the creation of compelling data visualizations and reports.
Purva Sholapurkar
EA Licence Number: 17S8727
Registration ID is R22109878
Disclaimer: The company is committed to ensuring the privacy and security of your information. By submitting this form, you consent to the collection, processing, and retention of the information you provide. The data collected (which may include your contact details, educational background, work experience and skills) will be used solely for the purpose of evaluating your qualifications for the position you're applying for. Your data will be stored securely and retained for the duration necessary to fulfill our hiring process. If you are not selected for the position, your data will be kept on file for a limited period in case future opportunities arise. You have the right to access, correct, or delete your data at any time by contacting us at Quess Singapore | A Leading Staffing Services Provider in Singapore (quesscorp.sg)
