![]() ![]() Our team were given access to AWS S3 via IAM user roles under our AWS account.Ģ. ![]() In building a solution for the first business use case to ingest data extracted from an external cloud and upload the csv files into folders within our Amazon S3 bucket (our data lake). Reading the database developer guide, testing with a sandbox environment and also working in a production environment. What is our minimum viable product? What are our blockers? IAM permissions and stopping and starting. With this greenfield project using my consulting hat, I helped our stakeholders develop the first use case, iterate and develop the next one and so forth. Using a Miro board to sketch the high level architecture (it's ok to draw what you think might be the final solution and then consult and iterate) collaborating with platform teams and also consulting with AWS to develop a solution using AWS Cloud Formation template. By understanding the current state we wanted to create a vision of the end state and how to improve customer experience for NSW citizens. Working in a sprint fashion, this greenfield government project included test, build and deploy. One of our initial questions on the journey with AWS included how do we store all of this data? How do we create a data lake to also bring in external data from another data lake? In a busy contact centre we serve customers and generate petabytpes of data in structured and un-structured data formats. The AWS services used to build our analytics roadmap were: With lake house architecture we integrated a data lake, data warehouse and business intelligence with security, performance, elasticity, data velocity, cost effectiveness and governance to design a unified analytics platform for our organization-wide data architecture roadmap.Īmazon Redshift has massively parallel processing (MPP) for fast and complex queries and can analyze all our data. ![]() We built a well-architected modern data analytics pipeline with a fully-managed, petabyte-scale cloud data warehouse using Amazon Redshift under the Department of Customer Service using AWS Organization. Our vision to become the world’s most customer-centric government agency.Service NSW is part of Department of Customer Service.It was at an AI startup and in Public Sector that I used Amazon services including Amazon S3, Amazon Sagemaker, AWS Glue, Amazon Redshift and Amazon QuickSight. I have consulted and instructed in data analytics and data science for government, financial services and startups for 6 years. I am an AWS Community Builder in data, a data scientist and also a Business Performance Analyst at Service NSW. Getting Started with AWS - On the way to cloudīuild a data warehouse with AWS Redshift - Part 1īuild a data warehouse with AWS Redshift - Part 2 If you are reading this post for the first time, you may wish to read the last three tutorials: In this final tutorial for AWS Redshift I will explain how we can bake our cake and eat it too by loading data into the Redshift cluster through to creating data visualizations with AWS QuickSight. Building a cloud data warehouse and business intelligence modernization
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |